Support calendar-based chunking

[erimatnor]

Hypertables can now be configured to create chunks that align with the start and end of days, weeks, months, and years in the local time zone. This includes days and months that vary in length due to daylight savings or month of the year.

This "calendar-based chunking" is achieved by anchoring chunk ranges at a user-configurable "origin" and calculating chunk ranges using local time zones and in units of, e.g., variable-length days and months. Therefore, a day-sized chunk can sometimes be 23 or 25 hours if it covers a daylight savings change.

Currently, calendar-based chunking is guarded by a GUC, and turned off by default to preserve the existing behavior. To use calendar-based chunking, the GUC must be turned on and the hypertable configured with an Interval-type chunk interval. Existing hypertables are not affected by the GUC setting.

The default origin is set to '2001-01-01 00:00' because that is the start of a new year and a Monday, so it also aligns with the start of a week (ISO). This means that chunk intervals set to 1 week will lead to chunks that start on a Monday and ends on a Sunday.

The origin-based approach was chosen because of the flexibility it gives; setting a different origin allows, e.g., daily chunks to start at noon instead of midnight. It also makes supporting chunk intervals of multiple months easy, as opposed to a truncation-based approach (e.g., date_trunc()), which only works with singular days, weeks, or months.

Implementation-wise, a challenge of the origin-based approach is to calculate a chunk range for a point in the future from the given origin. Since, e.g., a 1 month interval varies in size depending on which month it is, simple fixed-size interval arithmetic are not possible to calculate the N:th chunk range from the origin. Instead, the approach taken is to break down the calculations into full month, day, and sub-day units. But this only works for intervals that are non-fractional units of months, days, etc. As a fallback for arbitrary intervals, the range for a particular chunk is calculated from the origin by iteratively adding intervals until the desired point in time is covered. This iterative approach is optimized by (under-)estimating the number of intervals to the desired point, and then iterating from there.

Since the iterative approach works for all types of intervals, the question is whether this approach is good enough for all cases, and the "broken-down" calculations are not needed. However, for this change, both approaches exist together, although this decision can be revisited in a future change.

Changes include:

• Add interval column to dimension catalog for INTERVAL type storage
• Add partition_origin parameter to by_range() for origin specification
• Create chunk_range.c/h for calendar-based interval calculations
• Add GUC timescaledb.enable_calendar_chunking (default off)
• Update dimension handling to support both fixed and calendar intervals
• Update SQL API functions to support calendar intervals
• Add calendar_chunking regression test

The PR is divided into two commits. The first commit introduces the origin parameter:

Add support for specifying an origin point for aligning chunk boundaries. This allows chunks to be aligned to a specific reference point instead of the Unix epoch (or zero for integer dimensions).

Changes:

• Add interval_origin column to dimension catalog table
• Add origin parameter to create_hypertable(), set_chunk_time_interval(),
  set_partitioning_interval(), and add_dimension() SQL functions
• Add time_origin and integer_origin columns to dimensions view
• Modify chunk boundary calculation to use origin when specified
• Add type validation for origin (integer vs timestamp compatibility)
• Add chunk_origin test for origin parameter functionality

Disable-check: commit-count

Closes: #1500

Example usage:

SET timescaledb.enable_calendar_chunking = true;
SET
SHOW timezone;
     TimeZone
------------------
 Europe/Stockholm
(1 row)

-- Start with monthly chunks
create table hyper (time timestamptz, temp float)
with (tsdb.hypertable, tsdb.chunk_interval='1 month');
NOTICE:  using column "time" as partitioning column
CREATE TABLE
select hypertable_name,time_interval, time_origin
from timescaledb_information.dimensions where hypertable_name = 'hyper';
 hypertable_name | time_interval |         time_origin
-----------------+---------------+------------------------------
 hyper           | 1 mon         | Mon Jan 01 00:00:00 2001 CET
(1 row)

insert into hyper values ('2016-01-14 09:32:03', 1.0), ('2016-02-28 17:30:01', 2.0);
INSERT 0 2
select chunk_name, range_start, range_end, range_end::timestamptz - range_start::timestamptz as range
from timescaledb_information.chunks where hypertable_name = 'hyper';
     chunk_name     |         range_start          |          range_end           |  range
--------------------+------------------------------+------------------------------+---------
 _hyper_11_13_chunk | Fri Jan 01 00:00:00 2016 CET | Mon Feb 01 00:00:00 2016 CET | 31 days
 _hyper_11_14_chunk | Mon Feb 01 00:00:00 2016 CET | Tue Mar 01 00:00:00 2016 CET | 29 days
(2 rows)

-- Switch to weekly chunks
SELECT set_partitioning_interval('hyper', interval '1 week');
 set_partitioning_interval
---------------------------

(1 row)

insert into hyper values ('2016-03-01 11:56:19', 3.0), ('2016-03-09 10:23:43', 4.0);
INSERT 0 2

-- Note that weekly chunks align with start of week while monthly align with start of month. 
-- There's a transition chunk of 6 days because of the "conflict" between weekly and monthly
-- when transitioning. Also notice that February has 29 days because 2016 is a leap year.
select chunk_name, range_start, range_end, range_end::timestamptz - range_start::timestamptz as range
from timescaledb_information.chunks where hypertable_name = 'hyper';
     chunk_name     |         range_start          |          range_end           |  range
--------------------+------------------------------+------------------------------+---------
 _hyper_11_13_chunk | Fri Jan 01 00:00:00 2016 CET | Mon Feb 01 00:00:00 2016 CET | 31 days
 _hyper_11_14_chunk | Mon Feb 01 00:00:00 2016 CET | Tue Mar 01 00:00:00 2016 CET | 29 days
 _hyper_11_15_chunk | Tue Mar 01 00:00:00 2016 CET | Mon Mar 07 00:00:00 2016 CET | 6 days
 _hyper_11_16_chunk | Mon Mar 07 00:00:00 2016 CET | Mon Mar 14 00:00:00 2016 CET | 7 days
(4 rows)

-- Switching to daily chunks
SELECT set_partitioning_interval('hyper', interval '1 day');
 set_partitioning_interval
---------------------------
(1 row)

insert into hyper values 
('2016-03-15 12:22:18', 5.0), 
('2016-03-27 18:10:54', 6.0), 
('2016-03-28 03:03:33', 7.0);;
INSERT 0 3

-- Note the daylight savings transition on March 27.
select chunk_name, range_start, range_end, range_end::timestamptz - range_start::timestamptz as range
from timescaledb_information.chunks where hypertable_name = 'hyper';
     chunk_name     |          range_start          |           range_end           |  range
--------------------+-------------------------------+-------------------------------+----------
 _hyper_26_33_chunk | Fri Jan 01 00:00:00 2016 CET  | Mon Feb 01 00:00:00 2016 CET  | 31 days
 _hyper_26_34_chunk | Mon Feb 01 00:00:00 2016 CET  | Tue Mar 01 00:00:00 2016 CET  | 29 days
 _hyper_26_35_chunk | Tue Mar 01 00:00:00 2016 CET  | Mon Mar 07 00:00:00 2016 CET  | 6 days
 _hyper_26_36_chunk | Mon Mar 07 00:00:00 2016 CET  | Mon Mar 14 00:00:00 2016 CET  | 7 days
 _hyper_26_37_chunk | Tue Mar 15 00:00:00 2016 CET  | Wed Mar 16 00:00:00 2016 CET  | 1 day
 _hyper_26_38_chunk | Sun Mar 27 00:00:00 2016 CET  | Mon Mar 28 00:00:00 2016 CEST | 23:00:00
 _hyper_26_39_chunk | Mon Mar 28 00:00:00 2016 CEST | Tue Mar 29 00:00:00 2016 CEST | 1 day
(7 rows)

[codecov]

Codecov Report

❌ Patch coverage is 84.62567% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.55%. Comparing base (902c268) to head (23055c2).

Files with missing lines	Patch %	Lines
src/dimension.c	84.11%	26 Missing and 38 partials ⚠️
src/chunk_range.c	82.28%	18 Missing and 30 partials ⚠️
src/with_clause/create_table_with_clause.c	87.50%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9119      +/-   ##
==========================================
+ Coverage   82.42%   82.55%   +0.12%     
==========================================
  Files         244      246       +2     
  Lines       47953    48597     +644     
  Branches    12235    12438     +203     
==========================================
+ Hits        39525    40117     +592     
- Misses       3553     3608      +55     
+ Partials     4875     4872       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

@fabriziomello, @natalya-aksman: please review this pull request.

Powered by pull-review

[erimatnor]

Going to work on tests for continuous aggregates and other things.

[akuzm]

This "calendar-based chunking" is achieved by anchoring chunk ranges at a user-configurable "origin" and calculating chunk ranges using local time zones and in units of, e.g., variable-length days and months.

Is it the same thing as time_bucket, i.e. generates the same intervals for chunks if configured with the same bucket width?

[erimatnor]

This "calendar-based chunking" is achieved by anchoring chunk ranges at a user-configurable "origin" and calculating chunk ranges using local time zones and in units of, e.g., variable-length days and months.

Is it the same thing as time_bucket, i.e. generates the same intervals for chunks if configured with the same bucket width?

Yes, it is similar. But there are also differences, including (but probably not limited to):

• For chunks we need to calculate a range (start, end) and not just bucket start. So that needs to handle ranges that cover, e.g, daylight savings changes, etc.
• Support for arbitrary intervals, e.g., '1 month 1 day' which time_bucket() doesn't support.
• No errors, for, e.g., overflows. You don't want to throw errors at insert time when calculating ranges so the implementation is a bit different doing saturating adds and clamping ranges at extremes, etc.
• For sub-day (non variable) time intervals, chunk ranges are similar to date_bin() while time_bucket() can sometime produce different ranges/buckets.

Btw, when comparing to time_bucket(), I also found a potential issue with time_bucket(): #9136