Add new chunks to hypertable publication

[arajkumar]

Problem: Adding a hypertable to a publication automatically includes all existing chunks, but newly created chunks are not automatically added to the publication.

This commit fixes the problem by adding newly created chunks to all publications that the hypertable belongs to, while respecting the hypertable’s row and column filtering rules.

This ensures that logical replication works correctly for partitioned data without requiring manual chunk management.

Added a new GUC timescaledb.enable_chunk_auto_publication to control behaviour of this change.

By default timescaledb.enable_chunk_auto_publication is disabled, it could be modified without restarting the database.

[codecov]

Codecov Report

❌ Patch coverage is 88.88889% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.54%. Comparing base (4d2e91a) to head (0f42a72).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
src/chunk.c	88.63%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9030      +/-   ##
==========================================
+ Coverage   82.42%   82.54%   +0.11%     
==========================================
  Files         243      243              
  Lines       47914    47930      +16     
  Branches    12229    12233       +4     
==========================================
+ Hits        39492    39562      +70     
- Misses       3555     3557       +2     
+ Partials     4867     4811      -56     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

@pnthao, @akuzm: please review this pull request.

Powered by pull-review

[arajkumar]

CC: @gayyappan

[gayyappan]

also need a test after calling compress_chunk. we should not see a new entry in the publication table.
( this also means that direct modifications to the compressed chunk will not be visible to the publication's consumer).

[arajkumar]

Done.

[gayyappan]

we need to make sure this doesn't happen if the chunk is a foreign table (OSM). I believe that doesn't go through this path. (I need to check the other paths that create a chunk, but wanted to raise that). The best place to add a test for OSM specific chunks is chunk_utils_internal.sql

[arajkumar]

OSM chunks have two issues:
1. For a pre-existing publication on a hypertable, when tiering is enabled, it creates a child foreign table. This child table may be skipped when adding to the publication.
2. For a hypertable that is already tiered, adding it to a publication fails immediately because foreign tables cannot be added to a publication. I think this needs to be solved in upstream Postgres, or perhaps we could override the publishing behavior on our side(?). I want to try hacking PG code.

[arajkumar]

Added a test for case 1.

Case 2 is an upstream issue, where they don't exclude child foreign tables.

https://github.com/postgres/postgres/blob/493eb0da31be4520252e1af723342dc7ead0c3e5/src/backend/commands/publicationcmds.c#L1773-L1780

I will try fixing upstream. Other option would be overriding CREATE/ALTER publication from our extension and changing it's behavior.

[arajkumar]

One more thing, CREATE PUBLICATION .. FOR TABLE supports ONLY option, that will skip all child tables. This can be used on our internal context.

[gayyappan]

Yes, we use the ONLY workaround on the lake side.

[fabriziomello]

@arajkumar @gayyappan does it remove the chunk from the publication if it is dropped either by DROP TABLE statements, drop_chunks API or retention policy?

[gayyappan]

@arajkumar @gayyappan does it remove the chunk from the publication if it is dropped either by DROP TABLE statements, drop_chunks API or retention policy?

Yes, it should be auto-dropped. @arajkumar would be good to add a test in chunk_publication.sql using drop_chunks API. You might also consider combining chunk_publication.sql and the compression test into 1 sql test file. ( Just a suggestion).

[arajkumar]

@fabriziomello @gayyappan Added tests covering drop_chunk, DROP TABLE _chunk, TRUNCATE hypertable, merge_chunk, and split_chunk.

[arajkumar]

The failure is from the following step.

  - name: Check that no new internal program errors are introduced
      # We collect the same messages when uploading to the CI checks database,
      # but it runs on different conditions (not on forks and not on flaky check),
      # and requires different data (job date), so we do it separately here.
      # The jq --exit-code option is broken with select() on jq-1.6 which we see
      # on some machines, so we use grep instead (see https://github.com/jqlang/jq/issues/1139).
      if: always() && contains(matrix.name, 'Flaky')
      run: |
        ! jq 'select(
                (.state_code == "XX000" and .error_severity != "LOG")
                or (.message | test("resource was not closed"))
            ) | [.message, .func_name,  .statement] | @tsv
        ' -r postmaster.json | grep .

[arajkumar]

@fabriziomello @gayyappan PTAL.

[arajkumar]

@gayyappan Here is a test that covers drop chunk in 3 scenarios.

1. drop_chunk SQL api
2. DROP TABLE chunk_table
3. TRUNCATE TABLE

[gayyappan]

Excellent test case coverage. Thank you @arajkumar

[arajkumar]

@fabriziomello @akuzm Could you please review this PR? Thank you.

[svenklemm]

Please add a GUC to control this behaviour

[arajkumar]

Please add a GUC to control this behaviour

@svenklemm Thanks for reviewing this PR.

Added a GUC timescaledb.enable_chunk_auto_publication, enabled by default.

[svenklemm]

this should be off by default since we dont support logical replication for hypertables and this needs additional infrastructure to work correctly

[arajkumar]

@svenklemm Ty! now it is false by default.