Feature: Add diskquota extension to gpcontrib #1490
+26,037
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: hzhang2 <hzhang2@vmware.com>
This patch introduces isolation2 test framework to diskquota. We are able to write concurrent test cases for diskquota in future.
* Fix incorrect relation size for AO tables * Rename function to make it more consistent * Remove try-catch when stat a file * Add concurrent write test case for AO table
1. add relation_cache_lock 2. add relation_cache and relid_cache in SHM 3. add relation_cache test case Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com> Co-authored-by: Xing Guo <higuoxing+github@gmail.com>
…_committed_relation_from_cache() (#96) Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com>
Co-authored-by: hzhang2 <hzhang2@vmware.com>
* Fix test case bug: Drop test table after testing diskquota.relation_size(), because the relation_entry of the table in this test case will affect the next test case. Co-authored-by: hzhang2 <hzhang2@vmware.com>
* Add support for dispatching blockmap to segment servers. This patch adds support for dispatching blockmap to segment servers. The basic idea is that, we iterate over the active relations' oid, check that whether the relation's owner/tablespace/namespace is in one of the blockmap entries dispatched from diskquota worker from QD. If the relation should be blocked, we then add its relfilenode together with the relfilenodes of toast relation, toast index relation, appendonly auxiliary relations, appendonly auxiliary relations' indexes to the global blockmap on segment servers. This patch is a prerequisite of the hard-limit feature. This patch also introduces diskquota.blockmap view. User is able to query the current blocking status by 'SELECT * FROM diskquota.blockmap'. Co-authored-by: Xuebin Su <sxuebin@vmware.com> Co-authored-by: Hao Zhang <hzhang2@vmware.com>
This PR fixes incorrect JOIN condition in test_blackmap.sql. The relfilenodes are not always different across segments. Hence, we should add an additional JOIN condition to the test case or it will produce unstable results.
Add calculate_table_size() to calculate any table's size, including uncommitted table. 1. Monitor active table in master. 2. Replace relation_open() with SearchSysCache(), pg_appendonly and pg_index to fetch Form_pg_class of table and index, to avoid DEADLOCK. 3. Use DiskQuotaRelationCacheEntry to calculate table size, instead of pg_table_size(), to avoid DEADLOCK. Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com> Co-authored-by: Xing Guo <higuoxing+github@gmail.com>
This patch adds support for adding uncommitted relations to blackmap on segment servers. Most of the codes share similar logic with adding committed relations to blackmap. Test will be added in the next following commits.
The extension file with (segno=0, column=1) is not traversed by ao_foreach_extent_file(), we need to handle the size of it additionally. Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xing Guo <higuoxing+github@gmail.com>
This patch adds support for dispatching blackmap to segments. This patch also introduces two UDFs: diskquota.enable_hardlimit() diskquota.disable_hardlimit() User is able to enable and disable the hardlimit feature by using those UDFs. Co-authored-by: hzhang2 <hzhang2@vmware.com>
This PR is trying to fix CI building failure by ignoring distribution notice in CTAS statements. Co-authored-by: Hao Zhang <hzhang2@vmware.com>
…full (#112) * Fix bug The relation_cache_entry of temporary table, created during vacuum full, is not be removed after vacuum full. This table will be treated as an uncommitted table, although it has been dropped after vacuum full. And its table size still remains in diskquota.table_size, which causes quota size to be larger than real status. Use RelidByRelfilenode() to check whether the table is committed, and remove its relation_cache_entry. Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xing Guo <higuoxing+github@gmail.com> Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com>
The SQL statement is not equal to the expected output in test_vacuum.sql. Co-authored-by: hzhang2 <hzhang2@vmware.com>
…e index on ao_table` is committed (#113) We can not calculate the size of pg_aoblkdir_xxxx before `create index on ao_table` is committed. 1. Lack of the ability to parse the name of pg_aoblkdir_xxxx. 2. pg_aoblkdir_xxxx is created by `create index on ao_table`, which can not be searched by diskquota_get_appendonly_aux_oid_list() before index's creation. Solution: 1. parse the name begin with `pg_aoblkdir`. 2. When blkdirrelid is missing, we try to fetch it by traversing relation_cache. Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xing Guo <higuoxing+github@gmail.com> Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com>
Co-authored-by: Xuebin Su (苏学斌) <12034000+xuebinsu@users.noreply.github.com> Co-authored-by: Xing Guo <higuoxing@gmail.com> Co-authored-by: Hao Zhang <hzhang2@vmware.com>
Consider a user session that does a DELETE followed by a VACUUM FULL to reclaim the disk space. If, at the same time, the bgworker loads config by doing a SELECT, and the SELECT begins before the DELETE ends, while ends after the VACUUM FULL begins: bgw: ---------[ SELECT ]-----------> usr: ---[ DELETE ]-[ VACUUM FULL ]--> then the tuples deleted will be marked as RECENTLY_DEAD instead of DEAD. As a result, the deleted tuples cannot be removed by VACUUM FULL. The fix lets the user session wait for the bgworker to finish the current SELECT before starting VACUUM FULL .
When doing VACUUM FULL, The table size may not be updated if the table's oid is pulled before its relfilenode is swapped. This fix keeps the table's oid in the shared memory if the table is being altered, i.e., is locked in ACCESS EXCLUSIVE mode. Co-authored-by: Xuebin Su <sxuebin@vmware.com>
Currently, diskquota.pause() only takes effect on quota checking. Bgworkers still go over the loop to refreshing quota even if diskquota is paused. This wastes computation resources and can cause flaky issues. This fix makes bgworkers skip refreshing quota when the user pauses diskquota entirely to avoid those issues. Table sizes can be updated correctly after resume.
ci: create rhel8 release build. Signed-off-by: Sasasu <i@sasa.su> Co-authored-by: Xuebin Su <sxuebin@vmware.com>
) Fix the bug caused by #220: After the user removes any extension, the bgworker in the current database will be stopped.
- skip add last version to the current build. - skip upgrade test.
Co-authored-by: Chen Mulong <chenmulong@gmail.com> Co-authored-by: Xing Guo <admin@higuoxing.com>
Merge all types of quota info maps into one in each bgworker.
The package name is incorrect when releasing package in Rocky9/RHEL9 platforms (<package>-<version>-unknown_x86_64.tar.gz). This patch fix it to <package>-<version>-rhel9_x86_64.tar.gz. Cherry-pick https://github.com/pivotal/timestamp9/pull/41
- DDL message is only used on master, so it is unnecessary to allocate the memory on segments. - The launcher shmem should not be initialized on segments.
refresh_rejectmap looks for a tuple using SearchSysCacheCopy1 which retrieves a copy of the tuple allocating memory for it. However, refresh_rejectmap didn't free these tuple copies after use. If lots of oids were passed, diskquota could work incorrectly because of huge memory leak. This patch frees these tuples and prevents memory leaks.
fix: update actions Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Diskquota calculates sizes and stores information in the diskquota.table_size table periodically with a pause in diskquota.naptime, 2 seconds by default. If we restart the cluster during this pause, then diskquota will lose all changes that have occurred since the last save to the diskquota.table_size table. We could create temporary tables, wait when it will be flushed to diskquota.table_size table, restart the cluster, and diskquota would remember the information about the temporary tables. Or we could delete the tables, restart the cluster, and again diskquota will remember information about the deleted tables. This happens because at the start of the cluster, diskquota remembers all the information written to the diskquota.table_size table, but does not check that some tables may have already been deleted. As a solution, we invalidate diskquota.table_size during diskquota worker start in addition to pg_class validation. The remaining problem: the incorrect table size cannot be refreshed until the corresponding table becomes an active table. Solution: call diskquota.init_table_size_table().
tuhaihe
force-pushed
the
add-diskquota
branch
2 times, most recently
from
c916aae to
92de2e8
Compare
leborchuk
approved these changes
Dec 24, 2025
Contributor
leborchuk
left a comment
leborchuk
left a comment
There was a problem hiding this comment.
Checked it on my development ubuntu installation, all simple tests works fine
Here the output from examples
postgres=# select * from diskquota.show_fast_schema_quota_view;
schema_name | schema_oid | quota_in_mb | nspsize_in_bytes
-------------+------------+-------------+------------------
s1 | 17157 | 1 | 98304
my-ship-it
reviewed
Jan 23, 2026
Contributor
my-ship-it
left a comment
my-ship-it
left a comment
There was a problem hiding this comment.
Thank you for your contributions that enable Cloudberry to support more components. Is it possible for us to retain the previous commit history?
Member
Author
NP. Let me bring them back for a complete commit history. |
Cleanup the invalid files after importing diskquota extension from upstream. The specific changes include: 1. Remove .github directory: The upstream GitHub Actions workflows are no longer applicable in the Cloudberry repository. 2. Remove .gitmessage, .editorconfig, .clang-format: Code style and commit templates should follow Cloudberry's main repository standards. 3. Remove SECURITY.md: Legacy security policy. 4. Remove concourse directory: Legacy CI scripts. This cleanup makes the extension structure cleaner and more consistent with other contrib modules.
tuhaihe
force-pushed
the
add-diskquota
branch
2 times, most recently
from
710b95c to
8270730
Compare
Integrate diskquota extension into Apache Cloudberry build system and
adapt the codebase for Cloudberry 2.0+ (PostgreSQL 14 based).
Main changes:
Build system integration:
* Add new Makefile for building with Cloudberry source tree
```
make
make install
make installcheck
make clean
```
* Update gpcontrib/Makefile to include diskquota in build and installcheck
* Simplify CMakeLists.txt by removing GP6 version conditionals
* Add PG_SRC_DIR availability check for isolation2 tests
Code modernization (remove GP6 compatibility code):
* Remove GP_VERSION_NUM < 70000 conditionals throughout codebase
* Replace deprecated APIs:
heap_open -> table_open,
heap_beginscan_catalog -> table_beginscan_catalog,
heap_endscan -> table_endscan, etc.
* Replace init_ps_display() with set_ps_display() for process status
* Replace StrNCpy() with snprintf() for safer string handling
* Remove WaitForBackgroundWorkerShutdown() polyfill (now in core)
* Remove MemoryAccounting_Reset() calls (removed in GP7+)
* Update tuple descriptor attribute access from pointer to direct access
Documentation:
* Rewrite README.md for Apache Cloudberry with updated build instructions
Other improvements:
* Update extension comment to be more descriptive
* Ensure postgres.h is included first in all source files
CI:
* add ic-diskquota to `build-cloudberry.yml` workflow
* For `build-deb-cloudberry.yml`, the installation and configure prefix
are not consisent, which results in the test error. Will add
ic-diskquota test back once updating the deb workflow.
See: https://lists.apache.org/thread/1zd80r1hvpwwh5fjd5yqgcc7sr4f27qr
Add diskquota extension license information to comply with Apache release requirements. Changes: - Add diskquota entry to top-level LICENSE file under Greenplum section - Create licenses/LICENSE-diskquota.txt with PostgreSQL License text - Add gpcontrib/diskquota/** to pom.xml excludes for apache-rat checks The diskquota extension is licensed under the PostgreSQL License, originally developed by Pivotal Software and VMware.
tuhaihe
force-pushed
the
add-diskquota
branch
from
8270730 to
de212a4
Compare
Member
Author
Hi @my-ship-it PTAL again. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #ISSUE_Number
What does this PR do?
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions