Skip to content

Optimize 1.7 compression by using velocity-native compressor and reduce memory allocations inside chunk rewriting #647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Beaness wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Optimize 1.7 compression by using velocity-native compressor and reduce memory allocations inside chunk rewriting #647

Beaness wants to merge 4 commits into from

Conversation

@Beaness
Copy link
Contributor

@Beaness Beaness commented Dec 17, 2025

image Should improve chunk rewriting performance

@Beaness Beaness changed the title Optimize 1.7 compression by using velocity-native compressor Optimize 1.7 compression by using velocity-native compressor and reduce memory allocations inside chunk rewriting Dec 17, 2025
@Beaness
Copy link
Contributor Author

Beaness commented Dec 17, 2025

Latest commit has the additional following changes:

  • Dropped ExtendedBlockStorage to prevent having extra memory byte array allocations when it can be written directly to the output byte array
  • Allocate only one byte array in bulk chunk map where all chunks are serialized to

@Beaness
Don't include velocity native as shaded dependency, add fallback and …
9cf34ae 
…only use velocity compression if available
@Beaness
Change chunk compression back to default java as chunks are serialize…
5154df1 
…d on heap byte arrays so native compression has no use
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Beaness