Skip to content

Conversation

@inqueue
Copy link
Member

@inqueue inqueue commented Jan 27, 2026

Set the maximum memory flag (-m) to a supported value supported by pbzip2. Fixes a warning when decompressing bz2 corpus files:

2026-01-26 20:00:17,22 ActorAddr-(T|:56141)/PID:34824 esrally.utils.io WARNING Failed to decompress [/Users/jbryan/.rally/benchmarks/data/github_archive0/documents-20211001-1k.json.bz2] with [['pbzip2', '-d', '-k', '-m10000', '-c', '/Users/jbryan/.rally/benchmarks/data/github_archive0/documents-20211001-1k.json.bz2']]. Error [b'pbzip2: *ERROR: Memory usage size Min: 1MB and Max: 2000MB!  Aborting...\n']. Falling back to standard library.

The upper limit according to pbzip2.cpp source is 2000MB.

3569                                         if ((maxMemory < 1000000) || (maxMemory > 2000000000))
3570                                         {
3571                                                 fprintf(stderr,"pbzip2: *ERROR: Memory usage size Min: 1MB and Max: 2000MB!  Aborting...\n");
3572                                                 return 1;
3573                                         } 

@inqueue inqueue requested a review from gbanasiak January 27, 2026 15:37
Copy link
Contributor

@gbanasiak gbanasiak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@inqueue inqueue enabled auto-merge (squash) January 27, 2026 16:17
@inqueue inqueue merged commit 8ce95d7 into elastic:master Jan 27, 2026
15 checks passed
@fressi-elastic
Copy link
Contributor

This closes issue #2030

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants