It upgrades Elastic search client and `urlib3` libraries. #2024

fressi-elastic · 2026-01-13T14:59:20Z

This PR upgrades the Elasticsearch client libraries and urllib3 to newer major versions, updating from Elasticsearch 8.x to 9.x and urllib3 1.x to 2.x. The changes include API parameter updates, refactoring of header handling logic, and deprecation of certain track dependencies.

Changes:

It sets the minimal Elasticsearch server version to 8.0.0 and update documentation.
It update python package requirements:
- elasticsearch[async] from 8.6.1 to 9.2.1
- elastic-transport from 8.4.1 to 9.2.1
- urllib3 from 1.26.19 to 2.6.3
- docker from 6.0.0 to 7.1.0
It renames maxsize parameter to connections_per_node in client initialization to match the new API
It refactors header handling logic into reusable functions and improved MIME type compatibility handling
It replaces obsolete docker-compose command with docker compose to avoid conflicts with new urlib3 version.
It fixes it/docker_dev_image_test.py and update elasticsearch server to version 9.2.4.

This PR replaces the now obsolete #2003 and it requires PR elastic/rally-tracks#1000

…pgrade of elastisearch client libraries.

Copilot

Pull request overview

This PR upgrades the Elasticsearch client libraries and urllib3 to newer major versions, updating from Elasticsearch 8.x to 9.x and urllib3 1.x to 2.x. The changes include API parameter updates, refactoring of header handling logic, and deprecation of certain track dependencies.

Changes:

Updated elasticsearch[async] from 8.6.1 to 9.2.1, elastic-transport from 8.4.1 to 9.2.1, and urllib3 from 1.26.19 to 2.6.3
Renamed maxsize parameter to connections_per_node in client initialization to match the new API
Refactored header handling logic into reusable functions and improved MIME type compatibility handling

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pyproject.toml	Updated dependency versions for elasticsearch, elastic-transport, urllib3, and pip
esrally/client/factory.py	Changed `maxsize` to `connections_per_node` parameter
tests/client/factory_test.py	Updated test to use `connections_per_node` parameter
esrally/client/common.py	Refactored header combining and MIME type compatibility logic into dedicated functions
esrally/client/synchronous.py	Updated to use new header handling functions and added endpoint_id/path_parts parameters
esrally/client/asynchronous.py	Updated RallyAsyncElasticsearch signature and header handling to use new functions
esrally/track/loader.py	Added deprecation handling for elasticsearch and elastic-transport dependencies in tracks
esrally/utils/versions.py	Added type hints to is_version_identifier function
it/init.py	Updated test distributions to include 9.2.4 and removed 6.8.0/ARM logic
tests/client/common_test.py	Removed test file (likely obsolete after refactoring)
tests/driver/runner_test.py	Added typing import and @typing.no_type_check decorator to test

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

esrally/client/common.py

…ocker-compose`.

…ocker-compose` (second part).

Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

esrally/track/loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fressi-elastic · 2026-01-20T14:55:21Z

I am waiting for review for the following Elasticsearch PR fix that should block this PR build failures: elastic/elasticsearch#140947

It should correct the following error:
https://github.com/elastic/rally/actions/runs/21173420503/job/60895685787?pr=2024#step:11:530

fressi-elastic · 2026-01-20T15:28:19Z

PR elastic/elasticsearch#140947 has been merged. Now current Elasticsearch should compile again.

gareth-ellis

LGTM - Have you run any comparisons with the old and new clients, and validated whether performance is equivalent?

esrally/client/asynchronous.py

gbanasiak

A few areas of concern:

How does this change affect the range of supported Elasticsearch versions? This needs to be reflected in Rally documentation.
If the intention is to drop support for ES 6.8 and 7.x as part of client upgrade, we should release one more Rally version before merging this update IMO. It's been a while since we released.
What is the test plan here? Go to nightly staging, and then nightly production, and revert if things go south? I would prefer to test this more extensively before the merge (e.g. by running all nightlies in test mode).
Why _ProductChecker is kept in synchronous.py? This client upgrade is an opportunity to simplify, because the following no longer holds:

Because Rally needs to support versions of ES >= 6.8.0, we resurrect the previous
logic for determining the authenticity of the server, which does not rely exclusively
on this header.

esrally/track/loader.py

…al parent method implementations.

fressi-elastic · 2026-01-22T11:10:47Z

How does this change affect the range of supported Elasticsearch versions? This needs to be reflected in Rally documentation.

It should now match versions supported by Elasticsearch Python client 9.x

If the intention is to drop support for ES 6.8 and 7.x as part of client upgrade, we should release one more Rally version before merging this update IMO. It's been a while since we released.

OK.

What is the test plan here? Go to nightly staging, and then nightly production, and revert if things go south? I would prefer to test this more extensively before the merge (e.g. by running all nightlies in test mode).

It would be nice to push this to a branch used by the nightly staging CI jobs, so we would eventually see it working or not. Is master branch used by both staging and production nightlies?

Why _ProductChecker is kept in synchronous.py? This client upgrade is an opportunity to simplify, because the following no longer holds:

Because Rally needs to support versions of ES >= 6.8.0, we resurrect the previous
logic for determining the authenticity of the server, which does not rely exclusively
on this header.

I removed the old logic and matched the code in Rally as much as possible with the official client, with the only exception of the support for serverless and the automatic addition of application/*headers when body is given. It also helps with Elasticsearch 9 integration adding the x-ndjson mime type in a bulk request as it is being more strict that the version 8 on this aspect.

…bility with `options` method.

…g tracks preparation.

… version to 9.2.4

gbanasiak

I've noticed ES Python client docs say 9.x is not compatible with ES 8.x, so I started wondering if we shouldn't conservatively upgrade the client to 8.19 instead? This would still allow us to get rid of old urllib3 version. What exactly fails when using 9.x client against 8.x cluster?

What is the purpose of _ProductChecker in synchronous.py now that we're dropping support for ES 6.8 and 7.x?

Have you checked if we actually have to keep overriding _perform_request() in RallySyncElasticsearch and RallyAsyncElasticsearch? See note below.

gbanasiak · 2026-01-29T10:06:07Z

esrally/client/asynchronous.py

        if not self.is_serverless:
-            if versions.is_version_identifier(self.distribution_version) and (
-                versions.Version.from_string(self.distribution_version) >= versions.Version.from_string("8.0.0")
-            ):
-                _mimetype_header_to_compat("Accept", request_headers)
-                _mimetype_header_to_compat("Content-Type", request_headers)
+            # Converts all parts of a Accept/Content-Type headers
+            # from application/X -> application/vnd.elasticsearch+X
+            # see https://github.com/elastic/elasticsearch/issues/51816
+            mimetype_headers_to_compat(request_headers, self.distribution_version)


Do we need if not self.is_serverless condition? I don't see anything like it in 9.2.1 version of ES Python client, see https://github.com/elastic/elasticsearch-py/blob/2022b70b5c2139c3c5d9c47d038f022a1cb1f7ca/elasticsearch/_sync/client/_base.py#L307-L308 which does support serverless I presume.

If this goes away, can we get rid of _perform_request() completely?

Not sending compatible-with on serverless is more correct, that's a missing feature in the Python client. It's harder to fix because we don't know if something is serverless when talking to it initially, so the only thing we can do is add an opt-in serverless mode.

In the new client there are new features like supporting open telemetry. I decided to keep it and have the client code as much similar to the official version to make its maintenance simpler for the next upgrades. _perform_request is the actual new implementation in the base client, it is being wrapped to support open telemetry and I keep it the same. is_serverless condition makes things bit more complex but required to avoid sending an unsupported header to serverless. Anyway the header is only send if the server version is known that could not be the case (and it should work anyway I guess). The fact no CI job was broken by testing it with either 8 and 9 family give me confidence this approach could actually work and it is enough simple.

fressi-elastic · 2026-01-29T13:53:47Z

LGTM - Have you run any comparisons with the old and new clients, and validated whether performance is equivalent?

I never validated performances.

fressi-elastic · 2026-01-29T14:05:00Z

esrally/client/asynchronous.py

-        # from application/X -> application/vnd.elasticsearch+X
-        # see https://github.com/elastic/elasticsearch/issues/51816
-        # Not applicable to serverless
+        if body is not None:


I think this block is only required to avoid updating our tracks to include these headers. I wonder why the headers handling wasn't moved to an higher level to leave this client patching much simpler. @pquentin What do you think?

fressi-elastic · 2026-01-29T14:06:03Z

esrally/client/asynchronous.py

+        if body is not None:
+            # It ensures content-type and accept headers are set.
+            mimetype = "application/json"
+            if path.endswith("/_bulk"):


API protocol 9 is more picky and requires application/x-ndjson for bulk requests.

fressi-elastic · 2026-01-29T14:07:14Z

esrally/client/asynchronous.py


            raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(message=message, meta=meta, body=resp_body)

+        # 'X-Elastic-Product: Elasticsearch' should be on every 2XX response.


This is intended to keep the code much similar to the official client.

fressi-elastic · 2026-01-29T14:09:51Z

esrally/client/common.py

+        return
+
+    major_version = 8
+    if versions.is_version_identifier(distribution_version):


It only uses compatibility version 9 when it does knows server version is 9, else it always uses version 8.

…de-2

Concerns lifted after internal discussion.

fressi-elastic force-pushed the esclient-upgrade-2 branch 3 times, most recently from ee26164 to 1e8aeeb Compare January 13, 2026 17:47

Upgrade Elastic search client libraries and urlib3.

1421ad0

fressi-elastic force-pushed the esclient-upgrade-2 branch from 1e8aeeb to 1421ad0 Compare January 13, 2026 18:34

Try enabling client compatibility mode to support ES version 8

de1c485

fressi-elastic force-pushed the esclient-upgrade-2 branch from 553574d to de1c485 Compare January 14, 2026 13:03

fressi-elastic added 8 commits January 19, 2026 11:34

Update client compatibility version handling.

1011049

Drop support for distribution version < 8, add support for version 9

31a59e3

Remove obsolete unit test.

169050a

Change track dependency installation: add pip to venv and deprecate u…

494a501

…pgrade of elastisearch client libraries.

It removes unnecessary changes.

478846c

Skip installing dependencies, when all have been removed.

be8fe13

Use mimetype "application/x-ndjson" as default for bulk requests.

2a05ff1

Rewrite how headers are being reconbined in perform_request method

46c6f19

fressi-elastic force-pushed the esclient-upgrade-2 branch from b9c230c to 46c6f19 Compare January 19, 2026 17:39

fressi-elastic requested a review from Copilot January 19, 2026 19:03

Copilot AI reviewed Jan 19, 2026

View reviewed changes

esrally/client/common.py Show resolved Hide resolved

fressi-elastic added 2 commits January 19, 2026 22:27

Update requests version.

e9018e7

It uses docker compose command instead of the now deprecated old `d…

933bc1d

…ocker-compose`.

fressi-elastic changed the title ~~Upgrade Elastic search client libraries and urlib3.~~ It upgrades Elastic search client and urlib3 libraries. Jan 20, 2026

fressi-elastic added 2 commits January 20, 2026 04:37

It removes unnecessary dependency.

38aafd8

It uses docker compose command instead of the now deprecated old `d…

24f6479

…ocker-compose` (second part).

fressi-elastic force-pushed the esclient-upgrade-2 branch from b1a7027 to 24f6479 Compare January 20, 2026 03:38

fressi-elastic requested a review from Copilot January 20, 2026 03:43

Copilot AI reviewed Jan 20, 2026

View reviewed changes

esrally/track/loader.py Outdated Show resolved Hide resolved

esrally/track/loader.py Outdated Show resolved Hide resolved

fressi-elastic and others added 4 commits January 20, 2026 04:47

Corrected capitalization of 'rally' to 'Rally'.

79e299b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Simplify slice dependency[len(deprecated) : len(deprecated) + 1]

4ec0a24

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Refactor and test function _filter_requirements.

63dabea

Restore docker as Python dependency.

5c86570

fressi-elastic marked this pull request as ready for review January 20, 2026 14:55

fressi-elastic requested review from a team, gareth-ellis and gbanasiak January 20, 2026 14:57

fressi-elastic enabled auto-merge (squash) January 20, 2026 15:30

fressi-elastic disabled auto-merge January 20, 2026 15:30

gareth-ellis approved these changes Jan 22, 2026

View reviewed changes

esrally/client/asynchronous.py Outdated Show resolved Hide resolved

gbanasiak previously requested changes Jan 22, 2026

View reviewed changes

esrally/track/loader.py Outdated Show resolved Hide resolved

Revert wrong typo correction.

b67e108

fressi-elastic mentioned this pull request Jan 22, 2026

Upgrade Elasticsearch client version to 9.2.0 #2003

Closed

It reverts some changes to let client classes better match the origin…

002cabf

…al parent method implementations.

fressi-elastic added 3 commits January 22, 2026 12:18

Use compatibility mode version 8 when server version is unknown.

45309bd

Update minimal supported version in CLI and project documentation.

d86e421

Set constructor hosts parameter default value to None for compati…

1f7eaa4

…bility with `options` method.

fressi-elastic mentioned this pull request Jan 22, 2026

It removes dependencies that are conflicting with rally requirements. elastic/rally-tracks#1000

Merged

fressi-elastic added 2 commits January 22, 2026 14:09

Remove workaround to forbid installation of deprecated packages durin…

73ea086

…g tracks preparation.

It fixes it/docker_dev_image_test.py error handling and update server…

25db777

… version to 9.2.4

fressi-elastic requested a review from gbanasiak January 22, 2026 20:29

gbanasiak reviewed Jan 29, 2026

View reviewed changes

fressi-elastic commented Jan 29, 2026

View reviewed changes

fressi-elastic added 2 commits January 29, 2026 16:41

Merge branch 'master' of github.com:elastic/rally into esclient-upgra…

b725db4

…de-2

Remove _ProductChecker class

6b1132b


		raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(message=message, meta=meta, body=resp_body)

		# 'X-Elastic-Product: Elasticsearch' should be on every 2XX response.

It upgrades Elastic search client and urlib3 libraries. #2024

Are you sure you want to change the base?

It upgrades Elastic search client and urlib3 libraries. #2024

Uh oh!

Conversation

fressi-elastic commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

fressi-elastic commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fressi-elastic commented Jan 20, 2026

Uh oh!

gareth-ellis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gbanasiak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fressi-elastic commented Jan 22, 2026

Uh oh!

gbanasiak left a comment

Choose a reason for hiding this comment

Uh oh!

gbanasiak Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

pquentin Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

fressi-elastic Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

fressi-elastic commented Jan 29, 2026

Uh oh!

fressi-elastic Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

fressi-elastic Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

fressi-elastic Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

fressi-elastic Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

It upgrades Elastic search client and `urlib3` libraries. #2024

It upgrades Elastic search client and `urlib3` libraries. #2024

fressi-elastic commented Jan 13, 2026 •

edited

Loading

fressi-elastic commented Jan 20, 2026 •

edited

Loading