From 3528e67ed4497a3ee0eec92443647a26300ff56a Mon Sep 17 00:00:00 2001 From: Edoardo Tenani Date: Fri, 5 Dec 2025 18:43:43 +0100 Subject: [PATCH 1/5] document setting nofile limit on different systems --- .../observability/apm/apm-server/binary.md | 56 ++++++++++++++++++- .../observability/apm/apm-server/systemd.md | 10 ++++ .../apm/apm-server/tail-based-sampling.md | 8 +++ 3 files changed, 73 insertions(+), 1 deletion(-) diff --git a/solutions/observability/apm/apm-server/binary.md b/solutions/observability/apm/apm-server/binary.md index fca7859a40..f871febfa9 100644 --- a/solutions/observability/apm/apm-server/binary.md +++ b/solutions/observability/apm/apm-server/binary.md @@ -61,6 +61,8 @@ curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-{{versio tar xzvf apm-server-{{version.stack}}-linux-x86_64.tar.gz ``` +See [modifying the `nofile` ulimit](#modifying-nofile-ulimit). + $$$apm-mac$$$ **Mac:** @@ -926,4 +928,56 @@ It’s possible to embed your APM Server configuration in a custom image. Here i ```dockerfile FROM docker.elastic.co/apm/apm-server:9.0.0 COPY --chmod=0644 --chown=1000:1000 apm-server.yml /usr/share/apm-server/apm-server.yml -``` \ No newline at end of file +``` + +#### Modifying `nofile` ulimit [ulimit-on-docker] + +Limits can be set from the command line using `--ulimit=soft:hard`, see [Set ulimits in container (--ulimit)](https://docs.docker.com/reference/cli/docker/container/run/#ulimit). + +```sh +docker run -d \ + -p 8200:8200 \ + --name=apm-server \ + --user=apm-server \ + --volume="$(pwd)/apm-server.docker.yml:/usr/share/apm-server/apm-server.yml:ro" \ + docker.elastic.co/apm/apm-server:9.0.0 \ + --strict.perms=false -e \ + --ulimit=524287:524287 \ + -E output.elasticsearch.hosts=["elasticsearch:9200"] <1> <2> +``` + +1. Substitute your {{es}} hosts and ports. +2. If you are using {{ech}}, replace the `-E output.elasticsearch.hosts` line with the Cloud ID and elastic password using the syntax shown earlier. + + +## Modify the `nofile` ulimit [modify-nofile-ulimit] + +When run as a standalone binary APM Server will inherit the `nofile` limit from the user running the process. On most system this is configured to `1024`. This limit is too low for higher throughput scenarios or when using Tail Based Sampling. + +To chose the new limit, consider these guidelines: +- there is no system performance impact of using a limit of ``; +- a limit of `1024` would suffice for low throughput use cases; +- the major contributor to open files will be the number of incoming connections; +- Tail Based Sampling is file based, when enabling it the number of open files will be higher in proportion to the throughput and sampling policies. + +To configure the limit for your user, you need to know the username you will run APM Server process with. + +```sh +whoami +``` + +Edit `/etc/security/limits.conf` with root privileges: + +```sh +sudo nano /etc/security/limits.conf +``` + +Add the following lines to set soft and hard limits for your user: + +```text +apm-server soft nofile 524287 <1> +apm-server hard nofile 524287 <1> +``` + +1. Replace `apm-server` with the username you will run APM Server process with. + diff --git a/solutions/observability/apm/apm-server/systemd.md b/solutions/observability/apm/apm-server/systemd.md index b668571995..c93ea691fe 100644 --- a/solutions/observability/apm/apm-server/systemd.md +++ b/solutions/observability/apm/apm-server/systemd.md @@ -88,6 +88,16 @@ systemctl restart apm-server It is recommended that you use a configuration management tool to include drop-in unit files. If you need to add a drop-in manually, use `systemctl edit apm-server.service`. :::: +#### Configuring the NOFILE limit [configuring-nofile-limit] + +::::{note} +There should be no need to manually configure this limit when running APM Server. +:::: + +In systemd the `LimitNOFILE` defaults are set to `1024` (soft) and `524288` (hard) and most Linux systems with systemd will not change these values or reduce them drastically. Golang since 1.19 (see [golang/go#46279](https://github.com/golang/go/issues/46279)) automatically bump the process limit up to the available hard limit. This means that by default APM Server runs with the limit set to the hard limit value by the Operating System is being run on, generally `524287` on a recent system. There should be no reason to change this limit, as back-pressure from too many open files will happen from memory usage. + +For guidelines on the value to set this value to see [Modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit). + #### Configuration file ownership [apm-config-file-ownership] On systems with POSIX file permissions, the APM Server configuration file is subject to ownership and file permission checks. These checks prevent unauthorized users from providing or modifying configurations that are run by APM Server. diff --git a/solutions/observability/apm/apm-server/tail-based-sampling.md b/solutions/observability/apm/apm-server/tail-based-sampling.md index 7faede1e07..c4bacae96e 100644 --- a/solutions/observability/apm/apm-server/tail-based-sampling.md +++ b/solutions/observability/apm/apm-server/tail-based-sampling.md @@ -20,6 +20,14 @@ Most options on this page are supported by all APM Server deployment methods whe Enhanced privileges are required to use tail-based sampling. For more information, refer to [Create a tail-based sampling role](/solutions/observability/apm/create-assign-feature-roles-to-apm-server-users.md#apm-privileges-tail-based-sampling). :::: +::::{note} +If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this may result in a `too many open files` error. Please see [configuring the NOFILE limit](/solutions/observability/apm/systemd.md#configuring-nofile-limit). +:::: + +::::{note} +If you are running the binary standalone (not using the provided dev or rpm packages or the docker images) you need to adjust the `nofile` limit based on your throughput requirements. See [modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit). +:::: + Tail-based sampling configuration options. :::::::{tab-set} From 81deb75881349ca11216e572598a300e9a103b88 Mon Sep 17 00:00:00 2001 From: Edoardo Tenani Date: Fri, 5 Dec 2025 18:48:25 +0100 Subject: [PATCH 2/5] document updating limits on running process --- solutions/observability/apm/apm-server/binary.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/solutions/observability/apm/apm-server/binary.md b/solutions/observability/apm/apm-server/binary.md index f871febfa9..999f56c062 100644 --- a/solutions/observability/apm/apm-server/binary.md +++ b/solutions/observability/apm/apm-server/binary.md @@ -981,3 +981,16 @@ apm-server hard nofile 524287 <1> 1. Replace `apm-server` with the username you will run APM Server process with. +To update the `nofile` ulimit of a running process you need to know the PID + +```sh +pgrep -f apm-server +``` + +Then apply the new limits: + +```sh +prlimit --pid PID --nofile=524287:524287 <1> +``` +1. Replace `PID` with your APM Server process PID. + From 1db9f0b8f022696a0945965ce14743302fca83e7 Mon Sep 17 00:00:00 2001 From: Edoardo Tenani Date: Tue, 9 Dec 2025 09:57:13 +0100 Subject: [PATCH 3/5] accept docs suggestions --- .../observability/apm/apm-server/binary.md | 18 +++++++++--------- .../observability/apm/apm-server/systemd.md | 4 ++-- .../apm/apm-server/tail-based-sampling.md | 4 ++-- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/solutions/observability/apm/apm-server/binary.md b/solutions/observability/apm/apm-server/binary.md index 999f56c062..9d52f0015b 100644 --- a/solutions/observability/apm/apm-server/binary.md +++ b/solutions/observability/apm/apm-server/binary.md @@ -61,7 +61,7 @@ curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-{{versio tar xzvf apm-server-{{version.stack}}-linux-x86_64.tar.gz ``` -See [modifying the `nofile` ulimit](#modifying-nofile-ulimit). +Refer to [modifying the `nofile` ulimit](#modifying-nofile-ulimit). $$$apm-mac$$$ **Mac:** @@ -932,7 +932,7 @@ COPY --chmod=0644 --chown=1000:1000 apm-server.yml /usr/share/apm-server/apm-ser #### Modifying `nofile` ulimit [ulimit-on-docker] -Limits can be set from the command line using `--ulimit=soft:hard`, see [Set ulimits in container (--ulimit)](https://docs.docker.com/reference/cli/docker/container/run/#ulimit). +Limits can be set from the command line using `--ulimit=soft:hard`, refer to [Set ulimits in container (--ulimit)](https://docs.docker.com/reference/cli/docker/container/run/#ulimit) in the Docker documentation. ```sh docker run -d \ @@ -952,15 +952,15 @@ docker run -d \ ## Modify the `nofile` ulimit [modify-nofile-ulimit] -When run as a standalone binary APM Server will inherit the `nofile` limit from the user running the process. On most system this is configured to `1024`. This limit is too low for higher throughput scenarios or when using Tail Based Sampling. +When run as a standalone binary APM Server inherits the `nofile` limit from the user running the process. On most system this is configured to `1024`. This limit is too low for higher throughput scenarios or when using Tail Based Sampling. To chose the new limit, consider these guidelines: - there is no system performance impact of using a limit of ``; - a limit of `1024` would suffice for low throughput use cases; -- the major contributor to open files will be the number of incoming connections; -- Tail Based Sampling is file based, when enabling it the number of open files will be higher in proportion to the throughput and sampling policies. +- the major contributor to open files is the number of incoming connections; +- Tail Based Sampling is file based, when enabling it the number of open files is higher in proportion to the throughput and sampling policies. -To configure the limit for your user, you need to know the username you will run APM Server process with. +To configure the limit for your user, you need to know the username you run APM Server process with. ```sh whoami @@ -979,9 +979,9 @@ apm-server soft nofile 524287 <1> apm-server hard nofile 524287 <1> ``` -1. Replace `apm-server` with the username you will run APM Server process with. +1. Replace `apm-server` with the username you run APM Server process with. -To update the `nofile` ulimit of a running process you need to know the PID +To update the `nofile` ulimit of a running process you need to know the process ID (PID): ```sh pgrep -f apm-server @@ -992,5 +992,5 @@ Then apply the new limits: ```sh prlimit --pid PID --nofile=524287:524287 <1> ``` -1. Replace `PID` with your APM Server process PID. +1. Replace `PID` with your APM Server process ID. diff --git a/solutions/observability/apm/apm-server/systemd.md b/solutions/observability/apm/apm-server/systemd.md index c93ea691fe..1545d12540 100644 --- a/solutions/observability/apm/apm-server/systemd.md +++ b/solutions/observability/apm/apm-server/systemd.md @@ -94,9 +94,9 @@ It is recommended that you use a configuration management tool to include drop-i There should be no need to manually configure this limit when running APM Server. :::: -In systemd the `LimitNOFILE` defaults are set to `1024` (soft) and `524288` (hard) and most Linux systems with systemd will not change these values or reduce them drastically. Golang since 1.19 (see [golang/go#46279](https://github.com/golang/go/issues/46279)) automatically bump the process limit up to the available hard limit. This means that by default APM Server runs with the limit set to the hard limit value by the Operating System is being run on, generally `524287` on a recent system. There should be no reason to change this limit, as back-pressure from too many open files will happen from memory usage. +In systemd the `LimitNOFILE` defaults are set to `1024` (soft) and `524288` (hard) and most Linux systems with systemd do not change these values or reduce them drastically. Golang, starting from version 1.19 (refer to [golang/go#46279](https://github.com/golang/go/issues/46279)), automatically bump the process limit up to the available hard limit. This means that by default APM Server runs with the limit set to the hard limit value by the Operating System is being run on, generally `524287` on a recent system. There should be no reason to change this limit, as back-pressure from too many open files happens through memory usage. -For guidelines on the value to set this value to see [Modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit). +For guidelines on the value to set this value to refer to [modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit). #### Configuration file ownership [apm-config-file-ownership] diff --git a/solutions/observability/apm/apm-server/tail-based-sampling.md b/solutions/observability/apm/apm-server/tail-based-sampling.md index c4bacae96e..7ecaf3da57 100644 --- a/solutions/observability/apm/apm-server/tail-based-sampling.md +++ b/solutions/observability/apm/apm-server/tail-based-sampling.md @@ -21,11 +21,11 @@ Enhanced privileges are required to use tail-based sampling. For more informatio :::: ::::{note} -If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this may result in a `too many open files` error. Please see [configuring the NOFILE limit](/solutions/observability/apm/systemd.md#configuring-nofile-limit). +If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this can result in a `too many open files` error. Refert to [configuring the NOFILE limit](/solutions/observability/apm/systemd.md#configurin-nofile-limit) for further instructions. :::: ::::{note} -If you are running the binary standalone (not using the provided dev or rpm packages or the docker images) you need to adjust the `nofile` limit based on your throughput requirements. See [modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit). +If you are running the binary standalone (not using the provided dev or rpm packages or the docker image) you need to adjust the `nofile` limit based on your throughput requirements. Refer to [modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit). :::: Tail-based sampling configuration options. From 2639663bab3e41caa6e04286ecbd2f5e936b9ac3 Mon Sep 17 00:00:00 2001 From: Edoardo Tenani Date: Tue, 9 Dec 2025 10:09:54 +0100 Subject: [PATCH 4/5] fix links --- solutions/observability/apm/apm-server/binary.md | 2 +- solutions/observability/apm/apm-server/tail-based-sampling.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/solutions/observability/apm/apm-server/binary.md b/solutions/observability/apm/apm-server/binary.md index 9d52f0015b..abfdcbb369 100644 --- a/solutions/observability/apm/apm-server/binary.md +++ b/solutions/observability/apm/apm-server/binary.md @@ -61,7 +61,7 @@ curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-{{versio tar xzvf apm-server-{{version.stack}}-linux-x86_64.tar.gz ``` -Refer to [modifying the `nofile` ulimit](#modifying-nofile-ulimit). +Refer to [modifying the `nofile` ulimit](#modify-nofile-ulimit). $$$apm-mac$$$ **Mac:** diff --git a/solutions/observability/apm/apm-server/tail-based-sampling.md b/solutions/observability/apm/apm-server/tail-based-sampling.md index 7ecaf3da57..326fc3ef12 100644 --- a/solutions/observability/apm/apm-server/tail-based-sampling.md +++ b/solutions/observability/apm/apm-server/tail-based-sampling.md @@ -21,7 +21,7 @@ Enhanced privileges are required to use tail-based sampling. For more informatio :::: ::::{note} -If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this can result in a `too many open files` error. Refert to [configuring the NOFILE limit](/solutions/observability/apm/systemd.md#configurin-nofile-limit) for further instructions. +If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this can result in a `too many open files` error. Refert to [configuring the NOFILE limit](/solutions/observability/apm/apm-server/systemd.md#configurin-nofile-limit) for further instructions. :::: ::::{note} From ced3c6c394ad8f542807a8381304437eaf3a67ef Mon Sep 17 00:00:00 2001 From: Edoardo Tenani Date: Tue, 9 Dec 2025 10:14:45 +0100 Subject: [PATCH 5/5] fix links --- solutions/observability/apm/apm-server/tail-based-sampling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/solutions/observability/apm/apm-server/tail-based-sampling.md b/solutions/observability/apm/apm-server/tail-based-sampling.md index 326fc3ef12..a3a48d7ee0 100644 --- a/solutions/observability/apm/apm-server/tail-based-sampling.md +++ b/solutions/observability/apm/apm-server/tail-based-sampling.md @@ -21,7 +21,7 @@ Enhanced privileges are required to use tail-based sampling. For more informatio :::: ::::{note} -If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this can result in a `too many open files` error. Refert to [configuring the NOFILE limit](/solutions/observability/apm/apm-server/systemd.md#configurin-nofile-limit) for further instructions. +If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this can result in a `too many open files` error. Refert to [configuring the NOFILE limit](/solutions/observability/apm/apm-server/systemd.md#configuring-nofile-limit) for further instructions. :::: ::::{note}