CPU throttling on PromProxy node with 1300+ agents

[mhussain584]

Hi Paul,
Wishes
Hope you are doing well

Again thanks for your help with release of 1.23.1 last time which fixed my issue.

Now, i've encountered another issue, where i am sending metrics of 1300+ agents with like 500+ dummy agents, prometheus-proxy service is experiencing CPU throttling

Current Node Resource

8 Cores 
32 GB memory

I've upgraded to 2.1.0 for both agent and proxy, still the throttling is evident

PromPorxy logs

  16:33:30.299 INFO  [CallLogging.kt:48] - 200 OK: GET - /blmxx04_hssidsmac_virtual_09_10_147_233_111_hs_metrics - ip-100-65-70-119.ec2.internal [DefaultDispatcher-worker-31]
16:33:30.302 INFO  [CallLogging.kt:48] - 200 OK: GET - /blmxx06_hssidsmac_virtual_09_10_147_233_117_hs_metrics - ip-100-65-70-119.ec2.internal [DefaultDispatcher-worker-36]
16:33:30.305 INFO  [CallLogging.kt:48] - 503 Service Unavailable: GET - /blmxx03_hssidsmac_virtual_15_10_147_233_105_hs_metrics - ip-100-65-70-119.ec2.internal [DefaultDispatcher-worker-62]
16:33:30.305 INFO  [ProxyHttpConfig.kt:103] -  Throwable caught: ClosedByteChannelException [DefaultDispatcher-worker-7]
io.ktor.utils.io.ClosedByteChannelException: Broken pipe
  at io.ktor.utils.io.CloseToken$wrapCause$1.invoke(CloseToken.kt:16)
  at io.ktor.utils.io.CloseToken$wrapCause$1.invoke(CloseToken.kt:16)
  at io.ktor.utils.io.CloseToken.wrapCause(CloseToken.kt:21)
  at io.ktor.utils.io.CloseToken.wrapCause$default(CloseToken.kt:16)
  at io.ktor.utils.io.ByteChannel.getClosedCause(ByteChannel.kt:61)
  at io.ktor.utils.io.ByteReadChannelOperationsKt.rethrowCloseCauseIfNeeded(ByteReadChannelOperations.kt:543)
  at io.ktor.utils.io.ByteChannel.flush(ByteChannel.kt:94)
  at io.ktor.utils.io.CloseHookByteWriteChannel.flush(CloseHookByteWriteChannel.kt)
  at io.ktor.utils.io.ByteReadChannelOperationsKt.copyTo(ByteReadChannelOperations.kt:206)
  at io.ktor.utils.io.ByteReadChannelOperationsKt$copyTo$2.invokeSuspend(ByteReadChannelOperations.kt)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
  at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:233)
  at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:175)
  at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:147)
  at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:470)
  at kotlinx.coroutines.CancellableContinuationImpl.resumeImpl$kotlinx_coroutines_core(CancellableContinuationImpl.kt:504)
  at kotlinx.coroutines.CancellableContinuationImpl.resumeImpl$kotlinx_coroutines_core$default(CancellableContinuationImpl.kt:493)
  at kotlinx.coroutines.CancellableContinuationImpl.resumeWith(CancellableContinuationImpl.kt:359)
  at io.ktor.utils.io.ByteChannel$Slot$Task$DefaultImpls.resume(ByteChannel.kt:239)
  at io.ktor.utils.io.ByteChannel$Slot$Read.resume(ByteChannel.kt:242)
  at io.ktor.utils.io.ByteChannel.closeSlot(ByteChannel.kt:177)
  at io.ktor.utils.io.ByteChannel.flushAndClose(ByteChannel.kt:133)
  at io.ktor.utils.io.ByteWriteChannelOperationsKt$writer$job$1.invokeSuspend(ByteWriteChannelOperations.kt:184)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
  at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
Caused by: io.ktor.utils.io.ClosedWriteChannelException: Broken pipe
  at io.ktor.utils.io.ByteChannel$writeBuffer$1.invoke(ByteChannel.kt:54)
  at io.ktor.utils.io.ByteChannel$writeBuffer$1.invoke(ByteChannel.kt:54)
  at io.ktor.utils.io.CloseToken.wrapCause(CloseToken.kt:21)
  at io.ktor.utils.io.CloseToken.throwOrNull(CloseToken.kt:26)
  at io.ktor.utils.io.ByteChannel.getWriteBuffer(ByteChannel.kt:54)
  at io.ktor.utils.io.ByteWriteChannelOperationsKt.writeByte(ByteWriteChannelOperations.kt:19)
  at io.ktor.http.cio.internals.CharsKt.writeIntHex(Chars.kt:110)
  at io.ktor.http.cio.ChunkedTransferEncodingKt.writeChunk(ChunkedTransferEncoding.kt:167)
  at io.ktor.http.cio.ChunkedTransferEncodingKt.access$writeChunk(ChunkedTransferEncoding.kt:1)
  at io.ktor.http.cio.ChunkedTransferEncodingKt.encodeChunked(ChunkedTransferEncoding.kt:138)
  at io.ktor.http.cio.ChunkedTransferEncodingKt$encodeChunked$2.invokeSuspend(ChunkedTransferEncoding.kt)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
  at kotlinx.coroutines.EventLoop.processUnconfinedEvent(EventLoop.common.kt:65)
  at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:243)
  at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:147)
  at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:470)
  at kotlinx.coroutines.CancellableContinuationImpl.resumeImpl$kotlinx_coroutines_core(CancellableContinuationImpl.kt:504)
  at kotlinx.coroutines.CancellableContinuationImpl.resumeImpl$kotlinx_coroutines_core$default(CancellableContinuationImpl.kt:493)
  at kotlinx.coroutines.CancellableContinuationImpl.resumeWith(CancellableContinuationImpl.kt:359)
  at io.ktor.utils.io.ByteChannel$Slot$Task$DefaultImpls.resume(ByteChannel.kt:236)
  at io.ktor.utils.io.ByteChannel$Slot$Read.resume(ByteChannel.kt:242)
  at io.ktor.utils.io.ByteChannel.flushWriteBuffer(ByteChannel.kt:389)
  at io.ktor.utils.io.ByteChannel.flush(ByteChannel.kt:96)
  at io.ktor.utils.io.ByteChannel.flushAndClose(ByteChannel.kt:128)
  ... 7 common frames omitted
Caused by: io.ktor.utils.io.ClosedWriteChannelException: Broken pipe
  at io.ktor.utils.io.ByteChannel$writeBuffer$1.invoke(ByteChannel.kt:54)
  at io.ktor.utils.io.ByteChannel$writeBuffer$1.invoke(ByteChannel.kt:54)
  at io.ktor.utils.io.CloseToken.wrapCause(CloseToken.kt:21)
  at io.ktor.utils.io.CloseToken.throwOrNull(CloseToken.kt:26)
  at io.ktor.utils.io.ByteChannel.getWriteBuffer(ByteChannel.kt:54)
  at io.ktor.utils.io.ByteReadChannelOperationsKt.copyTo(ByteReadChannelOperations.kt:175)
  at io.ktor.utils.io.ByteReadChannelOperationsKt$copyTo$1.invokeSuspend(ByteReadChannelOperations.kt)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
  at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113)
  at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
  at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
  ... 2 common frames omitted
Caused by: java.io.IOException: Broken pipe
  at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
  at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62)
  at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
  at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:97)
  at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53)
  at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532)
  at io.ktor.network.sockets.CIOWriterKt$attachForWritingDirectImpl$1.invokeSuspend$lambda$1(CIOWriter.kt:42)
  at io.ktor.utils.io.core.ByteReadPacketExtensions_jvmKt.read(ByteReadPacketExtensions.jvm.kt:30)
  at io.ktor.network.sockets.CIOWriterKt$attachForWritingDirectImpl$1.invokeSuspend(CIOWriter.kt:78)
  ... 8 common frames omitted
16:33:30.306 INFO  [CallLogging.kt:48] - 200 OK: GET - /blmxx03_hssidsmac_virtual_04_10_147_234_71_hs_metrics - ip-100-65-70-119.ec2.internal [DefaultDispatcher-worker-7]

Container resource usage

Prom-agent.conf (Just a snippet)

proxy {
  admin.debugEnabled = true

  admin.enabled: true
  metrics.enabled: true

  http.requestLoggingEnabled: true
}

agent {

  proxy.hostname: "promproxy-us-east-1.aws-prod.examplecom:50051"
  admin.enabled: true
  metrics.enabled: true

  pathConfigs: [
    {
      name: "Agent System metrics"
      path: blmxx_agent_sys_metrics
      url: "http://10.147.92.199:9100/metrics"
    },
    {
      name: "Agent metrics"
      path: blmxx_agent_metrics
      labels: "{\"cloud\": \"blmxx\"}"
      url: "http://localhost:8083/metrics"
    },
    {
      name: "blmxx01.iosserver-5xxxxxxxx38.app_metrics"
      path: "blmxx01_iosserver_app_metrics_10_147_234_98_9091"
      labels: "{\"cloud\": \"blmxx01\", \"service\": \"iosserver\", \"ip\": \"10.147.234.98\"}"
      url: "http://10.147.234.98:9091/metrics"
    },
    {
      name: "blmxx01.iosserver-5xxxxxxxx38.sys_metrics"
      path: "blmxx01_iosserver_sys_metrics_10_147_234_98_9100"
      labels: "{\"cloud\": \"blmxx01\", \"service\": \"iosserver\", \"ip\": \"10.147.234.98\"}"
      url: "http://10.147.234.98:9100/metrics"
    }
,
    {
      name: "blmxx01.iosserver-5xxxxxxxx38VIRTUAL_01"
      path: "blmxx01_iosserver_virtual_01_10_147_234_98_hs_metrics"
      labels: "{\"cloud\": \"blmxx01\", \"service\": \"iosserver\", \"ip\": \"10.147.234.98\", \"cradle\": \"01\"}"
      url: "http://10.147.234.98/VIRTUAL/01/hs/metrics"
    },
    {
      name: "blmxx01.iosserver-5xxxxxxxx38.VIRTUAL_02"
      path: "blmxx01_iosserver_virtual_02_10_147_234_98_hs_metrics"
      labels: "{\"cloud\": \"blmxx01\", \"service\": \"iosserver\", \"ip\": \"10.147.234.98\", \"cradle\": \"02\"}"
      url: "http://10.147.234.98/VIRTUAL/02/hs/metrics"
    },
    {
      name: "blmcm01.iosserver-5xxxxxxxx38.VIRTUAL_03"
      path: "blmxx01_iosserver_virtual_03_10_147_234_98_hs_metrics"
      labels: "{\"cloud\": \"blmxx01\", \"service\": \"iosserver\", \"ip\": \"10.147.234.98\", \"cradle\": \"03\"}"
      url: "http://10.147.234.98/VIRTUAL/03/hs/metrics"
    },
    {
      name: "blmxx01.iosserver-5xxxxxxxx38.VIRTUAL_04"
      path: "blmxx01_iosserver_virtual_04_10_147_234_98_hs_metrics"
      labels: "{\"cloud\": \"blmxx01\", \"service\": \"iosserver\", \"ip\": \"10.147.234.98\", \"cradle\": \"04\"}"
      url: "http://10.147.234.98/VIRTUAL/04/hs/metrics"
    },
    {
      name: "blmxx01.iosserver-5xxxxxxxx38.VIRTUAL_05"
      path: "blmxx01_iosserver_virtual_05_10_147_234_98_hs_metrics"
      labels: "{\"cloud\": \"blmxx01\", \"service\": \"iosserver\", \"ip\": \"10.147.234.98\", \"cradle\": \"05\"}"
      url: "http://10.147.234.98/VIRTUAL/05/hs/metrics"
    },
    {
      name: "blmxx0.iosserver-5xxxxxxxx38.VIRTUAL_06"
      path: "blmxx01_iosserver_virtual_06_10_147_234_98_hs_metrics"
      labels: "{\"cloud\": \"blmxx01\", \"service\": \"iosserver\", \"ip\": \"10.147.234.98\", \"cradle\": \"06\"}"
      url: "http://10.147.234.98/VIRTUAL/06/hs/metrics"
    },
    {

Your help on this will be appreciated

Thanks

[pambrose]

Weird. So with fewer agents you are not seeing problems?

[mhussain584]

Can you RE-open the case i've accidently closed it

[mhussain584]

Weird. So with fewer agents you are not seeing problems?

Yes , with fewer agents it works fine

Could it be due to unlimited coroutines getting created for large number of paths leading to this?

Does the proxy fans out to all registered paths with unbounded coroutines?

Probably the dispatcher is getting overwhelmed due to these thousands of coroutines, i've even tried to upscale the node to 16 cores but capacity doesn't seem to be the issue

Also, i've tried to run a benchmark (ab), with single path and it works fine , looks like hundreds of paths / endpoints (coroutines) for agents are saturating it

➜  ~ ab -n 1500 -c 100 http://promproxy-us-east-1.aws-prod.example.com:8080/metrics
This is ApacheBench, Version 2.3 <$Revision: 1913912 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking promproxy-us-east-1.aws-prod.example.com (be patient)
Completed 150 requests
Completed 300 requests
Completed 450 requests
Completed 600 requests
Completed 750 requests
Completed 900 requests
Completed 1050 requests
Completed 1200 requests
Completed 1350 requests
Completed 1500 requests
Finished 1500 requests


Server Software:        Ktor/3.1.1
Server Hostname:        promproxy-us-east-1.aws-prod.example.com
Server Port:            8080

Document Path:          /metrics
Document Length:        13 bytes

Concurrency Level:      100
Time taken for tests:   11.734 seconds
Complete requests:      1500
Failed requests:        0
Non-2xx responses:      1500
Total transferred:      382500 bytes
HTML transferred:       19500 bytes
Requests per second:    127.84 [#/sec] (mean)
Time per request:       782.251 [ms] (mean)
Time per request:       7.823 [ms] (mean, across all concurrent requests)
Transfer rate:          31.83 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      356  361   4.5    359     384
Processing:   357  368  11.6    363     420
Waiting:      357  367  10.7    363     419
Total:        714  729  13.4    723     777

Percentage of the requests served within a certain time (ms)
  50%    723
  66%    726
  75%    735
  80%    740
  90%    750
  95%    758
  98%    765
  99%    772
 100%    777 (longest request)
➜  ~

[pambrose]

Are you running the proxy in a docker container? I am wondering if we need to adjust the jvm flags.

[mhussain584]

Yes , I am running it in a docker container

[mhussain584]

What JVM conf should be the appropriate?

  -e JAVA_OPTS="-Xms4g -Xmx8g -XX:+UseG1GC -XX:MaxGCPauseMillis=100"

Or may be

docker run --rm \
  -p 8082:8082 -p 8092:8092 -p 50051:50051 -p 8080:8080 \
  --env ADMIN_ENABLED=true \
  --env METRICS_ENABLED=true \
  --env JAVA_TOOL_OPTIONS="-Xms4g -Xmx8g -XX:+UseG1GC -XX:MaxGCPauseMillis=100" \
  pambrose/prometheus-proxy:2.1.0

[pambrose]

Coroutines are lightweight, so I am not worried about that.

My guess is the issue is either with the JVM settings or the gRPC settings. I see the failure in the stacktrace above, but do we have any other clues about what the failure is?

[pambrose]

And when you say there is throttling, are you referring to the failure being seen at a sertain level of clients, i.e., not something more specific than that?

[mhussain584]

I am able to see those error traces when the CPU scores over 100% and stays there for a while, other than i cannot find any other exception , i can take your help to validate JVM settings, anything else that i am missing

/app $ java -XX:+PrintFlagsFinal -version | grep HeapSize
   size_t ErgoHeapSizeLimit                        = 0                                         {product} {default}
   size_t HeapSizePerGCThread                      = 43620760                                  {product} {default}
   size_t InitialHeapSize                          = 515899392                                 {product} {ergonomic}
   size_t LargePageHeapSizeThreshold               = 134217728                                 {product} {default}
   size_t MaxHeapSize                              = 8229224448                                {product} {ergonomic}
   size_t MinHeapSize                              = 8388608                                   {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 7602480                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 122027880                              {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 122027880                              {pd product} {ergonomic}
   size_t SoftMaxHeapSize                          = 8229224448                             {manageable} {ergonomic}
openjdk version "17.0.14" 2025-01-21
OpenJDK Runtime Environment (build 17.0.14+7-alpine-r0)
OpenJDK 64-Bit Server VM (build 17.0.14+7-alpine-r0, mixed mode, sharing)
/app $

[mhussain584]

And when you say there is throttling, are you referring to the failure being seen at a sertain level of clients, i.e., not something more specific than that?

The failure is not particular with certain clients, as i said earlier, there are 1300+ endpoints out of which some are dummy and its not that the failure is occurring on the dummy ones

[pkazi]

Hi @pambrose
I work with @mhussain584 , so know the context behind this issue.
Just 1 correction here - We have only 1 prom-agent, but it has 1300+ different pathConfigs, (not 1300 agents)

About your question, we haven't checked if failures are seen on certain number of clients.
It was working fine with around 75 pathConfigs, today we added 1250+ more pathConfigs, and saw this issue.
Prometheus gives scrape error as -

Get "http://promproxy-us-east-1.aws-prod.example.com:8080/blmxxx_hssidsmac_virtual_11_10_147_232_104_hs_metrics": context deadline exceeded

[pambrose]

Hi @pkazi,

That error looks helpful. Do we know who is giving that message?
When I google "context deadline exceeded" I see all the usual suspects. It doesn't sound JVM or gRPC related, though.

Does this look like a possibility?
https://signoz.io/guides/context-deadline-exceeded-prometheus/

[pambrose]

Also, if we are unable to get to the bottom of this, would it be possible to run multiple prom-agents and distribute the 1300+ pathConfigs across them? Do not get me wrong, I want to identify the underlying issue.