Benchmark tools [kernel-images] #89
Conversation
Mesa Description
curl http://localhost:444/dev/benchmark?components=all
{
"elapsed_seconds":23.927925,
"errors":[],
"results":{
"cdp":{
"concurrent_connections":5,
"direct_endpoint":{
"endpoint_url":"http://localhost:9223",
"latency_ms":{"p50":0.16,"p95":0.346,"p99":0.586},
"scenarios":[
{"category":"Runtime","description":"Evaluate simple JavaScript expression","latency_ms":{"p50":0.159,"p95":0.328,"p99":0.568},"name":"Runtime.evaluate","operation_count":7303,"success_rate":100,"throughput_ops_per_sec":1460.5398}
// omitted ...
],
"throughput_msgs_per_sec":26280.918
},
"latency_ms":{"p50":0.352,"p95":0.576,"p99":0.843},
"memory_mb":{"baseline":1.1285477,"per_connection":0.7372101},
"message_size_bytes":{"avg":100,"max":200,"min":50},
"proxied_endpoint":{
"endpoint_url":"http://localhost:9222",
"latency_ms":{"p50":0.352,"p95":0.576,"p99":0.843},
"scenarios":[
{"category":"Runtime","description":"Evaluate simple JavaScript expression","latency_ms":{"p50":0.349,"p95":0.573,"p99":0.907},"name":"Runtime.evaluate","operation_count":3602,"success_rate":100,"throughput_ops_per_sec":720.31824}
// omitted ...
],
"throughput_msgs_per_sec":12958.929
},
"proxy_overhead_percent":50.690727,
"throughput_msgs_per_sec":12958.929
},
"recording":{
"avg_encoding_lag_ms":67.676765,
"concurrent_recordings":1,
"cpu_overhead_percent":0,
"disk_write_mbps":0.015905762,
"frame_rate_impact":{"before_recording_fps":22.500116,"during_recording_fps":23.468586,"impact_percent":-4.304292},
"frames_captured":131,
"frames_dropped":0,
"memory_overhead_mb":0.050559998
},
"webrtc_live_view":{
"bitrate_kbps":{"audio":0,"total":690.22424,"video":690.22424},
"codecs":{"audio":"unknown","video":"video/VP8"},
"concurrent_viewers":1,
"connection_state":"connected",
"cpu_usage_percent":0,
"frame_latency_ms":{"p50":40.002,"p95":1995.6,"p99":1995.6},
"frame_rate_fps":{"achieved":22.222351,"max":25.005001,"min":0.50110245,"target":30},
"frames":{"corrupted":0,"decoded":400,"dropped":20,"key_frames_decoded":16,"received":400},
"ice_connection_state":"connected",
"jitter_ms":{"audio":0,"video":1},
"memory_mb":{"baseline":19.667969,"per_viewer":19.667969},
"network":{"available_outgoing_bitrate_kbps":0,"bytes_received":0,"bytes_sent":0,"rtt_ms":1},
"packets":{"audio_lost":0,"audio_received":0,"loss_percent":0,"video_lost":0,"video_received":1502},
"resolution":{"height":1080,"width":1920}
}
},
"startup_timing":{
"phase_summary":{"fastest_ms":2,"fastest_phase":"shm_setup","slowest_ms":19259,"slowest_phase":"pulseaudio_start"},
"phases":[
{"duration_ms":2,"name":"shm_setup","percentage":0.010383677},
{"duration_ms":5,"name":"scale_to_zero_disable","percentage":0.025959192},
{"duration_ms":32,"name":"user_dirs_setup","percentage":0.16613883},
{"duration_ms":35,"name":"log_aggregator_start","percentage":0.18171434},
{"duration_ms":313,"name":"supervisord_start","percentage":1.6250454},
{"duration_ms":2620,"name":"xorg_start","percentage":13.602616},
{"duration_ms":5220,"name":"mutter_start","percentage":27.101397},
{"duration_ms":7405,"name":"dbus_start","percentage":38.445564},
{"duration_ms":12909,"name":"chromium_start","percentage":67.02144},
{"duration_ms":16095,"name":"neko_start","percentage":83.56264},
{"duration_ms":19069,"name":"kernel_api_start","percentage":99.00317},
{"duration_ms":19259,"name":"pulseaudio_start","percentage":99.98962}
],
"total_startup_time_ms":19261
},
"system":{"arch":"amd64","cpu_count":16,"memory_total_mb":8,"os":"linux"},
"timestamp":"2025-10-31T18:55:12.601706786-07:00"
}Next Steps :
Description generated by Mesa. Update settings |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| } | ||
|
|
||
| return (float64(deltaTotal) / 100.0) // Convert clock ticks to percentage | ||
| } |
There was a problem hiding this comment.
Bug: Incorrect CPU percent calculation in benchmarks
The CalculateCPUPercent function incorrectly calculates CPU usage. It divides the change in total CPU clock ticks (deltaTotal) by an arbitrary 100.0. This doesn't account for elapsed time or total available CPU time, leading to meaningless CPU usage percentages in benchmarks.
![mesa-dot-dev[bot]](https://avatars.githubusercontent.com/in/1050077?s=60&v=4){kind=small}
There was a problem hiding this comment.
Performed full review of b51da91...8cba71d
Analysis
-
Incorrect CPU Percentage Calculation: The
CalculateCPUPercentfunction inserver/lib/benchmarks/cpu_linux.gouses a flawed calculation method that will produce entirely inaccurate CPU metrics, invalidating benchmark results. -
Misleading Hardcoded Fallback Values: When WebRTC stats collection fails, the system returns hardcoded estimates (28.0 FPS, 35ms latency) instead of errors, masking real failures and potentially providing false confidence in results.
-
Race Condition in WebSocket Communication: The code uses non-null assertions when sending WebSocket data which could fail if the connection closes between the connection check and the send operation.
-
Inconsistent API Design and Silent Error Handling: Several functions ignore errors from stat collection and have confusing signatures (like accepting duration parameters that are ignored).
Tip
Help
Slash Commands:
/review- Request a full code review/review latest- Review only changes since the last review/describe- Generate PR description. This will update the PR body or issue comment depending on your configuration/help- Get help with Mesa commands and configuration options
20 files reviewed | 0 comments | Edit Agent Settings
| this.onConnected() | ||
| // Start WebRTC stats collection | ||
| if (this._peer) { | ||
| this._webrtcStatsCollector.start(this._peer) |
There was a problem hiding this comment.
@raiden-staging just double checking my understanding--the client would always be collecting these stats and sending them to the server, and then when you hit the /dev/benchmark endpoint it observes the incoming metrics from all connected webrtc clients and computes stats?
2 participants
curl http://localhost:444/dev/benchmark?components=all{ "elapsed_seconds":23.927925, "errors":[], "results":{ "cdp":{ "concurrent_connections":5, "direct_endpoint":{ "endpoint_url":"http://localhost:9223", "latency_ms":{"p50":0.16,"p95":0.346,"p99":0.586}, "scenarios":[ {"category":"Runtime","description":"Evaluate simple JavaScript expression","latency_ms":{"p50":0.159,"p95":0.328,"p99":0.568},"name":"Runtime.evaluate","operation_count":7303,"success_rate":100,"throughput_ops_per_sec":1460.5398} // omitted ... ], "throughput_msgs_per_sec":26280.918 }, "latency_ms":{"p50":0.352,"p95":0.576,"p99":0.843}, "memory_mb":{"baseline":1.1285477,"per_connection":0.7372101}, "message_size_bytes":{"avg":100,"max":200,"min":50}, "proxied_endpoint":{ "endpoint_url":"http://localhost:9222", "latency_ms":{"p50":0.352,"p95":0.576,"p99":0.843}, "scenarios":[ {"category":"Runtime","description":"Evaluate simple JavaScript expression","latency_ms":{"p50":0.349,"p95":0.573,"p99":0.907},"name":"Runtime.evaluate","operation_count":3602,"success_rate":100,"throughput_ops_per_sec":720.31824} // omitted ... ], "throughput_msgs_per_sec":12958.929 }, "proxy_overhead_percent":50.690727, "throughput_msgs_per_sec":12958.929 }, "recording":{ "avg_encoding_lag_ms":67.676765, "concurrent_recordings":1, "cpu_overhead_percent":0, "disk_write_mbps":0.015905762, "frame_rate_impact":{"before_recording_fps":22.500116,"during_recording_fps":23.468586,"impact_percent":-4.304292}, "frames_captured":131, "frames_dropped":0, "memory_overhead_mb":0.050559998 }, "webrtc_live_view":{ "bitrate_kbps":{"audio":0,"total":690.22424,"video":690.22424}, "codecs":{"audio":"unknown","video":"video/VP8"}, "concurrent_viewers":1, "connection_state":"connected", "cpu_usage_percent":0, "frame_latency_ms":{"p50":40.002,"p95":1995.6,"p99":1995.6}, "frame_rate_fps":{"achieved":22.222351,"max":25.005001,"min":0.50110245,"target":30}, "frames":{"corrupted":0,"decoded":400,"dropped":20,"key_frames_decoded":16,"received":400}, "ice_connection_state":"connected", "jitter_ms":{"audio":0,"video":1}, "memory_mb":{"baseline":19.667969,"per_viewer":19.667969}, "network":{"available_outgoing_bitrate_kbps":0,"bytes_received":0,"bytes_sent":0,"rtt_ms":1}, "packets":{"audio_lost":0,"audio_received":0,"loss_percent":0,"video_lost":0,"video_received":1502}, "resolution":{"height":1080,"width":1920} } }, "startup_timing":{ "phase_summary":{"fastest_ms":2,"fastest_phase":"shm_setup","slowest_ms":19259,"slowest_phase":"pulseaudio_start"}, "phases":[ {"duration_ms":2,"name":"shm_setup","percentage":0.010383677}, {"duration_ms":5,"name":"scale_to_zero_disable","percentage":0.025959192}, {"duration_ms":32,"name":"user_dirs_setup","percentage":0.16613883}, {"duration_ms":35,"name":"log_aggregator_start","percentage":0.18171434}, {"duration_ms":313,"name":"supervisord_start","percentage":1.6250454}, {"duration_ms":2620,"name":"xorg_start","percentage":13.602616}, {"duration_ms":5220,"name":"mutter_start","percentage":27.101397}, {"duration_ms":7405,"name":"dbus_start","percentage":38.445564}, {"duration_ms":12909,"name":"chromium_start","percentage":67.02144}, {"duration_ms":16095,"name":"neko_start","percentage":83.56264}, {"duration_ms":19069,"name":"kernel_api_start","percentage":99.00317}, {"duration_ms":19259,"name":"pulseaudio_start","percentage":99.98962} ], "total_startup_time_ms":19261 }, "system":{"arch":"amd64","cpu_count":16,"memory_total_mb":8,"os":"linux"}, "timestamp":"2025-10-31T18:55:12.601706786-07:00" }Next Steps :
onkernel/nekoBenchmark tools [neko])v3.0.8-v1.3.1onkernel/kernel-imagesBenchmark tools [kernel-images]). Notes :Dockerfilealready set to ghcr.io/onkernel/neko/base:3.0.8-v1.3.1onkernel/neko:v3.0.8-v1.3.1tag release and build completion, would use new neko build without needing changes[ @Sayan- @rgarcia ]
Note
Introduces a full benchmarking suite (CDP, WebRTC, recording) exposed via
/dev/benchmark, adds client-side WebRTC stats reporting, container/server startup timing, and regenerates OpenAPI and proxy integrations.GET /dev/benchmarkreturning consolidated results (BenchmarkResults) with system info, per-component metrics, and startup timing./tmp/kernel_startup_timing.json./json/versionand adds bench tests.GetStderr()for profiling.webrtc-stats-collectorand sends periodicbenchmark/webrtc_statsover WS; integrates start/stop with ICE state; updates events/messages.ghcr.io/onkernel/neko/base:3.0.8-v1.3.1and wires artifacts.Written by Cursor Bugbot for commit 8cba71d. This will update automatically on new commits. Configure here.