Panic when sorting > 600,000 rows

[o11ika]

I am running query on a table with about 600,000 rows

SELECT
*
FROM node_cpu_guest_seconds_total
ORDER BY greptime_timestamp DESC ;
LIMIT 5;

and it's crushes the database:

name = "GreptimeDB"
operating_system = "Ubuntu 22.4.0 [64-bit]"
crate_version = "1.0.0-beta.2"
explanation = """
Panic occurred in file '/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-buffer-56.2.0/src/buffer/scalar.rs' at line 70
"""
cause = "offset overflow"
method = "Panic"
backtrace = """
0: 0xaaaae8764fd8 - human_panic::report::Report::new::ha542f0d0febcaf94
1: 0xaaaae8766400 - human_panic::handle_dump::h72ed77e41fc10e8b
2: 0xaaaae839cc78 - human_panic::setup_panic::{{closure}}::h7f57250ba85124f5
3: 0xaaaae2f25a2c - common_telemetry::panic_hook::set_panic_hook::{{closure}}::h0bf8bf589b3c80a1
4: 0xaaaaecca7b24 - std::panicking::panic_with_hook::h9beef5d363b58ff6
5: 0xaaaaecca7924 - std::panicking::panic_handler::{{closure}}::h57f2e6165cc323d5
6: 0xaaaaecca0938 - std::sys::backtrace::__rust_end_short_backtrace::h21778420051a9152
7: 0xaaaaecc8b1f4 - __rustc[d556568c0434a7c8]::rust_begin_unwind
8: 0xaaaae02c79e4 - core::panicking::panic_fmt::ha13a2c95e2b2469c
9: 0xaaaae02c7570 - core::option::expect_failed::hba972a4ce20c70f1
10: 0xaaaae061eec0 - arrow_buffer::buffer::scalar::ScalarBuffer::new::hf2e2c0027d4a8a5b
11: 0xaaaae06214d8 - <arrow_array::array::byte_array::GenericByteArray as arrow_array::array::Array>::slice::h1af95bb062c46d25
12: 0xaaaae0611e48 - <alloc::vec::Vec as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter::h316ec8ed7505c3ac
13: 0xaaaaeab1b9b4 - query::part_sort::PartSortStream::can_stop_early::h6624b4ca652c04a5
14: 0xaaaaeab1af7c - query::part_sort::PartSortStream::split_batch::hebde24ccd9f13e98
15: 0xaaaaeab36a6c - <query::part_sort::PartSortStream as futures_core::stream::Stream>::poll_next::hdbd8f415e328d956
16: 0xaaaaeab15410 - query::window_sort::WindowedSortStream::poll_next_inner::hc5100b50c4eff573
17: 0xaaaaeab3e288 - <query::window_sort::WindowedSortStream as futures_core::stream::Stream>::poll_next::h15aa7250e79d3126
18: 0xaaaae587c05c - datafusion_common_runtime::trace_utils::trace_future::{{closure}}::h425adecebfd0a4b8
19: 0xaaaae573bc74 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h115fb526e918f51c
20: 0xaaaae5738b14 - tokio::runtime::task::core::Core<T,S>::poll::h99a107d93a02f26d
21: 0xaaaae5899d30 - tokio::runtime::task::harness::Harness<T,S>::poll::h7bb5eb13e37b1222
22: 0xaaaaed3898c8 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::he65e62774513afaa
23: 0xaaaaed386b8c - tokio::runtime::context::scoped::Scoped::set::h2fd6a111e630dea3
24: 0xaaaaed376b60 - tokio::runtime::context::runtime::enter_runtime::h0e9fe36edf51fece
25: 0xaaaaed3880b4 - tokio::runtime::scheduler::multi_thread::worker::run::h8b1af75cd53889de
26: 0xaaaaed365094 - <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll::h11ff3f0c751fd3ad
27: 0xaaaaed36b814 - tokio::runtime::task::core::Core<T,S>::poll::h2f77b6e58d883360
28: 0xaaaaed35e980 - tokio::runtime::task::harness::Harness<T,S>::poll::haea972f7da1a7cf6
29: 0xaaaaed363e38 - tokio::runtime::blocking::pool::Inner::run::hb37680b35790f597
30: 0xaaaaed381ff4 - std::sys::backtrace::__rust_begin_short_backtrace::h94229eb01607291e
31: 0xaaaaed382a70 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hdee688acfde5a32d
32: 0xaaaaecc9e514 - std::sys::thread::unix::Thread::new::thread_start::haf2767e8965915ad
33: 0xffff86b40398 -
34: 0xffff86ba9e9c -
35: 0x0 -
"""

[switz]

seeing a similar panic:

 │ Executive Summary                                                                                  │
     │                                                                                                    │
     │ GreptimeDB is experiencing repeated panic crashes in the datanodes due to a bug in the query       │
     │ engine's sorting logic. The cluster remains operational but with degraded stability.               │
     │                                                                                                    │
     │ ---                                                                                                │
     │ Issue #1: Arrow-Buffer Offset Overflow (CRITICAL)                                                  │
     │                                                                                                    │
     │ Description                                                                                        │
     │                                                                                                    │
     │ Datanodes are panicking with "offset overflow" errors in the arrow-buffer library during query     │
     │ execution.                                                                                         │
     │                                                                                                    │
     │ Error                                                                                              │
     │                                                                                                    │
     │ panicked at arrow-buffer-56.2.0/src/buffer/scalar.rs:70:52:                                        │
     │ offset overflow                                                                                    │
     │                                                                                                    │
     │ Stack Trace                                                                                        │
     │                                                                                                    │
     │ query::part_sort::PartSortStream::can_stop_early                                                   │
     │ query::part_sort::PartSortStream::split_batch                                                      │
     │ query::window_sort::WindowedSortStream::poll_next_inner                                            │
     │ <arrow_array::array::primitive_array::PrimitiveArray<T>>::slice                                    │
     │ arrow_buffer::buffer::scalar::ScalarBuffer<T>::new                                                 │
     │                                                                                                    │
     │ Impact                                                                                             │
     │                                                                                                    │
     │ - Continuous panics in both datanodes (greptimedb-datanode-0, greptimedb-datanode-1)               │
     │ - Multiple panics per second during query load                                                     │
     │ - Pods remain running (panics are caught, not full crashes)                                        │
     │ - Query reliability is degraded                                                                    │
     │                                                                                                    │
     │ Root Cause                                                                                         │
     │                                                                                                    │
     │ Bug in GreptimeDB's PartSortStream component when slicing primitive arrays during the              │
     │ "can_stop_early" optimization in sorted queries. Likely triggered by:                              │
     │ - Tables with many columns (~95 columns in opentelemetry_traces)                                   │
     │ - Specific query patterns involving sorting

[discord9]

Hi there, thanks for reporting this! I haven't been able to reproduce the error yet. Could you hop into the Greptime community so we can discuss the details and get this sorted out faster?

[yihong0618]

Patch: #7458