Possible data race in Rust runtime

[daviddrysdale]

On trying a test run under Thread Sanitizer (cf #778) there's a possible data race between wait_on_channels by the logging node, and channel_write from the Wasm node.

% export TSAN_OPTIONS="halt_on_error=1 report_atomic_races=0"
% RUSTFLAGS="-Z sanitizer=thread" cargo test --manifest-path=./examples/abitest/tests/Cargo.toml --target x86_64-unknown-linux-gnu -- --nocapture
    Finished test [unoptimized + debuginfo] target(s) in 0.09s
     Running target/x86_64-unknown-linux-gnu/debug/deps/abitest_tests-70de4023c24259b6

running 1 test
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Z sanitizer=thread --target wasm32-unknown-unknown --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
--- stdout
___.wasm
lib___.rlib
___.wasm
lib___.a
/usr/local/google/home/drysdale/.rustup/toolchains/nightly-2020-02-06-x86_64-unknown-linux-gnu
debug_assertions
proc_macro
sanitize="thread"
target_arch="wasm32"
target_endian="little"
target_env=""
target_has_atomic="16"
target_has_atomic="32"
target_has_atomic="64"
target_has_atomic="8"
target_has_atomic="ptr"
target_has_atomic_load_store="16"
target_has_atomic_load_store="32"
target_has_atomic_load_store="64"
target_has_atomic_load_store="8"
target_has_atomic_load_store="ptr"
target_os="unknown"
target_pointer_width="32"
target_thread_local
target_vendor="unknown"

--- stderr
error: ThreadSanitizer only works with the `x86_64-unknown-linux-gnu` or `x86_64-apple-darwin` target

warning: dropping unsupported crate type `dylib` for target `wasm32-unknown-unknown`

warning: dropping unsupported crate type `proc-macro` for target `wasm32-unknown-unknown`

error: aborting due to previous error


error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Z sanitizer=thread --target wasm32-unknown-unknown --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
--- stdout
___.wasm
lib___.rlib
___.wasm
lib___.a
/usr/local/google/home/drysdale/.rustup/toolchains/nightly-2020-02-06-x86_64-unknown-linux-gnu
debug_assertions
proc_macro
sanitize="thread"
target_arch="wasm32"
target_endian="little"
target_env=""
target_has_atomic="16"
target_has_atomic="32"
target_has_atomic="64"
target_has_atomic="8"
target_has_atomic="ptr"
target_has_atomic_load_store="16"
target_has_atomic_load_store="32"
target_has_atomic_load_store="64"
target_has_atomic_load_store="8"
target_has_atomic_load_store="ptr"
target_os="unknown"
target_pointer_width="32"
target_thread_local
target_vendor="unknown"

--- stderr
error: ThreadSanitizer only works with the `x86_64-unknown-linux-gnu` or `x86_64-apple-darwin` target

warning: dropping unsupported crate type `dylib` for target `wasm32-unknown-unknown`

warning: dropping unsupported crate type `proc-macro` for target `wasm32-unknown-unknown`

error: aborting due to previous error


2020-03-27 15:04:46,664 DEBUG [oak_runtime::node::wasm] new_instance: discovering frontend-config frontend_oak_main
2020-03-27 15:04:47,633 DEBUG [oak_runtime::node::wasm] new_instance: starting frontend-config frontend_oak_main
2020-03-27 15:04:47,635 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:47,735 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:47,836 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:47,937 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,037 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,138 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,239 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,340 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,440 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,541 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,642 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,647 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047240), I32(1047248)])
2020-03-27 15:04:48,647 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047240 1047248
2020-03-27 15:04:48,649 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 0 RuntimeArgs([I32(1049284), I32(14), I32(1395328), I32(8), I64(-5089638740390981732)])
2020-03-27 15:04:48,649 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create: 1049284 14 1395328 8 13357105333318569884
2020-03-27 15:04:48,649 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create config_name is: logging-config
2020-03-27 15:04:48,650 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create entrypoint is: oak_main
2020-03-27 15:04:48,652 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(-5089638740390981732)])
2020-03-27 15:04:48,652 DEBUG [oak_runtime::runtime] wait_on_channels: channels not ready, parking thread Thread { id: ThreadId(4), name: None }
2020-03-27 15:04:48,652 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 13357105333318569884
2020-03-27 15:04:48,699 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:48,699 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 0 RuntimeArgs([I32(1049428), I32(14), I32(1049442), I32(16), I64(1791648940387253873)])
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create: 1049428 14 1049442 16 1791648940387253873
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create config_name is: backend-config
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create entrypoint is: backend_oak_main
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] new_instance: discovering backend-config backend_oak_main
2020-03-27 15:04:48,742 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,843 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,944 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,044 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,145 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,245 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,346 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,442 DEBUG [oak_runtime::node::wasm] new_instance: starting backend-config backend_oak_main
2020-03-27 15:04:49,444 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(1791648940387253873)])
2020-03-27 15:04:49,445 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 1791648940387253873
2020-03-27 15:04:49,446 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:49,446 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:49,446 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,460 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 4 RuntimeArgs([I64(1676672801125846535), I32(1049552), I32(0), I32(1507416), I32(1)])
2020-03-27 15:04:49,460 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_write: 1676672801125846535 1049552 0 1507416 1
2020-03-27 15:04:49,467 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(-2987603100449681604)])
2020-03-27 15:04:49,467 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 15459140973259870012
2020-03-27 15:04:49,471 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:49,472 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:49,473 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 0 RuntimeArgs([I32(1049428), I32(14), I32(1049442), I32(16), I64(-6684026489260364187)])
2020-03-27 15:04:49,474 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create: 1049428 14 1049442 16 11762717584449187429
2020-03-27 15:04:49,474 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create config_name is: backend-config
2020-03-27 15:04:49,474 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create entrypoint is: backend_oak_main
2020-03-27 15:04:49,474 DEBUG [oak_runtime::node::wasm] new_instance: discovering backend-config backend_oak_main
2020-03-27 15:04:49,547 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,647 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,748 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,849 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,949 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,050 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,137 DEBUG [oak_runtime::node::wasm] new_instance: starting backend-config backend_oak_main
2020-03-27 15:04:50,139 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(-6684026489260364187)])
2020-03-27 15:04:50,139 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 11762717584449187429
2020-03-27 15:04:50,141 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:50,141 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:50,151 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,152 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 3 RuntimeArgs([I32(1046680), I32(1046688)])
2020-03-27 15:04:50,152 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: channel_create: 1046680 1046688
2020-03-27 15:04:50,154 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 0 RuntimeArgs([I32(1050108), I32(14), I32(1059780), I32(8), I64(2293680208635920034)])
2020-03-27 15:04:50,154 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: node_create: 1050108 14 1059780 8 2293680208635920034
2020-03-27 15:04:50,155 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: node_create config_name is: logging-config
2020-03-27 15:04:50,155 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: node_create entrypoint is: oak_main
2020-03-27 15:04:50,155 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 4 RuntimeArgs([I64(1277197695412101525), I32(1049552), I32(0), I32(1507416), I32(1)])
2020-03-27 15:04:50,156 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_write: 1277197695412101525 1049552 0 1507416 1
2020-03-27 15:04:50,162 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(-8944433438557824175)])
2020-03-27 15:04:50,163 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 9502310635151727441
2020-03-27 15:04:50,164 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 2 RuntimeArgs([I64(2293680208635920034)])
2020-03-27 15:04:50,164 DEBUG [oak_runtime::runtime] wait_on_channels: channels not ready, parking thread Thread { id: ThreadId(7), name: None }
2020-03-27 15:04:50,165 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: channel_close: 2293680208635920034
2020-03-27 15:04:50,167 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:50,167 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:50,169 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 0 RuntimeArgs([I32(1049428), I32(14), I32(1049442), I32(16), I64(6553477005041356551)])
2020-03-27 15:04:50,169 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create: 1049428 14 1049442 16 6553477005041356551
2020-03-27 15:04:50,169 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create config_name is: backend-config
2020-03-27 15:04:50,169 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create entrypoint is: backend_oak_main
2020-03-27 15:04:50,170 DEBUG [oak_runtime::node::wasm] new_instance: discovering backend-config backend_oak_main
2020-03-27 15:04:50,219 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 5 RuntimeArgs([I64(7374366863468297640), I32(1179672), I32(2), I32(1046604), I32(1179712), I32(2), I32(1046608)])
2020-03-27 15:04:50,220 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: channel_read: 7374366863468297640 1179672 2 1046604 1179712 2 1046608
2020-03-27 15:04:50,251 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,352 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 4 RuntimeArgs([I64(-3905051140228563756), I32(1179904), I32(163), I32(1), I32(0)])
2020-03-27 15:04:50,352 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,352 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: channel_write: 14541692933480987860 1179904 163 1 0
2020-03-27 15:04:50,352 DEBUG [oak_runtime::runtime] wait_on_channels: thread Thread { id: ThreadId(7), name: None } re-woken
2020-03-27 15:04:50,415 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 6 RuntimeArgs([I32(1179728), I32(1)])
2020-03-27 15:04:50,415 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: wait_on_channels: 1179728 1
2020-03-27 15:04:50,415 DEBUG [oak_runtime::runtime] wait_on_channels: channels not ready, parking thread Thread { id: ThreadId(5), name: None }
==================
WARNING: ThreadSanitizer: data race (pid=233640)
  Write of size 8 at 0x7b0800070980 by thread T6:
    #0 free /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:706 (abitest_tests-70de4023c24259b6+0x44d48)
    #1 alloc::alloc::dealloc /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/alloc.rs:103 (abitest_tests-70de4023c24259b6+0x233715)
    #2 <alloc::alloc::Global as core::alloc::AllocRef>::dealloc /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/alloc.rs:174 (abitest_tests-70de4023c24259b6+0x2339e8)
    #3 alloc::sync::Arc<T>::drop_slow /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/sync.rs:743 (abitest_tests-70de4023c24259b6+0x158767)
    #4 <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/sync.rs:1249 (abitest_tests-70de4023c24259b6+0x158e70)
    #5 core::ptr::drop_in_place /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ptr/mod.rs:174 (abitest_tests-70de4023c24259b6+0x16c93d)
    #6 oak_runtime::runtime::Runtime::wait_on_channels /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:259 (abitest_tests-70de4023c24259b6+0x147d58)
    #7 oak_runtime::node::logger::logger /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/logger.rs:84 (abitest_tests-70de4023c24259b6+0x24a439)
    #8 <oak_runtime::node::logger::LogNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/logger.rs:60 (abitest_tests-70de4023c24259b6+0x24781c)
    #9 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204c40)
    #10 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba704)
    #11 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9514)
    #12 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd23f)
    #13 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #14 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d9684)
    #15 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba0b0)
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16630d)
    #17 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

  Previous atomic write of size 8 at 0x7b0800070980 by thread T4 (mutexes: read M17, write M53):
    #0 __tsan_atomic64_fetch_sub /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:647 (abitest_tests-70de4023c24259b6+0x885c1)
    #1 core::sync::atomic::atomic_sub /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/sync/atomic.rs:2314 (abitest_tests-70de4023c24259b6+0x6900b9)
    #2 core::sync::atomic::AtomicUsize::fetch_sub /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/sync/atomic.rs:1654 (abitest_tests-70de4023c24259b6+0x1bb109)
    #3 <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/sync.rs:1214 (abitest_tests-70de4023c24259b6+0x158e2f)
    #4 core::ptr::drop_in_place /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ptr/mod.rs:174 (abitest_tests-70de4023c24259b6+0x16c93d)
    #5 oak_runtime::runtime::Runtime::channel_write::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:315 (abitest_tests-70de4023c24259b6+0x16260f)
    #6 oak_runtime::runtime::channel::ChannelMapping::with_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/channel.rs:224 (abitest_tests-70de4023c24259b6+0x136e22)
    #7 oak_runtime::runtime::Runtime::channel_write /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:266 (abitest_tests-70de4023c24259b6+0x147f58)
    #8 oak_runtime::node::wasm::WasmInterface::channel_write /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:315 (abitest_tests-70de4023c24259b6+0x22269e)
    #9 <oak_runtime::node::wasm::WasmInterface as wasmi::host::Externals>::invoke_index /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:609 (abitest_tests-70de4023c24259b6+0x2283cc)
    #10 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:151 (abitest_tests-70de4023c24259b6+0x204872)
    #11 wasmi::runner::Interpreter::run_interpreter_loop /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:307 (abitest_tests-70de4023c24259b6+0x1da649)
    #12 wasmi::runner::Interpreter::start_execution /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:215 (abitest_tests-70de4023c24259b6+0x1d983f)
    #13 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:146 (abitest_tests-70de4023c24259b6+0x204902)
    #14 wasmi::module::ModuleInstance::invoke_export /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/module.rs:639 (abitest_tests-70de4023c24259b6+0x21cecc)
    #15 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:841 (abitest_tests-70de4023c24259b6+0x1f151d)
    #16 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204bd0)
    #17 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba774)
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9584)
    #19 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd192)
    #20 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #21 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d95fa)
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba490)
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16636d)
    #24 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

  Mutex M17 (0x7b1400006360) created at:
    #0 pthread_rwlock_wrlock /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1366 (abitest_tests-70de4023c24259b6+0x48596)
    #1 std::sys::unix::rwlock::RWLock::write /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys/unix/rwlock.rs:71 (abitest_tests-70de4023c24259b6+0x247311)
    #2 std::sys_common::rwlock::RWLock::write /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/rwlock.rs:48 (abitest_tests-70de4023c24259b6+0x204ae9)
    #3 std::sync::rwlock::RwLock<T>::write /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sync/rwlock.rs:267 (abitest_tests-70de4023c24259b6+0x1fbcab)
    #4 oak_runtime::runtime::channel::ChannelMapping::new_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/channel.rs:163 (abitest_tests-70de4023c24259b6+0x140d07)
    #5 oak_runtime::runtime::Runtime::new_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:179 (abitest_tests-70de4023c24259b6+0x146fe4)
    #6 oak_runtime::runtime::Runtime::run /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:110 (abitest_tests-70de4023c24259b6+0x14616c)
    #7 oak_runtime::config::configure_and_run /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/config.rs:119 (abitest_tests-70de4023c24259b6+0x1b7f7b)
    #8 abitest_tests::tests::test_abi /usr/local/google/home/drysdale/src/oak/examples/abitest/tests/src/tests.rs:55 (abitest_tests-70de4023c24259b6+0xb937a)
    #9 abitest_tests::tests::test_abi::{{closure}} /usr/local/google/home/drysdale/src/oak/examples/abitest/tests/src/tests.rs:45 (abitest_tests-70de4023c24259b6+0xb6214)
    #10 core::ops::function::FnOnce::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0xb6437)
    #11 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x50533e)

  Mutex M53 (0x7b0c0001cbc0) created at:
    #0 pthread_mutex_init /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1225 (abitest_tests-70de4023c24259b6+0x472dd)
    #1 std::sys::unix::mutex::Mutex::init /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys/unix/mutex.rs:50 (abitest_tests-70de4023c24259b6+0x22f482)
    #2 std::sys_common::mutex::Mutex::init /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/mutex.rs:32 (abitest_tests-70de4023c24259b6+0x1db299)
    #3 std::sync::mutex::Mutex<T>::new /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sync/mutex.rs:173 (abitest_tests-70de4023c24259b6+0x194f83)
    #4 oak_runtime::runtime::channel::Channel::new /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/channel.rs:103 (abitest_tests-70de4023c24259b6+0x1403b3)
    #5 oak_runtime::runtime::channel::ChannelMapping::new_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/channel.rs:164 (abitest_tests-70de4023c24259b6+0x140d8d)
    #6 oak_runtime::runtime::Runtime::new_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:179 (abitest_tests-70de4023c24259b6+0x146fe4)
    #7 oak_runtime::node::wasm::WasmInterface::channel_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:222 (abitest_tests-70de4023c24259b6+0x22188d)
    #8 <oak_runtime::node::wasm::WasmInterface as wasmi::host::Externals>::invoke_index /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:594 (abitest_tests-70de4023c24259b6+0x227489)
    #9 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:151 (abitest_tests-70de4023c24259b6+0x204872)
    #10 wasmi::runner::Interpreter::run_interpreter_loop /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:307 (abitest_tests-70de4023c24259b6+0x1da649)
    #11 wasmi::runner::Interpreter::start_execution /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:215 (abitest_tests-70de4023c24259b6+0x1d983f)
    #12 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:146 (abitest_tests-70de4023c24259b6+0x204902)
    #13 wasmi::module::ModuleInstance::invoke_export /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/module.rs:639 (abitest_tests-70de4023c24259b6+0x21cecc)
    #14 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:841 (abitest_tests-70de4023c24259b6+0x1f151d)
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204bd0)
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba774)
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9584)
    #18 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd192)
    #19 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #20 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d95fa)
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba490)
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16636d)
    #23 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

  Thread T6 (tid=234002, running) created by thread T4 at:
    #0 pthread_create /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967 (abitest_tests-70de4023c24259b6+0x45c9b)
    #1 std::sys::unix::thread::Thread::new /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libstd/sys/unix/thread.rs:68 (abitest_tests-70de4023c24259b6+0x970fdd)
    #2 std::thread::Builder::spawn /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:386 (abitest_tests-70de4023c24259b6+0x1ba832)
    #3 std::thread::spawn /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:619 (abitest_tests-70de4023c24259b6+0x1b8ccd)
    #4 <oak_runtime::node::logger::LogNode as oak_runtime::node::Node>::start /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/logger.rs:58 (abitest_tests-70de4023c24259b6+0x249bd4)
    #5 oak_runtime::runtime::RuntimeRef::node_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:487 (abitest_tests-70de4023c24259b6+0x14909e)
    #6 oak_runtime::node::wasm::WasmInterface::node_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:199 (abitest_tests-70de4023c24259b6+0x221422)
    #7 <oak_runtime::node::wasm::WasmInterface as wasmi::host::Externals>::invoke_index /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:536 (abitest_tests-70de4023c24259b6+0x225d36)
    #8 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:151 (abitest_tests-70de4023c24259b6+0x204872)
    #9 wasmi::runner::Interpreter::run_interpreter_loop /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:307 (abitest_tests-70de4023c24259b6+0x1da649)
    #10 wasmi::runner::Interpreter::start_execution /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:215 (abitest_tests-70de4023c24259b6+0x1d983f)
    #11 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:146 (abitest_tests-70de4023c24259b6+0x204902)
    #12 wasmi::module::ModuleInstance::invoke_export /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/module.rs:639 (abitest_tests-70de4023c24259b6+0x21cecc)
    #13 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:841 (abitest_tests-70de4023c24259b6+0x1f151d)
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204bd0)
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba774)
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9584)
    #17 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd192)
    #18 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #19 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d95fa)
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba490)
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16636d)
    #22 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

  Thread T4 (tid=233996, running) created by thread T2 at:
    #0 pthread_create /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967 (abitest_tests-70de4023c24259b6+0x45c9b)
    #1 std::sys::unix::thread::Thread::new /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libstd/sys/unix/thread.rs:68 (abitest_tests-70de4023c24259b6+0x970fdd)
    #2 std::thread::Builder::spawn /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:386 (abitest_tests-70de4023c24259b6+0x1ba8fa)
    #3 std::thread::spawn /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:619 (abitest_tests-70de4023c24259b6+0x1b8e1d)
    #4 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:824 (abitest_tests-70de4023c24259b6+0x22bc1d)
    #5 oak_runtime::runtime::RuntimeRef::node_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:487 (abitest_tests-70de4023c24259b6+0x14909e)
    #6 oak_runtime::node::wasm::WasmInterface::node_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:199 (abitest_tests-70de4023c24259b6+0x221422)
    #7 <oak_runtime::node::wasm::WasmInterface as wasmi::host::Externals>::invoke_index /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:536 (abitest_tests-70de4023c24259b6+0x225d36)
    #8 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:151 (abitest_tests-70de4023c24259b6+0x204872)
    #9 wasmi::runner::Interpreter::run_interpreter_loop /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:307 (abitest_tests-70de4023c24259b6+0x1da649)
    #10 wasmi::runner::Interpreter::start_execution /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:215 (abitest_tests-70de4023c24259b6+0x1d983f)
    #11 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:146 (abitest_tests-70de4023c24259b6+0x204902)
    #12 wasmi::module::ModuleInstance::invoke_export /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/module.rs:639 (abitest_tests-70de4023c24259b6+0x21cecc)
    #13 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:841 (abitest_tests-70de4023c24259b6+0x1f151d)
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204bd0)
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba774)
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9584)
    #17 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd192)
    #18 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #19 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d95fa)
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba490)
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16636d)
    #22 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

SUMMARY: ThreadSanitizer: data race /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/alloc.rs:103 in alloc::alloc::dealloc
==================
error: test failed, to rerun pass '--lib'
drysdale@ring:~/src/oak{master}:

[blaxill]

@daviddrysdale any tips on how to investigate this further? I have a suspicion this is complaining there is a race on the Arc/Weak that corresponds to WaitingThreads, which could be a false positive.

[daviddrysdale]

Yeah, it looks to be related to the various layers of indirection involved in passing around a &Arc<Thread> (a reference to an atomic-reference-counted-pointer, which internally contains another atomic-reference-counted-pointer); interesting in particular that one access is a "write" and one is an "atomic write". Need to dig deeper to figure out whether it's a false positive or not, though.

[blaxill]

The reference there is unnecessary, that seems to get rid of the data race warning. I think the original complaint was that the reference was being using in a closure and the sanitizer couldn't guarantee this wasn't an issue - not sure. It doesn't seem to mind it with a copy though

However, there are now some potential lock inversion warnings im looking into.

[blaxill]

Ok the data race warning is a false positive due to Arc that is fixed in nightlies newer than 3/19 (c.f. rust-lang/rust#65097). Unfortunately there are components missing so updating the Docker image at this time would be a bad idea (clippy and rls are missing for example).

There were some lock inversion warnings that seem legitimate that I have fixed in #784, this can be verified by

• running RUSTFLAGS="-Z sanitizer=thread" cargo -Zbuild-std test --manifest-path=./examples/abitest/tests/Cargo.toml --target x86_64-unknown-linux-gnu -- --nocapture
  (Note I believe -Zbuild-std, to instrument the standard library, is required).
• Using a nightly compiler newer than 2020.03.19

and seeing there are no errors/warnings.

[daviddrysdale]

BTW, what were the lock inversion warnings? (I didn't spot any inversions (#779) from code-reading, but that's not a particularly reliable way to find them.)

[blaxill]

The warnings corresponded to one instance of inversion against your list "runtime.channels -> runtime.nodes". One other change in #784 was not strictly necessary but reduces lock scope.

[daviddrysdale]

Ah yes, I see it now, thanks for the pointer.