Skip to content

Conversation

@maxammann
Copy link
Contributor

@maxammann maxammann commented Oct 20, 2023

This adds experimental support for generating coverage using LLVM and cargo-llvm-cov. Note that replacing cargo-llvm-cov would be simple. It might simplify stuff even as cargo-llvm-cov basically just invokes llvm-profdata and llvm-cov.

How to use

Suppose you have a setup like in the examples directory of the repository. Then:

  1. Install cargo-llvm-cov: cargo install cargo-llvm-cov
  2. Install LLVM tools (they might get installed by cargo-llvm-cov automatically, we need to test that): rustup toolchain install nightly --component llvm-tools-preview
  3. Run cargo test
  4. Optionally, fuzz the project.
  5. Execute cargo-test-fuzz test-fuzz target --no-instrumentation --replay corpus --coverage

Coverage is generated as HTML in target/llvm-cov/html.

image

TODO:

  • Should --coverage imply --no-instrumentation?
  • Should --coverage be compatible with fuzzing? (probably not, even though it might just work)
  • Make sure that llvm-tools-preview and cargo-llvm-cov get installed or promt for installation.
  • On macOS I get the following warning during generation: warning: 4 functions have mismatched data
  • During previous experiments I got such coverage data:
    image
    I suspect this is due to the fact that test-fuzz injects code into the function-to-test here:
    #write_concretizations_and_args

    Either move that code somewhere else, or put it into a function, ad a no_coverage instruction and call that from the target function.

Details

The follwing commands are executes to generate coverage. We should validate them:

     Running `/Users/max/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-profdata merge -sparse -f /Users/max/projects/examples/target/examples-profraw-list -o /Users/max/projects/examples/target/examples.profdata`
     Running `/Users/max/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-cov show -format=html -instr-profile=/Users/max/projects/examples/target/examples.profdata -object target/debug/deps/alloc-34551e9cc1ff6b3f -object target/debug/deps/arc-bf8fdeaeed680770 -object target/debug/deps/assert-ba6618cd737ff666 -object target/debug/deps/associated_type-7ed59268473251eb -object target/debug/deps/auto_concretize_0-736514e466198bc3 -object target/debug/deps/auto_generate-7287f7108f5b00a9 -object target/debug/deps/conversion-c5a93588844a6594 -object target/debug/deps/debug-ff5617e8189a0951 -object target/debug/deps/default-79bdc830b898ec0f -object target/debug/deps/from-2bc06bf9dd96fea9 -object target/debug/deps/generic-23602c4ce1352038 -object target/debug/deps/hello_world-594474f770d22b32 -object target/debug/deps/hello_world-cff0c92d5749ded3 -object target/debug/deps/lifetime-f34199c00924285b -object target/debug/deps/manual_leak-7a7f8bfdea2a3f10 -object target/debug/deps/mut_ref-295c3cd2737a4d53 -object target/debug/deps/parse_duration-d93f2ce3d20bd782 -object target/debug/deps/qwerty-72231f74eb29ea0a -object target/debug/deps/rename-503984417115eb3f -object target/debug/deps/return_type-3b4f1b217dc76985 -object target/debug/deps/serde-26f50433033e3e5d -object target/debug/deps/test_fuzz_impl-2a426bdf99ecc53a -object target/debug/deps/unserde-4a2d01352b369939 -object target/debug/hello-world -ignore-filename-regex 'test-fuzz|/rustc/([0-9a-f]+|[0-9]+\.[0-9]+\.[0-9]+)/|^/Users/max/projects/examples(/.*)?/(tests|examples|benches)/|^/Users/max/projects/examples/target($|/)|^/Users/max/\.cargo/(registry|git)/|^/Users/max/\.rustup/toolchains($|/)' -show-instantiations=false -show-line-counts-or-regions -show-expansions -show-branches=count -Xdemangler=/Users/max/.cargo/bin/cargo-llvm-cov -Xdemangler=llvm-cov -Xdemangler=demangle -output-dir=/Users/max/projects/examples/target/llvm-cov/html`

fixes #293

@CLAassistant
Copy link

CLAassistant commented Oct 21, 2024

CLA assistant check
All committers have signed the CLA.

@smoelius
Copy link
Collaborator

@maxammann Is it okay if I take over this PR?

@maxammann
Copy link
Contributor Author

100%

Thank you!

@smoelius smoelius marked this pull request as ready for review December 29, 2025 20:42
@smoelius smoelius self-requested a review as a code owner December 29, 2025 20:42
@smoelius smoelius force-pushed the coverage branch 3 times, most recently from dec5429 to 951613e Compare December 30, 2025 01:25
@smoelius smoelius force-pushed the coverage branch 6 times, most recently from 85fe0c1 to eafddd9 Compare January 11, 2026 21:02
@smoelius smoelius force-pushed the coverage branch 5 times, most recently from 34521ae to 8c909b7 Compare January 14, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add tooling for generating coverage

4 participants