Proc-macro dependencies being rebuilt in `dev` mode

[dpc]

I've recently upgraded Rust from 1.67.0 to 1.71.0, and noticed that in our "build workspace" derivation, some dependencies are being re-compiled, even though they were compiled in "build workspace deps only" derivation.

I confirmed that just switching between 1.67.0 and 1.71.0 fixes/introduces the problem.

It looks like only build.rs (and deps that depend on them) are being rebuilt, and from the initial debugging session I suspect something about tracking which deps require rebuild changed between these two toolchain (well... probably cargo) versions.

Notably the files restored in /build/source/target/ from zstd (I use the zstd mode) archive produced by deps-only have different timestamps than originals: they are all reset to the default unix timestamp, while the original ones had the original timestamps. I suspect something about how cargo tracks these changed.

[dpc]

1.68.0 is unaffected, and 1.69 is affected.

And indeed some freshness work was done rust-lang/cargo@0.68.0...0.69.0

[dpc]

Ironically the change was done there to display reasons for rebuilds, but the reason for rebuilding affected crates is not being displayed.

rust-lang/cargo#11407

[ipetkov]

Do you have a minimal repro I can take a look at?

What does cargo say about which files have changed specifically?

[dpc]

Do you have a minimal repro I can take a look at?

Will try to minimize when I find some time.

What does cargo say about which files have changed specifically?

I can't get it to tell me anything, even with -vv. It just prints the Compiling proc-macros2 etc., and executes the build script.

[ipetkov]

You can also try running it with CARGO_LOG=cargo::core::compiler::fingerprint=trace and see if that shows anything interesting

[dpc]

I created a quick-start example based project where I've added syn as a depdency (which brings also proc-macro2), and couldn't reprod, even bringing more and more little details of the affected project.

CARGO_LOG of affected vs example shows a important difference:

Affected:

[2023-08-23T18:58:43Z DEBUG cargo::core::compiler::fingerprint] old local fingerprints deps "/nix/store/8xlxixbffyyra5w4xv8fabyfcwynfdh0-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/proc-macro2-1.0.66" precalculated="1.0.66"

Not affected:

quick-start-build> [2023-08-23T19:18:54Z DEBUG cargo::core::compiler::fingerprint] new local fingerprints deps "/nix/store/wzs1f02h3xw6i0ql0a12r6kbla1savr2-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/proc-macro2-1.0.66"

which reading the code suggests that in the affected build cargo somehow lost the information proc-macro2 does output rerun-if-changed=build.rs every time and is thus 'new-style' custom build.

[dpc]

I see that it affects another projects, so maybe I'm not just crazy. I reported it on cargo issue tracker, maybe I can get some help.

[dpc]

Important observation: I have 3 stages of building

• deps only
• build workspace
• run tests

The "run tests" internally, very early runs the same command that "build workspace" runs (in case this script is invoked from a dev shell). And there there is no rebuilds. Which I just realized, rules out the storing&restoring of target directory itself being an issue.

[dpc]

Not doing .cargo copying in self.mkDummySrc didn't seem to change anything.

[dpc]

Getting rid of installCargoArtifactsMode = true; does not remove the problem.

Getting rid of any differences between args passed to buildDepsOnly and building workspace does not remove the problem.

(I'm basically stripping improvements/optimizations and see if any fixes it).

[dpc]

The problem did not happen in release build!