misc coercion cleanups and handle safety correctly

[BoxyUwU]

r? lcnr

"remove normalize call"

Fixes #132765

If the normalization fails we would sometimes get a TypeError containing inference variables created inside of the probe used by coercion. These would then get leaked out causing ICEs in diagnostics logic

"leak check and lub for closure<->closure coerce-lubs of same defids"

Fixes rust-lang/trait-system-refactor-initiative#233

fn peculiar() -> impl Fn(u8) -> u8 {
    return |x| x + 1
}

the |x| x + 1 expr has a type of Closure(?31t) which we wind up inferring the RPIT to. The CoerceMany ret_coercion for the whole peculiar typeck has an expected type of RPIT (unnormalized). When we type check the return |x| x + 1 expr we go from the never type to Closure(?31t) which then participates in the ret_coercion giving us a coerce-lub(RPIT, Closure(?31t)).

Normalizing RPIT gives us some Closure(?50t) where ?31t and ?50t have been unified with ?31t as the root var. resolve_vars_if_possible doesn't resolve infer vars to their roots so these wind up with different structural identities so the fast path doesn't apply and we fall back to coercing to a fn ptr. cc #147193 which also fixes this

New solver probably just gets more inference variables here because canonicalization + generally different approach to normalization of opaques. Idk :3

FCP worthy stuffy

there are some other FCP worthy things but they're in my FCP comment which also contains some analysis of the breaking nature of the previously listed changes in this PR: #148602 (comment)

[rustbot]

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

This PR changes rustc_public

cc @oli-obk, @celinval, @ouz-a

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

Some changes occurred to constck

cc @fee1-dead

[BoxyUwU]

Hi @rust-lang/types. I've recently gone over our implementation of coercions and discovered a few things which I'd like to change which need to go through an FCP. I'm not really sure if it makes sense to split this into three separate FCPs or not so I've just kept it all together. Let me know if you'd rather me split them apart :)

I am not including lang on this FCP as this is either clear bug fixes that make our behaviour more consistent and agree with what is already in the reference, or its just incredibly minor type checking stuff :)

@rfcbot merge

Leak Checks in coerce-lub to fn-pointers

Background Context

For misc context on the leak checking see a previous FCP by lcnr: #119820. Also note that due to not having implied bounds on binders the leak check can incorrectly produce errors that would be satisfied if we took implied bounds into account.

leak check fndef lub fndef

When performing a coerce-lub between two FnDefs we first attempt to see if they have some common supertype (infcx.lub). If there is a common supertype we return that instead of coercing the FnDefs to fn-pointers.

macro_rules! lub {
    ($lhs:expr, $rhs:expr) => {
        if true { $lhs } else { $rhs }
    };
}

fn foo<T: ?Sized>() {}

// This performs a `coerce-lub(foo<?lhs>, foo<?rhs)` operation, which
// will infer `?lubbed_ty = foo<?lhs>, ?lhs=?rhs`.
let r: /* ?lubbed_ty */ = lub!(foo::<_ /* ?lhs */>, foo::<_ /* ?rhs */>);

// assert that `r` is a ZST/`FnDef`
unsafe { std::mem::transmute::<_, ()>(r) }

On stable we do not leak check after this infcx.lub operation. This can lead us to incorrectly believe two FnDefs have a mutual supertype and avoid coercing to fn-pointers.

This FCP proposes that we start leak checking there. See this test for a code example that will go from error->compile with this change: tests/ui/coercion/leak_check_fndef_lub.rs

This is theoretically breaking. There could exist deadcode which performs a coerce-lub of two unequal-by-binders FnDefs and then only typechecks due to the lubbed type being a FnDef.

With this change we would now (correctly) coerce those FnDefs to fn-pointers resulting in an error. See this test for a code example that will go from compile->error with this change: tests/ui/coercion/leak_check_fndef_lub_deadcode_breakage.rs

I believe this theoeretical breakage to be acceptable. I do not expect this to be encountered in practice let alone often, crater results agree with this showing no regressions due to this change.

If someone does wind up encountering this I still think this is acceptable. This feels like a clear bug fix to me and certainly falls under "allowable" inference breakage.

perform closure lub closure

When performing a coerce-lub between two closures we now handle it the same way we do FnDefs (including the proposed leak check addition). We first try to determine if the two closure types have a mutual supertype and if so we return that instead of coercing both to a fn-pointer.

I don't believe this is observable on stable but can be observed on nightly via type_alias_impl_trait hacks under the new trait solver: tests/ui/coercion/lub_closures_before_fnptr_coercion.rs

Safety and Target Features in coerce-lub

Background Context

See the reference header describing target features for an explanation of how target features interact with the safety of the function: r-attributes.codegen.target_feature

safe target feature fns are fn not unsafe fn

On stable when performing a coerce-lub operation between closures and FnDefs we do not treat the FnDef's signature as being safe if it has target features enabled. This differs from normal coerce operations.

This differs from normal coerce operations where when coercing an FnDef to a fn-pointer we will produce a safe fn-pointer if the target features of the FnDef are also enabled in the current body:

#[target_feature(enable = "avx512")]
fn my_fn() {}

#[target_feature(enable = "avx512")]
fn test() {
    let a: fn() = my_fn;
}

It also differs from other coerce-lub operations when coercing between a FnDef and a fn-pointer, which does correctly allow the FnDef to coerce to a safe fn-pointer.

This FCP proposes that when acquiring the signature of a FnDef during a coerce-lub operation, we give a safe signature if all of the target features of the FnDef are enabled in the current body.

See this test for examples of the proposed behaviour: tests/ui/coercion/lub_coercion_handles_safety.rs

This is theoretically a breaking change as there could be code that relies on us incorrectly coercing safe target feature functions to unsafe fnpointers when we will now coerce them to safe fn-pointers.

I expect this to be quite unlikely as safe target features are a relatively new addition to the language (1.86.0) and also a somewhat niche feature.

The crater runs confirm this showing no breakage due to this change.

If someone does wind up encountering this it still feels acceptable to me as it is a clear bug fix and certainly falls under "allowable" inference breakage.

fn->unsafe fn in coerce-lub of fndefs and closures

On stable when performing a coerce-lub operation between closures and FnDefs we do not allow coercing closures or FnDefs of safe functons to unsafe fn-pointers directly.

This differs from normal coerce operations where when coercing n safe FnDef/closure to an unsafe fn-pointer we will produce an unsafe fn-pointer:

fn my_fn() {}

fn test() {
    let a: unsafe fn() = my_fn;
}

It also differs from other coerce-lub operations when coercing between a safe FnDef/closure and a unsafe fn-pointer, which does correctly allow the FnDef/closure to coerce to an unsafe fn-pointer.

This FCP proposes that after acquiring the signatures of both sides of the coerce-lub operation, if they have mismatched safeties we coerce the safe signature to an unsafe signature.

See this test for examples of the proposed behaviour: tests/ui/coercion/lub_coercion_handles_safety.rs

I don't expect this to be able to break code as any code involving coerce-lubs of signatures with differing safeties is currently an error on stable.

Removing an unnecessary normalize call

We currently have a redundant normalize call inside of coercion logic to re-normalize an already normalized type. This was mostly a code tidyup but happend to fix an ICE.

This is theoretically breaking as re-normalizing already normalized types is not necessarily a no-op under the old solver due to ambiguous aliases inside of binders. I don't expect this to really be encountered in practice and this is backed up by the lack of crater regressions. If there are regressions we can always just re-add a normalize call somewhere.

Crater Regressions

There were no crater regressions detected for this set of changes. The crater run can be found here: #147565#issuecomment-3492623733. This crater run had an additional change part of it which does cause regressions and is not included in this PR. An analysis of those regressions can be found here: #147565#issuecomment-3486416652, which should hopefully show that those regressions are not a problem without the additional changes that PR has.

[rust-rfcbot]

Team member @BoxyUwU has proposed to merge this. The next step is review by the rest of the tagged team members:

• @BoxyUwU
• @jackh726
• @lcnr
• @oli-obk
• @spastorino

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[BoxyUwU]

I've gone and allowed ReifyFnPointer coercions to coerce safe fndefs to unsafe fn pointers directly instead of needing to compose two coercions. This made handling safety in coerce-lub simpler/nicer.

[lcnr]

@rfcbot reviewed

[rust-rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[lcnr]

@bors r+ rollup=never

[bors]

📌 Commit 2f95ecf has been approved by lcnr

It is now in the queue for this repository.

[bors]

⌛ Testing commit 2f95ecf with merge 97b131c...

[bors]

☀️ Test successful - checks-actions
Approved by: lcnr
Pushing 97b131c to main...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 864339a (parent) -> 97b131c (this PR)

Test differences

Show 27 test diffs

Stage 1

• [crashes] tests/crashes/132765.rs: pass -> [missing] (J1)
• [ui] tests/ui/coercion/hr_alias_normalization_leaking_vars.rs: [missing] -> pass (J1)
• [ui] tests/ui/coercion/issue-88097.rs: pass -> [missing] (J1)
• [ui] tests/ui/coercion/issue-88097.rs#current: [missing] -> pass (J1)
• [ui] tests/ui/coercion/issue-88097.rs#next: [missing] -> pass (J1)
• [ui] tests/ui/coercion/leak_check_fndef_lub.rs: [missing] -> pass (J1)
• [ui] tests/ui/coercion/leak_check_fndef_lub_deadcode_breakage.rs: [missing] -> pass (J1)
• [ui] tests/ui/coercion/lub_coercion_handles_safety.rs: [missing] -> pass (J1)
• [ui] tests/ui/coercion/structural_identity_dependent_reborrows.rs: [missing] -> pass (J1)

Stage 2

• [ui] tests/ui/coercion/hr_alias_normalization_leaking_vars.rs: [missing] -> pass (J0)
• [ui] tests/ui/coercion/issue-88097.rs: pass -> [missing] (J0)
• [ui] tests/ui/coercion/issue-88097.rs#current: [missing] -> pass (J0)
• [ui] tests/ui/coercion/issue-88097.rs#next: [missing] -> pass (J0)
• [ui] tests/ui/coercion/leak_check_fndef_lub.rs: [missing] -> pass (J0)
• [ui] tests/ui/coercion/leak_check_fndef_lub_deadcode_breakage.rs: [missing] -> pass (J0)
• [ui] tests/ui/coercion/structural_identity_dependent_reborrows.rs: [missing] -> pass (J0)
• [ui] tests/ui/coercion/lub_coercion_handles_safety.rs: [missing] -> pass (J2)
• [crashes] tests/crashes/132765.rs: pass -> [missing] (J3)
• [ui] tests/ui/coercion/lub_coercion_handles_safety.rs: [missing] -> ignore (only executed when the architecture is x86_64) (J4)

Additionally, 8 doctest diffs were found. These are ignored, as they are noisy.

Job group index

• J0: aarch64-apple, aarch64-gnu, aarch64-gnu-llvm-20-1, aarch64-msvc-1, arm-android, armhf-gnu, dist-i586-gnu-i586-i686-musl, i686-gnu-1, i686-gnu-nopt-1, i686-msvc-1, test-various, x86_64-gnu, x86_64-gnu-debug, x86_64-gnu-gcc, x86_64-gnu-llvm-20, x86_64-gnu-llvm-20-2, x86_64-gnu-llvm-21-2, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-1, x86_64-msvc-1
• J1: x86_64-gnu-llvm-20-3, x86_64-gnu-llvm-21-3
• J2: test-various, x86_64-gnu, x86_64-gnu-debug, x86_64-gnu-gcc, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-1, x86_64-msvc-1
• J3: aarch64-apple, aarch64-gnu, aarch64-gnu-llvm-20-1, aarch64-msvc-1, arm-android, armhf-gnu, dist-i586-gnu-i586-i686-musl, i686-gnu-1, i686-gnu-nopt-1, i686-msvc-1, test-various, x86_64-gnu, x86_64-gnu-gcc, x86_64-gnu-llvm-20-2, x86_64-gnu-llvm-21-2, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-1, x86_64-msvc-1
• J4: aarch64-apple, aarch64-gnu, aarch64-gnu-llvm-20-1, aarch64-msvc-1, arm-android, armhf-gnu, dist-i586-gnu-i586-i686-musl, i686-gnu-1, i686-gnu-nopt-1, i686-msvc-1, x86_64-gnu-llvm-20, x86_64-gnu-llvm-20-2, x86_64-gnu-llvm-21-2

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 97b131c90059e6122f22562ab4d78d50160daeff --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. dist-x86_64-apple: 6019.5s -> 8029.3s (+33.4%)
2. aarch64-msvc-2: 4631.3s -> 5384.7s (+16.3%)
3. dist-aarch64-apple: 7974.5s -> 6845.2s (-14.2%)
4. dist-apple-various: 4285.3s -> 3795.3s (-11.4%)
5. i686-gnu-2: 5472.2s -> 6025.4s (+10.1%)
6. aarch64-gnu: 7085.3s -> 6382.6s (-9.9%)
7. x86_64-gnu-aux: 6854.3s -> 7518.7s (+9.7%)
8. x86_64-gnu-nopt: 6747.9s -> 7373.7s (+9.3%)
9. x86_64-mingw-1: 10180.7s -> 9304.0s (-8.6%)
10. aarch64-apple: 8798.0s -> 9520.6s (+8.2%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (97b131c): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.2%	[-0.5%, -0.1%]	13
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results (primary 2.1%, secondary -2.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.1%	[2.1%, 2.1%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.4%	[-2.5%, -2.4%]	2
All ❌✅ (primary)	2.1%	[2.1%, 2.1%]	1

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 472.4s -> 470.371s (-0.43%)
Artifact size: 386.85 MiB -> 386.79 MiB (-0.02%)