Extend ProbingRequestFactory to support Models and Async calls

[meshulamer]

This change updates the IProbingRequestFactory's CreateRequest API to use DestinationState and ClusterState instead of DestinationModel, ClusterModel and makes the method async.

Simplifies scenarios where consumers need to access or update health state without relying on internal models.

Allows access to the ID and allows overloads to rely on ConditionalWeakTable mappings on the objects.

This change originated from a discussion with @MihaZupan, and subsequent filing of bug #2890

[meshulamer]

@dotnet-policy-service agree

[MihaZupan]

We might as well include a CancellationToken argument while we're at it

[meshulamer]

Adding CancellationToken means we should probably also support explicit cancelling in the probe creation flow, which should be treated as a separate case from ActiveHealthProbeConstructionFailedOnCluster, i.e.:

 catch (OperationCanceledException) when (!cts.IsCancellationRequested)
{
    Log.ActiveHealthProbingSkippedOnDestination(_logger, destination.DestinationId, cluster.ClusterId);
    return null;
}

What are your thoughts on that?

[MihaZupan]

Are you referring to just decreasing the log severity in case of cancellations, or also not reporting the exception as part of the probe result?
If it's the latter, IMO we should be including the exception and leave it up to the policy to decide how to handle it.

[meshulamer]

Both (decreasing log severity and not reporting exception)

I was considering this approach because I was thinking that OperationCanceledException via an external token signals an intentional skip rather than an error - consistent with how cancellation tokens are typically used for cooperative cancellation.

If the factory wanted the policy to treat it as a failure, it could throw a different exception type. Using OperationCanceledException signals "skip quietly" - avoiding inflated error metrics for expected behavior (e.g., circuit breaker patterns, rate limiting, conditional probing).

Main pros of this approach:

1. The edge case only applies when a user explicitly implements a custom IProbingRequestFactory that uses cancellation - the default factory doesn't do this, so built-in policies don't need to handle it
2. The factory author controls the behavior via exception type choice
3. It aligns with cancellation semantics: "don't do this, and don't worry about it"

That said, I see the value in giving the policy visibility into skipped probes.
Maybe including cancelled probes in results (with the exception) and lowering to Debug-level logging is a good middle-ground.

Another similar suggestion would be to add a flag on Probing Results which would mark explicit cancellation