Finished workloads show ADMITTED=False and empty RESERVED IN when listing workloads

[IrvingMg]

What happened:

After a workload successfully finishes, kubectl get workloads shows ADMITTED: False and RESERVED IN: empty. As shown below:

NAME                         QUEUE        RESERVED IN   ADMITTED   FINISHED   AGE
job-sample-job-8xmgl-34ac3   user-queue                 False      True       5m59s

This happens in both single-cluster and multi-cluster setups.

What you expected to happen:

Workloads should show ADMITTED: True or False, whichever value applies, rather than always False, and should show RESERVED IN: for historical and audit purposes.

How to reproduce it (as minimally and precisely as possible):

1. E2E_RUN_ONLY_ENV=true make kind-image-build test-e2e
2. kubectl apply -f https://kueue.sigs.k8s.io/examples/admin/single-clusterqueue-setup.yaml
3. kubectl create -f ./examples/jobs/sample-job.yaml

Anything else we need to know?:

I believe this is an undesired side effect of #7724, which clears the admission status when a workload finishes.

Environment:

• Kubernetes version (use kubectl version):
• Kueue version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[IrvingMg]

/cc @mbobrovskyi @mimowo

[mimowo]

This looks expected to me, I think this behavior was always underspecified and confusing. We got multiple asks from users why QuotaReserved=True for finished workloads.

Ok maybe we could somehow display still the information where the workload was reserving. I think this would solve itself if we decoupled the Reserved in column into two CQ and QuotaReserved

[IrvingMg]

Ok maybe we could somehow display still the information where the workload was reserving. I think this would solve itself if we decoupled the Reserved in column into two CQ and QuotaReserved

I agree. I think having the cluster queue in its own column would be better, and I also suggest keeping the real value in the ADMITTED column. This could help users quickly understand when a workload was not actually admitted. Even though we can use the FINISHED column to see this, I am not sure if it is enough. Still, having both columns might make things less confusing.

Additionally, I think we should update our documentation with the new expected outputs, like in https://kueue.sigs.k8s.io/docs/tasks/run/jobs/

[mimowo]

I agree. I think having the cluster queue in its own column would be better, and I also suggest keeping the real value in the ADMITTED column

What is "real" value here? I'm thinking the simplest model is one column per condition: QuotaReserved, Admitted, Finished. Plus column for CQ and LQ.

[IrvingMg]

What is "real" value here?

By “real value,” I mean that the ADMITTED column should show True if the workload was admitted, or False otherwise. But currently, it seems to always show ADMITTED: False after a workload finishes, which seems confusing to me.

[mimowo]

Oh, I think we now also transition "Admitted" to "False" for finished workloads, so this columns shows the "real" value, but the undelying value of the condition (status) is changed in 0.15.

[IrvingMg]

Yes, as I understand it, the transition of ADMITTED to False is also happening in #7724. Currently, we see ADMITTED: True while the workload is running, but after it finishes, it transitions to False. I think it would avoid confusion to preserve it as True. WDYT?

[mimowo]

Hm, I think Admitted=True might also be confusing as Admitted=False.

We could draw some similarities from Pods.

So for Pods when they complete they transition Ready=False, with reason PodCompleted. OTOH, they keep PodScheduled=True. The reasoning to transition Ready=False when Pod completes is that Ready is a "liveness" condition rather than lifecycle milestone (terminal). PodScheduled and Initialized are "lifecycle milestones" which never change.

In Kueue the model is more complex. QuotaReserved and Admitted aren't really "lifecycle milestones" (terminal), because they can change back and forth. So they are more like "liveness" point-in-time conditions.

[IrvingMg]

I see. Then it sounds like having a Phase column might be useful, similar to the status column we have in kueuectl. The values could be Pending | QuotaReserved | Admitted | Finished. This way, the workload information would look like:

NAME        QUEUE       CLUSTER QUEUE    PHASE      AGE                                                                                                                                      
my-wl-123   user-queue  cluster-queue    Finished   5m                                                                                                                                       
job-abc     dev-queue   cluster-queue    Admitted   2m  

However, I think this would require an API change, because we would need to add a new Phase field to the workload schema.

[mimowo]

Yes, I remember @tenzen-y has some reservations about adding the "Phase" field to workload, but this usecase of integration with kubectl would make it very handy. wdyt @tenzen-y ?