Problem Found When Using HAMi + P800 GPUs for Resource Sharing

[121812]

After starting 4 Pods, when creating additional Pods, they end up in the UnexpectedAdmissionError state.
kubectl describe pod shows:

Annotations:      hami.io/bind-phase: allocating
                  hami.io/bind-time: 1766199067
                  hami.io/vgpu-node: xxxhost.node
                  hami.io/vgpu-time: 1766199067
                  hami.io/xpu-devices-allocated: 00000000-03-00.0,XPU,24576,25:;
                  hami.io/xpu-devices-to-allocate: 00000000-03-00.0,XPU,24576,25:;
Status:           Failed
Reason:           UnexpectedAdmissionError
Message:          Pod was rejected: Allocate failed due to rpc error: code = Unknown desc = empty id from pod annotation, which is unexpected
...
Events:
  Type     Reason                    Age   From            Message
  ----     ------                    ----  ----            -------
  Normal   Scheduled                 57s   hami-scheduler  Successfully assigned default/xpu-test-644668fd7b-dgcff to xxxhost.node
  Normal   FilteringSucceed          58s   hami-scheduler  find fit node(xxxhost.node), 0 nodes not fit, 1 nodes fit(xxxhost.node:9.69)
  Normal   BindingSucceed            58s   hami-scheduler  Successfully binding node [xxxhost.node] to default/xpu-test-644668fd7b-dgcff
  Warning  UnexpectedAdmissionError  58s   kubelet         Allocate failed due to rpc error: code = Unknown desc = empty id from pod annotation, which is unexpected

xpu-device-plugin Logs:

I1220 02:51:11.756934     248 server.go:256] Allocate: &AllocateRequest{ContainerRequests:[]*ContainerAllocateRequest{&ContainerAllocateRequest{DevicesIDs:[11-2],},},}
I1220 02:51:11.760858     248 util.go:130] node annotation key is hami.io/mutex.lock, value is 2025-12-20T02:51:11Z,default,xpu-test-644668fd7b-ndgp2 
I1220 02:51:11.762571     248 util.go:351] checklist is [map[XPU:hami.io/xpu-devices-to-allocate]], annos is [map[hami.io/bind-phase:allocating hami.io/bind-time:1766199070 hami.io/vgpu-node:xxxhost.node hami.io/vgpu-time:1766199069 hami.io/xpu-devices-allocated:00000000-03-00.0,XPU,24576,25:; hami.io/xpu-devices-to-allocate:00000000-03-00.0,XPU,24576,25:;]]
I1220 02:51:11.762596     248 util.go:326] Start to decode container device 00000000-03-00.0,XPU,24576,25:
I1220 02:51:11.762603     248 util.go:346] Finished decoding container devices. Total devices: 1
I1220 02:51:11.762609     248 util.go:373] Decoded pod annos poddevices map[XPU:[[{0 00000000:03:00.0 XPU 24576 25}]]]
I1220 02:51:11.762620     248 server_hami.go:171] pod annotation decode vaule is map[XPU:[[{Idx:0 UUID:00000000:03:00.0 Type:XPU Usedmem:24576 Usedcores:25}]]]
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
2025/12/20 02:51:11 has numa is: false, numa node is: 0
I1220 02:51:11.958755     248 xpu.go:193] device set sriov vf num, busID: 00000000:03:00.0, mode: 2, dmemory: 98304, cmemory: 24576, free: %!b(bool=false), split: 4
I1220 02:51:11.958774     248 server_hami.go:144] uuid not match xpuId: 00000000:63:00.0, containerId: 00000000:03:00
I1220 02:51:11.958781     248 server_hami.go:144] uuid not match xpuId: 00000000:65:00.0, containerId: 00000000:03:00
I1220 02:51:11.958786     248 server_hami.go:144] uuid not match xpuId: 00000000:83:00.0, containerId: 00000000:03:00
I1220 02:51:11.958791     248 server_hami.go:144] uuid not match xpuId: 00000000:85:00.0, containerId: 00000000:03:00
I1220 02:51:11.958797     248 server_hami.go:144] uuid not match xpuId: 00000000:A3:00.0, containerId: 00000000:03:00
I1220 02:51:11.967755     248 util.go:200] Node lock released, node: xxxhost.node, err: <nil>

GPU Status:
All eight P800 GPUs appear to be functioning normally.
However, it seems that only 6 virtual devices are being created.

2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 0
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 1
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 2
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 3
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 4
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 5
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 6
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 7
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 8
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 9
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 10
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 11
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 12
2025/12/20 02:54:27 'kunlunxin.com/vxpu' device marked healthy: 13

Pod Resource Request and xpu-smi Output:
The Pod requests only 1 device, but actually seems to have 2 mapped in:

resources:                                                                                                                                                                         
  limits:                                                                                                                                                                          
    kunlunxin.com/vxpu: "1"                                                                                                                                                        

(py38) root@xxxhost:/workspace# xpu-smi 
Sat Dec 20 10:56:03 2025       
+-----------------------------------------------------------------------------+
| XPU-SMI               Driver Version: 5.0.21.40    XPU-RT Version: 5.0.21   |
|-------------------------------+----------------------+----------------------+
| XPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | XPU-Util  Compute M. |
|                               |             L3-Usage |            SR-IOV M. |
|===============================+======================+======================|
|   0  P800 OAM           N/A   | 00000000:05:00.0 N/A |                    0 |
| N/A   41C  N/A     89W / 400W |  48932MiB / 98304MiB |      0%      Default |
|                               |     64MiB /    96MiB |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  P800 OAM           N/A   | 00000000:85:00.0 N/A |                    0 |
| N/A   41C  N/A     87W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  XPU   XI   CI        PID   Type   Process name                  XPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A        43      C   ...onda/envs/py38/bin/python    48996MiB |
+-----------------------------------------------------------------------------+

Versions:
hami:v2.7.1
vxpu-device-plugin:v1.0.0

[archlitchi]

Is it related to this PR: #1549

[121812]

Is it related to this PR: #1549

Not related to this PR

[ouyangluwei163]

1. Please provide xpu-smi info.
2. Please provide annotation(the hami.io/xpu-devices-allocated) information for the created xpu pod.

[121812]

1. Please provide xpu-smi info.
2. Please provide annotation(the hami.io/xpu-devices-allocated) information for the created xpu pod.

[root@xxxnode ~]# xpu-smi 
Tue Dec 23 16:24:21 2025       
+-----------------------------------------------------------------------------+
| XPU-SMI               Driver Version: 5.0.21.40    XPU-RT Version: 5.0.21   |
|-------------------------------+----------------------+----------------------+
| XPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | XPU-Util  Compute M. |
|                               |             L3-Usage |            SR-IOV M. |
|===============================+======================+======================|
|   0  P800 OAM           N/A   | 00000000:03:00.0 N/A |                    0 |
| N/A   37C  N/A     85W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |              Enabled |
+-------------------------------+----------------------+----------------------+
|   1  P800 VF            N/A   | 00000000:03:00.1 N/A |                  N/A |
| N/A   37C  N/A     85W / 400W |  45604MiB / 49152MiB |      0%      Default |
|                               |     48MiB /    48MiB |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  P800 VF            N/A   | 00000000:03:00.2 N/A |                  N/A |
| N/A   37C  N/A     85W / 400W |      0MiB / 49152MiB |      0%      Default |
|                               |      0MiB /    48MiB |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  P800 OAM           N/A   | 00000000:05:00.0 N/A |                    0 |
| N/A   41C  N/A     89W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |              Enabled |
+-------------------------------+----------------------+----------------------+
|   4  P800 VF            N/A   | 00000000:05:00.1 N/A |                  N/A |
| N/A   41C  N/A     89W / 400W |   2410MiB / 24576MiB |      0%      Default |
|                               |     24MiB /    24MiB |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  P800 VF            N/A   | 00000000:05:00.2 N/A |                  N/A |
| N/A   41C  N/A     89W / 400W |      0MiB / 24576MiB |      0%      Default |
|                               |      0MiB /    24MiB |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  P800 VF            N/A   | 00000000:05:00.3 N/A |                  N/A |
| N/A   41C  N/A     89W / 400W |      0MiB / 24576MiB |      0%      Default |
|                               |      0MiB /    24MiB |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  P800 VF            N/A   | 00000000:05:00.4 N/A |                  N/A |
| N/A   41C  N/A     89W / 400W |      0MiB / 24576MiB |      0%      Default |
|                               |      0MiB /    24MiB |                  N/A |
+-------------------------------+----------------------+----------------------+
|   8  P800 OAM           N/A   | 00000000:63:00.0 N/A |                    0 |
| N/A   37C  N/A     87W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |             Disabled |
+-------------------------------+----------------------+----------------------+
|   9  P800 OAM           N/A   | 00000000:65:00.0 N/A |                    0 |
| N/A   41C  N/A     88W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |             Disabled |
+-------------------------------+----------------------+----------------------+
|  10  P800 OAM           N/A   | 00000000:83:00.0 N/A |                    0 |
| N/A   38C  N/A     86W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |             Disabled |
+-------------------------------+----------------------+----------------------+
|  11  P800 OAM           N/A   | 00000000:85:00.0 N/A |                    0 |
| N/A   41C  N/A     87W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |             Disabled |
+-------------------------------+----------------------+----------------------+
|  12  P800 OAM           N/A   | 00000000:A3:00.0 N/A |                    0 |
| N/A   38C  N/A     86W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |             Disabled |
+-------------------------------+----------------------+----------------------+
|  13  P800 OAM           N/A   | 00000000:A5:00.0 N/A |                    0 |
| N/A   41C  N/A     88W / 400W |      0MiB / 98304MiB |      0%      Default |
|                               |      0MiB /    96MiB |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  XPU   XI   CI        PID   Type   Process name                  XPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    1   N/A  N/A    367980      C                                    1628MiB |
|    1   N/A  N/A    555618      C   python                           14024MiB |
|    4   N/A  N/A    214463      C   python                           1434MiB |
+-----------------------------------------------------------------------------+


[root@xxxnode ~]# kubectl  get  nodes -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      csi.volume.kubernetes.io/nodeid: '{"cephfs.csi.ceph.com":"xxxnode.szlj","rbd.csi.ceph.com":"xxxnode.szlj"}'
      flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"2e:fb:e9:4c:a5:4a"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 10.213.205.87
      hami.io/node-handshake-xpu: Reported_2025.12.23 08:27:56
      hami.io/node-register-vxpu: 00000000:03:00.0-0,00000000:03:00.1-1,00000000:03:00.2-2,00000000:05:00.0-3,00000000:05:00.1-4,00000000:05:00.2-5,00000000:05:00.3-6,00000000:05:00.4-7,00000000:63:00.0-8,00000000:65:00.0-9,00000000:83:00.0-10,00000000:85:00.0-11,00000000:A3:00.0-12,00000000:A5:00.0-13
      hami.io/node-register-xpu: '[{"id":"00000000-03-00.0","count":4,"devmem":98304,"devcore":100,"type":"XPU-P800
        OAM","health":true},{"index":1,"id":"00000000-05-00.0","count":4,"devmem":98304,"devcore":100,"type":"XPU-P800
        OAM","health":true},{"index":2,"id":"00000000-63-00.0","count":4,"devmem":98304,"devcore":100,"type":"XPU-P800
        OAM","health":true},{"index":3,"id":"00000000-65-00.0","count":4,"devmem":98304,"devcore":100,"type":"XPU-P800
        OAM","health":true},{"index":4,"id":"00000000-83-00.0","count":4,"devmem":98304,"devcore":100,"type":"XPU-P800
        OAM","health":true},{"index":5,"id":"00000000-85-00.0","count":4,"devmem":98304,"devcore":100,"type":"XPU-P800
        OAM","health":true},{"index":6,"id":"00000000-A3-00.0","count":4,"devmem":98304,"devcore":100,"type":"XPU-P800
        OAM","health":true},{"index":7,"id":"00000000-A5-00.0","count":4,"devmem":98304,"devcore":100,"type":"XPU-P800
        OAM","health":true}]'
      node.alpha.kubernetes.io/ttl: "0"
      volumes.kubernetes.io/controller-managed-attach-detach: "true"