1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling..

[zwt-1234]

What happened:
worker:
replicas: 1
vgpu:
enabled: true # <- 新增这一行，开启 HAMi vGPU
resources:
requests:
cpu: "2"
memory: "8Gi"
limits:
nvidia.com/gpu: "4"
nvidia.com/gpucores: 10 # 占用 15% GPU 核心
nvidia.com/gpumem-percentage: 5 # 占用 10% 显存
ports:

• containerPort: 30001
  protocol: TCP

Command:
xinference-worker
Args:
-e
http://service-supervisor:9997
--host
$(POD_IP)
--worker-port
30001
--log-level
debug
Limits:
nvidia.com/gpu: 4
nvidia.com/gpucores: 10
nvidia.com/gpumem-percentage: 5
Requests:
cpu: 2
memory: 8Gi
nvidia.com/gpu: 4
nvidia.com/gpucores: 10
nvidia.com/gpumem-percentage: 5
Environment:
POD_IP: (v1:status.podIP)
XINFERENCE_PROCESS_START_METHOD: <set to the key 'XINFERENCE_PROCESS_START_METHOD' of config map 'xinferencecluster-distributed-config'> Optional: false
XINFERENCE_MODEL_SRC: <set to the key 'XINFERENCE_MODEL_SRC' of config map 'xinferencecluster-distributed-config'> Optional: false
XINFERENCE_HOME: <set to the key 'XINFERENCE_HOME' of config map 'xinferencecluster-distributed-config'> Optional: false
Mounts:
/dev/shm from shm (rw)
/opt/xinference from xinference-home (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-flwq4 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
xinference-home:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: xinferencecluster-distributed-home-pvc
ReadOnly: false
shm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: 128Gi
kube-api-access-flwq4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/arch=amd64
Tolerations: node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
nvidia.com/gpu:NoSchedule op=Exists
Events:
Type Reason Age From Message

---

Warning FailedScheduling 67s hami-scheduler 0/4 nodes are available: 1 NodeUnfitPod, 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 2 node unregistered. preemption: 0/4 nodes are available: 1 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod..
Warning FilteringFailed 67s hami-scheduler no available node, 3 nodes do not meet

nvidia.com/gpu: "4" If there are more than two, an error will be reported
What you expected to happen: Enable and allocate VGPU normally

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

• The output of nvidia-smi -a on your host
• Your docker or containerd configuration file (e.g: /etc/docker/daemon.json)
• The hami-device-plugin container logs
• The hami-scheduler container logs
• The kubelet logs on the node (e.g: sudo journalctl -r -u kubelet)
• Any relevant kernel output lines from dmesg

Environment:

• HAMi version:
• nvidia driver or other AI device driver version:
• Docker version from docker version
• Docker command, image and tag used
• Kernel version from uname -a
• Others:

[zwt-1234]

你好 这个为什么在增加完vgpu超过两个就会报这个问题

[wawa0210]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Hello, why does this problem appear after adding more than two vgpu?

[zwt-1234]

我有两张卡 看到日志已经分割了各十个vgpu 但是在nvidia.com/gpu: "4"配置添加为3的时候pod就会输出Warning FailedScheduling 67s hami-scheduler 0/4 nodes are available: 1 NodeUnfitPod, 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 2 node unregistered. preemption: 0/4 nodes are available: 1 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod..
Warning FilteringFailed 67s hami-scheduler no available node, 3 nodes do not meet

[zwt-1234]

机器人检测到正文语言不是英语，已自动翻译。👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

您好，为什么添加两个以上的虚拟GPU后会出现这个问题？

看到日志事件输出已划分十个vgpu 但是我在pod的配置中添加为nvidia.com/gpu：“4”就会输出报错：Warning FailedScheduling 67s hami-scheduler 0/4 nodes are available: 1 NodeUnfitPod, 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 2 node unregistered. preemption: 0/4 nodes are available: 1 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod..
Warning FilteringFailed 67s hami-scheduler no available node, 3 nodes do not meet
超过2个就会报这个错

[zwt-1234]

看到调度器pod日志中已经将两张卡分割成各十张 但我的服务pod只能分到两张 什么原因

[wawa0210]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

I see that the two cards have been divided into ten cards each in the scheduler pod log. But my service pod can only be divided into two cards. Why?

[zwt-1234]

我的服务pod只能分到两个gpu 超过两个就会报错1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling..

[zwt-1234]

Hello, what is the reason for this situation and how can it be resolved? Can you help me take a look

[zwt-1234]

It's okay to allocate one or two cards, but if there are more than two, an error will be reported

[Shouren]

It's okay to allocate one or two cards, but if there are more than two, an error will be reported

@zwt-1234 How many physical GPUs exists on the node? As far as i know, HAMi can't allocate more GPUs to a single container than the number of physical GPUs on that node.

[zwt-1234]

宿主机只有两个物理GPU

分配一两张卡是可以的，但如果超过两张，系统会报错。

@zwt-1234节点上有多少个物理GPU？据我所知，HAMi分配给单个容器的GPU数量不能超过该节点上物理GPU的数量。

节点上有两个物理GPU

[wawa0210]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The host has only two physical GPUs

It is OK to allocate one or two cards, but if there are more than two, the system will report an error.

@zwt-1234 How many physical GPUs are there on a node? As far as I know, HAMi cannot assign more GPUs to a single container than the number of physical GPUs on that node.

There are two physical GPUs on the node

[zwt-1234]

It's okay to allocate one or two cards, but if there are more than two, an error will be reported

@zwt-1234 How many physical GPUs exists on the node? As far as i know, HAMi can't allocate more GPUs to a single container than the number of physical GPUs on that node.

There are two physical GPUs on the node, can adding replicas allocate more

[Shouren]

There are two physical GPUs on the node, can adding replicas allocate more

@zwt-1234 What is the replicas means? Does it mean the number of physical GPUs or the number of Pod ?