Failed to attach binaries ISO for VM when create kubernetes cluster

[tungntvn23]

ISSUE TYPE

• Bug Report

COMPONENT NAME

API, Kubernetes Service

CLOUDSTACK VERSION

Cloudstack 4.14.0.0

CONFIGURATION

Advanged Networking, Vmware vSphere

OS / ENVIRONMENT

• Vmware vSphere 6.5
• CentOS 7

SUMMARY

After upgrade from 4.13 to 4.14 and enable kubernetest feature then i cannot create kubernetes cluster.

STEPS TO REPRODUCE

• Login to UI.
• Kubernetes service tab to click Add Kubernetes Cluster.
• Fill all information.
• Click OK

EXPECTED RESULTS

• Kubernetes cluster create successfully.
• State is OK

ACTUAL RESULTS

• Kubernetes cluster create failed.
• State is error.
• Error message:
  "Failed to attach binaries ISO for VM: test-master in the Kubernetes cluster name: test"

Note: i try manual attach binaries iso to vm test-master and get successfully.

[tungntvn23]

Attach management log

Message: Unable to find VM in vSphere to execute AttachIsoCommand, vmName: i-2-1465-VM
 Also check if your guest os is a supported version
2020-09-09 14:50:46,473 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-14:ctx-014abef7 job-114062 ctx-78218c74) (logid:528413cf) Failed to attach binaries ISO for VM: test-master in the Kubernetes cluster name: test
com.cloud.utils.exception.CloudRuntimeException: Failed to attach iso
	at com.cloud.template.TemplateManagerImpl.attachIso(TemplateManagerImpl.java:1217)
	at jdk.internal.reflect.GeneratedMethodAccessor956.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy195.attachIso(Unknown Source)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.attachIsoKubernetesVMs(KubernetesClusterActionWorker.java:320)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.attachIsoKubernetesVMs(KubernetesClusterActionWorker.java:331)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startKubernetesClusterOnCreate(KubernetesClusterStartWorker.java:530)
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1057)
	at org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd.execute(CreateKubernetesClusterCmd.java:273)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:603)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:551)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[shwstppr]

@tungntvn23 Can you please try deploying cluster with Root disk controller=pvscsi and NIC adapter type=Vmxnet3 for CoreOS template? We have had some issue with ISOs and CoreOS VM
apache/cloudstack-documentation#151

[tungntvn23]

@shwstppr thank you, but issue happen randomly. Sometime it's OK but sometime not OK.

[shwstppr]

@tungntvn23 I'll have a look. Is it possible for you to test out a particular PR?

[tungntvn23]

@shwstppr I setting rootDiskController and NIC but it still failed attach iso. I try disable DRS from VMware cluster so it worked fine.

[shwstppr]

@tungntvn23 Will have to check how DRS is affecting attaching ISO. If you have time can you please check if attaching ISO working fine for you everytime for a normal CoreOS VM (outside cks cluster) with DRS set to enabled?
Also, ACS 4.16 could see a new template k8s cluster node VMs which should prevent such issue.

[tungntvn23]

@shwstppr yes, i try many time to attach iso for normal CoreOS VM with DRS enabled and get result OK.