Issue during docker build in microclimate jenkins pipeline

[arjunsbabu]

Issue type: Bug
Reported in: 1.10 installed on ICP 3.1.0
Description of issue: If the pipeline is initiated docker build is failing with the following error

Step 11/17 : COPY . .
 ---> 39e33e89aed0
Step 12/17 : RUN chown -R node:node /usr/src/app
 ---> Running in afcc3b39aea2
**OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 40\"": unknown**
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

Workaround (if possible): I came to understand the link opencontainers/runc#1740 it could be an issue with page memory and for fixing the issue
we need to run "echo 1 > /proc/sys/vm/drop_caches" in the worker node hosting the microclimate jenkins pod. This issue is happening more than 3 times per day and each time we need to run the above command. I have created a cronjob for the same to run every 5 hours but cannot stop the issue . While checking the node i can see the node is not fully utilized ie we have enough memory available.

May i get some recommendation from Microclimate team for fixing the issue permanently. ?

[sishida]

Hello! Could you please try to update to the newest version of Microclimate?
https://github.com/IBM/charts/blob/master/stable/ibm-microclimate/RELEASENOTES.md
If problems persist after the update, please let us know! Thank you!

[arjunsbabu]

@sishida we cant do it because of microclimate limitation. We can only use ICp 3.1.0 because that is a requirement for another chart ( Watson assistant 1.1.0 ) . Microclimate 1.10 is the highest supported on ICp 3.1.0 . So we cannot proceed with highest version of MC without updating ICp
So let us know how to avoid such issues in MC 1.10.
Is it a known issue ?

[sishida]

I have notified my team of this situation. I will let you know what we find out.

[arjunsbabu]

@sishida any update

[sishida]

@arjunsbabu My team has asked another team to look into this issue. Hopefully we can get answers from them.

[arjunsbabu]

@sishida thanks for the update

[jagraj]

I and Arjun had a call on this issue and here is more information.

HA Cluster configuration.

Masters - 3 nodes
Management - 2 nodes
Proxy - 2 nodes
VA - 1 node

Workers for Watson Assistant - 4 nodes
Other workers shared - 3 nodes (Microclimate, Health coach, Cognitive virtual Assistant, Redis)
Worker config - (2 CPU cores, 8 GB RAM)

This issue happens in pre-prod but not in production environment. Customer restarted one of the node where we were seeing this problem and after that we have not seen this problem so far.

The next action item is to reproduce this problem in one of customer POC environment and I requested to capture monitoring information on these three worker nodes where Microclimate is running.