Exception in long running Java11/MySQL8 management server

[yadvr]

For a long-running Java11 management server (based on simulator) running latest master/4.14 with mysql 8.x library the following was seen: http://primate-qa.cloudstack.cloud:8080/client/

2020-03-24 00:50:17,751 ERROR [c.c.u.d.T.Transaction] (Network-Scavenger-1:ctx-7e038324) (logid:eee53f1b) Unexpected exception:
java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142)
        at com.cloud.utils.db.TransactionLegacy.getStandaloneConnectionWithException(TransactionLegacy.java:213)
        at com.cloud.utils.db.TransactionLegacy.getStandaloneConnection(TransactionLegacy.java:222)
        at com.cloud.utils.db.DbUtil.getConnectionForGlobalLocks(DbUtil.java:61)
        at com.cloud.utils.db.DbUtil.getGlobalLock(DbUtil.java:199)
        at com.cloud.utils.db.GlobalLock.lock(GlobalLock.java:154)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$NetworkGarbageCollector.runInContext(NetworkOrchestrator.java:2887)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)

This could hint at an potential long running GC/leak issue or could be solely env issue (in which case could be closed).

ISSUE TYPE

• Bug Report

COMPONENT NAME

Management server

CLOUDSTACK VERSION

master/4.14-snapshot

[yadvr]

@andrijapanicsb @DaanHoogland cc @borisstoyanov @PaulAngus marking it as a blocker, do we have any long-running env (running 7-14+ days). Close this if unable to reproduce.

[yadvr]

FYI - I checked mysql was still running only mgmt server was not working (unable to login etc) and the exceptions were seen. On restart mgmt server works OK.

[weizhouapache]

it might be caused by mysql setting
for example wait_timeout

[andrijapanicsb]

@DaanHoogland did you tell me you had spun Trillian env for this one - can't find it?

[andrijapanicsb]

@DaanHoogland we are NOT running the 8.x library, are we? Not sure our test-env makes any sense...?

[yadvr]

Env details:

# lsb_release  -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic
# uname -a
Linux primate-develop 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
# java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing)
# mysql --version
mysql  Ver 14.14 Distrib 5.7.29, for Linux (x86_64) using  EditLine wrapper

@andrijapanicsb @DaanHoogland please triage and decide severity/priority

[andrijapanicsb]

@DaanHoogland I believe we agreed that this one can be unmarked as blocker (not blocker), as we have not hit this in our long-running idle test env - and in busy env it will be even harder to hit this, if it's due to any timeouts and such.

We'll have the much wider testing during the RC1, so I'm marking this one as NOT blocker, and moving it to... 4.15?

[DaanHoogland]

moved it to 4.15, as discussed with Rohit privately, it might be simulator specific. We must mark it in the release notes that so/what we think about it and hope test proovers give us wisdom.