Cloudstack Management Improper shutdown - causes mgmt to not start after reboot of server

[rvalle]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Cloudstack Manager

CLOUDSTACK VERSION

4.13.0.0

CONFIGURATION

OS / ENVIRONMENT

Centos 7, following quick installation guide.

SUMMARY

I believe a Cloudstack Management Server shutdown will shutdown mysql and the Cloudstack Manager in no particular order, which results in instability.

STEPS TO REPRODUCE

Install the Cloudstack Management server followign the quick install guide,
when finished, reboot the system.
A lot of exceptions appear in the log files during the Manager shutdown pointing at the database not being available.

The system may become "broken" after restart.

EXPECTED RESULTS

The server should be rebooted cleanly, all components (manager, mysql, etc) in the right order, no exceptions thrown.

ACTUAL RESULTS

Lots of exceptions during shutdown.
Non working system after reboot.

[rvalle]

I am pretty sure the installed Systemd scripts don't observe the order in which components should be shutdown, can anybody confirm?

I will get 2 pastebins with the exceptions during the shutdown and after restart.

[rvalle]

Exceptions during shutdown: https://pastebin.com/YWCJqEir
Exceptions after restart: https://pastebin.com/RWRPCSd1

[DaanHoogland]

@rvalle can you confirm this with 4.14 as well?

[rvalle]

@DaanHoogland should be feasible. are there packages in any centos repo?

I can see I have frozen my playbooks at 4.13.

Let me check with 4.14

[rvalle]

I cannot find 4.14 in the release repo. Is there a nightly build repo? If there is a repo I can run my tests easily. And could also check 4.11 LTS

[DaanHoogland]

@rvalle 4.14 is not released and would be a snapshot of master. if you can't build yourself there are nightly builds around.

[rvalle]

@DaanHoogland I am currently migrating/extending an ansible collection of roles to manage my cloud form opennebula to cloudstack. The interesting thing is that I did setup a cloud Continuous Integration infrastructure. On every push a 3 node cloudstack cluster is created and tested .

Let me advance a little bit more on this front and I would be happy to integrate this CI with a Nightly Builds or Travis that generates packages that I can take. We can discuss.

[rvalle]

Systemd seems to support 2 approaches,

• The obvious one, to fix the provided Systemd files for CS Manager.
• The second one, is to supplement existing configuration files with additional files that fine tune its behaviour.
  The second one is useful in the (unlikely) case we needed to modify mysql unit file or in case we want to patch installation based on versions of CS Manager with this bug.

[DaanHoogland]

I think the best solution will be somewhere in the middle. @rvalle, are you taking the time to propose a change?

[rvalle]

@DaanHoogland I can certainly put together a PR for option 1, to add the dependency to Mysql the mysql service.

As you tagged the issue I believe it is critical. In particular with a reboot right after installation. By the log files, I believe you are using a Java Persistence framework, and probably it is likely that will have a cache layer.

I still think that it is useful do document the second approach somewhere, becuse it is likely that any deployment will have additional dependencies.