Garbage snapshots are left on Primary and Secondary storage after a volume snapshot is deleted

[andrijapanicsb]

ISSUE TYPE

• Bug Report

COMPONENT NAME

snapshots handling

CLOUDSTACK VERSION

4.11.2, 4.13.0.0 tested

CONFIGURATION

Xs and NFS or KVM and NFS/Ceph (have not tested VMware)

OS / ENVIRONMENT

SUMMARY

When a snapshot is deleted (manually, or as part of the scheduled snap policy), snapshot is NOT deleted from Primary Storage (except KVM + Ceph, which is being fixed in #3615) and is also NOT deleted from Secondary Storage.
DB entries are inconsistent/partially cleared.

STEPS TO REPRODUCE

XS+NFS tested as following
Make snap (XS plus NFS), delete snap - snap is left on Primary Storage AND on Secondary Storage.
Same goes for scheduled snap policy. "Primary" store ref row is deleted from the snapshots_store_ref but not really deleted on Primary Store. "image" store ref row is NOT deleted from snapshots_store_ref and image is NOT deleted on Secondary Store (after muiltiple runs of storage clean up thread i.e. storage.cleanup.delay=60, storage.cleanup.delay=60.

KVM+Ceph tested as per the updates in #3615 and will be tracker here.

NOTE: XenserverSnapshotStrategy is NOT handling only XS, also some KVM
Pay attention to the managed storage.
This whole "garbage" problem was not there in 4.8 at least (possibly also 4.9).

EXPECTED RESULTS

snapshot is deleted in both primary and secondary storage (after the cleanup thread runs)

ACTUAL RESULTS

garbage is left, left and right

[andrijapanicsb]

@PaulAngus @rhtyd @weizhouapache @shwstppr can we progress this one for the 4.14 - I originally planned this one for 4.13.1 but I'm afraid it will not be fixed before the 4.14 - so we want to avoid it being fixed in 4.13.1 and still broken in 4.14

[GabrielBrascher]

Updating the current status of the issue (see fix at PR #3649).

The code was not changed in the meantime; however, the delay was due to a lack of time and resources (from my side and the reviewers). Yesterday I was able to deploy a XenServer cluster and run some tests and assert that the PR fixes this issue for XenServer + NFS as well.