rbd snapshot delete failed

[div8cn]

ISSUE TYPE

• Bug Report

COMPONENT NAME

CLOUDSTACK VERSION

4.12/4.13

CONFIGURATION

snapshot.backup.to.secondary =false
kvm.snapshot.enabled =true

OS / ENVIRONMENT

system os: CentOS7
ceph : 12.2.11

SUMMARY

When the main storage is RBD and snapshot.backup.to.secondary = false

When deleting volume snapshots, only records in the database are deleted, and snapshots are not deleted on the main storage.

mgt log

2019-09-07 23:27:00,118 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-2e407b61) (logid:445cbea8) ===START=== 192.168.254.3 -- GET command=deleteSnapshot&id=0b50eb7e-4f42-4de7-96c2-1fae137c8c9f&response=json&=1567869534480
2019-09-07 23:27:00,139 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-2e407b61 ctx-679fd276) (logid:445cbea8) CIDRs from which account 'Acct[2f96c108-9408-11e9-a820-0200582b001a-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2019-09-07 23:27:00,204 DEBUG [c.c.a.ApiServer] (qtp504527234-17:ctx-2e407b61 ctx-679fd276) (logid:445cbea8) Retrieved cmdEventType from job info: SNAPSHOT.DELETE
2019-09-07 23:27:00,217 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-2:ctx-f0843047 job-1378) (logid:c34a368a) Add job-1378 into job monitoring
2019-09-07 23:27:00,219 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp504527234-17:ctx-2e407b61 ctx-679fd276) (logid:445cbea8) submit async job-1378, details: AsyncJobVO {id:1378, userId: 2, accountId: 2, instanceType: Snapshot, instanceId: 13, cmd: org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd, cmdInfo: {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"1237","id":"0b50eb7e-4f42-4de7-96c2-1fae137c8c9f","ctxDetails":"{"interface com.cloud.storage.Snapshot":"0b50eb7e-4f42-4de7-96c2-1fae137c8c9f"}","ctxAccountId":"2","uuid":"0b50eb7e-4f42-4de7-96c2-1fae137c8c9f","cmdEventType":"SNAPSHOT.DELETE","":"1567869534480"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 2200502468634, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2019-09-07 23:27:00,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-f0843047 job-1378) (logid:1cee5097) Executing AsyncJobVO {id:1378, userId: 2, accountId: 2, instanceType: Snapshot, instanceId: 13, cmd: org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd, cmdInfo: {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"1237","id":"0b50eb7e-4f42-4de7-96c2-1fae137c8c9f","ctxDetails":"{"interface com.cloud.storage.Snapshot":"0b50eb7e-4f42-4de7-96c2-1fae137c8c9f"}","ctxAccountId":"2","uuid":"0b50eb7e-4f42-4de7-96c2-1fae137c8c9f","cmdEventType":"SNAPSHOT.DELETE","":"1567869534480"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 2200502468634, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2019-09-07 23:27:00,221 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-2e407b61 ctx-679fd276) (logid:445cbea8) ===END=== 192.168.254.3 -- GET command=deleteSnapshot&id=0b50eb7e-4f42-4de7-96c2-1fae137c8c9f&response=json&=1567869534480
2019-09-07 23:27:00,305 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 1-8660140608456756853: Routing from 2199066247173
2019-09-07 23:27:00,305 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-2:ctx-f0843047 job-1378 ctx-f50e25a4) (logid:1cee5097) Can't find snapshot on backup storage, delete it in db

STEPS TO REPRODUCE

Delete volume snapshots

View snapshots in main storage (ceph) (rbd snap list cloudstack/xxxxx)

Delete volume snapshots

In main storage (ceph), view snapshots (rbd snap list cloudstack/xxxx)

EXPECTED RESULTS

The snapshot just deleted has disappeared on the main storage

ACTUAL RESULTS