Volume snapshot listing failing when originating primary pool removed

[DagSonsteboSB]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Volume snapshots in ACS4.11.2

CLOUDSTACK VERSION

• ACS 4.11.2

CONFIGURATION

• Any incorporating volume snapshots

OS / ENVIRONMENT

• N/A

SUMMARY

• Volume snapshot present as far as CloudStack and Secondary storage goes
• Original primary storage pool ID has been removed
• listSnapshots will fail with:
  Error: (HTTP 530, error code 9999)

2019-04-05 14:08:18,176 ERROR [c.c.a.ApiServer] (qtp788117692-312:ctx-6993074e ctx-97c28410) (logid:b9e936a0) unhandled exception executing api command: [Ljava.lang.String;@3ab4697e
java.lang.NullPointerException

STEPS TO REPRODUCE

• As above, take volume snapshot, then delete original primary storage pool.
• Confirm the snapshot has two entries in cloud.snapshot_store_ref - one for primary and one for image.
• Ensure the primary pool is marked as deleted.
• Do a listSnapshots API call against the snapshot and observe above errors.
• Now change the snapshot_store_ref entry for primary for the snapshot in question to a non-deleted primary pool.
• Do the listSnapshots API call again and observe this working.

EXPECTED RESULTS

• listSnapshots returning results

ACTUAL RESULTS

• Error as above.

[nvazquez]

Error is NullPointerException on StorageSubsystemSnapshotStrategy, as the pool is removed, this line returns an empty pool: https://github.com/apache/cloudstack/blob/4.11.2.0/engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/StorageSystemSnapshotStrategy.java#L976

2019-04-05 13:48:39,507 ERROR [c.c.a.ApiServer] (qtp788117692-12:ctx-817b96c3 ctx-5c399d14) (logid:9277d33d) unhandled exception executing api command: [Ljava.lang.String;@2829384
java.lang.NullPointerException
        at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:978)
        at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
        at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
        at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
        at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
        at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:134)
        at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:536)
        at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:732)

Possible workarounds without deleting the snapshot:

• Remove the entry on snapshot_store_ref with store_role = Primary for conflicting snapshot.
• Update the pool_id column on snapshot_store_ref for conflicting snapshot to a non-removed pool id.

[yadvr]

@shwstppr can you pick this up?

[shwstppr]

working on this

[shwstppr]

Unable to reproduce this on the master,

• Create disk and create snapshot for it [There are 2 entries in snapshot_store_ref]
• Put originating primary storage pool on maintenance and try to force remove
  Error: (HTTP 530, error code 4250) Cannot delete pool Primary as there are associated non-destroyed vols for this pool
• Destroy created disk or migrate created disk to a different primary storage
  It deletes snapshot entry from snapshot_store_ref where store_role=Primary
  Removing primary storage pool and list snapshots thereafter doesn't give NPE.

Any pointers here @nvazquez

[yadvr]

@shwstppr how about if you remove the volume as well (but not its full snapshots) and then delete the primary storage pool?

[DaanHoogland]

@shwstppr did you abandon this? any intermediate results?

[shwstppr]

@DaanHoogland was not able to reproduce this earlier. Let me try again today

[yadvr]

@shwstppr if nothing else, you can still follow the code where NPE was seen and make code around that defensive.

[yadvr]

Ping @shwstppr @DaanHoogland @andrijapanicsb

[shwstppr]

@rhtyd @DaanHoogland tried to reproduce with 4.13 but couldn't

• Created snapshot, noticed two entries in snapshot_store_ref

select * from snapshot_store_ref\G;
*************************** 1. row ***************************
                id: 2
          store_id: 1
       snapshot_id: 1
           created: 2020-02-07 10:43:47
      last_updated: NULL
            job_id: NULL
        store_role: Image
              size: 5368709120
     physical_size: 197120
parent_snapshot_id: 0
      install_path: snapshots/2/4/0dfc98e6-af66-4636-9dac-2175c473b7b9
             state: Destroyed
      update_count: 4
           ref_cnt: 0
           updated: 2020-02-07 10:55:48
         volume_id: 8
*************************** 2. row ***************************
                id: 3
          store_id: 3
       snapshot_id: 2
           created: 2020-02-07 10:56:03
      last_updated: NULL
            job_id: NULL
        store_role: Primary
              size: 5368709120
     physical_size: 5368709120
parent_snapshot_id: 0
      install_path: /mnt/ff714bd3-b656-38f9-a210-350e71acf93b/bc6cda75-8359-4c5b-9b7f-178dc098c7ba/c131c60d-d6eb-4f39-9d71-855089244bfa
             state: Ready
      update_count: 2
           ref_cnt: 0
           updated: 2020-02-07 10:56:03
         volume_id: 8
*************************** 3. row ***************************
                id: 4
          store_id: 1
       snapshot_id: 2
           created: 2020-02-07 10:56:03
      last_updated: NULL
            job_id: NULL
        store_role: Image
              size: 5368709120
     physical_size: 197120
parent_snapshot_id: 0
      install_path: snapshots/2/8/c131c60d-d6eb-4f39-9d71-855089244bfa
             state: Ready
      update_count: 2
           ref_cnt: 0
           updated: 2020-02-07 10:56:04
         volume_id: 8

• Deleted volume as without deleting/migrating was not being able to delete originating primary storage
• Deleted primary storage, primary storage entry gets removed from snapshot_store_ref

> select * from snapshot_store_ref\G;
*************************** 1. row ***************************
                id: 2
          store_id: 1
       snapshot_id: 1
           created: 2020-02-07 10:43:47
      last_updated: NULL
            job_id: NULL
        store_role: Image
              size: 5368709120
     physical_size: 197120
parent_snapshot_id: 0
      install_path: snapshots/2/4/0dfc98e6-af66-4636-9dac-2175c473b7b9
             state: Destroyed
      update_count: 4
           ref_cnt: 0
           updated: 2020-02-07 10:55:48
         volume_id: 8
*************************** 2. row ***************************
                id: 4
          store_id: 1
       snapshot_id: 2
           created: 2020-02-07 10:56:03
      last_updated: NULL
            job_id: NULL
        store_role: Image
              size: 5368709120
     physical_size: 197120
parent_snapshot_id: 0
      install_path: snapshots/2/8/c131c60d-d6eb-4f39-9d71-855089244bfa
             state: Ready
      update_count: 2
           ref_cnt: 0
           updated: 2020-02-07 10:56:04
         volume_id: 8
2 rows in set (0.00 sec)

• List snapshots worked fine,

> list snapshots 
{
  "count": 1,
  "snapshot": [
    {
      "account": "admin",
      "created": "2020-02-07T10:56:02+0000",
      "domain": "ROOT",
      "domainid": "215d2545-4987-11ea-9a78-1e0021010871",
      "id": "424f92d7-298e-46ff-9c13-6cd7cc820979",
      "intervaltype": "MANUAL",
      "name": "test12",
      "physicalsize": 197120,
      "revertable": false,
      "snapshottype": "MANUAL",
      "state": "BackedUp",
      "tags": [],
      "virtualsize": 5368709120,
      "volumeid": "8b7bee30-c20b-4ed1-9a0a-fc930b1016d7",
      "volumename": "DATA-3",
      "volumetype": "DATADISK",
      "zoneid": "025f0bd8-942f-44ea-9f94-c2c7851c935a"
    }
  ]
}

Should I try with 4.11?

[yadvr]

@shwstppr that's fine, no need. Can you however do a manual static code analysis and add defensive checks that avoid/handle potential NPEs if any:

2019-04-05 13:48:39,507 ERROR [c.c.a.ApiServer] (qtp788117692-12:ctx-817b96c3 ctx-5c399d14) (logid:9277d33d) unhandled exception executing api command: [Ljava.lang.String;@2829384
java.lang.NullPointerException
        at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:978)
        at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
        at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
        at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
        at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
        at org.apache.cloudstack.storage.snaps

Otherwise, let's close this as not reproducible.