PVs shows with wrong storageClass after upgrade from 0.25.0 to 0.26.1

[hreinking]

What happened?

We have vclusters since 0.14 (or earlier), which some were created with fake-pv=true. Before upgrading a vcluster called vcluster--sshd-gateway, there are 2 PVs labeled as fake-pv:

kubectl get pv -l vcluster.loft.sh/fake-pv=true | grep sshd-gateway
pvc-081794f4-0f6d-4937-a6ec-ff60a5640d6d   10Gi       ROX            Delete           Bound         vcluster--sshd-gateway/slac-apps-x-dev-x-vcluster--sshd-gateway                                        wekafs--sdf-k8s01            <unset>                          721d
pvc-0bf51f69-eb2a-4fcf-be6b-612a00ecebfc   10Gi       ROX            Delete           Bound         vcluster--sshd-gateway/slac-apps-x-prod-x-vcluster--sshd-gateway                                       wekafs--sdf-k8s01            <unset>                          707d

then, after upgrading to 0.26.1 (we tried first 0.26.0), we see:

pvc-081794f4-0f6d-4937-a6ec-ff60a5640d6d   10Gi       ROX            Delete           Bound         vcluster--sshd-gateway/slac-apps-x-dev-x-vcluster--sshd-gateway                                                          vcluster-wekafs--sdf-k8s01-x-vcluster--sshd-gateway--c4bd828f58   <unset>                          721d
pvc-0bf51f69-eb2a-4fcf-be6b-612a00ecebfc   10Gi       ROX            Delete           Bound         vcluster--sshd-gateway/slac-apps-x-prod-x-vcluster--sshd-gateway                                                         vcluster-wekafs--sdf-k8s01-x-vcluster--sshd-gateway--c4bd828f58   <unset>                          707d

which is a non-existing storageClass, causing a fair amount of failed APIServer Events.

What did you expect to happen?

After upgrading, PV should keep the original storageClass that it had prior to the upgrade.

How can we reproduce it (as minimally and precisely as possible)?

1. Create a vCluster with --set sync.fake-persistentvolumes.enabled: true. For us they were created with minimum values in version v0.7.1
2. Create a PV+PVC in the vcluster context without specifying the storageClass (so it defaults)
3. the PV and PVC should show the correct storageClass both at the vcluster and parent context
4. Upgrade the vcluster 1 minor at a time up to 0.25.0.
5. Upgrade from 0.25.0 to 0.26.

Anything else we need to know?

Our default storageClass has always been wekafs--sdf-k8s01.

Within the vcluster context the PVs looks as they should:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS        VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-081794f4-0f6d-4937-a6ec-ff60a5640d6d   10Gi       ROX            Delete           Bound    dev/slac-apps          wekafs--sdf-k8s01   <unset>                          721d
pvc-0bf51f69-eb2a-4fcf-be6b-612a00ecebfc   10Gi       ROX            Delete           Bound    prod/slac-apps         wekafs--sdf-k8s01   <unset>                          707d

sync.fake-persistentvolumes.enabled: true was disabled in release 0.14.

Host cluster Kubernetes version

```console $ kubectl version Client Version: v1.32.5 Kustomize Version: v5.5.0 Server Version: v1.32.5 ```

vcluster version

```console $ vcluster --version vcluster version 0.27.0 ```

VCluster Config

# Sync describes how to sync resources from the virtual cluster to host cluster and back.
sync:
  toHost:
    ingresses:
      enabled: true
    volumeSnapshots:
      enabled: true
    volumeSnapshotContents:
      enabled: true
    persistentVolumes:
      enabled: true
  fromHost:
    csiDrivers:
      enabled: true
    csiNodes:
      enabled: true
    csiStorageCapacities:
      enabled: true
    storageClasses:
      enabled: true
    ingressClasses:
      enabled: true
    nodes:
      enabled: true
controlPlane:
  distro:
    k8s:
      enabled: true
      version: "${K8S_VERSION}"
      imagePullPolicy: ""
      image:
        registry: ghcr.io
        repository: "loft-sh/kubernetes"
        tag: "$v1.32.5"
      apiServer:
        enabled: true
backingStore:
    database:
      embedded:
        enabled: false
      external:
        enabled: false
  backingStore:
    etcd:
      deploy:
        enabled: true
          highAvailability:
            replicas: 3
  statefulSet:
    image:
    repository: "loft-sh/vcluster-oss"
policies:
  resourceQuota:
    enabled: false
  limitRange:
    enabled: false
telemetry:
  enabled: false

[neogopher]

Hi @hreinking 👋 ,

Thanks for writing-in.
I tried to reproduce the issue by following the steps that you provided but was not able to do so. The persistent volumes always have the correct storageclass name after the upgrade.
The only time I could get the storage class with a translated name (like vcluster-wekafs--sdf-k8s01-x-vcluster--sshd-gateway--c4bd828f58) is by having sync.toHost.storageClasses.enabled=true and then subsequently creating the storage class inside the virtual context and PV-PVC. But I see this is not possible in your case since you have your PVs with ~700d+.

Do you happen to remember if there ever was a misconfiguration of the storageclass sync setting during the upgrade? Also do you still see the 'vcluster-wekafs--sdf-k8s01-x-vcluster--sshd-gateway--c4bd828f58' storage class on the host cluster? If so you would be able to check its creation timestamp to determine if it was created before or after the upgrade.

[hreinking]

Hi @neogopher !

You made my hesitate for a sec.... No, we are not syncing back to parent cluster (as we call it), I've just double check:

sync:
  toHost:
    # StorageClasses defines if storage classes created within the virtual cluster should get synced to the host cluster.
    storageClasses:
      # Enabled defines if this option should be enabled.
      enabled: false

We've been deleting and recreating (and rsync'ing) the PV+PVC's, so that one in particular is no longer present, but we still have some such us:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                              STORAGECLASS        VOLUMEATTRIBUTESCLASS   REASON   AGE    VOLUMEMODE
pvc-2c56c04f-1aa2-4f2f-a67e-1521746d5424   1000Gi     RWO            Delete           Bound    vcluster--usdf-prompt-processing/data-prompt-processing-zookeeper-0-x-kafka-x-vcluste-21eefe01ba   wekafs--sdf-k8s01   <unset>                          535d   Filesystem

and the manifest is:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi.weka.io
    vcluster.loft.sh/object-host-name: pvc-2c56c04f-1aa2-4f2f-a67e-1521746d5424
    vcluster.loft.sh/object-kind: /v1, Kind=PersistentVolume
    vcluster.loft.sh/object-name: pvc-2c56c04f-1aa2-4f2f-a67e-1521746d5424
    vcluster.loft.sh/object-uid: 527c32c7-94d7-4a27-a158-6cde003862db
    volume.kubernetes.io/provisioner-deletion-secret-name: csi-wekafs-api-secret
    volume.kubernetes.io/provisioner-deletion-secret-namespace: csi-wekafs
  creationTimestamp: "2024-03-25T20:08:56Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-provisioner.volume.kubernetes.io/finalizer
  labels:
    vcluster.loft.sh/fake-pv: "true"
  name: pvc-2c56c04f-1aa2-4f2f-a67e-1521746d5424
  resourceVersion: "3624133557"
  uid: 37f1dfa1-a642-42b5-b46e-afc4e7e8eaf1
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1000Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: data-prompt-processing-zookeeper-0-x-kafka-x-vcluste-21eefe01ba
    namespace: vcluster--usdf-prompt-processing
    resourceVersion: "1351277848"
    uid: 2c56c04f-1aa2-4f2f-a67e-1521746d5424
  csi:
    controllerExpandSecretRef:
      name: csi-wekafs-api-secret
      namespace: csi-wekafs
    controllerPublishSecretRef:
      name: csi-wekafs-api-secret
      namespace: csi-wekafs
    driver: csi.weka.io
    nodePublishSecretRef:
      name: csi-wekafs-api-secret
      namespace: csi-wekafs
    nodeStageSecretRef:
      name: csi-wekafs-api-secret
      namespace: csi-wekafs
    volumeAttributes:
      capacityEnforcement: HARD
      filesystemName: sdf-k8s01
      mountOptions: acl,writecache
      permissions: "0777"
      storage.kubernetes.io/csiProvisionerIdentity: 1708017119564-612-csi.weka.io
      volumeType: dir/v1
    volumeHandle: dir/v1/sdf-k8s01/csi-volumes/pvc-2c56c04f-1aa2-4f2f-a67e-1521746d5424-7aad6a7006f9f653cdedfbf881bd3d31b5bc629c
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.csi.weka.io/accessible
          operator: In
          values:
          - "true"
        - key: topology.csi.weka.io/global
          operator: In
          values:
          - "true"
  persistentVolumeReclaimPolicy: Delete
  storageClassName: wekafs--sdf-k8s01
  volumeMode: Filesystem
status:
  phase: Bound

Don't know if I mentioned it already, but it also happened with other storageClasses (zfs-localpv and local-path-provisioner), so even though I would have been tempted to blame weka-csi, it also happened with other csi operators.

Let'm know if I can provide more info. From 135 affected PVs, we are down to 12, but still might be helpful (for someone else) to not go through this pain....

Thanks!