Someone tidies up a namespace that has not been used in a while. A few PVCs go with it. Nobody thinks twice, because a PersistentVolumeClaim sounds like bookkeeping: a claim, a request, a bit of Kubernetes paperwork.<\/p>\n\n\n\n
Twenty seconds later the underlying disk no longer exists in AWS. Not detached. Deleted. The reclaim policy on the StorageClass was That is the sharp edge of Kubernetes persistent volumes<\/strong>, and it is a naming problem as much as a technical one. Three objects sit between your pod and a real disk, the relationships between them are not obvious from the names, and the defaults are tuned for ephemeral test clusters rather than for anything holding data you care about.<\/p>\n\n\n\n This is short and covers the parts that bite: reclaim policies, what access modes actually mean, why volumes pin pods to a zone, and the difference between a StatefulSet and a Deployment that matters in practice.<\/p>\n\n\n\n The mental model that prevents most of this:<\/p>\n\n\n\n The PVC is the object your team touches, and it is the one whose name most understates its power. Deleting a PVC can delete a disk. That behaviour comes from the StorageClass, so check what yours says before you need to know:<\/p>\n\n\n\n For anything holding real data, define your own class with The trade-off is real and worth stating: Existing volumes keep whatever policy they were created with, so changing the class does nothing retroactively. Patch them individually:<\/p>\n\n\n\n It means one node<\/em>. Multiple pods scheduled onto the same node can mount the same ReadWriteOnce volume and write to it simultaneously. The Kubernetes documentation says this explicitly, and almost everyone reads the name and assumes otherwise.<\/p>\n\n\n\n For a database, two writers on one filesystem is corruption. If you need genuine exclusivity, there is an access mode for it, stable since Kubernetes 1.29 and available on CSI volumes:<\/p>\n\n\n\n The same access mode causes a second, more visible problem. Put a ReadWriteOnce volume on a Deployment with the default rolling update strategy and you get a deadlock: the new pod cannot start because the volume is still attached to the old pod’s node, and the old pod will not terminate until the new one is ready. The rollout sits there until it times out.<\/p>\n\n\n\n That buys you downtime on every deploy, which is the honest cost of a single-writer volume behind a Deployment. If you cannot accept the downtime, you need either shared storage that supports ReadWriteMany or an application that handles its own replication, which is usually a StatefulSet.<\/p>\n\n\n\n Block storage from a cloud provider lives in one availability zone. A pod using it can only ever be scheduled onto a node in that zone. Nothing in your Deployment spec says so, and the symptom is a pod stuck in The preventive setting is The practical difference is not ordering guarantees or DNS names, useful as those are. It is that a StatefulSet gives each replica its own<\/em> volume, created from a template, with a stable name that survives rescheduling. A Deployment with a PVC gives every replica the same<\/em> volume, which is only correct when you have one replica.<\/p>\n\n\n\n So: one instance with a disk can be a Deployment with StatefulSet PVCs historically survived everything, including deleting the StatefulSet, which is safe and leaves litter. Since Kubernetes 1.32 you can state the policy explicitly, and both fields still default to That pairing is the sensible default for most workloads. Be careful with Volumes grow, they do not shrink.<\/strong> Expansion is stable and works by editing the claim’s requested size, provided the StorageClass allows it and the driver supports it. There is no supported path back. Over-provisioning a PVC is a permanent decision until you migrate the data to a smaller one by hand.<\/p>\n\n\n\n A volume snapshot is not a database backup.<\/strong> Snapshotting a running database’s disk gives you a crash-consistent image: the equivalent of pulling the power cable. Most engines recover from that, and “most” is not a word you want in a recovery plan. Use the database’s own backup mechanism for the database, and treat volume snapshots as infrastructure convenience rather than your restore path. Whichever you choose, restore one before you need to.<\/p>\n\n\n\n Pod stuck in Pending.<\/strong> Read the events at the bottom of PVC stuck in Pending.<\/strong> Rollout hangs forever on a stateful app.<\/strong> ReadWriteOnce plus rolling update. Switch to PVC will not delete.<\/strong> A finalizer is holding it because something still mounts it. Find the consuming pod rather than removing the finalizer by hand, which is how you end up with a PV bound to a claim that no longer exists.<\/p>\n\n\n\n Volume is full but the PVC says it has room.<\/strong> You expanded the PVC and the filesystem did not follow. Most CSI drivers resize the filesystem on the next pod restart; if yours has not, restart the pod and check again.<\/p>\n\n\n\n If the reclaim policy is Yes, if they are on the same node. ReadWriteOnce restricts access to one node, not one pod. Use ReadWriteOncePod when you need a single writer enforced.<\/p>\n\n\n\n Almost always zone affinity: the volume is in one availability zone and no schedulable node is there. The events in You can, with a mature operator that handles failover, backups and upgrades. Whether you should depends on whether your team wants to operate a database platform or just use a database. A managed service is often the better answer and there is no shame in it.<\/p>\n\n\n\n No. Expansion is supported, reduction is not. The only route down is a new smaller volume and a data migration.<\/p>\n\n\n\n Kubernetes storage defaults assume your cluster is disposable. Reclaim policies delete disks, access modes permit more sharing than their names suggest, and volumes constrain scheduling in ways nothing surfaces until a pod will not start.<\/p>\n\n\n\n Go and run Storage is where most Kubernetes migrations get uncomfortable, and it is usually the last thing anyone designs. Work I take on:<\/p>\n\n\n\n Send me the output of Delete<\/code>, which is the default on essentially every managed cluster, and deleting the claim cascaded straight through to the volume.<\/p>\n\n\n\nThree objects, one disk<\/h2>\n\n\n\n
\n
# Read the RECLAIMPOLICY column. On managed clusters it usually\n# says Delete, and nothing anywhere warns you about that.\nkubectl get storageclass<\/code><\/pre>\n\n\n\nRetain<\/code> and use it deliberately:<\/p>\n\n\n\napiVersion: storage.k8s.io\/v1\nkind: StorageClass\nmetadata:\n name: gp3-retain\nprovisioner: ebs.csi.aws.com\nreclaimPolicy: Retain # PV outlives the PVC\nvolumeBindingMode: WaitForFirstConsumer\nallowVolumeExpansion: true\nparameters:\n type: gp3<\/code><\/pre>\n\n\n\nRetain<\/code> means orphaned volumes accumulate and quietly cost money, and a released PV will not automatically rebind to a new claim. You are choosing a cleanup chore over a data loss risk. That is the right trade for a database and the wrong one for a scratch cache.<\/p>\n\n\n\nkubectl patch pv \"$PV_NAME\" \n -p '{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}'<\/code><\/pre>\n\n\n\nReadWriteOnce does not mean one pod<\/h2>\n\n\n\n
spec:\n accessModes:\n - ReadWriteOncePod # enforced by the scheduler<\/code><\/pre>\n\n\n\nspec:\n strategy:\n type: Recreate # old pod goes first, then the new one<\/code><\/pre>\n\n\n\nVolumes pin pods to a zone<\/h2>\n\n\n\n
Pending<\/code> after a node replacement or a scale-up that happened to land elsewhere.<\/p>\n\n\n\nkubectl describe pod<\/code> tells you, in among the scheduler events, that no node matched the volume’s node affinity. It is easy to miss if you are scanning for something more dramatic.<\/p>\n\n\n\nvolumeBindingMode: WaitForFirstConsumer<\/code>, which is in the StorageClass above. It delays provisioning until a pod is actually scheduled, so the volume gets created in a zone where the pod can run rather than the other way round. Set it on every class you create. The immediate-binding alternative provisions a volume somewhere convenient and then constrains your scheduler forever.<\/p>\n\n\n\nStatefulSet or Deployment<\/h2>\n\n\n\n
Recreate<\/code>. Anything that scales and keeps per-replica state should be a StatefulSet.<\/p>\n\n\n\nRetain<\/code>:<\/p>\n\n\n\nspec:\n persistentVolumeClaimRetentionPolicy:\n whenDeleted: Retain # deleting the StatefulSet keeps the data\n whenScaled: Delete # scaling down cleans up the extra replicas<\/code><\/pre>\n\n\n\nwhenScaled: Delete<\/code> on anything where you scale down temporarily and expect the data back when you scale up again.<\/p>\n\n\n\nTwo one-way doors<\/h2>\n\n\n\n
Troubleshooting<\/h2>\n\n\n\n
kubectl describe pod<\/code>. Volume node affinity conflict means a zone mismatch. No storage class means the cluster has no default and the claim did not name one.<\/p>\n\n\n\nkubectl describe pvc<\/code> names the provisioner error. With WaitForFirstConsumer<\/code> this is normal until a pod is scheduled, so check whether anything is actually trying to consume it before debugging the driver.<\/p>\n\n\n\nRecreate<\/code>.<\/p>\n\n\n\nCommon mistakes<\/h2>\n\n\n\n
\n
volumeBindingMode: Immediate<\/code>, then debugging Pending pods for an hour.<\/li>\nBest practices<\/h2>\n\n\n\n
\n
Retain<\/code> for anything you would miss, and the default for everything else.<\/li>\nWaitForFirstConsumer<\/code> on every class you define.<\/li>\nallowVolumeExpansion: true<\/code> from the start, because adding it later does not help existing volumes.<\/li>\nRecreate<\/code> for everything else.<\/li>\npersistentVolumeClaimRetentionPolicy<\/code> rather than relying on the default.<\/li>\n
\n\n\n\nFAQ<\/h2>\n\n\n\n
Does deleting a PVC delete my data?<\/h3>\n\n\n\n
Delete<\/code>, yes, and that is the default nearly everywhere. Run kubectl get storageclass<\/code> and read the RECLAIMPOLICY column before you assume otherwise.<\/p>\n\n\n\nCan two pods share a ReadWriteOnce volume?<\/h3>\n\n\n\n
Why is my pod Pending with a bound PVC?<\/h3>\n\n\n\n
kubectl describe pod<\/code> say so. WaitForFirstConsumer<\/code> prevents it happening again.<\/p>\n\n\n\nShould I run databases in Kubernetes at all?<\/h3>\n\n\n\n
Can I shrink a PersistentVolumeClaim?<\/h3>\n\n\n\n
\n\n\n\nThe one thing to remember<\/h2>\n\n\n\n
kubectl get storageclass<\/code> on your production cluster now. If the reclaim policy on the class holding your database says Delete<\/code>, you have found something worth fixing before lunch.<\/p>\n\n\n\nNeed a hand with stateful workloads?<\/h2>\n\n\n\n
\n
kubectl delete<\/code> would actually destroy.<\/li>\nkubectl get storageclass<\/code> and kubectl get pv<\/code>, and I will tell you what stands out.<\/p>\n\n\n\n