Deployments with failed mounts dont get deleted

[Phillezi]

Describe the bug
Deployments that experience a mount failure cannot be deleted. The deletion process appears to hang or fail due to issues with removing the associated PersistentVolumeClaim (PVC). It seems like there is a create storage job that still tries to create it everytime.

---

To Reproduce
Steps to reproduce the behavior:

1. Create a deployment with a volume mount that will fail (e.g., invalid NFS path or storage configuration).
2. Wait for the deployment to enter a failed or crash-loop state due to the mount issue.
3. Attempt to delete the deployment.
4. Observe that the deletion process fails or times out.

---

Expected behavior
Deployments should be deleted even if the PVC is missing, invalid, or cannot be mounted. The deletion logic should either skip over non-existent PVCs or handle stuck PVCs gracefully without blocking the overall deployment deletion.

---

Screenshots
N/A (the issue is visible in logs).

---

Additional context
This issue likely occurs because the deletion job waits indefinitely for a PVC that is either:

• already deleted,
• stuck in a terminating state, or
• never created successfully due to an earlier mount error.

Log excerpt:

{"level":"info","ts":1760083581.4472415,"logger":"default","msg":"failed job 069fe4da-3436-42c6-b991-b41432f8f545 (deleteDeployment), attempt: 1/5 delay: 30s \n\tdue to: failed to delete deployment. \n\tdue to: failed to delete k8s for deployment 05fd7745-860b-4c23-a1a2-c1a183943a1d. \n\tdue to: failed to delete k8s pvc root-philipzi-trino-coordinator-1. \n\tdue to: timeout waiting for pvc root-philipzi-trino-coordinator-1 to be deleted"}

Potential fix:
Add a safeguard in the deletion logic to detect non-existent or stuck PVCs / cancel all jobs related to it and proceed with deployment deletion without timing out.