ploop volumes hanging around after deletion

[jaxxstorm]

I create a pvc:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                 STORAGECLASS    REASON    AGE
pvc-d8faddb2-0d09-11e8-b2cc-001c42746431   10Gi       RWO            Delete           Bound     default/pagey-mongo   vzprovisioner             6s

And I can see it exists:

[root@tukd-srv-0080 k8s-volumes]# ploop info kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3/DiskDescriptor.xml
Opening delta /vstorage/tukd-pool-04/k8s-volumes/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3
Opening delta /vstorage/tukd-pool-04/k8s-volumes/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3
Adding delta dev=/dev/ploop26165 img=/vstorage/tukd-pool-04/k8s-volumes/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3 (rw)
Mounted /dev/ploop26165p1 at /vstorage/tukd-pool-04/k8s-volumes/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3.mnt fstype=ext4 data=',balloon_ino=12'
Unmounting file system at /vstorage/tukd-pool-04/k8s-volumes/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3.mnt
Unmounting device /dev/ploop26165
Opening delta /vstorage/tukd-pool-04/k8s-volumes/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3/kubernetes-dynamic-pvc-ff4f3069-8022-11e7-ab4f-001c42b2b9a3
   resource           Size           Used
  1k-blocks       20509264          45080
     inodes        1310720             12

However, once I delete the pvc:

kubectl delete -f pagey_aio.yaml
namespace "pagey" deleted
persistentvolumeclaim "pagey-mongo" deleted
ingress "pagey-admin" deleted
service "mongo" deleted
service "pagerbot-admin" deleted
service "pagerbot" deleted
deployment "mongo" deleted
deployment "pagerbot-admin" deleted
deployment "pagerbot" deleted

And I also ensure the pv is deleted:

kubectl delete pv pvc-d8faddb2-0d09-11e8-b2cc-001c42746431                                                                                   
persistentvolume "pvc-d8faddb2-0d09-11e8-b2cc-001c42746431" deleted

But the disk persists:

# ls -alh /vstorage/tukd-pool-04/k8s-volumes/kubernetes-dynamic-pvc-d8faddb2-0d09-11e8-b2cc-001c42746431
total 23M
drwx------ 1 root root   0 Feb  8 19:54 .
drwx------ 1 root root   0 Jul 17  2017 ..
-rwx------ 1 root root 841 Feb  8 19:54 DiskDescriptor.xml
-rwx------ 1 root root   0 Feb  8 19:54 DiskDescriptor.xml.lck
-rwx------ 1 root root 22M Feb  8 19:54 kubernetes-dynamic-pvc-d8faddb2-0d09-11e8-b2cc-001c42746431

What am I missing here?

[dmission]

Logs are appreciated. I suspect the problem could be due to pvc deletion before deployment. I.e. ploop was busy at that moment.

[avagin]

I think we need revoke leases for ploop files before removing them:
avagin@2759ea8

I built a docker image with these changes: docker.io/avagin/virtuozzo-provisioner:test

[jaxxstorm]

Controller logs:

E0211 22:13:38.571046       1 goroutinemap.go:166] Operation for "delete-pvc-5a9648b6-0df4-11e8-9f69-001c42746431[606b2c80-0df4-11e8-9f69-001c42746431]" failed. No retries permitted until 2018-02-11 22:13:39.571026659 +0000 UTC (durationBeforeRetry 1s). Error: recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil>) Call stack:
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:153
/usr/lib/golang/src/runtime/asm_amd64.s:514
/usr/lib/golang/src/runtime/panic.go:489
/usr/lib/golang/src/runtime/panic.go:63
/usr/lib/golang/src/runtime/signal_unix.go:290
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/virtuozzo-provisioner.go:274
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:1021
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:504
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:130
/usr/lib/golang/src/runtime/asm_amd64.s:2197
E0211 22:13:38.571089       1 goroutinemap.go:166] Operation for "delete-pvc-5a90ae0c-0df4-11e8-9f69-001c42746431[5c2b4195-0df4-11e8-9f69-001c42746431]" failed. No retries permitted until 2018-02-11 22:13:39.571077215 +0000 UTC (durationBeforeRetry 1s). Error: recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil>) Call stack:
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:153
/usr/lib/golang/src/runtime/asm_amd64.s:514
/usr/lib/golang/src/runtime/panic.go:489
/usr/lib/golang/src/runtime/panic.go:63
/usr/lib/golang/src/runtime/signal_unix.go:290
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/virtuozzo-provisioner.go:274
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:1021
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:504
/home/smelwani/go/src/github.com/virtuozzo/vzstorage-pd/vendor/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:130
/usr/lib/golang/src/runtime/asm_amd64.s:2197

[avagin]

@jaxxstorm What is a commit id for a source code what produced this backtrace?