go tests sometimes may fail #40
Description
Go tests may fail with the following output. Seems to be a bug with test files but not sure yet.
$ make docker-build
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/bin/controller-gen "crd:maxDescLen=0,trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
mkdir -p /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin
test -f /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh || curl -sSLo /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh
source /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh; fetch_envtest_tools /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin; setup_envtest_env /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin; go test ./... -coverprofile cover.out
/bin/sh: 1: source: not found
/bin/sh: 1: fetch_envtest_tools: not found
/bin/sh: 1: setup_envtest_env: not found
? github.com/paddleflow/paddle-operator [no test files]
? github.com/paddleflow/paddle-operator/api/v1 [no test files]
Running Suite: Controller Suite
===============================
Random Seed: 1618384660
Will run 1 of 1 specs
• Failure [1.193 seconds]
PaddleJob controller
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/controllers/paddlejob_controller_test.go:18
test wide-and-deep in service mode [It]
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/controllers/paddlejob_controller_test.go:98
Expected success, but got an error:
<*errors.StatusError | 0xc0003d75e0>: {
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "Operation cannot be fulfilled on paddlejobs.batch.paddlepaddle.org \"wide-and-deep-service\": the object has been modified; please apply your changes to the latest version and try again",
Reason: "Conflict",
Details: {
Name: "wide-and-deep-service",
Group: "batch.paddlepaddle.org",
Kind: "paddlejobs",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 409,
},
}
Operation cannot be fulfilled on paddlejobs.batch.paddlepaddle.org "wide-and-deep-service": the object has been modified; please apply your changes to the latest version and try again
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/controllers/paddlejob_controller_test.go:94
------------------------------
2021-04-14T15:17:51.090+0800 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2021-04-14T15:17:51.090+0800 INFO controller-runtime.manager.controller.paddlejob Starting EventSource {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "source": "kind source: /, Kind="}
2021-04-14T15:17:51.090+0800 INFO controller-runtime.manager.controller.paddlejob Starting EventSource {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "source": "kind source: /, Kind="}
2021-04-14T15:17:51.090+0800 INFO controller-runtime.manager.controller.paddlejob Starting EventSource {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "source": "kind source: /, Kind="}
2021-04-14T15:17:51.090+0800 INFO controller-runtime.manager.controller.paddlejob Starting EventSource {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "source": "kind source: /, Kind="}
2021-04-14T15:17:51.090+0800 INFO controller-runtime.manager.controller.paddlejob Starting Controller {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob"}
2021-04-14T15:17:51.090+0800 INFO controller-runtime.manager.controller.paddlejob Starting workers {"reconciler group": "batch.paddlepaddle.org", "reconciler kind": "PaddleJob", "worker count": 1}
2021-04-14T15:17:51.090+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "48"}
2021-04-14T15:17:51.104+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "50"}
2021-04-14T15:17:51.104+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"50"}, "reason": "Created", "message": "created wide-and-deep-service-ps-0"}
2021-04-14T15:17:51.109+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "52"}
2021-04-14T15:17:51.109+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"52"}, "reason": "Created", "message": "created wide-and-deep-service-ps-0"}
2021-04-14T15:17:51.111+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"52"}, "reason": "Created", "message": "created wide-and-deep-service-ps-1"}
2021-04-14T15:17:51.111+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "52"}
2021-04-14T15:17:51.117+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "56"}
2021-04-14T15:17:51.117+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"56"}, "reason": "Created", "message": "created wide-and-deep-service-ps-1"}
2021-04-14T15:17:51.121+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "56"}
2021-04-14T15:17:51.121+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"56"}, "reason": "Created", "message": "created wide-and-deep-service-ps-2"}
2021-04-14T15:17:51.127+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "61"}
2021-04-14T15:17:51.127+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"61"}, "reason": "Created", "message": "created wide-and-deep-service-ps-2"}
2021-04-14T15:17:52.132+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "65"}
2021-04-14T15:17:52.132+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"65"}, "reason": "Created", "message": "created wide-and-deep-service-worker-0"}
2021-04-14T15:17:52.139+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "69"}
2021-04-14T15:17:52.139+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"69"}, "reason": "Created", "message": "created wide-and-deep-service-worker-0"}
2021-04-14T15:17:52.142+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "69"}
2021-04-14T15:17:52.142+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"69"}, "reason": "Created", "message": "created wide-and-deep-service-worker-1"}
2021-04-14T15:17:52.148+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "73"}
2021-04-14T15:17:52.149+0800 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PaddleJob","namespace":"default","name":"wide-and-deep-service","uid":"595f6b08-9032-4111-8234-b7c88e38a0e3","apiVersion":"batch.paddlepaddle.org/v1","resourceVersion":"73"}, "reason": "Created", "message": "created wide-and-deep-service-worker-1"}
2021-04-14T15:17:52.166+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "77"}
2021-04-14T15:17:52.172+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "77"}
2021-04-14T15:17:52.179+0800 INFO controllers.PaddleJob Reconcile {"paddlejob": "default/wide-and-deep-service", "version": "78"}
Summarizing 1 Failure:
[Fail] PaddleJob controller [It] test wide-and-deep in service mode
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/controllers/paddlejob_controller_test.go:94
Ran 1 of 1 Specs in 11.493 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestAPIs (11.49s)
FAIL
coverage: 42.9% of statements
FAIL github.com/paddleflow/paddle-operator/controllers 11.536s
FAIL
make: *** [Makefile:21: test] Error 1
This is not very reproducible. Most times it just succeeds.
$ make test
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/home/zzy/go/src/github.com/Ruminateer/paddle-operator/bin/controller-gen "crd:maxDescLen=0,trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
mkdir -p /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin
test -f /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh || curl -sSLo /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh
source /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin/setup-envtest.sh; fetch_envtest_tools /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin; setup_envtest_env /home/zzy/go/src/github.com/Ruminateer/paddle-operator/testbin; go test ./... -coverprofile cover.out
/bin/sh: 1: source: not found
/bin/sh: 1: fetch_envtest_tools: not found
/bin/sh: 1: setup_envtest_env: not found
? github.com/paddleflow/paddle-operator [no test files]
? github.com/paddleflow/paddle-operator/api/v1 [no test files]
ok github.com/paddleflow/paddle-operator/controllers 6.290s coverage: 47.8% of statements