Docker swarm mesh networking not working

[yangm97]

The only relevant log I could find on docker was Jan 17 20:37:33 scw-f118f1 dockerd[3495]: time=“2018-01-17T20:37:33.047047682Z” level=error msg=“Failed to deserialize netlink ndmsg: Link not found"

Steps to reproduce:

• init a swarm
• join on scaleway node
• deploy something like docker service create
  --name my-web
  --publish published=8080,target=80
  --replicas 1
  nginx
• try to curl from a scaleway host which doesn’t have a copy of the service running
• curl will hang there, then say it has timed out

[Vad1mo]

what kernel do you use? I head no problem running swarm make sure you have the right kernel you can google for kubernetes/scaleway. Same kernel recommendations apply to swarm.

[yangm97]

@Vad1mo I tried most of the available kernels, same results every time.

[Vad1mo]

What's actually not working? What is the error message?

I had Swarm running without any problems a few months ago.

[yangm97]

https://docs.docker.com/engine/swarm/ingress/#publish-a-port-for-a-service

Suppose the scaleway node is in the same situation as node3, it should forward the requests to a node running the published service. What happens is some error in the communication between nodes and the client connection times out.

[Vad1mo]

when setting up swarm mode you should use the private IP address not the public ones.

[yangm97]

You do need to use the public IP when your cluster is located on more than one datacenter. In my case, I set up a swarm between amazon, azure and scaleway instances. Communication between amazon and azure nodes went just fine.

[Vad1mo]

yes because scaleway public IPs are NAT to private instance. This is a different to AWS/Azure. This might be the problem.

[yangm97]

They all use NAT. And I disabled all the three firewalls.

When running nmap, I found something interesting: while amazon and azure nodes would always keep the communication port open, the scaleway node would inconsistently open and close said port. Actually, you might be able to get some traffic a few moments after restarting, but the port will shut closed soon enough.

[Vad1mo]

do you see why the connection is closed? Any message/status code? HTTP or TCP level?

[yangm97]

I might have overlooked it, but the only message I saw on the docker daemon which could be relevant was the single line on the op. Now I regret not dumping the full log, as I already teared down this setup (but would bring up again if needed).

[doublemcz]

+1
I am not even in any cloud - We have hosted private cluster and nodes has been joined over private network (not any NAT)

[piec]

Apparently the issue is the kernel conf of the default images, there are missing modules that are needed for the overlay network. The “Failed to deserialize netlink ndmsg: Link not found" is unrelated
See:

• Services not being able to reach each other on ports exposed in underlying containers moby/moby#28168 (comment)
• https://adminswerk.de/docker-swarm-scaleway/
• https://github.com/scaleway/image-ubuntu/issues/78

[r3pek]

having exactly the same problem service ports are not published.
Using the private IP also isn't ideal since if you reboot your server, you kill the swarm because the IP address get's changed. EDIT: (or poweroff / poweron)

[r3pek]

BTW, to fix the network issue I had to change the bootscript to the 4.15.* one. The one with "Docker" in the name didn't work.