Swarm, troubleshooting

When the kitchen won’t come together.

Real problems from putting a Windows laptop and a Linux home server in one swarm, each with what solved it. Every fix below was run on those two machines; the panels are there to remember them by.

The setup A laptop with Windows 11 and Docker in WSL, with networkingMode=mirrored, as the manager (192.168.1.14). A Linux home server on the same network as the worker (192.168.1.16). New to swarms? Read the comic first.

The new cook knocks at the back door with a perfectly good ticket, and nobody inside hears it. The door is the firewall.

Problem 1

The join times out after 20 seconds

The manager at 192.168.1.14:2377 did not answer within 20 seconds. Usually a firewall on the manager blocks port 2377 (under WSL also the Hyper-V firewall), or its address cannot be reached from here.

What you see

The machine that joins waits, gives up, and stays in the state error. From that machine, port 2377 on the manager does not open.

Why

With mirrored networking, WSL shares the address of the laptop, but traffic coming in to WSL goes through the Hyper-V firewall. That firewall only lets through ping and mDNS for WSL; the swarm ports are not in it.

What solved it

Open the swarm ports for WSL, only for your own network. In PowerShell as administrator; the GUID is the one WSL uses for its virtual machine, and 192.168.1.0/24 is the home network.

$wsl = '{40E0AC32-46A5-438A-A0B2-2B479E8F2E90}'
New-NetFirewallHyperVRule -Name 'swarm-tcp' -DisplayName 'Docker Swarm TCP' -Direction Inbound -VMCreatorId $wsl -Protocol TCP -LocalPorts 2377,7946 -RemoteAddresses 192.168.1.0/24 -Action Allow
New-NetFirewallHyperVRule -Name 'swarm-udp' -DisplayName 'Docker Swarm UDP' -Direction Inbound -VMCreatorId $wsl -Protocol UDP -LocalPorts 7946,4789 -RemoteAddresses 192.168.1.0/24 -Action Allow

How to check

From the machine that wants to join, the port opens:

timeout 5 bash -c '</dev/tcp/192.168.1.14/2377' && echo open

The kitchen phone is dead, while the street outside is as busy as ever. Hang up and dial again: restart WSL.

Problem 2

WSL cannot reach anything, and commands hang

What you see

Pulling an image in WSL fails with "no route to host" while Windows itself is online. Inside WSL, even ss and ip hang. The swarm ports stay closed, whatever the firewall says.

Why

Mirrored networking can get stuck, for example after the laptop has slept or moved to another network. Nothing in the swarm is wrong; WSL is simply not on the network.

What solved it

Restart WSL. Everything in it stops for a moment, the engine and its containers too; the swarm itself is kept and comes back when dockerd starts.
```
wsl --shutdown
wsl -e true
```

How to check

Inside WSL, Docker Hub answers again (401 is the right answer here), and the manager listens on its ports:

curl -s -o /dev/null -w "%{http_code}\n" https://registry-1.docker.io/v2/
ss -ltn | grep -E ':(2377|7946)'

A cook who knocked while the door was shut keeps waiting at it, even after it opens. Send them home first, then let them in the right way.

Problem 3

A machine keeps trying to join

This machine keeps trying in the background; leaving the swarm stops that.

What you see

The machine that joined while the port was closed stays in the state error, "context deadline exceeded", even after the port is open.

Why

A join that failed does not start over by itself in a way you can rely on. The machine holds on to the attempt.

What solved it

On that machine, let go of the old attempt:
```
docker swarm leave --force
```
Then join again, with the address the others should use for it. In the app: Join a swarm… on that machine, through a manager that is also in the app; it fetches the token itself.
```
docker swarm join --token <worker token> --advertise-addr 192.168.1.16 192.168.1.14:2377
```

How to check

On the manager, the machine is Ready:

docker node ls

Orders passed through the hatch to the chef’s side get lost on the way. So the chef stops cooking and only gives orders: drain the WSL machine.

Problem 4

Half the requests time out

What you see

A service with a task on the laptop and one on the server answers every other request. The requests that the routing mesh sends to the other machine never come back, from either side.

Why

Between machines, containers talk over VXLAN on 4789/udp. Under mirrored networking that traffic never reaches WSL, even with the firewall rule: the VXLAN interface in WSL sends, but its receive counter stays at zero.

What solved it

Let the WSL machine give the orders and the servers do the cooking. Drain it: it stays a manager, and its tasks move to the server.
```
docker node update --availability drain <name of the laptop node>
```
Reach the service through the address of the server, not the laptop.

How to check

Every request answers, spread over the tasks:

for i in 1 2 3 4 5 6; do curl -s http://192.168.1.16:8088/ | grep Hostname; done

The price: with the laptop drained, the tasks have nowhere to go while the only server is off. How the problem showed itself: inside WSL, nsenter --net=/var/run/docker/netns/1-… ip -s link showed TX going up with every request and RX staying at 0.

Docker Desktop cooks in a kitchen without a street door. Whoever stands outside cannot get in, however good their ticket.

Problem 5

Nobody can join a swarm that Docker Desktop manages

What you see

A machine that joins a swarm whose manager is Docker Desktop never gets in and ends up in the state error.

Why

Docker Desktop runs in a virtual machine of its own and announces an address inside it, 192.168.65.x. No other machine can reach that address.

What solved it

Let a machine that others can reach be the manager: a server, or Docker in WSL with mirrored networking and the firewall rules above. The app says this when you start a swarm on Docker Desktop.

Cas goes home at nine every night. That is fine for a cook. A second chef who does the same would leave the kitchen without a majority.

Problem 6

A server that switches off every night

What you see

The home server goes off at nine in the evening. While it is off, the manager lists it as Down.

Why

Managers decide by majority. With two managers, one going away leaves no majority, and nobody can change the swarm until it is back.

What solved it

Keep a machine that switches off a worker; do not promote it. Its tasks move to another node if there is one, and it takes part again when it is back on.

How to check

The server has no manager status:

docker node ls

The ports, once more

What has to be open on every machine in the swarm, for the machines in your own network. Under WSL that is the Hyper-V firewall; on a Linux server with ufw, the same ports there.

Port	Between	For
2377/tcp	Every node to the managers	Joining, and managing the swarm
7946/tcp and udp	Every node to every node	Who is in the swarm, and where
4789/udp	Every node to every node	Traffic between containers on different machines

The app says most of this before you run into it.

Where the engine runs, which address others can reach, which ports a firewall has to let through, and what went wrong when a join does not get through.

What the app does with a swarm →Swarm, the comic →