systemd user service not started at boot: "Found ordering cycle ... Job deleted to break ordering cycle"

Symptoms #

A web front end, run as a systemd user service, was down after a reboot. The reverse proxy in front of it returned 502 and logged connection refused to the local port. The unit was inactive (dead) since boot, and nothing had tried to start it. The user journal from boot had this (unit names made generic):

textdefault.target: Found ordering cycle: myapp-web.service/start
  after myappd.service/start after default.target/start
  - after myapp-web.service
default.target: Job myapp-web.service/start deleted to break ordering cycle

Root cause #

Two user units:

• myappd.service: the daemon. WantedBy=default.target, and After=default.target.
• myapp-web.service: the web front end. WantedBy=default.target, Requires= and After=myappd.service.

WantedBy=default.target makes the target want the unit. A target unit also automatically adds After= for everything it Wants= or Requires= (systemd.target(5)), so default.target is ordered after myapp-web.service. With the explicit lines that gives:

textdefault.target  after  myapp-web.service  after  myappd.service  after  default.target

That is a loop. systemd breaks ordering cycles at boot by deleting one of the jobs in it, here the start job for myapp-web.service, so the web front end never started.

A unit that is WantedBy= a target must not also be After= that same target: the target only counts as reached once the units it wants are up.

Fix #

Remove After=default.target from the daemon's unit (~/.config/systemd/user/myappd.service):

ini[Unit]
Description=myapp daemon

[Service]
Type=simple
ExecStart=/path/to/myappd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Then:

bashsystemctl --user daemon-reload
systemctl --user start myapp-web.service   # pulls in myappd via Requires=

Both units came up, the local port answered, and the proxy stopped returning 502.

If a user service really needs ordering, order it after a more specific unit, not after the target that wants it. Note that services with default dependencies are already ordered After=basic.target (systemd.service(5)), so you do not need to add that yourself.

How it was found #

The proxy logged connection refused to the local port, so the back end was down rather than slow. systemctl --user status myapp-web.service showed inactive (dead) since boot, and the user journal for that boot showed the ordering-cycle message above.

References #

• systemd.target(5): automatic After= dependencies for units a target wants.
• systemd.service(5): implicit dependencies of service units.