Manager Unavailable

Symptom #

The console, Manager API, or zwallctl cannot reach the Manager. Typical operator observations:

• browser cannot load /login
• zwallctl ping / zwallctl login connection refused, TLS errors, or HTTP 503
• Agents cannot heartbeat because the Manager listener is down
• a load balancer marks the Manager unhealthy

Likely verified causes #

1. zwall-manager.service is inactive, failed, or restarting.
2. Listeners are not admitted: TLS files missing while server.tls.enabled is true, or bind/port mismatch.
3. PostgreSQL is down, unreachable, or database.ssl_mode does not match the server (example default require).
4. Operator curls or points zwallctl at the wrong port (example YAML 8443 vs zwallctl default http://localhost:8080).
5. Config path prefix mismatch: Enterprise unit uses -config /etc/zwall-manager/manager.yml; Community installer writes $CONFIG_DIR/manager.yml (often /etc/zcs); Virtualization Agent is /etc/zcs/agent.yml.
6. HTTP client against a TLS Manager, or cookie issuance without HTTPS (Secure HTTPS session required).
7. Liveness 503 because updater runtime mode is draining or updating (in-product updater is not the supported production upgrade path).
8. Readiness 503 with cluster: timeout when cluster is enabled. This is not a public HA product.

Diagnostic commands/endpoints #

Enterprise unit name is zwall-manager. Community setup also installs zwall-manager.service. Follow the unit that is actually running.

sudo systemctl status zwall-manager
sudo journalctl -u zwall-manager -n 200 --no-pager
sudo systemctl status postgresqlCopy

Health probes (unauthenticated):

# Example Manager listener from configs/manager.example.yml
curl -sS --cacert /etc/zcs/certs/manager.crt \
  https://manager.example.com:8443/health/live

curl -sS --cacert /etc/zcs/certs/manager.crt \
  https://manager.example.com:8443/health/readyCopy

If you only need to learn whether anything is listening, and TLS is the unknown, a one-shot curl -k distinguishes “TLS mismatch” from “connection refused”. Do not leave -k in monitoring.

Wrong-port check:

curl -sS http://127.0.0.1:8080/health/live
curl -sS https://127.0.0.1:8443/health/liveCopy

Optional operator CLI (uses the same Manager URL as zwallctl; default is http://localhost:8080):

zwallctl ping --manager-url https://manager.example.com:8443Copy

Do not treat GET /health or GET /api/v1/health as readiness. Prefer /health/live and /health/ready.

What each result means #

systemd #

`GET /health/live` #

`GET /health/ready` #

Timeout is health.readiness_timeout (code default 5s when set). 503 body uses status: "not_ready" and checks.

200 body includes at least status: "ready", timestamp, database: "ok". Extra keys (cluster, blocking_authority) are additive.

Port and TLS #

Path prefix #

Editing the wrong file does not change the running process.

Safe remediation #

1. If the unit is dead: fix the journal error, then sudo systemctl start zwall-manager. Do not combine this with -migrate on the running service (migration flags are exclusive Manager modes that then exit).
2. If live is 200 and ready is 503 with database checks: restore PostgreSQL, confirm database.ssl_mode (disable, require, verify-ca, verify-full; example require), and confirm Enterprise database_password systemd credential / Community DB_PASSWORD contract. See Database.
3. If ready reports listener_admission_incomplete: place cert and key at the configured server.tls.cert_file / key_file, or generate an operator certificate with scripts/generate_tls_certs.sh (see TLS). Restart the Manager after files exist.
4. If the client uses 8080: set --manager-url https://manager.example.com:8443 or ZWALL_MANAGER_URL. Do not change the Manager listen port only to match zwallctl folklore.
5. If HTTP vs HTTPS: use https:// and the operator trust store. Do not set Agent allow_insecure_remote_tls: true to “fix” the Manager.
6. If login returns Secure HTTPS session required: enable TLS and connect over HTTPS. Do not disable cookie security to force a session.
7. If live is 503 draining/updating: this is updater runtime, not a certified production upgrade. Do not document or perform cluster member failover. Wait for the process to leave that mode or revert the unsupported updater action on a lab host.
8. After any config change: Enterprise unit supports systemctl reload (HUP). Treat HUP as a reload request, not a certified live-apply of every key. Restart if readiness still fails.

Do-not-do warnings #

• Do not treat /health/live 200 as ready.
• Do not tell operators to fail over Manager cluster. Cluster gRPC example port 50051 is inter-manager, not a public HA product.
• Do not publish or copy unit Documentation= lab URLs (Hosting unit contains an internal lab URL; omit it).
• Do not mix Community source binaries with Enterprise verified-release units.
• Do not run -migrate against a live Manager as a “restart.” -migrate applies schema then exits.
• Do not use curl | sudo bash installers.
• Do not disable server.tls.enabled in production to make health checks easier.
• Do not put JWT, database password, or admin password on the curl command line.

Escalation condition #

Escalate to engineering / vendor when:

• live is 200, listeners should be admitted, TLS files exist and are readable by the service user, PostgreSQL HealthCheck succeeds from the same host, and ready still returns listener_admission_incomplete
• ready reports blocking_authority failure after the blocking pipeline is required and the database is otherwise healthy
• cluster checks fail and cluster.enabled was intentionally set — there is no certified customer HA runbook
• the unit crash-loops on credential custody after setup.sh --manager-secret-custody (that path is implemented but not finally security-accepted)

Related documentation #

• Health
• API health
• Service control
• Database troubleshooting
• TLS hardening
• Network exposure
• CLI troubleshooting
• Paths and ports

Edition #

SHARED probes and example ports. Enterprise unit user zwall-manager, empty capabilities, config /etc/zwall-manager/manager.yml. Community installer unit currently User=root with JWT_SECRET in the unit environment. See Manager authentication.

Status #

SUPPORTED for live/ready/systemd diagnosis. NOT_SUPPORTED: Manager cluster as customer HA. NOT_VERIFIED as a single canonical port or filesystem prefix (port mismatch and path-prefix inconsistency).