Health

Manager health and readiness

Purpose #

Prove whether the Manager process is alive and whether it is ready to serve traffic. These are different questions. A live process can still be unreadiness if PostgreSQL, listener admission, or (when enabled) cluster health has failed.

Audience #

Operators who start, restart, or load-balance the Manager, and anyone diagnosing “console down.”

Edition #

SHARED. Community and Enterprise expose the same four HTTP probes. Enterprise may add cluster and blocking-authority checks to readiness when those subsystems are enabled. Cluster itself is license-gated and not a public HA product.

Status #

SUPPORTED as unauthenticated process probes. Do not treat them as a Kubernetes product integration; source comments mention probe style only.

Prerequisites #

• Manager HTTP(S) listener reachable from the operator host
• The real listen URL (example YAML uses https://manager.example.com:8443; unconfigured zwallctl uses http://localhost:8080)

Required permissions #

None. GET /health/live, GET /health/ready, GET /health, and GET /api/v1/health are unauthenticated.

GET /metrics is not a health probe. When metrics.enabled is true it requires an administrator JWT/session and RoleAuth("admin").

Dependencies #

Readiness depends on:

• HTTP listener admission (listenerAdmissionReady and servers serving)
• PostgreSQL HealthCheck()
• Optional blocking-authority readiness when the blocking pipeline is required
• Optional cluster health when cluster.enabled is true (not a customer HA procedure)

Configuration #

zwall-manager.service does not wrap these probes. Operators, a local load balancer, or a watchdog call them.

Enterprise unit facts that affect whether probes can succeed:

• User=zwall-manager
• ExecStart=/opt/zwall-manager/bin/zwall-manager -config /etc/zwall-manager/manager.yml
• After=postgresql.service
• ReadWritePaths=/var/lib/zwall-manager /var/log/zwall-manager /run/zwall-manager

Community source installs often use /etc/zcs/manager.yml. Follow the unit actually installed.

Procedure #

Replace the URL with the Manager you actually run.

curl -k https://manager.example.com:8443/health/live
curl -k https://manager.example.com:8443/health/ready
curl -k https://manager.example.com:8443/healthCopy

Use -k only when you have not yet installed a trusted CA for the Manager certificate. Production scrapers should verify TLS.

To confirm the unit, not just the HTTP path:

systemctl is-active zwall-manager
journalctl -u zwall-manager -n 100 --no-pagerCopy

Exclusive Manager flags that do not start listeners (-migrate, -preflight-migrate, -check-schema, -migration-exec-preflight, -validate-runtime) will not keep /health/live up. Choose only one of those exclusive modes; they exit after the check.

Verification #

GET /health/live #

Liveness: the process is running and not in an updater drain.

200:

{"status": "alive", "timestamp": 1710000000}Copy

timestamp is Unix seconds.

503 when updater runtime mode is draining or updating:

{"status": "draining", "timestamp": 1710000000}Copy

(status is the snapshot mode: draining or updating.)

Do not treat HTTP 200 as “database healthy.”

GET /health/ready #

Readiness: listeners admitted and PostgreSQL health check passed. If cluster is enabled, cluster health is included.

200 includes at least:

503 uses the same object with status: "not_ready" and populated checks. Example failed check strings from source:

• manager: listener_admission_incomplete
• database: not_configured
• database: timeout
• database: plus the health-check error text
• cluster: timeout when cluster is enabled and the readiness timeout fires
• blocking_authority: plus state when the blocking pipeline is required and not ready

Additive keys on the same object include cluster and blocking_authority. Treat extra keys as additive, not as a second public schema.

GET /health and GET /api/v1/health #

Compatibility health (healthCheck). Prefer /health/live and /health/ready for automation.

Success JSON includes status: "ok", database, timestamp, version. 503 when draining/updating or when the database is not ok (status: "degraded" in that path). Exact keys besides the live/ready contract are not a second public schema.

Expected result #

A load balancer that routes on liveness only will send traffic at a Manager that cannot talk to PostgreSQL. Use readiness for that decision.

Failure modes #

Troubleshooting #

1. systemctl status zwall-manager — is the unit active?
2. journalctl -u zwall-manager -n 200 — config validation, JWT/database secret resolution, TLS file missing.
3. Call live, then ready. If live fails, do not debug PostgreSQL first.
4. Confirm you are not hitting zwallctl’s default http://localhost:8080 while the Manager listens on TLS 8443.
5. Enterprise production: YAML database.password and auth.jwt_secret must stay empty; missing systemd credentials prevent a ready server.

Security considerations #

• Probes are unauthenticated. Expose them to a local load balancer or operator network, not as an internet information leak if you can avoid it. They do not return secrets.
• Do not put administrator JWTs on health check command lines.
• /metrics is admin-equivalent. Never scrape it without the same care as /api/v1.

Limitations #

• Not a Kubernetes product. You may point kubelet-style probes at these URLs if you operate your own cluster; Zwall does not ship that integration.
• Live does not imply schema version correctness. Schema checks are -check-schema / -validate-runtime (Enterprise).
• Blocking-authority and cluster keys on readiness are additive. Do not build a public HA product story from them.

Related documentation #

• API: health probes
• Manager unavailable
• Service control
• Manager administration
• First verification