Zwall
Talk to an Engineer
Talk to an Engineer
Documentation
Docs Home › Troubleshooting › Agent Disconnected

Agent Disconnected

SharedStatus: SUPPORTED for heartbeat, timeout, token lifetime, and Hosting 409/400 contractsAudience: incident operator

Symptom

An Agent that previously registered is offline in the console, zwallctl list-agents, or GET /api/v1/agents. Heartbeats stop. Hosting status is stale. Virtualization maps stop updating.

Likely verified causes

  1. Agent process stopped (zwall-agent, zwall-hosting-agent, or loader).
  2. Manager unreachable (treat as Manager Unavailable first).
  3. Manager agents.timeout elapsed (example "2m") since last heartbeat. Example Agent heartbeat_interval: 30s.
  4. Agent JWT expired and refresh failed. Example: auth.agent_token_ttl: "8h", auth.agent_refresh_before: "20m", auth.agent_expired_grace: "2m".
  5. Clock skew. Hosting observation status CollectedAt must be within ±15 minutes of Manager time or the heartbeat is rejected.
  6. TLS: Agent verify_tls: true; allow_insecure_remote_tls: false.
  7. Hosting heartbeat contract violation → 400 Invalid hosting observation heartbeat (must be observation_only with Hosting profile identity; no active-control payload).
  8. Hosting binding missing → 403 Hosting observation registration is not authorized.
  9. Operator used POST /api/v1/agents/:id/restart or zwallctl restart --agent on a Hosting Agent → 409 Hosting observation Agents cannot receive restart commands.
  10. Heartbeat hostname does not match the authenticated Agent principal → 403 Access denied.
  11. Heartbeat without Agent JWT → 401 Agent authentication required.

Diagnostic commands/endpoints 

zwallctl login --manager-url https://manager.example.com:8443 --username admin
zwallctl list-agents
zwallctl status --agent <id>
GET /api/v1/agents
GET /api/v1/agents/:id
POST /api/v1/agents/heartbeat
POST /api/v1/agents/token/refresh

Heartbeat is Agent JWT. Enterprise may wrap capability + optional mTLS (heartbeatWithMTLS). Body hostname is protocol-compatible but must identify the authenticated principal.

On the node:

sudo systemctl status zwall-agent
sudo journalctl -u zwall-agent -n 200 --no-pager

# Hosting
sudo systemctl status zwall-hosting-ebpf-loader
sudo systemctl status zwall-hosting-agent
sudo journalctl -u zwall-hosting-agent -u zwall-hosting-ebpf-loader -n 200 --no-pager

Hosting admin (Enterprise):

GET /api/v1/agents/:id/hosting/status

Restart API (admin):

POST /api/v1/agents/:id/restart

zwallctl restart --agent ID calls that path. zwallctl start / stop are not remote implementations (CLI).

What each result means

Scroll sideways to see the full table. Use the arrow keys when focused.

ResultMeaning
Agent status offline after ~agents.timeout (example 2m)No successful heartbeat in the timeout window. Process, network, TLS, or token.
401 on heartbeatMissing/invalid Agent JWT or principal could not be reconciled.
403 Access denied on heartbeatBody hostname ≠ authenticated hostname.
400 Invalid hosting observation heartbeatHosting contract failed: profile not hosting, incomplete identity, observation_only false, HostingStatus missing, active-control fields present (XDP/TC caps, blocked IPs, VMs, rule_count, eBPF status, UDP protection), prohibited mutation counters, or CollectedAt outside ±15m.
403 Hosting observation registration is not authorizedNo authoritative Hosting registration binding.
409 Hosting profile transition is not pending/authorizedv1→v2 heartbeat without authenticated transition.
409 Hosting observation Agents cannot receive restart commandsprotection_model is hosting. Restart API is refused.
200 restart {"success":true,"message":"Restart command sent to agent"}Command recorded for a non-Hosting Agent. Handler does not guarantee an immediate SIGTERM from the Manager; the Agent acts on a later cycle.
TLS errors in Agent journalCertificate verification failed. Do not “fix” with allow_insecure_remote_tls except break-glass.
Hosting status runtime_available: falseNo persisted Hosting status yet (no successful observation heartbeat).
Token near token_expires_atAgent should call POST /api/v1/agents/token/refresh (CapabilityAuthLifecycle). If clock is wrong, refresh and heartbeat both fail.

Heartbeat required body fields: hostname, ip_address. Status if present must be online, offline, or error.

Safe remediation

  1. Confirm Manager ready (GET /health/ready).
  2. Confirm the Agent unit is active. Hosting: start loader first, then Agent. Stop Agent, then loader (--detach).
  3. Confirm server.manager_url matches the reachable HTTPS Manager (example https://manager.example.com:8443), not zwallctl’s default 8080.
  4. Keep verify_tls: true. Repair the Manager certificate chain.
  5. If the JWT is expired and the Agent cannot refresh: issue a new one-time pre-register for recovery with explicit agent_id matching hostname and IP. Do not leave auto_register on.
  6. Correct node clock (Hosting ±15m is product-enforced on CollectedAt; admin/Agent JWT lifetimes are the 8h / 20m / 2m example).
  7. Hosting: do not send Virtualization heartbeat fields. Do not use the restart API. Use:
sudo systemctl restart zwall-hosting-agent
  1. Virtualization: zwallctl restart --agent ID is allowed; it is still not a substitute for systemd if the Agent process is dead and cannot receive a later heartbeat action.
  2. After recovery, confirm list-agents and, for Hosting, GET .../hosting/status.

Do-not-do warnings

  • Do not use zwallctl start / stop as remote power control.
  • Do not restart Hosting Agents through the Manager restart API (409 is correct).
  • Do not set allow_insecure_remote_tls: true to mask a certificate problem.
  • Do not widen agents.timeout as a substitute for fixing heartbeats.
  • Do not treat Manager cluster as an Agent failover product. Example Agent manager_urls is cluster-oriented commentary; cluster is not certified HA.
  • Do not put Agent JWTs in tickets or command argv.

Escalation condition

Escalate when:

  • Agent process is running, Manager ready is 200, TLS verifies, clocks agree, token is unexpired, and heartbeats still 401/403
  • Hosting heartbeats that match the observation contract still return 400
  • Non-Hosting restart API returns 200 but the Agent never restarts and systemd shows the process never received a command — the handler currently logs a request and returns success without a guaranteed remote SIGTERM

Edition

SHARED heartbeat and timeout. Hosting observation heartbeat contract, Hosting status API, and Hosting 409 restart are ENTERPRISE_ONLY.

Status

SUPPORTED. Restart-via-API for Virtualization is “command sent,” not a certified remote process manager.