Zwall
Talk to an Engineer
Talk to an Engineer
Documentation
Docs Home › Hosting Agent › Troubleshooting

Troubleshooting

Hosting Agent troubleshooting

Purpose

Diagnose missing Hosting telemetry, permission-blocked collectors, loader/socket failures, and accidental enforcement-profile mix-ups without turning the node into a drop engine.

Audience

On-call operators with admin access to the Manager and root on the hosting node.

Edition

ENTERPRISE_ONLY.

Status

SUPPORTED diagnostics. If pass-only programs drop traffic, escalate as a product defect; do not load firewall.o on the hosting node.

Prerequisites

  • Admin JWT for GET /api/v1/agents/:id/hosting/status and /hosting/events
  • journalctl on the node
  • Knowledge of which unit the release installed (zwall-hosting-agent vs zwall-hosting-agent-v2)

Required permissions

Admin on the Manager. Root on the node for unit/socket/ACL inspection. Do not grant the Agent extra capabilities as a “fix.”

Procedure — first five checks

  1. systemctl is-active zwall-hosting-ebpf-loader
  2. systemctl is-active zwall-hosting-agent or zwall-hosting-agent-v2
  3. ls -l /run/zwall-hosting-ebpf/observer.sock
  4. Confirm YAML hosting.enabled: true, hosting.enforcement: false, agent.protection_model: hosting
  5. As admin: GET /api/v1/agents/:id/hosting/status

Heartbeat must show observation-only. If the Agent is listed as a Virtualization enforcement node, you are on the wrong unit.

Symptom table

Scroll sideways to see the full table. Use the arrow keys when focused.

SymptomLikely verified causesDiagnosticSafe remediation
No hosting status / 404Wrong Agent id; profile is not Hosting v2GET /api/v1/agents/:id; protection_model; unit nameEnroll a Hosting Agent; do not assign Hosting APIs to a virt Agent
Unit inactiveloader/Agent not started; start-limit burstsystemctl status; StartLimitBurst=3start loader first, then Agent
Agent fatals must run as rootHosting not enabled so root check applieshosting.enabledenable Hosting profile or you installed zwall-agent.service
Collectors permission_blockedfile ACL; v2 unit has only admHosting status permission_status; namei -l PATHgrant exact-path read to zwall-hosting. Do not chmod 777 /var/log. Do not add a customer-content group
source_not_presentpath not on this OScompare example path vs hostretarget hosting.sources[].path or accept absence (PHP-FPM / MariaDB alternate)
parser_unsupportedunknown source idstatus ParserVersion / iduse inventory ids only
No log events in Managersource not active; events are category-safe so “missing IPs” is expected/hosting/events projection fields; parser healthfix source state first; do not expect raw payload
No network telemetryloader not attached; socket missing; SHA-256 mismatchloader journal; ZWALL_HOSTING_EBPF_SHA256; socket pathstart loader; fix env digest; confirm --interfaces
Preflight INCOMPATIBLEcapabilities, Manager URL, probe idprinted reasonAgent must lack CAP_NET_ADMIN/CAP_NET_RAW; Manager /api/v1/health 200; probe id [A-Za-z0-9][A-Za-z0-9_-]{7,63}
Unexpected dropsVirtualization firewall programs on the nodesystemctl is-active zwall-agent; ebpf.program_pathstop zwall-agent.service; keep Hosting units only
Restart API 409Hosting cannot receive restart commandsPOST /api/v1/agents/:id/restartsystemctl restart the Hosting Agent unit
Rules API 409Hosting cannot receive rulesassign-rule responseassign rules to a Virtualization Agent
Agent refuses YAMLhybrid protection model, enforcement true, metrics enabled, virt/geoip/smtp/uid-gid flagsjournal validation errorrestore v2 example fail-closed keys (the hybrid example misleads vs Hosting identity)
Token/env leftoverbootstrap still in environmentsystemctl show-environmentHosting main unsets AGENT_TOKEN/AGENT_PRE_TOKEN after load; do not persist them in the unit

Diagnose missing telemetry (ordered)

  1. Unit and loader socket. If the socket is missing, collectors may still tail logs, but network observation will not.
  2. Source state. permission_blocked / source_not_present / parser_unsupported — fix ACL or path; do not claim the product is inactive.
  3. Parser health. Status includes parser bundle / per-source parser version. Mismatch after upgrade → run Hosting upgrade preflight, do not force updater campaigns.
  4. Heartbeat observation_only. If false, you are not on the Hosting contract.
  5. Manager ingest. Hosting events API filters IsHostingEventType. Security-log list may contain other Agents’ events; use the Hosting projection.
  6. Category-safe contract. Absence of customer IPs is success, not data loss in this API. Lossless delivery is not claimed.

Verification after repair

  • Both units active
  • Status runtime_available: true once a heartbeat with Hosting status has been accepted
  • Intended sources active / readable
  • Network drops remain zero
  • GET /api/v1/agents/:id/hosting/events returns projection rows without raw bodies

Expected result

A specific source state or unit failure, not a generic “Hosting is down.”

Failure modes that are not operator-fixable here

Scroll sideways to see the full table. Use the arrow keys when focused.

IssueClass
No universal ACL packno shipped universal ACL pack
hybrid example misleading vs Hostinghybrid example vs Hosting profile identity
Pass-only program drops packetsproduct defect — escalate
Kernel matrix for every hosting distroOS/kernel matrix not certified

Security considerations

  • Do not attach customer log excerpts to tickets.
  • Do not disable verify_tls.
  • Do not run the Agent as root “to read logs.” Grant the file, keep the user.

Limitations

Hosting restart/rule APIs returning 409 is correct behavior. Troubleshooting is not a reason to switch the node to Virtualization enforcement.