Troubleshooting
Use this index to pick the document that matches the symptom. Do not start from a component name if the symptom is ambiguous.
This page does not replace Hosting-only, Virtualization-only, or zwallctl-only books. Those remain the deeper component procedures.
How to use this index
- Match the symptom, not the first log line.
- Confirm edition (Community vs Enterprise) before following an Enterprise-only endpoint.
- Confirm Agent profile (Hosting observation vs Virtualization enforcement) before treating missing drops as a bug.
- Treat
GET /health/live200 as process-alive only. Readiness isGET /health/ready. - If the procedure would require an unpublished contract, stop at the named engineering gap. Do not invent Linux, eBPF, or HA folklore.
Symptom → document
Scroll sideways to see the full table. Use the arrow keys when focused.
| Symptom | Go to | Edition |
|---|---|---|
Console, API, or zwallctl ping cannot reach the Manager | Manager Unavailable | Shared |
/health/live 200 but /health/ready 503 | Manager Unavailable | Shared |
Login returns Secure HTTPS session required | Manager Unavailable and TLS | Shared |
zwallctl talks to port 8080 while the Manager listens on 8443 | Manager Unavailable and CLI | Shared |
| Agent will not register / pre-register | Agent Enrollment | Shared |
Community fourth Agent rejected (COMMUNITY_AGENT_LIMIT_REACHED) | Agent Enrollment | Community |
Enterprise Agent enroll returns 403 LicenseErrorResponse | Agent Enrollment | Enterprise |
Invalid Agent profile identity or Hosting vs Virtualization mix-up | Agent Enrollment | SHARED / Hosting ENTERPRISE |
| Agent shows offline / disconnected / no heartbeat | Agent Disconnected | Shared |
Hosting heartbeat 400 Invalid hosting observation heartbeat | Agent Disconnected | Enterprise |
POST /api/v1/agents/:id/restart returns 409 on a Hosting Agent | Agent Disconnected | Enterprise |
| No security logs, Hosting events, or collector status | Missing Telemetry | SHARED; Hosting ENTERPRISE |
Hosting collector permission_blocked or source_not_present | Missing Telemetry | Enterprise |
| Hosting observer socket missing | Missing Telemetry | Enterprise |
| Rule created but traffic is not dropped | Enforcement | SHARED; Hosting never enforces |
| Assigning a rule to a Hosting Agent returns 409 | Enforcement | Enterprise |
| Testing mode appears to flush or disable rules | Enforcement | Shared |
| eBPF not loaded / XDP attach failed / capability denied | eBPF | SHARED; Hosting loader ENTERPRISE |
GET /api/v1/dashboard/ebpf-health missing on Community | eBPF | COMMUNITY gap vs Enterprise |
| Manager will not start: database / SSL / migration | Database | Shared |
-migrate vs -preflight-migrate vs -check-schema confusion | Database | Shared |
| Backup script exists; restore is not a certified contract | Database | Shared |
| Cloudflare / GeoIP / RBL / notifications / ModSecurity / libvirt / Prometheus / unblock portal | Integrations | see that file |
| AI Preview screens missing, connection test fails, or “approved but not enforced” | AI Preview | ENTERPRISE_ONLY PREVIEW |
zwallctl start / stop prints not implemented | CLI | Shared |
Enterprise zwallctl secret-rotate says API rotation is retired | CLI | ENTERPRISE vs COMMUNITY |
| Hosting-only collector, loader, or ACL symptoms | Troubleshooting | Enterprise |
| Virtualization-only libvirt / XDP / VM symptoms | Troubleshooting | Shared |
First probes (any Manager incident)
These are unauthenticated process probes. They are not /api/v1 except the duplicate GET /api/v1/health.
sudo systemctl status zwall-manager
sudo journalctl -u zwall-manager -n 200 --no-pager
curl -sS https://manager.example.com:8443/health/live
curl -sS https://manager.example.com:8443/health/readyScroll sideways to see the full table. Use the arrow keys when focused.
| Result | Meaning | Next |
|---|---|---|
| Unit inactive / failed | Process not running | Manager Unavailable |
live 200 {"status":"alive",...} | Process up; not database-ready | still call ready |
live 503 draining or updating | Updater runtime mode; not a load-balancer “healthy” | do not treat as HA failover |
ready 200 status: ready | Listeners admitted and database check passed | Agent/console issues are elsewhere |
ready 503 listener_admission_incomplete | HTTP servers not serving (TLS/bind) | Manager Unavailable |
ready 503 database: not_configured|timeout|unavailable | PostgreSQL / ssl_mode / credentials | Database |
ready 503 cluster: timeout | Cluster enabled and timed out | not a public HA product; do not fail over cluster |
| connection refused on 8080 | zwallctl default URL, not the example Manager port | CLI |
| connection refused on 8443 | Manager not listening or wrong bind | Manager Unavailable |
Prefer the operator CA with curl --cacert. curl -k only distinguishes “TLS mismatch” from “nothing listening”; it is not a production health check.
Profile split that causes false incidents
Scroll sideways to see the full table. Use the arrow keys when focused.
| If you expected | But the node is | Typical false conclusion |
|---|---|---|
| Packet drop | Hosting Agent (protection_model: hosting, hosting.enforcement: false) | “Enforcement is broken” — Hosting cannot receive rules (409) |
| Hosting collectors | Virtualization Agent / zwall-agent.service | “No hosting status” |
Remote zwallctl start | Any edition | Command prints not implemented |
| Lossless event delivery | Any edition | Not a certified contract |
Engineering gaps that stop a complete procedure
Do not invent a workaround that pretends these are closed.
Scroll sideways to see the full table. Use the arrow keys when focused.
| ID | Why troubleshooting stops |
|---|---|
| OS/kernel matrix | No single certified OS/kernel matrix |
| Filesystem prefixes | Multiple filesystem prefixes (/etc/zcs, /etc/zwall-manager, /etc/zwall-hosting) |
| Manager URLs and ports | Example Manager port 8443 vs zwallctl default http://localhost:8080 |
| Backup/restore contract | Backup/restore scripts exist; not a certified recovery contract |
| Manager cluster HA | Manager cluster is not a public HA product |
| UID/GID filtering | UID/GID filtering is not a customer contract |
| ACME certification | ACME HTTP-01 + Cloudflare DNS-01 not certified |
| Event delivery guarantees | Event-delivery exactly-once is open |
| Secret custody acceptance | Secret-custody implemented, not finally security-accepted |
| Hosting collector ACLs | No shipped universal Hosting collector ACL |
| Remote start/stop | zwallctl start/stop not implemented |
| Stale agent path | Stale start-agent path /etc/zwall/agent.yml — do not copy |
| PostgreSQL version | Community README PostgreSQL 15+ vs Enterprise observed PostgreSQL 16 |
| Enterprise HTTP groups | Cloudflare HTTP group and unblock portal are Enterprise-only in V1 routers |