Missing Telemetry

Symptom #

The Manager console or API shows no (or incomplete) events:

• empty GET /api/v1/security-logs
• Hosting status missing, runtime_available: false, or collectors not active
• Hosting events list empty
• Virtualization traffic/VM views empty
• operators expected “every packet/log arrived once”

Likely verified causes #

1. Agent disconnected — no upload path (Agent Disconnected).
2. Wrong profile: Hosting collectors do not run on zwall-agent.service; Virtualization does not expose Hosting status APIs.
3. Hosting source state permission_blocked (file ACL) or source_not_present (path not on this OS). There is no shipped universal Hosting ACL.
4. Hosting loader not active; observer socket /run/zwall-hosting-ebpf/observer.sock missing. Pass-only programs do not drop; missing network telemetry is often “loader not attached,” not “observation off.”
5. Hosting events API is category-safe: no raw payload / customer IPs in the projection. Empty-looking events may still be collecting (collection_status: COLLECTING, review_state: COLLECTING).
6. Security-log ingest uses Agent capability auth (POST /api/v1/security-logs). Admin GET is a different path.
7. Event-delivery exactly-once is an open engineering gap. Gaps are not proof of a collector bug by themselves.
8. Virtualization: libvirt/KVM not present, wrong vm_bridge_name, or Agent not posting VM data.

Diagnostic commands/endpoints #

Any Agent:

zwallctl list-agents
zwallctl list-events --hours 24 --limit 100Copy

GET /api/v1/security-logs
GET /api/v1/security-logs/summary
GET /api/v1/dashboard/logsCopy

Admin auth. Agent ingest is POST /api/v1/security-logs (and /batch) with classified Agent auth, not the admin JWT.

Hosting (Enterprise; requires Hosting v2 profile):

GET /api/v1/agents/:id/hosting/status
GET /api/v1/agents/:id/hosting/events?limit=50Copy

Events projection fields: event_id, source, category, severity, timestamp, parser, contract_version, review_state. Limit 1–100 (default 50). Non-v2 or non-hosting → 404 Hosting v2 profile not available.

On the Hosting node:

sudo systemctl status zwall-hosting-ebpf-loader
sudo systemctl status zwall-hosting-agent
ls -l /run/zwall-hosting-ebpf/observer.sockCopy

Socket path is product-fixed: loader --socket default /run/zwall-hosting-ebpf/observer.sock; non-approved paths are rejected.

Virtualization:

sudo systemctl status zwall-agent
# libvirt is an operator tool, not a Zwall command
virsh -c qemu:///system list --allCopy

GET /api/v1/virtualization/vms
GET /api/v1/dashboard/ebpf-healthCopy

ebpf-health is Enterprise (DashboardHandler.GetEBPFHealth). Community V1 router has no ebpf-health route.

What each result means #

Security logs #

Hosting status / sources #

Source states verified in config/schema include active, available_but_disabled, source_not_present, permission_blocked, parser_unsupported, degraded, failed, stopped, unavailable.

Network telemetry: loader loads hosting_observe.o (pass-only). If the socket is missing, the Agent cannot poll observer events (hosting.network.socket_path in example).

Virtualization #

Safe remediation #

1. Confirm the Agent profile and unit.
2. Hosting: start loader, confirm socket, start Agent, then re-read Hosting status.
3. For permission_blocked: add a precise POSIX ACL/read grant for zwall-hosting on that file. Do not chmod 777 /var/log.
4. For source_not_present: fix the path or accept that the source is absent on this image.
5. Interpret Hosting events as category-safe investigation signals, not raw customer logs.
6. Virtualization: fix libvirt and Agent connectivity; do not enable Hosting enforcement flags.
7. If some events are missing under load: treat as the delivery gap (no lossless contract). Do not claim exactly-once.

Do-not-do warnings #

• Do not interpret Hosting pass-only as “observation is off” without checking the loader and socket.
• Do not load Virtualization firewall objects on a shared hosting node to “get telemetry.”
• Do not expect Hosting events JSON to include raw payloads or customer IPs.
• Do not claim lossless or exactly-once delivery.
• Do not use Hosting sources as automatic/provider AI input.
• Do not invent a universal Hosting ACL.
• Do not publish PPS or “every flow visible.”

Escalation condition #

Escalate when:

• loader is active, socket exists, source state is active, heartbeats succeed, and no category-safe events appear for an interval where the source is known to be writing
• pass-only Hosting programs drop traffic (product defect — do not “fix” by attaching firewall objects)
• Community operators need ebpf-health — that handler is not in the Community router (product gap, not an operator misconfig)

Related documentation #

• Hosting troubleshooting
• Hosting telemetry
• Signals and events
• Event model
• Enforcement
• eBPF
• Log sensitivity

Edition #

SHARED security logs. Hosting status/events, observer socket, and Hosting collector states are ENTERPRISE_ONLY. dashboard/ebpf-health is Enterprise.

Status #

SUPPORTED for diagnosis. The exactly-once delivery and universal-ACL issues remain product gaps.