Docs Home Virtualization Agent

Troubleshooting

Virtualization Agent troubleshooting

Purpose

Diagnose a Virtualization Agent that will not stay up, will not attach XDP/TC, will not list VMs, or will not apply rules — without converting the node into a Hosting profile or enabling incomplete UID/GID ABI.

Audience

On-call operators with root on the hypervisor and admin on the Manager.

Edition

SHARED. Enterprise adds GET /api/v1/dashboard/ebpf-health.

Status

SUPPORTED diagnostics. Kernel minimum is not certified as a single number.

Prerequisites

Unit name zwall-agent.service
Config /etc/zcs/agent.yml (Enterprise) unless Community used another path
Admin JWT / zwallctl with --manager-url set

Required permissions

Root on the node. Admin on the Manager.

Procedure — first checks

systemctl is-active zwall-agent
journalctl -u zwall-agent -n 200 --no-pager
id -u   # Agent must be root; unit User=root
curl -k https://manager.example.com:8443/health/ready
zwallctl login --manager-url https://manager.example.com:8443 --username admin
zwallctl list-agents

Confirm this is not zwall-hosting-agent.service. Hosting cannot receive rules or restart commands.

Symptom table

Scroll sideways to see the full table. Use the arrow keys when focused.

Symptom	Likely causes	Diagnostic	Safe remediation
Agent exits immediately	not root; eBPF preflight; bad config	journal; `min_kernel_version`; `strict_kernel_version_check`	run as root; fix YAML; do not pick a kernel from Community README vs 5.8.0 as “certified”
“must be run as root”	Hosting not enabled (correct for this profile)	`hosting.enabled`	run `zwall-agent.service` as root; do not enable Hosting to dodge root on a hypervisor
Agent missing in Manager	enrollment; TLS; `manager_url`; Community 4th Agent	Agent log; `GET /api/v1/agents`	Enrollment
No VMs	libvirt off; `virtualization.enabled` false; wrong `vm_bridge_name`	`virsh -c qemu:///system list --all`	enable KVM/libvirt; set URI; correct bridge
Rules not applying	sync failure; testing mode; Hosting target 409	`zwallctl list-rules`; `test-flush-status`; assign error	restore connectivity; disable testing mode only when intended; assign to this Agent id
Attach failure	XDP mode; driver; integrity	`xdp_attach_mode`; `ebpf-health`; dmesg	controlled `generic` fallback; never disable `verify_artifact_integrity` for convenience
UID/GID surprises	example YAML enables incomplete ABI	`enable_uid_gid_filtering`	set false until a stable customer contract is published
Maps empty after restart	pinned maps off; pin dir not writable	`/sys/fs/bpf`; unit `ReadWritePaths`	use shipped unit; `enable_pinned_maps: true`
`zwallctl start/stop` no-op	not implemented	printed message	systemd
Restart API “success” but process stays	restart is a request, not guaranteed SIGTERM	handler contract	`systemctl restart zwall-agent`
Edge mode oddities	`operating_mode: edge`	YAML	revert to `host`
Detection profiles ignored	example list vs map tags	detection.profiles list vs map mismatch	do not trust example `detection.profiles` list

Verification after repair

Verification. Agent online, eBPF health, VM/interface list on a populated libvirt host, testing mode understood before drops.

Expected result

A single primary cause from the table, with a systemd or API evidence line.

Failure modes that need engineering/owner

Scroll sideways to see the full table. Use the arrow keys when focused.

Issue	Class
Which kernel is supported	OS/kernel matrix not certified
UID/GID customer contract	UID/GID filtering not a customer contract
hybrid vs virtualization profile	hybrid example vs Hosting profile identity
SIGHUP vs restart	SIGHUP not certified live-apply
Edge mode	Edge mode not certified

Security considerations

Break-glass allow_unsigned_artifacts and allow_insecure_remote_tls are incident-only.
Do not attach hosting_observe.o to “fix” virt attach.
Do not collect customer packet payloads; this is map policy plus security logs.

Limitations

XDP earlier than netfilter can surprise operators who expected iptables counters to see the same packets. That coexistence is not certified.

Previous KVM and Libvirt Next Verification

Troubleshooting

Purpose #

Audience #

Edition #

Status #

Prerequisites #

Required permissions #

Procedure — first checks #

Symptom table #

Verification after repair #

Expected result #

Failure modes that need engineering/owner #