cilium
Cilium CNI operations: kube-proxy replacement, BPF configuration, Gateway API, Ingress Controller, LB-IPAM, L2 Announcements, upgrades, and troubleshooting on K3s/ARM64.
SKILL.mdskills/cilium/SKILL.md
GitHubCilium Skill
Cluster Context
This cluster uses Cilium with full kube-proxy replacement (--disable-kube-proxy on K3s).
Node: Raspberry Pi CM4, ARM64, Ubuntu 24.04.
Helm release: cilium in namespace kube-system.
Chart version pinned in: roles/install-cilium/defaults/main.yml.
Active Helm values
kubeProxyReplacement: "true"
k8sServiceHost: \"192.168.178.30\"
k8sServicePort: "6443"
rollOutCiliumPods: true # hash ConfigMap into pod template → auto rollout
operator.replicas: 1
operator.rollOutPods: true # same for operator
envoy.rollOutPods: true # same for envoy
bpf.masquerade: true # required on ARM64
loadBalancer.acceleration: disabled # no hardware offload on CM4
gatewayAPI.enabled: true
gatewayAPI.externalTrafficPolicy: Cluster # REQUIRED for L2 Announcements (see below)
l2announcements.enabled: true # Cilium handles ARP (MetalLB removed)
l2announcements.leaseDuration: 3s
l2announcements.leaseRenewDeadline: 1s
l2announcements.leaseRetryPeriod: 500ms
k8sClientRateLimit.qps: 32 # sized for L2 announcement leader election
k8sClientRateLimit.burst: 64
ingressController.enabled: true
ingressController.default: true
ingressController.enforceHttps: false # flip to true once cert-manager is up
ingressController.loadbalancerMode: shared
ingressController.service.externalTrafficPolicy: Cluster # REQUIRED for L2 Announcements
Critical: rollOutPods flags
Always set these three values in the Cilium Helm chart:
rollOutCiliumPods: true
operator.rollOutPods: true
envoy.rollOutPods: true
These inject a hash of cilium-config ConfigMap into pod template annotations.
Any helm upgrade that changes config triggers an automatic rolling restart of
all three components, making wait: true reliable.
Without these flags: helm upgrade updates the ConfigMap but pods keep
running with stale in-memory config. Silent deadlock — agent waits forever for
CRDs that the stale operator never registers. No obvious error message.
Critical: Envoy CRDs
ciliumenvoyconfigs.cilium.io and ciliumclusterwideenvoyconfigs.cilium.io
are registered by the operator ONLY when enable-envoy-config=true, which
is set automatically when gatewayAPI.enabled=true or
ingressController.enabled=true.
If the operator pod started before those flags were set (stale pod), it will never register the Envoy CRDs. The agent will then hang indefinitely:
Still waiting for Cilium Operator to register CRDs: [ciliumenvoyconfigs.cilium.io ...]
Fix: ensure operator.rollOutPods: true is set so the operator restarts on
every config change. If already stuck, delete the operator pod manually:
kubectl delete pod -n kube-system -l io.cilium/app=operator
# Wait for new pod, then check:
kubectl get crd | grep envoy
Critical: Helm stuck in pending-upgrade
If a helm upgrade is interrupted (Ctrl+C, timeout, context canceled), the
release may be left in pending-upgrade state. All subsequent upgrades silently
hang — no error, no output, just blocks indefinitely.
# Diagnose
helm history cilium -n kube-system
# Look for STATUS=pending-upgrade
# Fix: delete the stuck secret (vN = the pending-upgrade revision number)
kubectl delete secret sh.helm.release.v1.cilium.vN -n kube-system
# Verify state is clean (last revision should be "deployed")
helm history cilium -n kube-system
# Then re-run
ansible-playbook playbooks/bootstrap.yml -i inventory/hosts.ini \
--start-at-task "Add Cilium Helm repository"
Gateway API
Cilium 1.19 supports Gateway API v1.4.1 (standard channel).
Prerequisites (all already met in this cluster):
kubeProxyReplacement: truel7Proxy: true(default)- Gateway API CRDs installed BEFORE
gatewayAPI.enabled=trueis applied
Role order in bootstrap.yml:
install-gateway-api-crds → install-cilium → install-cilium-pools
What gets created automatically:
GatewayClassnamedcilium(controller:io.cilium/gateway-controller)- A
LoadBalancerService perGatewayresource (Cilium LB-IPAM assigns IP from pool)
GatewayClass status meanings:
Unknown→ operator/agent not yet running with new config, or no LB-IPAM pool definedTrue→ fully operational
kubectl get gatewayclass
kubectl get gateway -A
kubectl get httproute -A
LB-IPAM (Cilium Load Balancer IP Address Management)
MetalLB has been removed. Cilium's built-in LB-IPAM handles IP assignment.
LB-IPAM is always compiled in; it activates when the first CiliumLoadBalancerIPPool is created.
The pool lives in the install-cilium-pools role.
CiliumLoadBalancerIPPool
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
name: default-pool
spec:
blocks:
- cidr: "192.168.178.200/28" # .200 – .215 (16 IPs)
Pin a specific IP
metadata:
annotations:
lbipam.cilium.io/ips: "192.168.178.203"
Sharing an IP across multiple services (replaces metallb allow-shared-ip)
metadata:
annotations:
lbipam.cilium.io/sharing-key: "shared-gateway"
Verify pool and assignments
kubectl get ciliumloadbalancerippools
kubectl get svc -A | grep LoadBalancer
# Check EXTERNAL-IP column — should show assigned IP
L2 Announcements (ARP)
Cilium's L2 Announcements replace MetalLB's speaker for ARP on the LAN.
Leader election uses Kubernetes Lease objects in kube-system:
kubectl -n kube-system get lease | grep cilium-l2announce
CiliumL2AnnouncementPolicy
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
name: default-l2-policy
spec:
interfaces:
- ^eth.*$ # legacy wired NICs: eth0, eth1, etc.
- ^en.* # wired NICs: end1 / eno1 / ens* / enp* / enx*
- ^wl.* # Wi-Fi NICs: wlp4s0, wlan0, etc.
- ^end.*
- ^eno.*
- ^ens.*
- ^enp.*
- ^enx.*
externalIPs: false
loadBalancerIPs: true
Important: if the host NIC changes (for example end1 on newer nodes or a
Wi-Fi interface such as wlp4s0), the L2 announcement selector must include
it. A too-narrow ^eth[0-9]+ regex will leave the VIP advertised in Kubernetes
but unreachable from the LAN.
Workflow note
For long-running cluster changes, prove the desired state manually or with Helm first, then run Ansible in the background with redirected logs, and only after that re-run the same Ansible command normally.
Check the LAN side
ip neigh show 192.168.178.200
ip neigh show 192.168.178.203
arp -n 192.168.178.200
arp -n 192.168.178.203
.203 is the Pi-hole DNS VIP and .200 is the shared HTTP/HTTPS Gateway VIP.
If .203 answers but .200 does not, DNS is healthy and only the Gateway path is broken.
If .203 times out, fix the Pi-hole L2 announcement path first.
If the lease exists but ARP is FAILED or (incomplete), the policy is too
narrow or the wrong interface family is advertised. Update the interface regexes
and reapply the policy.
CRITICAL: externalTrafficPolicy must be Cluster
Cilium L2 Announcements are incompatible with externalTrafficPolicy: Local.
The L2 announcement leader may not be the same node as the endpoint — with Local,
traffic would be dropped. Always use Cluster on all LoadBalancer services.
Set in Helm values:
gatewayAPI.externalTrafficPolicy: Cluster
ingressController.service.externalTrafficPolicy: Cluster
Set on individual services:
spec:
externalTrafficPolicy: Cluster
Verify ARP is working
# From macOS/Linux laptop on the same LAN:
arp -n 192.168.178.200 # should show node MAC, not "(incomplete)"
arp -n 192.168.178.203
Why MetalLB was removed
MetalLB requires the kubernetes.io/service-name label on EndpointSlices to
associate slices with a service for L2 leader election. Cilium's Gateway API
creates EndpointSlices with gateway-specific labels but WITHOUT this label.
This is a fundamental incompatibility — MetalLB's speaker never elected a node
for the Gateway service, so ARP was never announced and traffic never reached
the node. Cilium LB-IPAM + L2 Announcements have no such requirement.
Ingress Controller
Cilium handles networking.k8s.io/v1 Ingress resources via Envoy.
shared mode (configured here): one cilium-ingress LoadBalancer Service
for all Ingress resources — saves LB-IPAM IPs. Use in homelab.
dedicated mode: one LB Service per Ingress — use only if per-Ingress IP assignment is required.
kubectl get ingressclass
kubectl get ingress -A
kubectl get svc cilium-ingress -n kube-system
externalTrafficPolicy
Both gatewayAPI and ingressController create LoadBalancer Services.
| Policy | Source IP | Notes |
|---|---|---|
Cluster |
Lost (SNAT) | Required for L2 Announcements |
Local |
Preserved | Incompatible with L2 Announcements |
Always use Cluster on this cluster. L2 Announcements docs explicitly state
incompatibility with Local — the announcement leader may differ from the
endpoint node, causing traffic to be dropped silently.
ztunnel — NOT for Cilium
ztunnel is the Istio Ambient Mesh L4 proxy. It is NOT part of Cilium.
Cilium has its own integrated service mesh via cilium-envoy + CiliumEnvoyConfig.
Never install ztunnel alongside Cilium.
Upgrade Procedure
- Check current version:
helm list -n kube-system - Check release notes: https://github.com/cilium/cilium/releases
- Update
cilium_versioninroles/install-cilium/defaults/main.yml - Diff before applying:
helm diff upgrade cilium cilium/cilium --version <new> -n kube-system - Re-run:
ansible-playbook playbooks/bootstrap.yml -i inventory/hosts.ini \ --start-at-task "Add Cilium Helm repository" - Verify:
kubectl rollout status ds/cilium -n kube-system kubectl rollout status deploy/cilium-operator -n kube-system kubectl get gatewayclass
Hubble Metrics & Observability
Cilium 1.19.2 includes comprehensive Hubble metrics for network observability.
Configuration
Hubble metrics are configured in the install-cilium role with:
hubble.enabled: true
hubble.metrics.enabled: ["httpV2:exemplars=true;sourceContext=identity;destinationContext=identity",
"drop:sourceContext=identity;destinationContext=identity",
"tcp:sourceContext=identity;destinationContext=identity",
"flow:sourceContext=identity;destinationContext=identity",
"icmp:sourceContext=identity;destinationContext=identity",
"policy:sourceContext=identity;destinationContext=identity",
"port-distribution:sourceContext=identity;destinationContext=identity"]
hubble.metrics.enableOpenMetrics: true
hubble.metrics.port: 9965
ServiceMonitor (Prometheus Integration)
The install-cilium-hubble-monitoring role creates a ServiceMonitor for Prometheus discovery:
- Tag:
networking-observability - Namespace:
kube-system - Port:
9965/metrics - Labels: Must include
release: kube-prometheus-stackfor discovery - OpenMetrics: Enabled for exemplars and trace correlation
# Deploy Hubble monitoring
make networking-observability
# Verify Prometheus targets
kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -- \
wget -qO- http://localhost:9090/api/v1/targets | grep hubble
Available Metrics
| Metric | Description | Labels |
|---|---|---|
hubble_drop_total |
Network drops by reason | reason, protocol, source, destination |
hubble_flows_processed_total |
Flow processing stats | type, subtype, verdict, protocol |
hubble_http_request_duration_seconds |
HTTP request histograms | method, reporter, source, destination |
hubble_tcp_flags_total |
TCP flag combinations | flag, family |
hubble_icmp_total |
ICMP message stats | family, type |
hubble_policy_verdict_total |
Network policy decisions | verdict, direction |
hubble_port_distribution_total |
Port usage distribution | protocol, port |
Critical Notes
⚠️ Never enable both http and httpV2 metrics simultaneously - causes CrashLoopBackOff.
✅ Use only httpV2 with exemplars for trace correlation.
⚠️ Idempotency: The ServiceMonitor is managed by a separate role with tag networking-observability.
Running networking tag alone won't disable Hubble monitoring.
Verification Commands
# Check metrics endpoint directly
kubectl port-forward -n kube-system svc/hubble-metrics 9965:9965 &
curl -s localhost:9965/metrics | grep hubble_
# Check ServiceMonitor labels
kubectl get servicemonitor cilium-hubble -n kube-system --show-labels
# Verify Prometheus discovery (should show 4 targets UP)
kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -- \
wget -qO- http://localhost:9090/api/v1/targets | python3 -c \
"import json,sys; [print(f'{t[\"scrapeUrl\"]} [{t[\"health\"]}]') for t in json.load(sys.stdin)['data']['activeTargets'] if 'hubble' in str(t)]"
Exemplars for Trace Correlation
When exemplars=true is enabled on httpV2 metrics, Hubble can correlate HTTP metrics
with distributed traces from Tempo, providing end-to-end observability.
Health Checks
# Pod status
kubectl get pods -n kube-system -l k8s-app=cilium
kubectl get pods -n kube-system -l io.cilium/app=operator
# Helm state
helm history cilium -n kube-system
helm get values cilium -n kube-system
# CRDs
kubectl get crd | grep -E "cilium|gateway"
# GatewayClass
kubectl get gatewayclass
# LB-IPAM pools and assignments
kubectl get ciliumloadbalancerippools
kubectl get svc -A | grep LoadBalancer
# L2 Announcement leases (leader election)
kubectl -n kube-system get lease | grep cilium-l2announce
# ARP (from laptop on same LAN)
arp -n 192.168.178.200
arp -n 192.168.178.203
# Connectivity test
kubectl run -it --rm pingtest --image=nicolaka/netshoot -- ping <pod-ip>
Troubleshooting
Agent stuck: "Still waiting for Cilium Operator to register CRDs"
→ Operator pod is stale. See Critical: Envoy CRDs section above.
helm upgrade hangs with no output
→ Stuck pending-upgrade. See Critical: Helm stuck section above.
GatewayClass stays Unknown
- Check operator is running with new config:
kubectl logs -n kube-system -l io.cilium/app=operator --tail=20 - Check Envoy CRDs exist:
kubectl get crd | grep envoy - Check LB-IPAM pool exists:
kubectl get ciliumloadbalancerippools
LoadBalancer service stays <pending> for EXTERNAL-IP
- Check
CiliumLoadBalancerIPPoolexists and pool has free IPs - Check
lbipam.cilium.io/ipsannotation matches an IP in the pool range - Check for conflicting
lbipam.cilium.io/sharing-keyannotations
ARP not resolving / IP not reachable from LAN
- Check L2 announcement lease exists:
kubectl -n kube-system get lease | grep cilium-l2announce - Verify
CiliumL2AnnouncementPolicyexists andinterfacesregex matches the node NIC - Verify
CiliumL2AnnouncementPolicyexists andinterfacesregex matches the node NIC (on this cluster it must includeend1) - Confirm
externalTrafficPolicy: Clusteron the service —Localsilently breaks L2 - Check Cilium agent logs for L2 announcement errors:
kubectl logs -n kube-system -l k8s-app=cilium --tail=50 | grep -i l2 - Run
tcpdump -i eth0 arpon the node to see if ARP replies are being sent
BPF programs: using TCX not legacy tc filter
Cilium uses TCX (BPF links) on eth0, NOT legacy tc filter.
tc filter show dev eth0 ingress returns nothing even when BPF is active.
Check actual attachment via: ls /sys/fs/bpf/cilium/devices/eth0/links/
Node stays NotReady after Cilium install
kubectl describe node | grep NetworkUnavailable
# Should flip to False with reason CiliumIsUp once agent is Running
kubectl delete pod -n kube-system -l k8s-app=cilium # force restart if stuck
kube-proxy replacement issues
sudo systemctl cat k3s | grep kube-proxy # verify --disable-kube-proxy is set
kubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxyReplacement
BPF masquerade not working (ARM64)
- Ensure
loadBalancer.acceleration: disabled— hardware offload not available on CM4 - Kernel must be 5.10+ (Ubuntu 24.04 ships 6.x — fine)
Useful Commands
# Live agent logs
kubectl logs -n kube-system -l k8s-app=cilium -f
# Live operator logs
kubectl logs -n kube-system -l io.cilium/app=operator -f
# Full helm diff before upgrade
helm diff upgrade cilium cilium/cilium --version <ver> -n kube-system
# ConfigMap (what the agent reads at runtime)
kubectl get configmap cilium-config -n kube-system -o yaml
# Inside cilium pod
kubectl -n kube-system exec ds/cilium -- cilium status
kubectl -n kube-system exec ds/cilium -- cilium monitor