cgroup

cgroup. Use when you need help with cgroup.

SKILL.md

GitHub

Cgroup Program Types

All cgroup programs attach to a cgroup directory and affect all sockets/processes within that cgroup (and its descendants unless overridden).

Related Guidance

  • For cross-kernel compatibility checks, see workflows/testing.md.
  • These hooks operate on real kernel objects such as cgroups, sockets, and sysctl state. If a context field or object behavior is unclear, consult Linux kernel source and kernel docs directly.

Quick Choice

  • Use cgroup_skb for coarse per-cgroup packet policy.
  • Use cgroup/connect or cgroup/bind for address rewrite and transparent redirect workflows.
  • Use cgroup/sendmsg / cgroup/recvmsg for UDP address rewriting.
  • Use cgroup/sockopt or cgroup/sysctl for policy over socket options and sysctls.
  • Use cgroup/dev for device access control.
  • Use sockops for TCP lifecycle insight and tuning.

Attach with bpftool:

bpftool cgroup attach /sys/fs/cgroup/<path> <attach_type> id <prog_id>
bpftool cgroup detach /sys/fs/cgroup/<path> <attach_type> id <prog_id>
bpftool cgroup list /sys/fs/cgroup/<path>

cgroup_skb — Packet Filtering per Cgroup

Filter ingress/egress packets for all sockets in a cgroup. Return 1 to pass, 0 to drop.

When to use it:

  • coarse packet allow/deny by cgroup
  • accounting or classification at the cgroup boundary
SEC("cgroup_skb/ingress")
int handle_ingress(struct __sk_buff *skb) {
    return SK_PASS;  // or 1
}

SEC("cgroup_skb/egress")
int handle_egress(struct __sk_buff *skb) {
    return SK_PASS;
}

Attach types: ingress, egress

Notes:

  • Cannot modify packet headers (read-only skb)
  • Use for per-cgroup network accounting or coarse filtering

cgroup/sock — Socket Lifecycle

Called on socket creation and release. Return 1 to allow, 0 to deny.

When to use it:

  • socket create/release policy
  • lightweight socket lifecycle observation at cgroup scope
SEC("cgroup/sock_create")
int handle_sock_create(struct bpf_sock *ctx) {
    // ctx->family, ctx->type, ctx->protocol available
    return 1;  // allow
}

SEC("cgroup/sock_release")
int handle_sock_release(struct bpf_sock *ctx) {
    return 1;
}

// Called after bind() completes — can inspect the bound address
SEC("cgroup/post_bind4")
int handle_post_bind4(struct bpf_sock *ctx) {
    return 1;
}

SEC("cgroup/post_bind6")
int handle_post_bind6(struct bpf_sock *ctx) {
    return 1;
}

Notes:

  • sock_create runs before the socket is fully initialized — limited fields available
  • Denying (return 0) causes the syscall to return -EPERM

cgroup/bind — Override Bind Address

Intercepts bind() calls. Can rewrite the address the socket binds to.

When to use it:

  • bind-time address policy
  • forcing or rewriting bind destinations
SEC("cgroup/bind4")
int handle_bind4(struct bpf_sock_addr *ctx) {
    // ctx->user_ip4, ctx->user_port — can be rewritten
    return 1;  // allow (after any rewrites)
}

SEC("cgroup/bind6")
int handle_bind6(struct bpf_sock_addr *ctx) {
    return 1;
}

Notes:

  • Write to ctx->user_ip4 / ctx->user_port to redirect the bind
  • Return 0 to block the bind entirely (-EPERM)

cgroup/connect — Intercept connect()

Intercepts connect() calls. Can transparently redirect connections to a different address (e.g., service mesh transparent proxy).

When to use it:

  • transparent proxying
  • service-mesh-style connect redirection
  • outbound connect policy by cgroup
SEC("cgroup/connect4")
int handle_connect4(struct bpf_sock_addr *ctx) {
    // rewrite destination: ctx->user_ip4, ctx->user_port
    return 1;
}

SEC("cgroup/connect6")
int handle_connect6(struct bpf_sock_addr *ctx) {
    return 1;
}

SEC("cgroup/connect_unix")
int handle_connect_unix(struct bpf_sock_addr *ctx) {
    return 1;
}

Notes:

  • This is how Cilium and Istio implement transparent proxying without iptables
  • Write ctx->user_ip4 and ctx->user_port to redirect to a different endpoint
  • Original destination is preserved in the socket for the proxy to query

cgroup/sendmsg — Intercept sendmsg()

Intercepts UDP sendmsg(). Can rewrite the destination address of outgoing datagrams.

When to use it:

  • UDP destination rewrite on send path
  • cgroup-scoped datagram policy
SEC("cgroup/sendmsg4")
int handle_sendmsg4(struct bpf_sock_addr *ctx) {
    return 1;
}

SEC("cgroup/sendmsg6")
int handle_sendmsg6(struct bpf_sock_addr *ctx) {
    return 1;
}

Notes:

  • Same bpf_sock_addr context as connect — field rewrites work identically
  • Only fires for UDP (connected and unconnected send)

cgroup/recvmsg — Intercept recvmsg()

Intercepts UDP recvmsg(). Can rewrite the apparent source address seen by the application.

When to use it:

  • UDP source rewrite on receive path
  • cgroup-scoped recvmsg mediation
SEC("cgroup/recvmsg4")
int handle_recvmsg4(struct bpf_sock_addr *ctx) {
    return 1;
}

SEC("cgroup/recvmsg6")
int handle_recvmsg6(struct bpf_sock_addr *ctx) {
    return 1;
}

cgroup/sockopt — Override Socket Options

Intercepts getsockopt() and setsockopt(). Can filter, replace, or synthesize socket option values.

When to use it:

  • enforcing or virtualizing socket options
  • policy that needs to inspect or override sockopt behavior
SEC("cgroup/getsockopt")
int handle_getsockopt(struct bpf_sockopt *ctx) {
    // ctx->level, ctx->optname, ctx->optval, ctx->optlen
    // rewrite ctx->optval to change what the app sees
    ctx->retval = 0;
    return 1;
}

SEC("cgroup/setsockopt")
int handle_setsockopt(struct bpf_sockopt *ctx) {
    // intercept and modify socket options being set
    return 1;
}

Notes:

  • ctx->optval points to a kernel buffer; use bpf_probe_read_kernel to read it
  • Set ctx->retval = 0 to override the return code seen by the application
  • Useful for enforcing socket option policies across a cgroup

cgroup/sysctl — Control sysctl Access

Intercepts reads and writes to sysctl variables for processes in a cgroup.

When to use it:

  • per-cgroup sysctl access control
  • policy around sysctl reads and writes in containerized environments
SEC("cgroup/sysctl")
int handle_sysctl(struct bpf_sysctl *ctx) {
    // ctx->write: 1 if write, 0 if read
    // ctx->current_value, ctx->new_value
    return 1;  // allow; return 0 to deny
}

Notes:

  • Return 0 to deny the sysctl access with -EPERM
  • ctx->current_value is a string representation
  • Useful for enforcing per-container sysctl policies

cgroup/dev — Device Access Control

Controls access to character and block devices for processes in a cgroup. Equivalent to cgroup v1 devices controller but more flexible.

When to use it:

  • device allow/deny policy for containers or service groups
  • replacing cgroup v1 device rules with BPF logic
SEC("cgroup/dev")
int handle_dev(struct bpf_cgroup_dev_ctx *ctx) {
    // ctx->access_type: BPF_DEVCG_ACC_READ, BPF_DEVCG_ACC_WRITE, BPF_DEVCG_ACC_MKNOD
    // ctx->major, ctx->minor: device numbers
    // ctx->type: BPF_DEVCG_DEV_CHAR or BPF_DEVCG_DEV_BLOCK
    return 0;  // 0 = deny, 1 = allow
}

Notes:

  • Return 1 to allow, 0 to deny (opposite of most cgroup programs)
  • Replaces cgroup v1 device whitelist/blacklist
  • Useful for container runtimes to restrict device access

sockops — TCP Connection Events

Receives callbacks at key TCP lifecycle events. Used to inspect and tune TCP socket parameters.

When to use it:

  • TCP lifecycle tuning
  • RTT/retransmission observation
  • congestion-control related policy or socket tuning
SEC("sockops")
int handle_sockops(struct bpf_sock_ops *ctx) {
    switch (ctx->op) {
    case BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB:
        // outgoing connection established — set TCP options
        bpf_sock_ops_cb_flags_set(ctx, BPF_SOCK_OPS_RETRANS_CB_FLAG);
        break;
    case BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB:
        // incoming connection established
        break;
    case BPF_SOCK_OPS_RETRANS_CB:
        // retransmission occurred
        break;
    }
    return 1;
}

Common ctx->op values:

Op When
BPF_SOCK_OPS_TCP_CONNECT_CB Active connect initiated
BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB Outgoing connection established
BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB Incoming connection established
BPF_SOCK_OPS_NEEDS_ECN ECN negotiation
BPF_SOCK_OPS_RETRANS_CB Retransmission (must be enabled via flags)
BPF_SOCK_OPS_STATE_CB TCP state change

Attaching:

bpftool cgroup attach /sys/fs/cgroup/unified sock_ops id <prog_id>

Notes:

  • Use bpf_sock_ops_cb_flags_set to opt into additional callbacks (retransmit, RTT, state)
  • Can call bpf_setsockopt to tune TCP parameters (window size, congestion control, etc.)
  • Cilium uses this for TCP acceleration between co-located pods

Real pattern: congestion control + TCP socket introspection

SEC("sockops")
int handle_sockops(struct bpf_sock_ops *ctx) {
    if (ctx->op != BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB &&
        ctx->op != BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB)
        return 1;

    // Switch to BBR on connections within the same /20 subnet
    if (ctx->family == AF_INET) {
        __u32 local  = bpf_ntohl(ctx->local_ip4);
        __u32 remote = bpf_ntohl(ctx->remote_ip4);
        if ((local & 0xfffff000) == (remote & 0xfffff000)) {
            char cong[] = "bbr";
            bpf_setsockopt(ctx, SOL_TCP, TCP_CONGESTION, &cong, sizeof(cong));
        }
    }

    // Opt into RTT callbacks for latency tracking
    bpf_sock_ops_cb_flags_set(ctx, ctx->bpf_sock_ops_cb_flags |
                                   BPF_SOCK_OPS_RTT_CB_FLAG);
    return 1;
}

// Access detailed TCP stats via bpf_tcp_sock
SEC("sockops")
int handle_rtt(struct bpf_sock_ops *ctx) {
    if (ctx->op != BPF_SOCK_OPS_RTT_CB) return 1;

    struct bpf_tcp_sock *tcp = bpf_tcp_sock(ctx->sk);
    if (!tcp) return 1;

    bpf_printk("srtt=%u loss=%u retrans=%u\n",
               tcp->srtt_us, tcp->lost_out, tcp->total_retrans);
    return 1;
}

Additional ctx->op values:

Op Enable flag
BPF_SOCK_OPS_RTT_CB BPF_SOCK_OPS_RTT_CB_FLAG
BPF_SOCK_OPS_RETRANS_CB BPF_SOCK_OPS_RETRANS_CB_FLAG
BPF_SOCK_OPS_STATE_CB BPF_SOCK_OPS_STATE_CB_FLAG
BPF_SOCK_OPS_RTO_CB BPF_SOCK_OPS_RTO_CB_FLAG