yugabytedb-k8s-submariner
Connect two Kubernetes clusters with Submariner (broker, gateway, Lighthouse service discovery) and deploy one independent YugabyteDB universe per cluster with the yugabyte Helm chart, so the universe
SKILL.mdyugabytedb-k8s/submariner/SKILL.md
GitHubYugabyteDB xCluster over Submariner
Two Kubernetes clusters (s1 = source, s2 = target) joined by Submariner. Each cluster runs its
own universe. Submariner gives pod-to-pod routing between the clusters through a gateway
node on each side, and Lighthouse publishes exported services under clusterset.local. xCluster
needs both: target tservers pull changes straight from source tservers, so every source pod
address must be resolvable and routable from the target.
kind-s1 (pods 10.30/16, svc 10.130/16) kind-s2 (pods 10.40/16, svc 10.140/16)
broker (submariner-k8s-broker ns)
gateway: s1-worker <════ IPsec / WireGuard / VXLAN tunnel ════> gateway: s2-worker
yb-master-0.s1.yb-masters.<ns>.svc.clusterset.local <— Lighthouse —> yb-master-0.s2...
yb-tserver-0.s1.yb-tservers.<ns>.svc.clusterset.local yb-tserver-0.s2...
source universe ─────────────── xCluster ───────────────> target universe
Bundled files (paths relative to this skill):
| File | Purpose |
|---|---|
assets/kind-s1.yaml, assets/kind-s2.yaml |
kind clusters (default kindnet CNI), unique pod/service CIDRs, one worker for the gateway |
scripts/submariner-up.sh |
Creates both clusters, deploys the broker, labels gateways, joins both clusters (idempotent) |
assets/rp-filter-ds.yaml |
DaemonSet that keeps rp_filter loose on every node (invariant 8) |
assets/values-s1.yaml, assets/values-s2.yaml |
yugabyte Helm values for each universe |
scripts/check-mcs-dns.sh |
Checks cross-cluster DNS and TCP to every master/tserver |
references/troubleshooting.md |
Symptoms → causes |
Verified end to end with yugabyte chart 2026.1.2, Submariner 0.25.0 (libreswan cable), kind 0.33
on macOS: tunnel up, both universes formed, cross-cluster DNS/TCP from gateway and
non-gateway nodes, and semi-automatic xCluster replicating rows with sub-second lag. Automatic-DDL mode was not verified (it needs a
backup/restore; see step 5). Before using with other versions, run helm show values yugabytedb/yugabyte and subctl join --help and check that the keys and flags still exist.
Invariants (most failures come from breaking one of these)
- Unique per cluster: Submariner
--clusterid(a DNS-1123 label), pod CIDR, service CIDR. Neither may overlap the other cluster's, or the node network. If they must overlap, deploy the broker with--globalnet(see below) instead. - CIDRs given to
subctl joinmatch the cluster:--clustercidr== kindpodSubnet,--servicecidr== kindserviceSubnet. subctl auto-detects them on most clusters; pass them explicitly anyway so a wrong guess fails loudly. multicluster.kubernetesClusterId==subctl join --clusterid. That ID is the<cluster>label in<pod>.<cluster>.<svc>.<ns>.svc.clusterset.local.masterAddresseslists exactly the masters that exist, using the clusterset names. Withmulticluster.createServiceExports: trueandoldNamingStyle: true, the chart sets each master's--server_broadcast_addressesto${HOSTNAME}.<clusterId>.yb-masters.${NAMESPACE}.svc.clusterset.local:7100.masterAddressesmust use the same names, or masters can't recognise themselves in the list.replicas.totalMasters== number of entries inmasterAddresses. WhenisMultiAz: true, the chart sets--replication_factorfromtotalMasters, which defaults to 3. If you setreplicas.master: 1and leavetotalMastersat its default, the universe expects RF 3 and never becomes healthy.- The namespace must be the same in both clusters. Lighthouse merges a ServiceExport only with exports of the same name in the same namespace, and the clusterset names embed it.
- kube-proxy runs in iptables mode. Submariner does not support IPVS. kind's default is iptables.
- Loose reverse-path filtering (
rp_filter=2) on every node. Pods on non-gateway nodes reach the other cluster overvx-submariner, but replies arrive from the gateway oneth0. The kernel applies the stricter ofnet.ipv4.conf.alland the interface value, and Submariner only loosensvx-submarinerand the CNI interface. So whenall=1(inherited from the host kernel), the replies are dropped. Only pods on the gateway node keep working, which hides the problem until a pod lands elsewhere. Submariner has no option for this.assets/rp-filter-ds.yaml(applied by the script) is a privileged DaemonSet that sets it on every node and keeps it set across node restarts. On real clusters, set it in the node OS (/etc/sysctl.d) or node image instead. ${NAMESPACE}/${HOSTNAME}in values stay literal. The pod's startup script expands them withenvsubst, so don't let your shell expand them (use'…'heredocs).
Workflow
1. Prerequisites
docker, kind, kubectl, helm. On kind, don't run a host-installed subctl: the script runs
subctl from the quay.io/submariner/subctl:<version> image on the docker kind network with kind's
internal kubeconfigs (kind get kubeconfig --internal). A host-run subctl deploy-broker records the
broker URL as https://127.0.0.1:<port>, which the other cluster's pods can't reach, and on macOS
the host binary may also fail with x509: certificate is not trusted or connect: no route to host.
For real clusters, install subctl on the host (curl -Ls https://get.submariner.io | VERSION=0.25.0 bash,
lands in ~/.local/bin; ask before installing it for the user). For two clusters (2 nodes each) plus two RF1 universes, give Docker Desktop at least about
6 CPU and 10 GiB of memory. On Linux hosts, raise fs.inotify.max_user_instances (for example to
512), or kind nodes fail with "too many open files".
For real (non-kind) clusters, skip kind. Open UDP 4500 (IPsec NAT-T) and 4490 (NAT discovery)
between the gateway nodes, and UDP 4800 between all nodes inside each cluster (non-gateway nodes
tunnel to their gateway on it). Keep --natt at its default (on) when the gateways sit behind
NAT, as on most clouds. Calico needs VXLAN mode; see references/troubleshooting.md.
2. Connect the clusters
scripts/submariner-up.sh # add --skip-verify to save several minutes
The script runs these steps (run them by hand if the user wants to see each one):
kind create cluster --name s1 --config assets/kind-s1.yaml
kind create cluster --name s2 --config assets/kind-s2.yaml
# subctl in a container on the kind network, with internal kubeconfigs (https://s1-control-plane:6443)
kind get kubeconfig --internal --name s1 > k1; kind get kubeconfig --internal --name s2 > k2
KUBECONFIG=k1:k2 kubectl config view --flatten > kubeconfig-internal
subctl() { docker run --rm --network kind --user "$(id -u)" -v "$PWD:/work" -w /work \
-e KUBECONFIG=/work/kubeconfig-internal -e HOME=/work quay.io/submariner/subctl:0.25.0 /subctl "$@"; }
subctl deploy-broker --context kind-s1 # writes ./broker-info.subm (contains the IPsec PSK)
kubectl --context kind-s1 apply -f assets/rp-filter-ds.yaml # loose rp_filter on every node
kubectl --context kind-s2 apply -f assets/rp-filter-ds.yaml
kubectl --context kind-s1 label node s1-worker submariner.io/gateway=true
subctl join broker-info.subm --context kind-s1 --clusterid s1 \
--clustercidr 10.30.0.0/16 --servicecidr 10.130.0.0/16 \
--natt=false --label-gateway=false
# same for s2: s2-worker, --clusterid s2, 10.40.0.0/16, 10.140.0.0/16
subctl show connections --context kind-s1 # STATUS: connected
subctl verify --context kind-s1 --tocontext kind-s2 --only connectivity,service-discovery --verbose
Flag notes:
- The broker is just a namespace of CRDs that both clusters sync through. It can live on either
cluster or a third one.
broker-info.submholds credentials and the IPsec PSK: don't commit it. --natt=false: kind nodes share the dockerkindnetwork, so there's no NAT between gateways.--cable-driver:libreswan(IPsec, default),wireguard, orvxlan(unencrypted). If the IPsec tunnel won't come up on a Docker Desktop host, rerun withCABLE_DRIVER=vxlan.--label-gateway=falsestops join from prompting for a gateway node; the label is applied first.- Lighthouse adds a
clusterset.localforward to the cluster's CoreDNS during join.
The gateway, route-agent and Lighthouse pods appear a minute or two after join returns (image
pulls); until then subctl show connections says "No gateways detected" or "No connections found".
Done when subctl show connections shows connected from both sides and subctl verify passes.
If not, run subctl diagnose all --context kind-s1 (and for kind-s2) before continuing.
3. Deploy the universes
NS=yb
helm repo add yugabytedb https://charts.yugabyte.com && helm repo update yugabytedb
for c in s1 s2; do
kubectl --context kind-$c create namespace $NS
helm install yb-$c yugabytedb/yugabyte -n $NS --kube-context kind-$c \
-f assets/values-$c.yaml --wait --timeout 15m
done
Points to tell the user about the values files:
- They are RF1 demo universes. For RF3, set
replicas.master: 3,replicas.tserver: 3,replicas.totalMasters: 3, and list three clusterset master addresses (yb-master-0/1/2.<c>.yb-masters…). Remove theresource/storageoverrides on real hardware. isMultiAz: trueis only there so the chart usesmasterAddressesandtotalMasters. Each cluster still runs a separate universe. Never put the other cluster's masters intomasterAddresses; that would build one stretched universe, not xCluster.- The chart's
ServiceExports usemulticluster.x-k8s.io/v1alpha1, which is what Submariner serves. Check withkubectl get crd serviceexports.multicluster.x-k8s.io. - The masters resolve their own clusterset name at startup. The chart's headless services set
publishNotReadyAddresses: true, and Lighthouse (0.16+) publishes not-ready pods for such services, so this works. On older Submariner the masters can deadlock waiting for DNS. - Check that the universe formed before moving on:
kubectl --context kind-s1 -n $NS exec yb-master-0 -c yb-master -- yb-admin --master_addresses yb-master-0.s1.yb-masters.$NS.svc.clusterset.local:7100 list_all_tablet_servers
4. Verify cross-cluster reachability
scripts/check-mcs-dns.sh yb 1 1 # namespace, masters, tservers
Each direction must resolve and connect to every remote master (7100) and tserver (9100).
kubectl get serviceexports -n yb -o yaml should show the Valid and Ready conditions as True,
and kubectl get serviceimports -n yb should list yb-masters and yb-tservers in both clusters.
If anything fails, stop here and go to references/troubleshooting.md. Don't continue to xCluster.
5. Configure xCluster
Run yb-admin from a master pod. The names below assume namespace yb and RF1:
SRC=yb-master-0.s1.yb-masters.yb.svc.clusterset.local:7100
TGT=yb-master-0.s2.yb-masters.yb.svc.clusterset.local:7100
src() { kubectl --context kind-s1 -n yb exec yb-master-0 -c yb-master -- yb-admin --master_addresses $SRC "$@"; }
tgt() { kubectl --context kind-s2 -n yb exec yb-master-0 -c yb-master -- yb-admin --master_addresses $TGT "$@"; }
Transactional, database-scoped xCluster, semi-automatic mode (YSQL; verified with this skill):
- Create the database and an identical schema on both universes, while the source is still empty:
sql() { kubectl --context kind-$1 -n yb exec yb-tserver-0 -c yb-tserver -- ysqlsh -h yb-tserver-0.$1.yb-tservers.yb.svc.clusterset.local -d $2 -c "$3"; }thensql s1 yugabyte 'CREATE DATABASE app',sql s1 app 'CREATE TABLE …', and the same fors2. src create_xcluster_checkpoint app_rg app. It reports whether a bootstrap is needed. For empty tables it only asks you to create the same objects on the target (done in step 1). If the source already has data, back up and restore the database to the target first.src setup_xcluster_replication app_rg $TGT- Verify:
tgt get_replication_status(no errors) andtgt get_xcluster_safe_time include_lag_and_skew(lag around 1 s). Insert rows on the source and count them on the target. - Later DDL: run it on the source first, then on the target (see the YugabyteDB semi-automatic docs for the order per statement type).
Automatic mode (automatic_ddl_mode, v2025.1+) always requires a full copy. Even for an empty
database, run create_xcluster_checkpoint app_rg app automatic_ddl_mode, then a distributed
backup of the source database restored to the target, then setup_xcluster_replication. Creating
the target database by hand isn't enough, because Postgres OIDs won't match. If you skip the
restore, setup still succeeds, but the first replicated DDL stalls: xCluster safe time freezes, the
table never appears on the target, and the stall also blocks other replication groups on the same
tservers. Recover with src drop_xcluster_replication <group> $TGT. A Helm-only deployment has no
built-in backup target, so use YBA (DrConfig via yb-k8s-operator) or set up distributed backups
before choosing automatic mode.
Table-level (non-transactional, or YCQL): get the table IDs with src list_tables include_table_id
and the source universe UUID with src get_universe_config, then run
tgt setup_universe_replication <source_uuid>_<name> $SRC <table_id,…>.
Planned switchover, failover and DR orchestration are out of scope. For YBA-managed DR, use
the yb-k8s-operator (DrConfig) or yba-api skills.
6. Teardown
helm uninstall yb-s1 -n yb --kube-context kind-s1; helm uninstall yb-s2 -n yb --kube-context kind-s2
kind delete cluster --name s1; kind delete cluster --name s2; rm -f broker-info.subm kubeconfig-internal
On real clusters, remove Submariner with subctl uninstall --context <ctx> instead of deleting the cluster.
PVCs survive helm uninstall. Delete them, or delete the namespace, before reinstalling with
different master addresses, because old master data pins the previous config.
Overlapping CIDRs: Globalnet
If the clusters' pod or service CIDRs overlap and can't be changed, deploy the broker with
subctl deploy-broker --globalnet (default global range 242.0.0.0/8; avoid it with Calico,
which uses it internally). Each headless-service pod then gets a global IP used for both ingress
and egress, and Lighthouse answers clusterset.local queries with those IPs, so the Helm values
don't change. Globalnet creates Services with externalIPs; disable the DenyServiceExternalIPs
admission plugin if it is on. IPv6 is not supported. Prefer non-overlapping CIDRs when you can
choose them: Globalnet adds NAT on every cross-cluster hop.
Reviewing a user's own config
When the user pastes their kind, subctl or Helm config, check each invariant above. The most common mistakes:
replicas.masteris lowered without settingreplicas.totalMasters, so RF stays at 3.masterAddresseslists more masters than exist, or usessvc.cluster.localnames while ServiceExports are on.kubernetesClusterIddoesn't matchsubctl join --clusterid.- The two clusters' CIDRs overlap without Globalnet, or overlap the node network.
- Different namespaces on the two sides, or kube-proxy in IPVS mode.
- On kind:
subctlrun from the host, sobroker-info.submpoints athttps://127.0.0.1:<port>. - Nodes left at
rp_filter=1, so pods off the gateway node can't reach the other cluster. - Automatic-DDL xCluster set up with a hand-created target database instead of a backup/restore.