argocd
ArgoCD GitOps operations: Application and ApplicationSet management, sync waves, health checks, and best practices for K3s homelab clusters.
SKILL.mdskills/argocd/SKILL.md
GitHubArgoCD Skill
Cluster Context
ArgoCD runs on a multi-node K3s cluster (ARM64 Raspberry Pi CM4).
Helm release: argocd in namespace argocd
Chart version pinned in: roles/install-argocd/defaults/main.yml
Chart: argo/argo-cd v9.4.17 (ArgoCD v3.3.6)
Current state
ArgoCD is exposed via type: ClusterIP + HTTPRoute on the shared Cilium Gateway.
Admin password: kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d
URL: https://argocd.cluster.home
Key Helm config:
dex.enabled: false— no SSO (single-user homelab)server.insecure: true— plain HTTP (TLS terminated at Gateway)redis-ha.enabled: false— single-nodeserver.service.type: ClusterIP
ARM64 CM4: repo-server probe tuning
The argocd-repo-server does Redis+GPG+git init on startup and its
/healthz?full=true endpoint takes 2-3 seconds on CM4 hardware.
Default liveness probe timeout was 1s → constant CrashLoopBackOff.
Fixed probes:
repoServer:
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
HTTPRoute (HTTP):
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: argocd-http
namespace: argocd
spec:
parentRefs:
- name: cluster-gateway
namespace: gateway
hostnames:
- "argocd.cluster.home"
rules:
- backendRefs:
- name: argocd-server
port: 80
GRPCRoute (CLI uses gRPC — required for argocd CLI):
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
name: argocd-grpc
namespace: argocd
spec:
parentRefs:
- name: cluster-gateway
namespace: gateway
hostnames:
- "argocd.cluster.home"
rules:
- backendRefs:
- name: argocd-server
port: 80
Note:
BackendTLSPolicyis in the ArgoCD chart but Cilium does not support it yet. Do not configureBackendTLSPolicy— it will be silently ignored or cause issues.
Note:
GRPCRoutesupport was confirmed inargo/argo-cdchart v9.4.17.
GitOps Pattern for this Cluster
~/projects/infra/
└── apps/ ← ArgoCD Application manifests (future)
├── argocd-apps/ ← App-of-apps or ApplicationSets
└── <service>/
└── application.yaml
Store Application manifests in Git — never create them through the UI. Use app-of-apps or ApplicationSet to avoid managing many Application resources.
Creating an Application
Minimal Application manifest:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/villadalmine/infra
targetRevision: main
path: apps/my-app
destination:
server: https://kubernetes.default.svc
namespace: my-app
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
ApplicationSet (preferred for multiple apps)
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: cluster-apps
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/villadalmine/infra
revision: main
directories:
- path: apps/*
template:
metadata:
name: '{{path.basename}}'
spec:
project: default
source:
repoURL: https://github.com/villadalmine/infra
targetRevision: main
path: '{{path}}'
destination:
server: https://kubernetes.default.svc
namespace: '{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Sync Waves (ordered deploys)
Use argocd.argoproj.io/sync-wave annotation to control order within an Application:
metadata:
annotations:
argocd.argoproj.io/sync-wave: "1" # lower = earlier
Typical wave order:
- Wave -1: Namespaces, CRDs
- Wave 0: Secrets, ConfigMaps
- Wave 1: Deployments, StatefulSets
- Wave 2: Jobs, post-deploy checks
Health Check
# All ArgoCD pods
kubectl get pods -n argocd
# All Applications and their sync status
kubectl get applications -n argocd
# App details
kubectl describe application <name> -n argocd
# Trigger manual sync
kubectl patch application <name> -n argocd \
--type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{}}}'
# Check current service type
kubectl get svc argocd-server -n argocd
Troubleshooting
repo-server CrashLoopBackOff on ARM64
→ Health check /healthz?full=true takes 2-3s on CM4 but default probe timeout was 1s.
→ Increase livenessProbe.timeoutSeconds to 5 and initialDelaySeconds to 30.
→ See roles/install-argocd/tasks/main.yml for the fix.
App stuck OutOfSync but diff shows nothing
- Check
argocd.argoproj.io/compare-optionsannotations - Helm chart may generate non-deterministic fields (e.g. random tokens) — use
ignoreDifferences kubectl get application <name> -n argocd -o yaml | grep -A10 ignoreDifferences
App Degraded / pods not starting
- Check Events:
kubectl describe application <name> -n argocd - Check pod status in target namespace:
kubectl get pods -n <namespace> - Sync may have succeeded but app health check fails — check health status field
ComparisonError — cannot get resource
- ArgoCD RBAC may lack permission for a CRD — check
argocd-application-controllerClusterRole - If CRDs installed after ArgoCD: restart controller
kubectl rollout restart deploy/argocd-application-controller -n argocd
UI not reachable at argocd.cluster.home (post-migration)
- Check HTTPRoute:
kubectl get httproute -n argocd - Check Gateway:
kubectl get gateway -n gateway - Check service is ClusterIP:
kubectl get svc argocd-server -n argocd - Check Pi-hole resolves
argocd.cluster.home→192.168.178.200
Useful Commands
# Helm values currently deployed
helm get values argocd -n argocd
# Force app refresh (re-read Git without waiting)
kubectl annotate application <name> -n argocd \
argocd.argoproj.io/refresh=hard --overwrite
# Delete app WITHOUT pruning resources (orphan)
kubectl patch application <name> -n argocd \
-p '{"metadata":{"finalizers":[]}}' --type merge
kubectl delete application <name> -n argocd
# App-controller logs (sync errors)
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100