Removing kubectl from CI Pipelines: A Slower, More Honest Deployment Model

Most teams adopt ArgoCD to speed up Kubernetes deployments. The real value is that it slows them down. The shift from a push-based CI/CD pipeline to a pull-based GitOps model introduces deliberate latency — a 45–60 second reconciliation lag — in exchange for eliminating an entire category of production incidents: configuration drift.

Context / Stakes

Consider a mid-size company running an internal developer platform: ~12 platform services, three environments (dev, staging, prod), and a single regional cluster serving ~80 internal engineers. The platform team is four engineers. In the classic model, the CI/CD pipeline builds a container image, pushes it to a registry, and runs kubectl apply directly against the cluster.

This appears fast on the surface. A developer merges a PR, observes a green checkmark, and assumes the change is live. But the platform becomes a support ticket machine. A team files a ticket: “my deployment broke.” A platform engineer investigates and finds a manual kubectl edit from weeks ago that was never committed to Git. The repository and the live state tell different stories. When a deployment fails, the question is not “what broke?” — it is “what is running, and who changed it?”

At this scale, the cost of a bad deployment is not downtime. It is developer trust. Every time a platform engineer says “we are not sure what is running,” another team loses faith in self-service. If the deployment model remains event-driven rather than state-driven, the platform team becomes a bottleneck.

The “Obvious” Solution

The standard approach is a push-based CI/CD pipeline. GitHub Actions, GitLab CI, or Jenkins builds the image, then executes kubectl apply or helm upgrade directly against the cluster. This works well for small teams, single environments, and low change rates. It is the industry default because it feels fast: one pipeline, one green checkmark, one deployed service.

This approach fails when the pipeline is forced to act as a cluster administrator. The CI system holds cluster credentials. Manual changes in the cluster are invisible to Git. Rollbacks require a platform engineer to hunt for the last known good image tag and re-run the pipeline. The pipeline becomes a deployment event, not a deployment state. When supporting dozens of internal developers, events are not auditable.

The Real Solution / Experiment

4.1 The Decision

The correct alternative is ArgoCD with a separate GitOps repository. The insight is that the problem is not deployment speed, but deployment honesty — the gap between what Git declares should be running and what is actually running. When the customers are internal developers, honesty is the product.

In this model, the CI pipeline builds the image and updates the image tag in the GitOps repository. ArgoCD, running inside the cluster, watches that repository and reconciles the live state. The pipeline never touches the cluster.

4.2 The Code

The GitOps repository should be separate from application code so that platform changes and application changes have independent review paths:

# gitops-repo/apps/api-gateway/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: api-gateway

resources:
  - ../../base

images:
  - name: platform/api-gateway
    newTag: v1.4.2  # CI bumps this line; ArgoCD handles the rest

replicas:
  - name: api-gateway
    count: 4

The ArgoCD Application resource that watches the repository:

# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-gateway-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: platform
  source:
    repoURL: https://github.com/acme/platform-gitops.git
    targetRevision: main
    path: apps/api-gateway/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: api-gateway
  syncPolicy:
    automated:
      prune: true        # delete resources removed from Git
      selfHeal: true     # revert manual cluster changes
    syncOptions:
      - CreateNamespace=true

The critical trade-offs here:

• selfHeal: true means Git wins. A manual kubectl edit is reverted within minutes. This is the anti-hotfix feature, and it is the point.
• prune: true means deleting a manifest from Git deletes the resource. Dangerous for stateful workloads, essential for honest state.
• The CI pipeline only writes to Git. It never holds cluster credentials. A compromised CI secret cannot touch the cluster.

4.3 The Benchmark

Unexpected cost: The ArgoCD sync adds a 50-second lag between “CI finished” and “deployment is live.” Developers notice. Teams must be retrained to stop watching the pipeline and start watching the ArgoCD UI. The pipeline is no longer the source of truth — Git is. The friction is real, but it is educational.

4.4 What Broke

The first failure mode appears when a senior engineer, responding to a traffic spike, manually scales a Deployment to 20 replicas. ArgoCD detects the divergence, flags the app as OutOfSync, and reverts the scale to the Git-defined count of 4. The engineer is frustrated: “I needed 20 replicas. I do not care about GitOps.”

The root cause is not technical — it is cultural. Engineering teams are trained to treat the cluster as mutable. GitOps treats it as read-only. The platform team’s role is not to block developers; it is to make the right way the easy way.

The fix is to add a Horizontal Pod Autoscaler to the base manifests and create a fast-path PR template for platform changes. A developer opens a PR to bump the replica count or HPA threshold, it auto-approves via CODEOWNERS if linting passes, and ArgoCD syncs within minutes. The technical fix takes a day. The culture shift takes a month.

Trade-offs

This trade-off is correct when the platform team is small enough that manual cluster access is a risk, and the internal customer base is large enough that drift is inevitable. It is a mistake when sub-minute deployment feedback is required for rapid iteration, or when the engineering culture is not ready to treat Git as the only source of truth.

When to Use This

Use this approach when:

• The platform serves > 50 internal developers who need self-service
• The platform team is < 5 engineers who cannot afford to be cluster firefighters
• Invalidation logic is complex (e.g., “deploy the gateway only after the auth service is healthy”)

Avoid it when:

• Sub-minute deployment feedback is required for rapid feature iteration
• There is no observability into cluster state (ArgoCD reveals drift; teams must act on it)
• The culture treats manual kubectl as a valid on-call playbook

Conclusion

The pipeline does not deploy the application. Git does. ArgoCD enforces the contract.