Video summary

What a CI/CD Pipeline Actually Does?

Main summary

Key takeaways

Technology

Main technological concepts and CI/CD analysis

Why a “green pipeline” can still fail in production

A commit can successfully build, test, and deploy to staging, yet production fails because the production environment runs a different artifact than what staging tested—even if the source commit and pipeline steps appear identical.

What actually gets deployed: artifact vs commit/recipe

A deploy ships a build output (“artifact”) created from the commit, including:

  • Your code
  • All dependencies
  • The base image
  • Anything added by the build toolchain/build machine

Artifacts can be:

  • Container images
  • JAR files
  • Binaries

The “recipe drift” problem (moving tags / changing ingredients)

Build instructions act like a recipe (e.g., a Dockerfile using FROM node:20). If a tag changes over time (maintainers release a patch and move the tag), rebuilding the same commit later can produce a different artifact.

Result: staging may have tested older dependencies, while production runs newly rebuilt ones.

Core rule to prevent it: build once, promote the artifact

  • After tests pass, the pipeline creates one artifact and stores it in a registry.
  • It uses an artifact digest (fingerprint) representing the exact contents.
  • No downstream stage rebuilds it.
  • Staging and production deploy the same digest (same bits), while differences come only from configuration.

Configuration vs artifact immutability

The artifact stays the same, while environment-specific settings vary, such as:

  • Database connection targets
  • Keys / secrets
  • Service addresses

So the same image reads different environment configuration per deployment.

Debugging shortcut enabled by artifact pinning

If the same artifact behaves differently in production, you can narrow the cause away from code differences and toward:

  • Environment configuration
  • Infrastructure
  • Data
  • Runtime state
  • Traffic patterns
  • External services

Staging limitations

Staging traffic is simulated; real user traffic can surface issues earlier than automated tests. The objective is to reduce blast radius until confidence is built with real traffic.

Deployment strategies and rollback behavior

Rolling deployment

Deploy the new artifact to a subset of servers (e.g., 2 at a time), keeping old + new versions running concurrently. Rollback requires deploying the old version again across the fleet.

Blue/green deployment

Two full production-like environments:

  • Blue serves users
  • Green is deployed and tested

Then routing is switched so 100% of traffic goes to Green. Switch back quickly if needed.

Tradeoff: higher cost (two environments).

Canary deployment (incremental traffic shifting)

  • Send a small portion of traffic (e.g., 1%) to the new artifact.
  • Compare live error rate/latency against the baseline (old version).
  • Increase gradually (e.g., 10% → 50% → 100%) if healthy.

Goal: validate against real traffic while affecting fewer users first.

Rollback as “deploying an older artifact digest”

If canary metrics look bad:

  • Redeploy the last known good artifact digest stored in the registry.

Rollback doesn’t require a special mechanism beyond the same artifact-promotion workflow. The older artifact is already proven under real production traffic.

Fix-forward guidance when pipelines are slow

  • If a pipeline is fast, you can commit a fix and redeploy a new artifact.
  • If production is actively failing and the pipeline takes too long, the recommended approach is:

    rollback first, then fix forward afterward This avoids waiting for rebuild time.

Database compatibility concern for rollback

If schema changes occur:

  • The new version might depend on new tables.
  • Rollback may fail if the old artifact can’t work with the new schema.

Emphasis: schema compatibility is required (not expanded; said it deserves a separate video).

Real-world incident example (safety/scale justification)

CrowdStrike (July 19, 2024) Windows crash event

  • A configuration update to the Falcon security agent was rolled out globally “at once.”
  • The update triggered a bug causing affected machines to crash (about 8.5 million).
  • Revert released ~78 minutes later, but many machines couldn’t download the fix because they crashed during boot.
  • Root cause analysis: introduce deployment rings starting with canary.
  • Scale reasoning: 1% of 8.5 million is 85,000—still large, but far less than the full blast radius.

Overall: what the pipeline is really for

Tests matter, but the bigger purpose is:

  • Tracking the same artifact from build to production
  • Ensuring recovery/rollback is fast and reliable

Key takeaway: “Promote the bits, not the recipe.”

Main speakers/sources

  • Speaker: Not specified in the provided subtitles.
  • External source example: CrowdStrike (Falcon security agent incident; referenced as a case study).

Original video