Video Summary - This One DevOps Question Changed Everything 💀| Must Watch Before Giving DevOps & Cloud Interviews 🚨

Main Ideas & Concepts Covered

Mock DevOps interview setup & community building
- The video is a mock interview organized by “DevOps and AI Cloud Labs” to help people prepare for real DevOps & cloud interviews.
- An experienced DevOps engineer (interviewee) is interviewed by an interviewer who asks typical DevOps questions.
DevOps role overview (AWS + Kubernetes + CI/CD)
- The interviewee has ~5 years of IT experience and currently works as an AWS DevOps engineer.
- Key responsibilities include:
  - CI/CD pipeline management and optimization using Jenkins
  - Infrastructure provisioning on AWS using Terraform
  - Deployments and orchestration using Kubernetes / EKS
  - Supporting microservices deployment using Docker
  - Cluster/node upgrades (weekly or biweekly)
  - Deployment support across dev / QA / production
  - Observability setup using Prometheus + Grafana (later mentions Grafana Cloud)

Jenkins CI/CD Pipeline (High-Level Stages + Controls)

The interviewer asks for a “from repo to deployment” view. The interviewee describes a multi-stage Jenkins pipeline (~6–7 stages):

Version/read & setup
- Read the application version
- Install dependencies
Static analysis / quality gate
- Perform static code analysis
- Run quality gate checks
- If code issues/bugs/third-party issues are found:
  - Stop the pipeline
  - Developer must address failures
Tests
- Run testing
- Include smoke tests and functional tests
Build
- Apply final quality gate checks based on predefined rules
- Build a Docker image
Vulnerability scanning
- Push image to AWS ECR
- Run Trivy scanning for vulnerabilities
- If vulnerabilities exist, handle scanning results via an ECR “image repro” flow (wording unclear, intent is to manage vulnerability outcome)
Deployment
- Deploy to AWS EKS (cluster) using Helm charts
- Use environment-specific variables (e.g., dev/QA)

Automation trigger

Pipeline triggers on merge request / pull request to main
- Developers create a feature branch PR to main
- Once approved/merged, the pipeline starts

Helm-Based Deployment Approach to EKS

Deployment method: Helm (charts/templates)
Environment targeting:
- Dynamically pass variables (e.g., --env=dev/QA or Helm values)
Target cluster: AWS EKS
Why Helm: enables deploying the same microservice into multiple environments by changing configuration values

Preventing “Deployment Succeeded but Pods Crash Looping” Issues

The interviewee describes validation/mitigation before promoting deployments across environments:

Synthetic analysis / SLO validation
Live testing
Measure traffic/performance using percentiles (P90/P95/P99 mentioned)
Monitor for serious issues such as:
- Severe runtime/incident errors (referred to as “OEM killer errors” — wording unclear)
- image pull errors
Only after expected behavior is confirmed do they move to QA/UAT/upper environments

The focus is framed as preventing bad deployments from progressing (rather than a single explicit “crashloop alert step,” though alerting is discussed elsewhere).

EKS Upgrade / Migration Planning (Kubernetes 1.33 → 1.34)

The interviewer asks for high-level steps to upgrade production EKS clusters:

Planning

Check release notes for the target Kubernetes version
Validate compatibility with cluster components
Confirm whether workloads will continue running and consider scheduler/queue-related concerns
Take etcd backups (etcd mentioned as the database)

Control plane / cluster update

Update the cluster via AWS Console

Worker node upgrade

Drain pods / avoid running pods on old nodes:
- “Call down” worker nodes
- Create new worker nodes
- Drain old nodes so pods reschedule to new nodes
- Keep new worker capacity ready
Run synthetic testing and performance validation (P95/P99 mentioned again)
After success, delete/retire the old worker group

Deployment strategy

Identified as blue-green deployment, which the interviewee accepts as the approach

Compatibility Handling for Cloud-Native Ingress Tools

Tools mentioned: Ingress Controller (and discussion references NGINX, cert-manager, etc.)
If release notes indicate an ingress controller is not compatible with Kubernetes 1.34, the interviewee would:
- Not update the incompatible component blindly
- Test in dev first
- Take backups/prepare rollback
- Plan changes based on observed issues and safe compatibility

Terraform Concepts

Terraform: Destroy vs Taint

destroy: destroys the resource
taint: marks a resource so the next apply forces recreation/update behavior (wording imperfect, but intent is to trigger replacement rather than always full removal)

Handling Secrets

Avoid hardcoding secrets in Terraform files
Use Terraform variable properties:
- sensitive variables/labels to mask values
Store secrets in AWS Systems Manager Parameter Store
Pipelines fetch secrets during execution

Kubernetes Ingress Flow (abc.com → Pod)

From the request-path perspective:

User hits abc.com
Traffic routes to:
- Route 53 (DNS)
- Application Load Balancer (ALB)
ALB traffic is handled/managed by the Ingress controller
Ingress rules perform path-based routing (e.g., /home)
Ingress → Service → Endpoints/Pods
- Service routes to endpoints, which map to backend pods

Monitoring / Alerting for Kubernetes Using Grafana

Observability tool: Grafana Cloud (also earlier mentions Prometheus/Grafana)
Metric/alert categories mentioned:
- Cluster-level counts (phrasing unclear; likely workload/health metrics)
- Pod counts and lifecycle states
  - track pods being alive
  - use liveness probe and readiness probe
- Detect issues such as:
  - ImagePullBackOff
  - CrashLoopBackOff
  - OOM events
- Alerts by namespace
- Service health checks:
  - endpoint existence
  - alerts when services lack endpoints
- Latency-based alerts:
  - latency across Ingress → Service → Pods
APM
- Interviewer asks what APM is used with Grafana Cloud; interviewee response is incomplete/uncertain (“need to look”).

Kubernetes Security Hardening Steps

The interviewee outlines steps to secure a “default-like” Kubernetes cluster:

Network & isolation
- Deploy cluster on a private subnet
- Use Kubernetes Network Policies to restrict traffic between namespaces/pods
Access control
- Use RBAC (role-based access control)
Pod/container security
- Use Security Context settings:
  - prevent running as root
- Mentions only what they know when asked about additional features
Secret handling & runtime injection
- Don’t hardcode secrets in YAML
- Fetch secrets from AWS Parameter Store
- Use init containers to retrieve/mount secrets
IRSA / service accounts for AWS access
- Create service accounts for pods that access AWS services
- Use OIDC / IRSA provider for secure AWS permissions

Cloud-Native Tools Mentioned

Helm
Ingress controller / ALB/load balancing (AWS-related)

The interviewer also asks specifically about cert-manager, NGINX, and Velero; the interviewee indicates limited/no experience with cert-manager and some of the others (states they haven’t worked on some).

Cost Optimization Practices (AWS)

The interviewee describes cost reduction across compute, network, and database:

Identify top cost drivers
- Use Resource Optimizer / Cost Explorer
- Find over-provisioned vs under-provisioned EC2 instances
Compute savings
- Stop workloads in dev/QA
- Stop workloads weekends and non-business hours
NAT Gateway optimization
- Review NAT gateway billing drivers (data transfer)
- Avoid unnecessary NAT deployment across multiple AZs
- Suggest VPC endpoints instead of NAT for certain traffic patterns
- Claimed cost reduction: ~25% in one ECS/private image-fetch example
RDS strategy optimization
- Avoid unnecessary active-active DR if not required
- Switch to active-passive DR where appropriate
- Match DR strategy to business needs to reduce ongoing costs

Production Incident Scenario (Pressured on a Saturday)

Situation

Production deployment engineer is the only person available
A senior person requests immediate deployment to reduce risk

Approach

First verify the incident
- Check logs and Kubernetes events (e.g., via kubectl)
- Determine if issues are node-related vs scheduler-related
- Inspect pod-level errors
If the previous version is safe but the current version has issues:
- Roll back quickly to the known-good version (e.g., via kubectl rollout-style actions)
- Keep the business running
- Investigate the failing version afterward
If both previous versions have the same issue, but the new version fixes it:
- Understand and validate the fix
- Rerun/trigger the pipeline if appropriate
- Even under pressure:
  - follow the process to avoid compliance issues
  - validate at key levels (ALB/ingress/app health)
  - proceed only after checks

Key lesson emphasized: Don’t skip process, even when pressured by management.

GitOps Usage (How They Apply GitOps Principles)

Infrastructure configs managed in GitHub
- Configuration files stored in a repository
Approval workflow
- PR required
- At least two approvers before proceeding
Pipeline gates
- After merge/approval:
  - additional approval required (manager + TL)
  - then Terraform apply runs

Speakers / Sources Featured

Interviewer: Ayush (asks questions; introduces the mock interview structure)
Interviewee: Arpitha / Vara (introduced as “Arpitha”; later referenced as “Vara”)
Organization/Source (context): DevOps and AI Cloud Labs

Tools / Technologies referenced

Jenkins, Docker, AWS ECR
Trivy
Kubernetes, AWS EKS, Helm
Prometheus, Grafana, Grafana Cloud
Terraform, AWS Parameter Store
Route 53, Application Load Balancer (ALB), Ingress controller
AWS Resource Optimizer / Cost Explorer
RBAC, Security Context, Network Policies
GitHub-based workflow (GitOps-like process)

This One DevOps Question Changed Everything 💀| Must Watch Before Giving DevOps & Cloud Interviews 🚨

Key takeaways