Ask an OpenShift Admin (Ep 23): DNS and cluster networking

Summary (OpenShift DNS + cluster networking, Ask an OpenShift Admin Ep 23)

Show purpose / context

• This episode focuses primarily on DNS in OpenShift (also applicable to Kubernetes), because DNS behavior in cluster environments can be a “black box” compared to typical DNS expectations.
• They structure the discussion around how service discovery works in Kubernetes/OpenShift and how DNS resolution differs between:
  • Node-level DNS
  • Pod-level DNS (via CoreDNS and the pod resolver configuration)

---

Pre-show updates and operational notes (not DNS-specific)

1) vSphere IPI node-reap data loss risk

• Reported issue: During vSphere IPI with worker node failures, the IPI process can delete/reap the VM. If that VM still has PVC-attached VMDKs, the PVCs may be destroyed with the VM, leading to data loss.
• Mitigation suggestion:
  • If using IPI with autoscaling, configure the autoscaler to include a delay (e.g., ~5 minutes) so workloads can be relocated before the VM is reclaimed.

2) CoreOS + vSphere packet loss in OpenShift SDN (VXLAN offload workaround)

• Observed issue: Inconsistent packet loss (e.g., ~1–5%) after deploying/upgrading to OpenShift 4.7 with CoreOS on vSphere.
• Suspected cause: Related to VMXNET drivers in kernel 4.7 / Rail 8.3 used by CoreOS.
• Workaround:
  • Disable VXLAN offload on VM network adapters (using sysctls / machine config / recurring configuration).
• Verification approach:
  • Test using oc debug or MachineConfig.

3) Reference architectures policy shift

• Red Hat stopped publishing reference architectures frequently due to:
  • resource/maintenance burden
  • partner variability
• Instead, Red Hat integrates guidance into documentation, such as:
  • recommended host practices
  • scalability/performance sections
  • rationale embedded within docs
• Partner-created reference architectures still exist (e.g., HPE/Dell/NetApp/Cisco) and are reviewed by Red Hat.
• Exception mentioned: The OpenShift on OpenStack reference architecture remains Red Hat-created because it’s part of a Red Hat stack.

4) Sizing/app resource guidance references

• They point to a blog post about application sizing principles (and related episodes on resource limits/ranges).
• Emphasis: differences between VM resource behavior and container constraints.

---

Main technical content: DNS in OpenShift

High-level DNS model in OpenShift/Kubernetes

• Kubernetes DNS is described as a layered design:
  1. Pod-level name resolution
  2. CoreDNS (cluster internal DNS service)
  3. Node-level resolver (/etc/resolv.conf)
  4. External DNS resolution via forwarding
• Key takeaway: knowing “DNS in general” is not the same as understanding “DNS in OpenShift/Kubernetes.”

---

Kubernetes/OpenShift service discovery (core reason DNS exists here)

• In Kubernetes/OpenShift, DNS primarily enables service discovery:
  • Applications reference other services by name (e.g., database) rather than hardcoding IPs.
• Implementation details:
  • Pods (microservices) are fronted by Kubernetes Services
  • CoreDNS serves those service names inside the cluster
• CoreDNS history:
  • DNS components evolved from SkyDNS → kube-dns → CoreDNS

---

CoreDNS deployment model and operator-managed configuration

• OpenShift deploys CoreDNS via a DNS operator.
• The DNS service uses the service network overlay (cluster DNS service address).
• Cluster defaults referenced:
  • Kubernetes internal domain: cluster.local
  • CoreDNS service Cluster IP comes from the service IP range configured in install-config.yaml
• Important caution:
  • Avoid cluster names/domains that conflict with “.local-style naming.” Conflicts (notably “anything dot local” issues) can cause troublesome DNS behavior.

---

How pod resolution works (pod → CoreDNS → node → external)

Pod resolver (/etc/resolv.conf)

• Each pod has /etc/resolv.conf pointing to the CoreDNS service IP.
• Pods use a search domain list, which triggers DNS lookups using different suffixes (e.g., namespace-based search, then internal/cluster domains, then external/internal behavior depending on configuration).
• For internal service names:
  • CoreDNS resolves them from Kubernetes service records

Forwarding behavior for external names

• If CoreDNS cannot resolve a name internally, it uses forwarding:
  • CoreDNS forwards “unknown” queries to the DNS servers listed in the node’s /etc/resolv.conf
• This forms the chain:
  • Pod checks /etc/hosts → pod resolver → CoreDNS → host resolver (/etc/resolv.conf on the node) → external DNS

---

DNS forwarding configuration in CoreDNS (plugin feature)

• CoreDNS supports extensible behavior via plugins, including forwarding.
• Forwarding can route specific domains (e.g., foo.com, bar.example.com) to custom upstream DNS servers, rather than relying only on the node resolver.
• Notable detail:
  • CoreDNS forwarding can specify ports (contrasted with classic BIND behavior).

---

Pod-to-node and name file precedence details

• DNS flow also includes precedence and local files:
  • Pod /etc/hosts entries (populated via OpenShift DNS node resolver)
  • Pod resolve.conf
  • Node resolve.conf as fallback when CoreDNS forwards
• DNS component on nodes:
  • Runs as a per-node component (described as DaemonSet-style)
  • The node-side container updates/manipulates pod host files (e.g., /etc/hosts) so common names may resolve without extra DNS hops where appropriate
• Static pods question:
  • They confirm CoreDNS also applies to static pods (e.g., etcd/control plane), since they participate in the same SDN and receive the same DNS configuration.

---

Troubleshooting guidance emphasized

• When encountering DNS failures:
  • Follow the resolution chain to find where it breaks.
• Important framing:
  • DNS problems often look like “DNS is broken,” but can be symptoms of a different component failure.
• They recommend using an OSI model approach for troubleshooting.

---

Q&A points (important behavioral implications)

External DNS / DHCP / node name lookups

• Question discussed: whether vSphere IPI DHCP must provide DNS records for nodes.
• Guidance:
  • For many setups, DHCP should provide/update DNS (soft-yes; strongly encouraged), especially if external systems must resolve node names.
  • For internal resolution, OpenShift/on-prem IPI may rely on mDNS for node name resolution.

Order changes of upstream name servers

• If upstream DNS order must change after installation:
  • Prefer machine config (or adjust DHCP order so leases refresh in the desired order).
• CoreDNS forwarding can also be used for domain-specific upstream selection.

Multi-zone IPI feasibility

• IPI cannot span multiple network zones for control plane vs workers because of Keepalived requirements, which need appropriate layer-2 adjacency as configured/supported.

vSphere multi-cluster / multi-DRS

• Mentioned as roadmap/possibly not fully supported yet.
• Some internal testing showed “mostly works” with an approximate 95% success on certain versions, with a referenced Jira issue for details.

Router / authentication operator degraded due to external lookup failure

• Home lab scenario: if upstream DNS becomes unavailable, authentication cluster operator becomes degraded.
• Hypothesis:
  • Degraded status likely reflects failure in an external DNS chain or router-related lookup health checks.
  • External endpoints (node ports / well-known endpoints) depend on external resolution, so when external DNS fails, operator health checks can fail.

---

Main speakers / sources

• Chris Short (Executive Producer, OpenShift TV) — host/producer
• Andrew Sullivan (Host, Red Hat) — moderator for office hours / Q&A
• Christian Hernandez — “DNS man” guest; main DNS technical explainer
• Referenced materials:
  • OpenShift documentation
  • DNS Operator / CoreDNS
  • Kubernetes DNS concepts