Ask an OpenShift Admin (Ep 27): Day 2 Operations, part 2

Summary (OpenShift Admin Office Hours Ep 27: Day 2 Operations, part 2)

Participants / format

• Hosted as an “Ask an OpenShift Admin” office hours session.
• Goal: the audience can ask any OpenShift/admin question.
  • If it can’t be answered live, it will be followed up in a blog post.

---

Key topic 1: vSphere-related upgrade block (OpenShift 4.7)

• Andrew shares visibility into Cincinnati blocked edges status for OpenShift 4.7.
• Finding: upgrades remain blocked for customers using vSphere due to a bug discovered in vSphere.
• Impact described as:
  • Holistic release behavior: upgrades are effectively “all or none” across platforms.
    • You can’t target “all platforms except vSphere.”
  • As of the session, it has been ~2 months and the issue is still blocked.
• Advice:
  • Track the linked Bugzilla (BZ).
  • Work is ongoing involving Red Hat and VMware/kernel developers.

---

Key topic 2: IPI on-prem built-in load balancer limitations + external LB options

What’s happening with on-prem IPI ingress

For on-prem IPI, OpenShift uses built-in VIPs during installation:

• VIPs such as *.apps and *.api
• VIP IPs managed via Keepalived to keep services reachable

However, the built-in “ingress” path doesn’t truly load balance across multiple HA router instances:

• Traffic to apps.<cluster> is effectively constrained so that most traffic goes to a single node hosting the router/HAProxy.
• Additional router instances are mainly for HA/failover, not traffic distribution.

Approaches discussed

1. Use an external ingress controller (F5, Citrix, Cisco ACI, etc.)

   • Install an appropriate operator (example: F5 ingress controller for Kubernetes) and use it instead of the default OpenShift ingress controller (HAProxy-based).
   • Benefit: the external appliance can provide real load balancing.

2. Use OpenStack documentation for external load balancer + “endpoint swing”

   • Described as documented/verified for OpenStack only.
   • Not confirmed for other install types (e.g., vSphere), so it sits in a “support gray area.”

3. Unofficial/unrecommended approach

   • Mentioned as possible, but explicitly not supported, so details weren’t emphasized.

Operational guidance

• For on-prem IPI, external LB with true distribution isn’t straightforward because the built-in behavior is closer to a single VIP that moves (via keepalived), not a multi-backend load balancer.
• Keepalived is positioned as “poor man’s F5” only for VIP continuity, not for true load distribution.

---

Key topic 3: DHCP reservations for day 2 stability (especially control plane)

Why DHCP matters

• DHCP is required for IPI because nodes need IPs dynamically.

Recommended: use DHCP reservations (especially control plane)

• Static DHCP reservations are encouraged, particularly for control plane nodes:
  • If control plane IPs change after shutdown/restart:
    • etcd and Cluster Operators can fail to reform
  • Losing quorum across multiple control-plane nodes can prevent recovery

Day-2 process

• After provisioning (even if you didn’t know IPs at install time because IPI uses MAC addresses):
  • Update your DHCP server to convert control plane node addresses into DHCP reservations.

Special note: bare metal IPI

• For bare metal IPI specifically, there’s automation that converts infinite DHCP lease behavior into static IP configuration.
  • Implemented via machine config logic (as cited by the speaker).

DNS clarifications

• DNS expectations can vary:
  • For IPI, internal resolution generally works (with mdns deployed).
  • For direct access/SSH using hostnames from outside the environment, external DNS visibility may matter.

---

Key topic 4: “Day 2 operations” for networking—secondary interfaces/storage networks

Common scenario

Many users need a secondary/tertiary NIC for storage traffic, such as:

• an isolated, non-routable storage network for NFS/CSI/iSCSI PV traffic

Options for configuring extra NICs

1. Configure interfaces at install time (preferred)

   • For UPI and non-integrated installs: use kernel-style ip= parameters for additional interfaces (including examples such as bonds/VLANs).
   • For UPI: installer supports persistent configuration via documented patterns.

2. For vSphere IPI: add extra NICs via machine sets + use NMState

   • Create machine sets / VM network device definitions so nodes receive the additional NICs.
   • NMState:
     • Standalone NMState is Technology Preview
     • When used via OpenShift Virtualization, it’s fully supported and can provide supported interface configuration management

3. Use Machine Config to set static IPs (possible but discouraged)

   • Static IPs differ by node, so this may require:
     • node-specific machine configs, or
     • many machine config pools
   • Operational downsides:
     • update/maintenance behavior depends on machine config pool settings (e.g., “max nodes down”)
     • a single-node pool forces very conservative disruption (nearly sequential reboots), increasing risk/overhead

“Last resort” alternative

• A custom script + machine config approach:
  • detect node identity and apply node-specific IP configuration
• Still not a full solution:
  • adding/removing nodes would require script/machine config updates and triggers reconfiguration/reboots.

---

Key topic 5: Resource management (cluster utilization vs requested resources)

• Clusters change dynamically (machine sets, autoscaling), so day 2 resource management is ongoing.
• Emphasis:
  • optimizing requests vs actual usage (example given: workloads request 10Gi but use less)
  • using metrics/observability ecosystems to reclaim or predict capacity
    • examples mentioned: Datadog, Turbonomic
  • in hyperscaler environments, improving utilization can reduce cost (targeting ~80–90% real CPU usage rather than very low utilization)

---

Q&A highlights

• Calico as SDN

  • Question: can Calico be installed as the SDN?
  • Response: yes—Calico can be deployed via manifests during install.
  • Same idea for IPI vs UPI: Calico manifests are inserted as part of the install flow.

• VMware OVA updates

  • Suggestion: update OVA at least once per major OpenShift release (e.g., 4.6 → 4.7).
  • Within a release: worker provisioning pulls the correct OS/image via installer/machine config/operator steps.

• Cluster DNS vs requirements

  • Reiterated: internal resolution generally works; external DNS depends on how users connect directly to nodes.

---

Main speakers / sources (as referenced in the subtitles)

• Andrew Sullivan (Red Hat) — primary technical speaker
• Chris (“Chris short” / host — OpenShift TV producer) — co-host and moderator