Ask an OpenShift Admin (Ep 51): Easy non-production test cluster setup

Summary — OpenShift non-production test cluster setup (Ep 51)

This episode focuses on practical ways to stand up easy non-production OpenShift test environments, with emphasis on tradeoffs in supportability, resource requirements, and realistic vs. simplified testing.

Key theme: non-production clusters are “marginally supported”

The hosts repeatedly caveat that most approaches discussed are for non-production and minimal-resource setups, so expectations should be tempered regarding official support.

---

Top of mind / related OpenShift & ecosystem items

• OpenShift Pipelines 1.6 released (Tekton-based CI/CD).
  • Interest in experimenting with pipelines to automatically rebuild containers when source/template images change.
• SRE/DevOps learning resources
  • A series of ebooks on DevOps culture and SRE practices (linked during the stream).
• Argo conference (online, free)
  • Hosted by Intuit; Christian Hernandez (Red Hat) presenting.
• OpenShift 4.9 network interface configuration guidance
  • Discussion of configuring primary vs secondary machine network interfaces via install config (networking.machineNetwork.subnet) and later changes.
  • Primary interface changes
    • Considered risky/hard to change.
    • Recommended path: node reload.
    • Brief mention of nmstate operator as tech preview.
  • Secondary interfaces
    • Options include:
      • Machine Config approach (supported, but considered awkward—often involves defining configs for many nodes in one MachineConfig).
      • nmstate operator for additional interfaces (tech preview, except when used with OpenShift Virtualization).
• OpenShift networking / DHCP
  • DHCP is required for IPI networking because OpenShift can’t directly manage external DNS/IP allocation.
  • DHCP (with dynamic DNS) is used to assign and register node identity.

---

Q&A / operational notes highlighted

• Restoring from etcd backups (VMware UPI)
  • Mentioned restoring etcd data via a script documented in the relevant guidance.
  • Clarified intent of etcd backups: not a full cluster disaster recovery mechanism.
  • Best use cases: scenarios like control plane node failure where resync scope is minimized.
• Accidentally deleting the openshift namespace
  • Suggestion: open a Red Hat support case.
  • Operators/templates/samples often recover if the namespace is recreated, but recovery isn’t guaranteed to be identical for all content.
• Single-node OpenShift storage gotcha
  • Single-node setups rely on the Local Storage Operator, so registry/PV readiness may require manual configuration (similar to bare metal).

---

Main tutorial content: “easy” non-production cluster setup options

1) Recommended minimal lab: CodeReady Containers (CRC)

Used to get an OpenShift-like API/console environment quickly for testing (especially developer/admin workflows).

Use cases emphasized

• Familiarize with OpenShift resource model (cluster operators, operators, etc.).
• Deploy and test operators and builds; create persistent volumes; run build configs.
• Reproduces “feels like production” workflows, but with reduced footprint/capabilities.

Current footprint / constraints (as described)

• CRC installs an OpenShift 4 environment with a virtual machine.
• Resource requirements mentioned during the demo:
  • ~4 vCPU and ~8GB RAM minimum, plus ~35GB disk baseline.
• No supported upgrade path for the CRC-style cluster (upgrade blocked).
• Requires hardware virtualization; nested virtualization may work if host/VM flags permit, but performance may suffer.
• Not suitable for features like OpenShift Virtualization that depend on nested/hardware virtualization.

---

2) Single-node OpenShift (assisted installer / CLI)

A “real” OpenShift cluster (more production-like than CRC, but heavier).

Resource guidance

• Assisted installer generally requires 8 CPUs and 32GB RAM.
• CLI-based single-node might run with fewer resources, but hosts suggested it’s still substantial:
  • Example guidance: try not below ~6 CPUs / 24GB,
  • likely more if you want to run anything beyond cluster bring-up.

When to choose it

• If you want closer-to-real OpenShift behavior for admin tasks.

Test realism warning

• Single-node updates may not behave the same as multi-node upgrades—so emulate production only as needed.

---

3) Compact cluster

A compromise: a 3-node layout where nodes can include both control plane + compute.

Why it’s useful

• More realistic than single-node for scenarios like machine/config operations (e.g., adding nodes).
• Potentially closer to multi-node scheduling/resource-request behavior.

Resource notes

• Discussed as requiring roughly 6 CPUs / 24GB RAM per node (can be reduced, but scheduler requests drive minimum “requested CPU” capacity).
• Even if CPU usage looks low, resource requests still affect scheduling capacity.
  • Example: scheduler must reserve CPU based on requests from system components (e.g., ingress/controller components and operator workloads).

---

Automation options for spinning up test clusters

The hosts covered multiple “infrastructure automation” paths, with a common theme: some are officially supported components used as building blocks, while others are partner/unsupported for production.

A) Hive (Cluster-as-a-Service approach)

• Deploy cluster infrastructure by defining a CRD instance that triggers OpenShift installation with parameters.
• Suggested for teams needing dynamic cluster provisioning.
• Best matched with IPI; on-prem can be harder due to external DNS/external environment dependencies.

B) Red Hat community / partner GitHub automation (Ansible-based)

• Official-ish repos: sets of Ansible playbooks for home-lab / PoC use.
• Strong caveat: automation may be not supported for production.

C) Helper Node (common for disconnected/no-DNS/no-DHCP bootstrap)

• Helper Node v1: Ansible-driven installation of services like NFS exports to back PersistentVolumes (example NFSv4 mention).
• Helper Node v2: containerized implementation using a helper-node-ctl workflow.
• Motivation: get clusters running quickly when enterprise services (DNS/DHCP/PXE) aren’t available—primarily for testing/disconnected scenarios.

D) Partner automation examples

• Nutanix Calm Blueprints: deploy OpenShift via Nutanix automation (in progress/productizing, as mentioned).
• VMware automation / vRealize referenced as another route.
• Equinix Metal / KVM-based scripting automation developed by Red Hat community folks:
  • described as creating multi-node clusters hosted internally,
  • including possible bare-metal/IPMI emulation.

---

Free/low-cost hands-on learning environments

• learn.openshift.com
  • Redesigned learning paths.
  • Includes interactive labs and an OpenShift 4.9 playground that launches a cluster experience quickly (brief-lived sessions).
  • Useful for operator exploration (e.g., trying oc get ... and observing CRC-backed behavior).
• OpenShift Sandbox
  • Mentioned separately as developer-focused; resets periodically.
• RH demo systems
  • Internal to Red Hat employees/partners; customers can request workshop access via their account team.

---

Licensing / entitlements (important takeaway)

• CRC: can use a developer subscription for basic use.
• Single-node / compact:
  • discussion of 60-day trial entitlements and developer entitlements enabling up to 16 OpenShift cores for non-production single-user use cases.
• Mentioned using Red Hat Console to enable entitlements.

---

Main speakers / sources

• Andrew Sullivan (Red Hat) — host / main explainer
• Johnny (Red Hat) — co-host (also drives CRC demo and architecture comparisons)
• Stephanie — producer (behind the scenes)