We Had One Rule...NEVER Restart This Server

Summary

A critical ordering service failed when the company’s long-running CentOS 7 server, legacy01, became unreachable. The server had been running for 2,561 days, had little documentation or monitoring, and was governed by an informal rule: never reboot it.

What the server did

Investigation revealed that an undocumented inventory gateway was running on legacy01 and listening on port 8443. The order service relied on it as a fallback for an older warehouse partner, reserving stock before orders could be completed. When the gateway could not be reached, order requests timed out and the processing queue grew.

The gateway process was still running even though its executable had been deleted. Linux can keep an executable’s contents available to a running process after the file is removed, but the program may no longer be restartable once that process stops. The service had been started manually years earlier, was not managed by systemd, and had no reliable restart procedure—explaining the warning against rebooting the server.

Incident investigation and resolution

• The VM was still powered on, and the gateway worked when tested locally; the application itself had not crashed.
• Network checks showed the server could not reach its gateway. Packet capture showed outbound traffic without useful responses.
• The outage began just after a live migration to another hypervisor.
• Comparing the hosts’ configurations exposed the cause: identically named port groups were assigned different VLANs—216 on the source host and 261 on the destination.
• The virtualization team corrected the destination VLAN. Network connectivity returned, the gateway became reachable, and order errors and queue depth declined.

Follow-up and lessons

The team documented the network setup, added gateway monitoring, and required migration checks to verify actual network connectivity—not just that a VM was powered on. They later rebuilt and migrated the gateway to supported infrastructure, adding deployment automation, monitoring, documentation, redundancy, and tested restart behavior. After confirming that nothing depended on legacy01, they shut it down safely.

Key technical takeaway: A VM being “running” does not guarantee that its application is reachable or healthy. The incident also illustrates the operational risk of undocumented services, manually launched processes, deleted binaries, and unverified network configuration.

Guides, reviews, or tutorials

No formal product review or step-by-step tutorial is presented. The video recounts an incident investigation and demonstrates Linux process behavior, network troubleshooting, and a VLAN configuration failure.

Main speakers and sources

• Narrator: The infrastructure engineer recounting the investigation.
• Mark: Senior engineer who knew the warning against rebooting the server.
• Anna: Backend engineer who identified the order service’s fallback dependency.
• Robert: Virtualization engineer who found and corrected the VLAN mismatch.

The account is presented by the BugX channel as a story. The video metadata describes such stories as inspired by real-world IT incidents and experiences, with identifying details fictionalized.