Operations and Lifecycle

Day-to-day operations

A data center is never finished. Servers arrive and leave every week, firmware needs updating, filters and batteries wear out, and every change is a chance to unplug the wrong cable. The Uptime Institute, which classifies data center designs, reports that human error accounts for more than 70% of downtime: a well-designed building run carelessly can fail more often than a modest one run with discipline.

Typical routine work in a data center team:

• Walk-throughs (rounds) on the floor: check for amber LEDs and alarms, unusual noise or heat, open rack doors, cables in walkways, water, and anything that does not match the documentation. Most monitoring is remote, but eyes and ears still catch a failing fan bearing or a loose floor tile.
• Monitoring through the building management system (power, cooling, fire, leaks; see environmental monitoring) and the IT monitoring, including the servers' out-of-band management controllers.
• Tickets: installs, replacements of failed disks and power supplies, cabling requests, visitor access.
• Shift handover: what is open, what is running, which alarms are known and which are new.
• Maintenance of facility equipment by qualified contractors: UPS and battery checks, generator test runs, cooling unit service, fire suppression inspections.

DCIM and documentation

DCIM (Data Center Infrastructure Management) is software that combines the inventory of everything in a data center — racks, devices, rack positions, power chains and cable connections — with live readings of power, temperature and other sensors, so that operators can see the current state and plan capacity. A typical DCIM tool shows:

• Floor plans and rack elevations: which device sits in which rack unit, with front and rear views.
• Power chains: which device power supply is plugged into which outlet of which rack PDU, on which A or B feed, from which UPS. This answers "what goes down if we switch off this breaker?" before someone does it.
• Connections: patch panel ports, switch ports and cable IDs, end to end.
• Readings: power per rack and per circuit, inlet temperatures, alarms, often collected from smart PDUs and sensors.
• Capacity: free rack units, remaining power and cooling per rack, row and room, free ports.

DCIM covers the physical layer. The configuration management database (CMDB) records the services and their dependencies; the two are usually linked, so a server's CMDB record points to its rack position and power chain, and a planned power maintenance can be translated into "these services are at risk".

Capacity planning

Capacity planning in a data center means tracking several resources at once, because the first one to run out limits everything else:

• Space: free racks and rack units.
• Power: per circuit, per rack and per room, keeping enough headroom that either the A or the B feed alone can carry the load (see power).
• Cooling: the heat the room and each rack can remove (see cooling).
• Network: free switch ports, fiber pairs between rooms, uplink bandwidth.
• Weight: racks full of batteries, storage or GPU servers can exceed what a raised floor or building floor is rated for.

Capacity that exists but cannot be used is called stranded: a rack with 20 free units but no power left, or a room with power but no cooling. Good planning keeps the resources in balance. This program estimates, for each resource in Kestrel Energy's own server room, how many months remain before it reaches the planning threshold of 80%:

# Kestrel Energy's server room: capacity, current use and monthly growth.
# (name, unit, capacity, used now, growth per month)
RESOURCES = [
    ("IT power", "kW", 120, 78, 1.6),
    ("Cooling", "kW", 130, 80, 1.6),
    ("Rack space", "U", 14 * 42, 391, 4),
    ("Leaf switch ports", "ports", 192, 151, 3),
]
THRESHOLD = 0.80     # plan new capacity before 80% is in use

print(f"{'Resource':<18}{'Used':>12}{'Share':>7}  Months until {THRESHOLD:.0%}")
for name, unit, capacity, used, growth in RESOURCES:
    share = used / capacity
    months = (THRESHOLD * capacity - used) / growth
    when = "already over" if months <= 0 else f"{months:.0f}"
    print(f"{name:<18}{used:>6} {unit:<5}{share:>7.0%}  {when}")

Resource                  Used  Share  Months until 80%
IT power              78 kW       65%  11
Cooling               80 kW       62%  15
Rack space           391 U        66%  20
Leaf switch ports    151 ports    79%  1

Space and power look comfortable, but the leaf switches are almost full: the next project will need a new pair of switches before it needs a new rack. The threshold is below 100% on purpose, because the fix has a lead time. A switch arrives in weeks; extra cooling takes months; a heavier grid connection can take years, especially where the grid is congested (see energy and sustainability). Capacity planning for services — headroom, forecasting and load tests — is on Capacity Planning and Performance.

Remote hands

Many organizations have equipment in a colocation facility, or in a site of their own that is not staffed around the clock. Remote hands is the service in which on-site technicians of the data center operator carry out simple physical tasks on a customer's equipment on request, such as power-cycling a server, swapping a labeled disk, reseating a cable or reading out a display, so that the customer does not have to travel. More involved work, such as installing servers or cabling, is often sold as smart hands, usually charged per quarter hour.

The technician does not know your systems, so the request must be impossible to misread:

• Site, room, cage, rack and rack unit, plus the device's asset tag or serial number and a photo or rack elevation if you have one.
• Exactly what to do, step by step, and what not to touch.
• What the technician should see to confirm the right device: a blinking locator LED (which you switch on remotely through the BMC), an amber fault light, a label.
• A contact who is reachable during the work, and a request for a photo afterward.

When the operating system and the network are down, the on-site fallback is a crash cart: a trolley with a monitor, keyboard and mouse (and often a laptop with serial cables) that is rolled to the rack and plugged into the server's local console ports. Out-of-band management has made it less common, but every data center still keeps one.

Changes on the floor

Physical work follows change enablement like any other change. Racking a new server in an empty position may be a standard change; anything that touches power distribution, cooling or core network is a normal change with a maintenance window. For risky facility work, data centers write a method of procedure (MOP): an exact, numbered list of steps, with checks after each step, the expected readings, and a rollback for each step. Two people work through it, one reading and checking, one acting.

• Check the power chain first. Before unplugging anything, confirm in DCIM and on the device that it has a second, live power supply — and that the other feed is actually healthy.
• Label before you unplug. Cables that are removed should go back in the same port.
• Firmware updates of BIOS/UEFI, BMC, controllers and drives follow the vendor's order and compatibility list, one host at a time in a cluster, after its workloads have been moved away.
• Hot-swappable parts (drives, power supplies, fans) can be replaced while the server runs, but only the failed one — pulling the healthy disk of a degraded RAID 1 mirror takes the server down.

The hardware lifecycle

Every device in a data center goes through the same asset lifecycle, and each step has a data center specific job:

• Plan and buy: check that the hardware is on the hardware compatibility list (HCL) of the hypervisor or operating system, and that the rack has the space, power, cooling and ports.
• Receive and test: check the delivery against the order, record serial numbers, update firmware to the approved versions, and run a burn-in test for a day or more, because many defective components fail early in their life.
• Install and record: rack, cable, label, and register in DCIM and the CMDB before the server goes into production.
• Operate and maintain: support contracts, spare parts, firmware and patch updates.

Hardware refresh

A hardware refresh is the planned replacement of servers, storage or network equipment that is reaching the end of its useful or supported life, typically in batches and on a cycle of several years. The triggers are concrete:

• The vendor's support or warranty ends, so failed parts and firmware fixes are no longer available.
• Newer hardware does the same work with far fewer servers and watts, which frees power and rack space (see the capacity section above) and can lower licensing costs that are counted per CPU core.
• The software stops supporting the hardware: a new hypervisor release drops older CPUs or controllers from its compatibility list.

Refreshes are planned a year or more ahead, because migrating hundreds of VMs, rebuilding storage and moving data takes time. In a virtualized cluster the new hosts join the cluster, workloads are moved with live migration, and the old hosts are drained and removed — without downtime for the services.

Decommissioning

Decommissioning is the controlled removal of a system from service: proving that nothing depends on it any more, retiring it through change management, removing it from every record and system, cleaning up its power and network connections, and handing its storage media to secure sanitization before the hardware is reused or disposed of. Done sloppily, it causes outages (a forgotten dependency) or data leaks (a disk sold with customer data on it).

1. Verify non-utilization. Check network traffic, logins, scheduled jobs and the CMDB dependencies over a meaningful period, including month-end and year-end. Agree the removal with the service owner.
2. Raise a change and settle what happens to the data: which backups or archives must be kept, and for how long (media retention).
3. Shut down, then wait. Power the system off but leave it in place for an agreed period, so it can be started again if something unexpected breaks.
4. Remove it from everything: monitoring, backup jobs, DNS, IP address management, load balancer pools, firewall rules, certificates, licenses, out-of-band management accounts.
5. Cable remediation: remove the power and network cables back to the PDU and patch panel, free the switch ports and update the cabling records. Abandoned cables block airflow and make the next change riskier.
6. Remove the hardware, update DCIM and the asset record, and send all storage media to sanitization.

Data sanitization

Data sanitization is making the data on storage media unrecoverable before the media is reused or leaves the organization, with a method chosen for the media type and the sensitivity of the data, and with verification and records that prove it was done. The basics — why formatting is not enough, what degaussing does and a table of methods per media type — are on the secure disposal section of the Security topic.

The reference is NIST SP 800-88 Rev. 2, Guidelines for Media Sanitization, finalized in September 2025; Rev. 1 from 2014 was withdrawn. Rev. 2 is organized around a sanitization program — a policy, roles, decisions per kind of data, verification and records — and leaves most technique details to IEEE 2883-2022, Standard for Sanitizing Storage, to other recognized specifications, or to the organization's own approved standard. IEEE 2883 sorts techniques into three categories:

• Clear: logical techniques such as overwriting all user-addressable storage; protects against simple recovery tools.
• Purge: techniques that defeat even laboratory recovery, such as the sanitize commands built into SATA, SAS and NVMe drives (overwrite, block erase, crypto erase).
• Destruct: shredding, disintegrating, melting or incinerating the media.

Cryptographic erase

Most data center drives and arrays encrypt everything they store. Cryptographic erase (crypto erase) sanitizes such media by securely destroying the encryption key, so the data that remains on the media can no longer be decrypted. It takes seconds instead of hours, works for flash media where overwriting cannot reach every cell, and lets a whole array be sanitized at once. It is only as good as the encryption: the data must have been encrypted from the start, with no copy of the key elsewhere, and the drive's implementation must be trustworthy. That is why policies combine it with verification, and often with a block erase as well.

An organization writes its decisions down as a policy that technicians can apply without debate. This program encodes such a policy for a batch of retiring media from Kestrel Energy's server room; note the rule that a broken drive is always destroyed, because it cannot run a sanitize command or prove the result:

# An example sanitization policy, as an organization might write it down
# under NIST SP 800-88 Rev. 2, using the IEEE 2883 categories.
POLICY = {
    # media type: (stays in the organization, leaves the organization)
    "HDD": ("Clear: overwrite all sectors, verify",
            "Purge: sanitize overwrite command, verify"),
    "SATA SSD": ("Purge: sanitize block erase, verify",
                 "Purge: sanitize block erase, verify"),
    "NVMe SSD": ("Purge: sanitize crypto erase, verify",
                 "Purge: sanitize crypto erase + block erase, verify"),
    "Self-encrypting drive": ("Purge: cryptographic erase, verify",
                              "Purge: cryptographic erase, verify"),
    "LTO tape": ("Clear: overwrite before reuse",
                 "Destruct: shred"),
}


def decide(media, leaves_org, working):
    if not working:
        # A drive that cannot run commands cannot prove it was sanitized.
        return "Destruct: shred, record serial number"
    return POLICY[media][1 if leaves_org else 0]


RETIRING = [
    ("SRV-0412 disk 1", "HDD", True, True),
    ("SRV-0412 disk 2", "HDD", True, False),
    ("ARR-02 shelf 3", "Self-encrypting drive", True, True),
    ("HOST-17 boot", "NVMe SSD", False, True),
    ("LIB-01 slot 44", "LTO tape", True, True),
    ("SRV-0388 cache", "SATA SSD", True, True),
]

for asset, media, leaves_org, working in RETIRING:
    where = "leaves" if leaves_org else "reused"
    print(f"{asset:<16} {media:<22} {where:<7} {decide(media, leaves_org, working)}")

SRV-0412 disk 1  HDD                    leaves  Purge: sanitize overwrite command, verify
SRV-0412 disk 2  HDD                    leaves  Destruct: shred, record serial number
ARR-02 shelf 3   Self-encrypting drive  leaves  Purge: cryptographic erase, verify
HOST-17 boot     NVMe SSD               reused  Purge: sanitize crypto erase, verify
LIB-01 slot 44   LTO tape               leaves  Destruct: shred
SRV-0388 cache   SATA SSD               leaves  Purge: sanitize block erase, verify

The policy here is an example, not a rule from the standard: what your organization requires depends on the sensitivity of the data and its own approved methods. Whatever the method, the record per device (serial number, method, date, who did it, verification result) is what makes sanitization auditable. When an external company destroys media, it issues a certificate of destruction listing the serial numbers, and the chain of custody from your floor to its shredder is documented.

Reuse, recycling and WEEE

Sanitized hardware can be repurposed internally (an old host becomes a lab or backup proxy), sold through an IT asset disposition (ITAD) company, or recycled. In the European Union the Waste Electrical and Electronic Equipment (WEEE) Directive, 2012/19/EU, requires that discarded equipment goes to proper collection and recycling, not to general waste (see e-waste). Use certified recyclers or ITAD partners, keep their certificates with the asset records, and remember that batteries from UPS systems and servers are a separate, hazardous waste stream.