Storage Fundamentals: Block, File and Object

Three ways to reach data

Every application eventually reads and writes bytes on a drive. What differs is the interface the storage offers to the server, and that interface decides what the storage can be shared with, how it scales, and what it costs. There are three: block, file and object. The Cloud Storage page describes the same three as cloud services; this page looks at them as you meet them on a data center floor, where you buy, rack and run the systems yourself.

The clearest way to tell them apart is to ask where the file system runs:

Block storage

A block storage system offers a server a raw volume, usually called a LUN (logical unit number), that looks exactly like a local disk. The server partitions it, formats it with NTFS, ext4, XFS or VMFS (the VMware cluster file system) and manages every file on it. The storage system only sees numbered blocks being read and written; it has no idea which file they belong to. Servers reach it over a storage area network (SAN) with Fibre Channel (FC), the Internet Small Computer Systems Interface (iSCSI) or NVMe over Fabrics (NVMe-oF); see SAN Protocols.

Block storage gives the lowest latency and the most control, which is why databases and virtual machine (VM) datastores live on it. The catch: two servers that each mount the same NTFS or ext4 volume will corrupt it, because each thinks it owns the file system. Sharing a volume safely needs a cluster file system (VMFS, for example) or an application that coordinates access itself.

File storage

A file storage system, usually a network-attached storage (NAS) appliance or a file server, runs the file system itself and shares folders over the network with the Network File System (NFS) or Server Message Block (SMB) protocol. Clients ask for files by path (/projects/2026/tariffs.xlsx), not for block numbers. Because one system owns the file system, it can handle locking, permissions and many simultaneous clients safely. Typical uses: home directories, department shares, shared content for web servers, and sometimes VM datastores over NFS. DAS, NAS and SAN covers NFS and SMB in detail.

Object storage and the S3 API

An object store keeps each piece of data as a whole object: the bytes, a key (its name) and metadata, in a flat namespace of buckets. There is no file system to mount and objects are written as a whole, not edited in place. In exchange, object stores scale to billions of objects and petabytes across many nodes, and they spread data over nodes with replication or erasure coding (see Software-Defined Storage and Hyperconverged Infrastructure).

Applications talk to an object store over HTTPS. The S3 API — the interface of Amazon Simple Storage Service (S3) — is the de facto standard: no standards body owns it, but on-premises object stores such as Ceph's RADOS Gateway (RGW), MinIO and most storage vendors' object platforms implement it. Backup software, log archives and analytics tools that "support S3" can therefore usually write to an object store in your own data center as well as to the cloud. Many S3-compatible stores also support Object Lock, which makes objects immutable for a set period — a key defense against ransomware for backups.

Which model for which workload

Many storage products offer more than one model. A "unified" array serves LUNs and NFS/SMB shares from the same drives, and some also offer an S3 endpoint. The choice is still made per workload: the billing database of an energy supplier wants a low-latency block volume; its contract scans fit a file share; ten years of smart-meter readings and the nightly backup copies fit an object store.

Capacity: raw, usable and effective

Storage is sold by raw capacity: the sum of all drive labels. What you can actually store is much less, and capacity planning goes wrong when the two are confused. Usable capacity is what is left for data after everything the system needs for itself has been taken off:

• Redundancy: RAID parity or mirrors in an array, or extra copies and erasure-coding chunks in software-defined storage. RAID 10 gives up half the drives, RAID 6 two drives per group.
• Hot spares: drives that sit empty until another one fails (see failures and rebuilds).
• Units: drive vendors count in decimal terabytes (TB, 10¹² bytes); most operating systems and arrays report binary tebibytes (TiB, 2⁴⁰ bytes). A 7.68 TB SSD shows up as about 6.98 TiB — 9% "missing" before anything else happens.
• Reserves: metadata, snapshot space and free-space headroom. Many arrays and file systems slow down or behave badly when nearly full, so vendors advise keeping a margin free — often 10–20%; the exact figure is in each product's documentation.

Effective capacity goes the other way: with deduplication and compression a system can store more logical data than its usable capacity. That figure depends entirely on the data, so plan on usable capacity and treat data reduction as a bonus (see Storage Arrays in Practice).

Raw to usable in Python

One shelf of 24 SSDs of 7.68 TB each, laid out three different ways. The script converts to TiB and keeps 10% free:

# One shelf of 24 SSDs, 7.68 TB each (decimal terabytes, as on the label).
DRIVE_TB = 7.68
DRIVES = 24
RESERVE = 0.10  # keep 10% free for metadata, snapshots and headroom

layouts = [
    # name, groups, drives per group, redundancy drives per group, hot spares
    ("RAID 10, 11 mirrored pairs", 11, 2, 1, 2),
    ("RAID 6, 2 groups of 11", 2, 11, 2, 2),
    ("RAID 6, 1 group of 23", 1, 23, 2, 1),
]

raw_tb = DRIVES * DRIVE_TB
print(f"Raw: {DRIVES} x {DRIVE_TB} TB = {raw_tb:.2f} TB")
for name, groups, per_group, redundancy, spares in layouts:
    assert groups * per_group + spares == DRIVES
    data_drives = groups * (per_group - redundancy)
    after_raid_tib = data_drives * DRIVE_TB * 10**12 / 2**40
    usable_tib = after_raid_tib * (1 - RESERVE)
    share = usable_tib * 2**40 / 10**12 / raw_tb
    print(f"{name:27} {data_drives:2} data drives  "
          f"{usable_tib:6.1f} TiB usable  ({share:.0%} of raw)")

Raw: 24 x 7.68 TB = 184.32 TB
RAID 10, 11 mirrored pairs  11 data drives    69.2 TiB usable  (41% of raw)
RAID 6, 2 groups of 11      18 data drives   113.2 TiB usable  (68% of raw)
RAID 6, 1 group of 23       21 data drives   132.0 TiB usable  (79% of raw)

The same 184 TB of flash yields anywhere from 69 to 132 TiB. The wide RAID 6 group looks best on paper, but a rebuild after a drive failure has to read all 22 surviving drives, and the group can lose only two drives in total. Real arrays make this trade-off for you with their own layouts; the point is to ask the vendor for usable capacity in TiB, after spares and reserves, not raw TB.

Performance: IOPS, block size, throughput and latency

Four numbers describe how fast storage is, and they only make sense together.

IOPS

IOPS (input/output operations per second) counts how many separate read and write requests a drive, volume or array completes per second. It is the number that matters for workloads with many small, scattered requests: databases, mailbox servers, and dozens of VMs sharing one datastore. A vendor's IOPS figure is usually measured with small (4 KiB) random reads, the most flattering case; your workload mixes reads and writes of different sizes and gets less.

Writes cost more than reads on protected storage. In classic RAID, one small random write turns into several drive operations: two for RAID 1 and 10 (both mirrors), four for RAID 5 (read data and parity, write both) and six for RAID 6. This "write penalty" is why a write-heavy database is a poor fit for RAID 5 on spinning disks. Modern flash arrays hide much of it behind a write cache and full-stripe writes, but it has not disappeared.

Block size

Block size (or I/O size) is the amount of data moved by one request. Databases typically use 8 KiB or 16 KiB pages, VMs booting read 4 KiB to 64 KiB, and backup or video software streams 256 KiB to 1 MiB at a time. Do not confuse it with the file system's allocation unit (cluster size), although the two are related.

Throughput and latency

Throughput, in MiB/s or MB/s, is IOPS multiplied by block size. It matters for large sequential transfers: backups, restores, video and analytics scans. Latency is the time one request takes, from the moment the server sends it until the answer arrives. Latency is what users feel: a database that waits 10 ms instead of 0.5 ms for every read is slow no matter how many IOPS the array promises.

The three are tied together by Little's Law (see Kanban for the same law applied to work items): the number of requests in flight equals IOPS times latency. To reach high IOPS at a given latency, the server must keep many requests outstanding at once — the queue depth. A single-threaded application that waits for each read before sending the next can never exceed 1 ÷ latency IOPS: 2,000 IOPS at 0.5 ms, however fast the array.

# Four workloads on the same array: what IOPS, block size and latency add up to.
workloads = [
    # name, IOPS, block size (KiB), average latency (ms)
    ("Billing database", 20_000, 8, 0.5),
    ("VDI boot storm", 30_000, 4, 1.0),
    ("Nightly backup", 1_600, 256, 4.0),
    ("Smart-meter archive", 400, 1024, 10.0),
]

print(f"{'Workload':20} {'IOPS':>7} {'Block':>8} {'MiB/s':>7} {'In flight':>10}")
for name, iops, block_kib, latency_ms in workloads:
    mib_per_s = iops * block_kib / 1024          # throughput = IOPS x block size
    in_flight = iops * latency_ms / 1000         # Little's Law: L = arrival rate x wait
    print(f"{name:20} {iops:7,} {block_kib:5} KiB {mib_per_s:7.1f} {in_flight:10.1f}")

Workload                IOPS    Block   MiB/s  In flight
Billing database      20,000     8 KiB   156.2       10.0
VDI boot storm        30,000     4 KiB   117.2       30.0
Nightly backup         1,600   256 KiB   400.0        6.4
Smart-meter archive      400  1024 KiB   400.0        4.0

The virtual desktop infrastructure (VDI) boot storm needs the most IOPS but the least bandwidth; the backup needs only 1,600 IOPS and the archive only 400, yet those two move the most data (400 MiB/s each). Sizing storage for one says nothing about the other, which is why a storage request should always state IOPS, block size, read/write ratio and the latency the application can tolerate.

What drives can do

A hard disk drive (HDD) has to move its head to the right track (seek) and then wait, on average, half a rotation for the data to pass under it. That mechanical delay caps a single HDD at a few hundred random IOPS, whatever its interface. Solid-state drives (SSDs) have no moving parts and reach tens of thousands to hundreds of thousands of IOPS (see Storage devices). The HDD figures below use half a rotation plus a typical average seek time; real drives vary.

This is why new primary storage is almost always flash, and why 7,200 rpm drives survive where capacity per euro matters more than IOPS: backup targets, archives and object stores. Reaching 20,000 random IOPS with 7,200 rpm drives would take about 250 of them; a handful of SSDs does it easily.

Hot, warm and cold data: storage tiers

Not all data is equally busy. Today's billing run is read thousands of times per hour; last year's invoices are read a few times a month; meter readings from 2019 are kept for legal reasons and may never be read again. Putting all of it on the fastest flash wastes money; putting all of it on slow disks makes the busy part slow.

A storage tier is a class of storage media with its own speed and cost per terabyte, chosen to match how hot the data on it is. Names and numbering differ per vendor, but a common split is:

Data can be placed on a tier by hand, moved by policy (for example "move files not opened for 180 days"), or moved automatically block by block inside an array; that last form is automated tiering. Cloud providers sell the same idea as storage classes (see Cloud Storage). Whatever the mechanism, watch the retrieval side: reading archive data back can take minutes (tape) to hours (some cloud archive classes), which matters if a restore depends on it.

Reading a storage request

Put together, a useful storage request from an application team answers six questions. When one is missing, ask before you size:

1. Access model: block, file or object — and if file, NFS or SMB.
2. Capacity: usable, today and in 12 to 36 months.
3. Performance: peak IOPS, block size and read/write ratio, or peak throughput for streaming work.
4. Latency: what the application tolerates (a database might need under 1 ms).
5. Protection: snapshots, replication, backup, and the recovery point and recovery time objectives (see Disaster Recovery and Site Resilience).
6. Sharing: one server, a cluster, or many clients.