CSV, JSON and Parquet

The file formats data travels in: CSV and its many dialects, JSON and JSON Lines, Parquet as a columnar format, and the classic traps with decimal commas, leading zeros, dates and encodings.

Every night at 02:30 KBS, Kestrel Energy's billing system, writes an export file. Lucía Romero opens it in Excel to build the monthly KPI pack; Joost Bakker wants to load it into an analytics store. Whether the numbers survive that trip depends less on the tools than on the file format and on how it is read. This page shows the formats with Python, pandas and DuckDB 1.5 (as of October 2026). How to read and write CSV and JSON in Python in general is on Files and Paths; JSON itself is explained on JSON and Other Data Formats.

CSV: simple until it isn't

Comma-separated values (CSV) is a table as plain text: one record per line, fields separated by a delimiter, usually a header line with column names. Every spreadsheet, database and programming language can read and write it, which is why exports are almost always CSV. Its simplicity is also the problem: a CSV file says nothing about its delimiter, its encoding or the types of its columns.

RFC 4180: the common form

RFC 4180 (October 2005) describes the CSV format and registers the media type text/csv. It is an Informational document, not a standard, and says so itself: "there is no formal specification in existence". It describes what most programs already did:

  • each record on its own line, ending in CRLF (carriage return + line feed);
  • an optional header line with the same number of fields;
  • fields separated by commas;
  • fields that contain a comma, a double quote or a line break are enclosed in double quotes;
  • a double quote inside such a field is written twice: "".

It says nothing about other delimiters, decimal separators, dates or data types, and for the character set it only offers an optional charset parameter on the media type. A revision that would have made UTF-8 the default (draft "rfc4180bis", last updated in August 2024) expired without becoming an RFC, so as of October 2026 RFC 4180 is still the only reference. Python's csv module follows these rules by default: 

import csv
import io

rows = [
    ["customer_id", "name", "note"],
    ["C-10005", "Bakkerij Korenschoof B.V.", "Ovens 04:00-07:00, closed Sunday"],
    ["C-10007", "Fietsenmakerij De Spaak", 'Wants "paper" invoices'],
    ["C-10010", "Drukkerij Inktvis B.V.", "Two meters:\nhall and office"],
]

buffer = io.StringIO()
csv.writer(buffer).writerows(rows)
text = buffer.getvalue()
print(text.replace("\r\n", "\\r\\n\n"), end="")
print()

for row in csv.reader(io.StringIO(text)):
    print(len(row), row)
customer_id,name,note\r\n
C-10005,Bakkerij Korenschoof B.V.,"Ovens 04:00-07:00, closed Sunday"\r\n
C-10007,Fietsenmakerij De Spaak,"Wants ""paper"" invoices"\r\n
C-10010,Drukkerij Inktvis B.V.,"Two meters:
hall and office"\r\n

3 ['customer_id', 'name', 'note']
3 ['C-10005', 'Bakkerij Korenschoof B.V.', 'Ovens 04:00-07:00, closed Sunday']
3 ['C-10007', 'Fietsenmakerij De Spaak', 'Wants "paper" invoices']
3 ['C-10010', 'Drukkerij Inktvis B.V.', 'Two meters:\nhall and office']

The writer added quotes only where they were needed and doubled the quotes around "paper". The last record spans two lines in the file, yet the reader returns it as one row with three fields. A script that splits lines on \n and fields on , gets both wrong, which is why you always read CSV with a real CSV parser.

Dialects, delimiters and decimal commas

A CSV dialect is the set of formatting choices a particular CSV file uses: delimiter, quote character, how quotes are escaped, line ending, and whether there is a header. Two files can both be "CSV" and still need different settings to be read correctly.

The most visible choice is the field delimiter, the character that separates one field from the next: a comma, a semicolon, a tab (often saved as .tsv) or a pipe |. It is tied to the decimal separator, the character between the whole and the fractional part of a number: a point in English (86.35) and a comma in Dutch, German and many other locales (86,35). In those locales the thousands separator is often a point: 1.204,75.

Excel takes the list separator it uses for CSV from the Windows region settings. Microsoft's own documentation notes that switching the decimal separator to a comma forces Excel to use a semicolon as the list separator. A Dutch Windows computer therefore writes semicolon files with decimal commas, and a CSV made on a laptop in Amsterdam can open as one long column on a laptop in Dublin. The KBS export follows the Dutch convention: semicolons, decimal commas and dates written as day-month-year. 

import csv
from datetime import datetime
from decimal import Decimal

with open("kbs-export.csv", "w", encoding="utf-8", newline="") as f:
    f.write(
        "invoice_id;ean;period;amount_eur;status\r\n"
        "INV-40026;871687400000000017;01-09-2026;86,35;open\r\n"
        "INV-40030;871687400000000024;01-09-2026;640,10;paid\r\n"
        "INV-40035;871687400000000031;01-09-2026;452,90;open\r\n"
    )

with open("kbs-export.csv", encoding="utf-8", newline="") as f:
    print(next(csv.reader(f)))  # the default dialect expects commas

with open("kbs-export.csv", encoding="utf-8", newline="") as f:
    dialect = csv.Sniffer().sniff(f.read(1024), delimiters=";,")
    print("delimiter:", repr(dialect.delimiter))

total = Decimal("0")
with open("kbs-export.csv", encoding="utf-8", newline="") as f:
    for row in csv.DictReader(f, delimiter=";"):
        amount = Decimal(row["amount_eur"].replace(".", "").replace(",", "."))
        period = datetime.strptime(row["period"], "%d-%m-%Y").date()
        total += amount
        print(row["invoice_id"], row["ean"], period.isoformat(), amount)
print("total", total)
['invoice_id;ean;period;amount_eur;status']
delimiter: ';'
INV-40026 871687400000000017 2026-09-01 86.35
INV-40030 871687400000000024 2026-09-01 640.10
INV-40035 871687400000000031 2026-09-01 452.90
total 1179.35

With the default comma dialect, the whole header comes back as a single field. csv.Sniffer can guess the delimiter from a sample, but in a script that runs every night it is safer to state it. The amount is converted by removing thousands points and replacing the decimal comma, and stored as a Decimal, because money should not become a floating-point number (see rounding errors). The EAN codes (the 18-digit codes that identify a connection to the grid) stay text.

Schema inference: tools guess the types

Because CSV carries no types, every tool that reads it into a table has to decide what each column is. Schema inference is that automatic guess: the reader samples the file, tries to parse each column as a number, a date or text, and picks a delimiter and header setting. It saves a lot of typing and is usually right, which is exactly why its mistakes go unnoticed. DuckDB calls its version the CSV sniffer: 

cat > kbs-export.csv <<'CSV'
invoice_id;ean;period;amount_eur;status;due_date
INV-40026;871687400000000017;01-09-2026;86,35;open;15-10-2026
INV-40030;871687400000000024;01-09-2026;640,10;paid;15-10-2026
INV-40035;871687400000000031;01-09-2026;452,90;open;15-10-2026
CSV
duckdb -c "SELECT * FROM 'kbs-export.csv';"
duckdb -c "SELECT * FROM read_csv('kbs-export.csv', decimal_separator=',',
           types={'ean': 'VARCHAR', 'amount_eur': 'DECIMAL(10,2)'});"
┌────────────┬────────────────────┬────────────┬────────────┬─────────┬────────────┐
│ invoice_id │        ean         │   period   │ amount_eur │ status  │  due_date  │
│  varchar   │       int64        │    date    │  varchar   │ varchar │    date    │
├────────────┼────────────────────┼────────────┼────────────┼─────────┼────────────┤
│ INV-40026  │ 871687400000000017 │ 2026-09-01 │ 86,35      │ open    │ 2026-10-15 │
│ INV-40030  │ 871687400000000024 │ 2026-09-01 │ 640,10     │ paid    │ 2026-10-15 │
│ INV-40035  │ 871687400000000031 │ 2026-09-01 │ 452,90     │ open    │ 2026-10-15 │
└────────────┴────────────────────┴────────────┴────────────┴─────────┴────────────┘
┌────────────┬────────────────────┬────────────┬───────────────┬─────────┬────────────┐
│ invoice_id │        ean         │   period   │  amount_eur   │ status  │  due_date  │
│  varchar   │      varchar       │    date    │ decimal(10,2) │ varchar │    date    │
├────────────┼────────────────────┼────────────┼───────────────┼─────────┼────────────┤
│ INV-40026  │ 871687400000000017 │ 2026-09-01 │         86.35 │ open    │ 2026-10-15 │
│ INV-40030  │ 871687400000000024 │ 2026-09-01 │        640.10 │ paid    │ 2026-10-15 │
│ INV-40035  │ 871687400000000031 │ 2026-09-01 │        452.90 │ open    │ 2026-10-15 │
└────────────┴────────────────────┴────────────┴───────────────┴─────────┴────────────┘

The sniffer found the semicolon, the header and even the day-month-year dates. It did not recognize the decimal comma: amount_eur came back as text, so a sum() on it would fail. And it made the EAN code a 64-bit integer, which happens to work for these values but would silently drop a leading zero in any code that had one. The second command states what the file is: decimal comma, EAN as text, amounts as exact decimals.

pandas behaves the same way: read_csv infers types unless you tell it otherwise. pandas 3.0 shows text columns with the new str data type (see Data Analysis with Python): 

import pandas as pd

with open("kbs-export.csv", "w", encoding="utf-8") as f:
    f.write(
        "invoice_id;ean;period;amount_eur;status\n"
        "INV-40026;871687400000000017;01-09-2026;86,35;open\n"
        "INV-40030;871687400000000024;01-09-2026;640,10;paid\n"
        "INV-40035;871687400000000031;01-09-2026;452,90;open\n"
    )

naive = pd.read_csv("kbs-export.csv")
print(naive.shape)

df = pd.read_csv(
    "kbs-export.csv",
    sep=";",
    decimal=",",
    dtype={"ean": "str"},
)
df["period"] = pd.to_datetime(df["period"], format="%d-%m-%Y")
print(df)
print(df.dtypes)

df.to_parquet("invoices.parquet", index=False)
print(pd.read_parquet("invoices.parquet").dtypes)
(3, 1)
  invoice_id                 ean     period  amount_eur status
0  INV-40026  871687400000000017 2026-09-01       86.35   open
1  INV-40030  871687400000000024 2026-09-01      640.10   paid
2  INV-40035  871687400000000031 2026-09-01      452.90   open
invoice_id               str
ean                      str
period        datetime64[us]
amount_eur           float64
status                   str
dtype: object
invoice_id               str
ean                      str
period        datetime64[us]
amount_eur           float64
status                   str
dtype: object

The default read gives three rows and one column: the whole line. With sep, decimal and a text type for the EAN the table is right, and the explicit date format avoids any guessing between day-month and month-day. Once written to Parquet, the types travel with the file: reading it back needs no options at all. pandas keeps amounts as float64, which is fine for analysis; billing calculations need exact decimals.

The classic traps

Long codes and leading zeros

Codes made of digits are not numbers (see levels of measurement), but every tool that infers types will try to make them numbers. Two things go wrong:

  • Leading zeros disappear. A customer number 004417 read as a number becomes 4417, and no longer matches anything.
  • Long codes lose digits. Excel stores numbers with 15 significant digits. An 18-digit EAN such as 871687400000000017 is displayed as 8.71687E+17 and stored as 871687400000000000: the last three digits are gone, and saving the file writes the damaged value back. A double-precision floating-point number, the type most tools use for decimals, holds about 15 to 17 significant digits and fails the same way. 
ean = "871687400000000017"

as_float = float(ean)
print(as_float)
print(int(as_float))
print(int(as_float) == int(ean))
8.716874e+17
871687400000000000
False

The mechanism differs from Excel's (binary rounding instead of cutting at 15 digits), but for this code the damage is identical. Two meters whose codes differ only in the last digits now have the same code, and a lookup silently matches the wrong one. Current Excel for Microsoft 365 and Excel 2024 have "Automatic Data Conversion" options under File > Options > Data that can turn off removing leading zeros, truncating long numbers to 15 digits, converting digits around the letter "E" to scientific notation, and converting a continuous string of letters and numbers (such as MARCH1) to a date. Ordinary dates such as 01-10-2026 are still converted. The options help, but they are an Excel setting, not part of the file: the next person to open the CSV may not have switched them off. The full story of what goes wrong in spreadsheets is on When a Spreadsheet Is Not Enough.

Dates

01-10-2026 is 1 October in the Netherlands and 10/01/2026 is 1 October in the United States; each one read with the other country's rule is a valid date in January. The UK, like the Netherlands, puts the day first. A reader that infers date formats can only be sure when a day above 12 appears; for a file that only has the first of each month, it guesses. Use ISO 8601 (2026-10-01) in every file you produce (see dates and times), and state the format when you read anyone else's.

Encodings and the byte order mark

A CSV file is bytes, and the reader has to know which character encoding turns them into text. Older Windows tools often write Windows-1252 (also called ANSI); almost everything else uses UTF-8. Reading one as the other produces mojibake or an error. Excel adds a twist: according to Microsoft, it opens a UTF-8 CSV correctly by double-clicking only if the file starts with a byte order mark (BOM); otherwise you import it through Data > Get Data > From Text/CSV. Many programs, in turn, do not expect that BOM: 

import csv

note = "Correctie één dag"
print(note.encode("utf-8").decode("cp1252"))

try:
    note.encode("cp1252").decode("utf-8")
except UnicodeDecodeError as error:
    print("UnicodeDecodeError:", error.reason)

with open("bom.csv", "w", encoding="utf-8-sig", newline="") as f:
    f.write("invoice_id;amount_eur\r\nINV-40026;86,35\r\n")

with open("bom.csv", "rb") as f:
    print(f.read(3))

with open("bom.csv", encoding="utf-8", newline="") as f:
    print(list(next(csv.DictReader(f, delimiter=";"))))

with open("bom.csv", encoding="utf-8-sig", newline="") as f:
    print(list(next(csv.DictReader(f, delimiter=";"))))
Correctie één dag
UnicodeDecodeError: invalid continuation byte
b'\xef\xbb\xbf'
['\ufeffinvoice_id', 'amount_eur']
['invoice_id', 'amount_eur']

UTF-8 bytes read as Windows-1252 give "één" instead of "één"; the other way round, Python refuses. The BOM is three bytes at the start of the file. Read with plain utf-8, it sticks to the first column name, so row["invoice_id"] raises a KeyError even though the header looks right on screen. Python's utf-8-sig encoding removes it when present and is harmless when it is not.

Other traps

  • Thousands separators: 1.204,75 is one thousand two hundred and four euros in Dutch notation, and 1.20475 to a reader that expects a decimal point.
  • Line endings: RFC 4180 uses CRLF, Linux tools LF. Most readers accept both; a stray CR at the end of the last field can still end up inside a value, such as "open\r".
  • Missing values: an empty field, NULL, - and n/a all mean "nothing" to a person and something different to each tool (see missing values).
  • Round trips through a spreadsheet: opening a CSV in Excel and saving it again can rewrite dates, codes, decimals and the encoding in one go. Keep the original file and never edit it.

JSON and JSON Lines

JSON (JavaScript Object Notation) is the format of web APIs and application logs, and it is semi-structured: nested objects, lists and optional fields. Its current standard, RFC 8259 (December 2017, Internet Standard STD 90), settles several things CSV leaves open. JSON exchanged between systems that are not part of a closed ecosystem "MUST be encoded using UTF-8", and implementations "MUST NOT add a byte order mark" to it. Names within an object "SHOULD be unique", which leaves duplicates to each parser. The syntax and the common pitfalls are on JSON and Other Data Formats.

For analytics the important limit is numbers. RFC 8259 notes that integers are only exactly interoperable between −(253) + 1 and 253 − 1, because many parsers, including JavaScript's built-in JSON.parse, store every number as a double-precision float. Python's parser keeps large integers exact; a parser that uses floats does what the second line below simulates: 

import json

message = '{"ean": 871687400000000017, "kwh": 9.13}'
print(json.loads(message)["ean"])
print(json.loads(message, parse_int=float)["ean"])
print("largest safe integer:", 2**53 - 1)
871687400000000017
8.716874e+17
largest safe integer: 9007199254740991

An 18-digit EAN is about a hundred times larger than the safe limit. Send identifiers as strings ("ean": "871687400000000017"), as meter-api does, and money as a string or in whole cents.

JSON Lines for data files

A single JSON document holding a million readings has to be parsed as a whole, and one broken byte makes the whole file unreadable. JSON Lines solves this for data files: each line is one complete JSON value, usually an object, so files can be appended to, split, streamed and processed line by line. The conventions at jsonlines.org are short: UTF-8, one valid JSON value per line, lines ending in \n, no BOM, and the extension .jsonl. A nearly identical convention is known as NDJSON (newline-delimited JSON). DuckDB reads it directly: 

cat > readings.jsonl <<'JSONL'
{"meter_id": "M-30002", "reading_date": "2026-09-01", "kwh_import": 8.76, "kwh_export": 3.207}
{"meter_id": "M-30005", "reading_date": "2026-09-01", "kwh_import": 5.612, "kwh_export": null}
{"meter_id": "M-30009", "reading_date": "2026-09-01", "kwh_import": 5.33}
JSONL
wc -l readings.jsonl
duckdb -c "SELECT * FROM read_json('readings.jsonl');"
3 readings.jsonl
┌──────────┬──────────────┬────────────┬────────────┐
│ meter_id │ reading_date │ kwh_import │ kwh_export │
│ varchar  │     date     │   double   │   double   │
├──────────┼──────────────┼────────────┼────────────┤
│ M-30002  │ 2026-09-01   │       8.76 │      3.207 │
│ M-30005  │ 2026-09-01   │      5.612 │       NULL │
│ M-30009  │ 2026-09-01   │       5.33 │       NULL │
└──────────┴──────────────┴────────────┴────────────┘

DuckDB inferred a table from the objects: the ISO date string became a date, and the third record, which has no kwh_export at all, got NULL just like the second one, which says null explicitly. Whether "absent" and "null" mean the same thing is a question for the producer of the file (see null versus missing).

Parquet: a columnar file format

Apache Parquet is an open, binary file format for analytical data that stores a table column by column, with the schema, the data types and statistics inside the file. It is the standard storage format of data lakes and lakehouses, and DuckDB, pandas, Spark, the cloud warehouses and most BI tools read it directly. The idea of storing by column, and why that suits analytics, is explained on columnar storage. The current format specification is parquet-format 2.14.0 (11 September 2026).

Inside a Parquet file

A Parquet file starts and ends with the four bytes PAR1. In between, the rows are cut into horizontal slices. A row group is one such slice: a block of rows whose values are stored column by column, one column chunk per column, each chunk split into pages. The schema, the location of every column chunk and statistics such as the minimum and maximum of each chunk are in the footer, which is written after the data so that a writer can produce the file in a single pass.

Layout of a Parquet file. It starts with the magic bytes PAR1. Then come row groups; each row group holds one column chunk per column (meter_id, interval_start, kwh), and each column chunk is made of pages. After the last row group comes the footer with the schema, the location of every column chunk and min/max statistics, then the footer length and PAR1 again. PAR1 Row group 0 rows 1–122,880 meter_id interval_start kwh Row group 1, 2, … same columns, next rows Footer schema · column chunk locations · min/max per chunk footer length · PAR1
Each shaded box is a column chunk; a reader fetches only the chunks of the columns and row groups a query needs, found through the footer.

Each column is stored with a physical type (32- or 64-bit integer, floating point, byte array) and an annotation that says how to interpret it: a string is a byte array annotated as UTF-8 text, a date is a 32-bit integer annotated as a date. Writing Kestrel's invoices to both formats and reading them back shows the difference: 

COPY invoices TO 'invoices.csv';
COPY invoices TO 'invoices.parquet' (FORMAT parquet);

-- What each file gives back when it is read again
SELECT column_name, c.column_type AS from_csv, p.column_type AS from_parquet
FROM (DESCRIBE SELECT * FROM 'invoices.csv') c
JOIN (DESCRIBE SELECT * FROM 'invoices.parquet') p USING (column_name)
ORDER BY column_name;

-- How Parquet stores them: a physical type plus an annotation
SELECT name, type AS physical_type, converted_type
FROM parquet_schema('invoices.parquet')
WHERE type IS NOT NULL;
┌─────────────┬──────────┬───────────────┐
│ column_name │ from_csv │ from_parquet  │
│   varchar   │ varchar  │    varchar    │
├─────────────┼──────────┼───────────────┤
│ amount_eur  │ DOUBLE   │ DECIMAL(10,2) │
│ contract_id │ VARCHAR  │ VARCHAR       │
│ due_date    │ DATE     │ DATE          │
│ invoice_id  │ VARCHAR  │ VARCHAR       │
│ period      │ DATE     │ DATE          │
│ status      │ VARCHAR  │ VARCHAR       │
└─────────────┴──────────┴───────────────┘
┌─────────────┬───────────────┬────────────────┐
│    name     │ physical_type │ converted_type │
│   varchar   │    varchar    │    varchar     │
├─────────────┼───────────────┼────────────────┤
│ invoice_id  │ BYTE_ARRAY    │ UTF8           │
│ contract_id │ BYTE_ARRAY    │ UTF8           │
│ period      │ INT32         │ DATE           │
│ amount_eur  │ INT64         │ DECIMAL        │
│ status      │ BYTE_ARRAY    │ UTF8           │
│ due_date    │ INT32         │ DATE           │
└─────────────┴───────────────┴────────────────┘

Through CSV, the exact DECIMAL(10,2) amount came back as an approximate DOUBLE, because the sniffer only saw text like 92.40. The Parquet file returned exactly the types that went in: the amount is stored as a 64-bit integer number of cents with a decimal annotation (precision 10, scale 2). The DESCRIBE results were joined on the column name and sorted alphabetically.

Why Parquet files are small and fast

Values in one column resemble each other: the same ten meter IDs, timestamps 15 minutes apart, readings with three decimals. Parquet exploits that with encodings such as dictionary encoding (store each distinct value once, then small numbers that refer to it), run-length encoding and delta encoding, and then compresses each page with a codec such as Snappy, gzip or Zstandard. DuckDB uses Snappy by default. A year of 15-minute readings for ten meters shows the effect: 

-- A year of 15-minute readings for 10 meters: 350,400 rows
CREATE TABLE readings AS
SELECT 'M-' || (30001 + m) AS meter_id,
       TIMESTAMP '2026-01-01' + INTERVAL 15 MINUTE * CAST(q AS INTEGER) AS interval_start,
       CAST((q * 7919 + m * 104729) % 997 / 1000 AS DECIMAL(6,3)) AS kwh
FROM range(10) AS t(m), range(35040) AS s(q)
ORDER BY interval_start, meter_id;

COPY readings TO 'readings.csv';
COPY readings TO 'readings.parquet' (FORMAT parquet);

SELECT filename, round(size / 1e6, 2) AS megabytes
FROM read_blob('readings.*')
ORDER BY filename;

-- The footer knows the range of every column in every row group
SELECT row_group_id, row_group_num_rows AS row_count,
       stats_min AS first_interval, stats_max AS last_interval
FROM parquet_metadata('readings.parquet')
WHERE path_in_schema = 'interval_start'
ORDER BY row_group_id;
┌────────────────────┬───────────┐
│      filename      │ megabytes │
│      varchar       │  double   │
├────────────────────┼───────────┤
│ ./readings.csv     │     11.91 │
│ ./readings.parquet │      0.54 │
└────────────────────┴───────────┘
┌──────────────┬───────────┬─────────────────────┬─────────────────────┐
│ row_group_id │ row_count │   first_interval    │    last_interval    │
│    int64     │   int64   │       varchar       │       varchar       │
├──────────────┼───────────┼─────────────────────┼─────────────────────┤
│            0 │    122880 │ 2026-01-01 00:00:00 │ 2026-05-08 23:45:00 │
│            1 │    122880 │ 2026-05-09 00:00:00 │ 2026-09-13 23:45:00 │
│            2 │    104640 │ 2026-09-14 00:00:00 │ 2026-12-31 23:45:00 │
└──────────────┴───────────┴─────────────────────┴─────────────────────┘

The readings are generated with simple arithmetic so that the example gives the same result every time. The Parquet file is about 22 times smaller than the CSV file. That ratio is high because these columns are very regular; on messier real data a factor of several is more typical, but Parquet is practically always much smaller.

The second result shows the row groups (122,880 rows each is DuckDB's default) and the time range the footer records for each. A query for September only needs row groups 1 and 2; the reader can skip row group 0 without reading it. Combined with reading only the columns a query mentions, that is why querying a Parquet file is often faster than querying the same data in CSV, even though the data has to be decompressed.

Choosing a format

CSV JSON Lines Parquet
Readable by people Yes, in any editor or spreadsheet Yes, in any editor No, binary; needs a tool
Types None: everything is text, types are guessed Strings, numbers, booleans, null; no date or exact decimal type Full: integers, decimals, dates, timestamps, nested
Nesting No Yes Yes
Size Large Largest: names repeated on every line Small: encoded and compressed per column
Append a record Yes Yes No: write a new file
Typical use Exports for people and spreadsheets Logs, events, API dumps Analytical storage, data lakes

A common pattern is to accept whatever the source produces, convert it to Parquet once with every reading option stated, and keep the original file next to it. All analysis then starts from the Parquet copy, whose types no tool has to guess again. For Kestrel that means the 02:30 KBS export lands as CSV and leaves the first pipeline step as Parquet; see Data Pipelines.