JSON, CSV and APIs

Why parse instead of grep

JSON (JavaScript Object Notation) and CSV are the formats in which systems hand data to scripts: an export from the billing system, the response of a ticketing API, the inventory of a cloud account. Both have rules that line-based tools do not know. JSON may arrive on one line or pretty-printed over fifty; a CSV field may contain a comma inside quotes. A grep that works today breaks the day the producer formats its output differently. A parser reads the structure and does not care about layout. The formats themselves are explained in JSON and Other Data Formats.

Most examples on this page use the same outage ticket, as an API might return it:

{"ticket":"INC-20419","priority":2,"tags":["outage","storm"],"site":{"city":"Utrecht"},"meters":[{"id":"M-1042","kwh":4521},{"id":"M-1043","kwh":312}]}

jq: JSON in Bash

jq is a command-line JSON processor: it reads JSON, applies a filter written in its own small language, and writes JSON (or plain text with -r). A filter is a path such as .site.city; [] walks through an array; | inside the filter passes results on, just like a shell pipe. Without -r, strings keep their JSON quotes:

cat > ticket.json <<'EOF'
{"ticket":"INC-20419","priority":2,"tags":["outage","storm"],"site":{"city":"Utrecht"},"meters":[{"id":"M-1042","kwh":4521},{"id":"M-1043","kwh":312}]}
EOF
jq '.ticket' ticket.json
jq -r '.ticket' ticket.json
jq -r '.site.city' ticket.json
jq -r '.tags[]' ticket.json
jq -r '.tags | join(",")' ticket.json
jq '.meters | map(.kwh) | add' ticket.json

"INC-20419"
INC-20419
Utrecht
outage
storm
outage,storm
4833

select() filters, and string interpolation "\(…)" or the formatters @csv and @tsv turn objects into lines that a shell loop can read. Values from the shell go in with --arg name value (always a string) or --argjson name value (parsed as JSON); never paste a shell variable into the filter text itself:

cat > ticket.json <<'EOF'
{"ticket":"INC-20419","priority":2,"tags":["outage","storm"],"site":{"city":"Utrecht"},"meters":[{"id":"M-1042","kwh":4521},{"id":"M-1043","kwh":312}]}
EOF
echo "Meters above 1000 kWh:"
jq -r '.meters[] | select(.kwh > 1000) | .id' ticket.json
echo "As CSV:"
jq -r '.meters[] | [.id, .kwh] | @csv' ticket.json
echo "One meter, chosen in the shell:"
meter=M-1043
jq -r --arg id "$meter" '.meters[] | select(.id == $id) | "\(.id): \(.kwh) kWh"' ticket.json
echo "Loop in Bash:"
jq -r '.meters[] | [.id, .kwh] | @tsv' ticket.json |
  while IFS=$'\t' read -r id kwh; do
    printf '  %s uses %5d kWh\n' "$id" "$kwh"
  done

Meters above 1000 kWh:
M-1042
As CSV:
"M-1042",4521
"M-1043",312
One meter, chosen in the shell:
M-1043: 312 kWh
Loop in Bash:
  M-1042 uses  4521 kWh
  M-1043 uses   312 kWh

Missing values and errors

A path that does not exist gives null, not an error. // supplies a default, and -e makes jq exit with status 1 when the last result is null or false, which a script can test. Invalid JSON is a real error: jq prints a parse error and exits with a status above 1 (4 in jq 1.6, as shown below):

cat > ticket.json <<'EOF'
{"ticket":"INC-20419","priority":2,"tags":["outage","storm"],"site":{"city":"Utrecht"},"meters":[{"id":"M-1042","kwh":4521},{"id":"M-1043","kwh":312}]}
EOF
jq -r '.assignee' ticket.json
jq -r '.assignee // "unassigned"' ticket.json
if ! jq -e '.assignee' ticket.json > /dev/null; then
  echo "Ticket has no assignee"
fi
echo '{"ticket": INC-20419}' > broken.json
jq '.ticket' broken.json
echo "jq exit status: $?"

null
unassigned
Ticket has no assignee
parse error: Invalid numeric literal at line 1, column 21
jq exit status: 4

Building JSON safely

JSON built by gluing strings breaks, or becomes an injection risk, as soon as a value contains a quote or a backslash. Let jq -n (no input) build it: every --arg value is escaped correctly:

summary='Storm damage at "Utrecht Noord" substation'
ticket=INC-20419
echo "Glued by hand (invalid JSON):"
echo "{\"ticket\": \"$ticket\", \"summary\": \"$summary\"}"
echo "Built with jq:"
jq -n --arg ticket "$ticket" --arg summary "$summary" --argjson priority 2 \
  '{ticket: $ticket, summary: $summary, priority: $priority}'

Glued by hand (invalid JSON):
{"ticket": "INC-20419", "summary": "Storm damage at "Utrecht Noord" substation"}
Built with jq:
{
  "ticket": "INC-20419",
  "summary": "Storm damage at \"Utrecht Noord\" substation",
  "priority": 2
}

ConvertFrom-Json: JSON as objects

ConvertFrom-Json is the PowerShell cmdlet that parses a JSON string into objects: JSON objects become PSCustomObjects, arrays become arrays, and numbers, booleans and null get matching .NET types. From then on everything from the object pipeline applies: dot notation, Where-Object, Measure-Object. Read a whole file with Get-Content -Raw, which returns one string instead of an array of lines:

'{"ticket":"INC-20419","priority":2,"tags":["outage","storm"],"site":{"city":"Utrecht"},"meters":[{"id":"M-1042","kwh":4521},{"id":"M-1043","kwh":312}]}' |
    Set-Content ticket.json

$t = Get-Content -Path ticket.json -Raw | ConvertFrom-Json
"Ticket:   $($t.ticket), priority $($t.priority) ($($t.priority.GetType().Name))"
"City:     $($t.site.city)"
"Tags:     $($t.tags -join ', ')"
"Meters:   $($t.meters.id -join ', ')"
"Total:    $(($t.meters | Measure-Object -Property kwh -Sum).Sum) kWh"
"Above 1000 kWh: $(($t.meters | Where-Object kwh -gt 1000).id)"
"Assignee: $($t.assignee ?? 'unassigned')"

Ticket:   INC-20419, priority 2 (Int64)
City:     Utrecht
Tags:     outage, storm
Meters:   M-1042, M-1043
Total:    4833 kWh
Above 1000 kWh: M-1042
Assignee: unassigned

Here the numbers really are numbers (Int64), so the comparison kwh -gt 1000 is numeric, unlike with CSV data. A missing property is $null, and the null-coalescing operator ?? (PowerShell 7 and later) supplies a default, like // in jq. -AsHashtable returns hash tables instead of objects, which is useful for keys that are not valid property names or that differ only in case.

One conversion surprises people: ConvertFrom-Json turns strings that look like ISO 8601 dates into DateTime values. That is convenient for date arithmetic, but a script that only passes the value on may change its format. Since PowerShell 7.5, -DateKind String keeps such values as text:

$json = '{"ticket":"INC-20419","opened":"2026-10-06T09:00:00Z"}'
$default = $json | ConvertFrom-Json
$asText  = $json | ConvertFrom-Json -DateKind String
"Default:          $($default.opened.GetType().Name)"
"-DateKind String: $($asText.opened.GetType().Name) '$($asText.opened)'"
$closed = [datetime]::Parse('2026-10-06T17:00:00Z').ToUniversalTime()
"Hours open until 17:00 UTC: $(($closed - $default.opened.ToUniversalTime()).TotalHours)"

Default:          DateTime
-DateKind String: String '2026-10-06T09:00:00Z'
Hours open until 17:00 UTC: 8

ConvertFrom-Json is also more lenient than the JSON standard: it accepts, for example, a trailing comma. That is fine for reading, but do not take "PowerShell could read it" as proof that another system will. Test-Json checks a string, optionally against a JSON Schema.

ConvertTo-Json and serialization depth

ConvertTo-Json goes the other way and turns objects into JSON text. Its serialization depth, the number of nested levels it writes out in full, is only 2 by default. Anything deeper is replaced by the name of its type, and since PowerShell 7.1 a warning says so. Pass -Depth whenever data is nested (the maximum is 100):

$report = [ordered]@{
    ticket = 'INC-20419'
    site   = [ordered]@{
        city   = 'Utrecht'
        feeder = [ordered]@{
            id    = 'F-12'
            cable = [ordered]@{ type = 'XLPE'; lengthKm = 3.2 }
        }
    }
}
$report | ConvertTo-Json -Compress
$report | ConvertTo-Json -Compress -Depth 5

WARNING: Resulting JSON is truncated as serialization has exceeded the set depth of 2.
{"ticket":"INC-20419","site":{"city":"Utrecht","feeder":{"id":"F-12","cable":"System.Collections.Specialized.OrderedDictionary"}}}
{"ticket":"INC-20419","site":{"city":"Utrecht","feeder":{"id":"F-12","cable":{"type":"XLPE","lengthKm":3.2}}}}

[ordered] keeps the keys in the order you wrote them; a plain hash table would write them in an unpredictable order. -Compress leaves out the indentation, which suits APIs and log lines. jq has no such limit: it always writes the full structure.

JSON Lines

JSON Lines (also called NDJSON, newline-delimited JSON) is a format with one complete JSON value per line. It combines the structure of JSON with the line orientation of logs: a file can be appended to line by line, and read, filtered or split with normal line tools. Many logging systems and bulk APIs use it. jq -c writes it, and jq reads it naturally because it processes one value after another; -s (slurp) collects all values into one array when you need totals:

cat > readings.jsonl <<'EOF'
{"meter":"M-1042","kwh":4521,"status":"ok"}
{"meter":"M-1043","kwh":312,"status":"estimated"}
{"meter":"M-1044","kwh":987,"status":"ok"}
EOF
jq -r 'select(.status != "ok") | .meter' readings.jsonl
jq -s 'map(.kwh) | add' readings.jsonl
echo "Back to one line per value:"
jq -c '{meter, kwh}' readings.jsonl

M-1043
5820
Back to one line per value:
{"meter":"M-1042","kwh":4521}
{"meter":"M-1043","kwh":312}
{"meter":"M-1044","kwh":987}

PowerShell reads the file line by line and converts each line separately:

@'
{"meter":"M-1042","kwh":4521,"status":"ok"}
{"meter":"M-1043","kwh":312,"status":"estimated"}
{"meter":"M-1044","kwh":987,"status":"ok"}
'@ | Set-Content readings.jsonl

$readings = Get-Content readings.jsonl | ForEach-Object { $_ | ConvertFrom-Json }
($readings | Where-Object status -ne 'ok').meter
($readings | Measure-Object -Property kwh -Sum).Sum
'Back to one line per value:'
$readings | ForEach-Object {
    [ordered]@{ meter = $_.meter; kwh = $_.kwh } | ConvertTo-Json -Compress
}

M-1043
5820
Back to one line per value:
{"meter":"M-1042","kwh":4521}
{"meter":"M-1043","kwh":312}
{"meter":"M-1044","kwh":987}

CSV in both shells

Import-Csv is the PowerShell cmdlet that reads a CSV file and returns one object per row, with the header line as property names. It understands quoting, so a field such as "Utrecht, Noord" stays one value. Every property is a string, as the pipeline page shows; convert before you compare. Export-Csv writes objects back; in PowerShell 7 it quotes every field unless you ask for -UseQuotes AsNeeded, and it no longer writes the #TYPE line that made -NoTypeInformation necessary in Windows PowerShell 5.1:

@'
MeterId,Site,Kwh
M-1042,"Utrecht, Noord",4521
M-1044,Delft,987
'@ | Set-Content readings.csv

$rows = Import-Csv -Path readings.csv
"Rows: $($rows.Count), site of first row: '$($rows[0].Site)'"
"Kwh is a $($rows[0].Kwh.GetType().Name)"

$rows | Select-Object MeterId, @{ Name = 'MWh'; Expression = { [int]$_.Kwh / 1000 } } |
    Export-Csv -Path mwh.csv
Get-Content mwh.csv
$rows | Export-Csv -Path asneeded.csv -UseQuotes AsNeeded
Get-Content asneeded.csv

Rows: 2, site of first row: 'Utrecht, Noord'
Kwh is a String
"MeterId","MWh"
"M-1042","4.521"
"M-1044","0.987"
MeterId,Site,Kwh
M-1042,"Utrecht, Noord",4521
M-1044,Delft,987

Bash has no CSV parser built in. awk -F, and cut -d, work only for simple CSV without quoted fields; on the file above they split "Utrecht, Noord" in two and every later column shifts. When the data may contain quotes, hand the parsing to a tool that knows CSV. Python's csv module is installed on almost every Linux server:

cat > readings.csv <<'EOF'
MeterId,Site,Kwh
M-1042,"Utrecht, Noord",4521
M-1044,Delft,987
EOF
echo "awk -F, (wrong):"
awk -F, 'NR > 1 { print "  " $1 " | " $2 " | " $3 }' readings.csv
echo "Python csv module (right):"
python3 -c '
import csv, sys
for row in csv.DictReader(open(sys.argv[1], newline="")):
    print("  " + " | ".join([row["MeterId"], row["Site"], row["Kwh"]]))
' readings.csv

awk -F, (wrong):
  M-1042 | "Utrecht |  Noord"
  M-1044 | Delft | 987
Python csv module (right):
  M-1042 | Utrecht, Noord | 4521
  M-1044 | Delft | 987

Dedicated tools such as Miller (mlr) and csvkit do the same from the command line, if you are allowed to install them. And if you control the export, ask for JSON or tab-separated values instead.

Calling web APIs

An API (application programming interface) over HTTP is how scripts talk to ticketing systems, monitoring, cloud platforms and network controllers. Bash scripts use curl for the request and jq for the response. Invoke-RestMethod is the PowerShell cmdlet that sends an HTTP request and converts a JSON (or XML) response into objects in one step; its sibling Invoke-WebRequest returns the raw response with status code and headers.

The requests below need a network and a real token, so they are not run here; the outputs are illustrative. Tomasz Nowak from the NOC (network operations center) fetches the open outage tickets from the ticketing API, with the token in an environment variable (on a shared server, keep it off curl's command line too; see Keeping secrets out of scripts):

curl -fsS --max-time 30 --retry 3 \
  -H "Authorization: Bearer $KESTREL_API_TOKEN" \
  -H 'Accept: application/json' \
  'https://api.kestrel.example/v1/tickets?status=open' |
  jq -r '.items[] | "\(.ticket)\t\(.priority)\t\(.summary)"'

INC-20419	2	Storm damage Utrecht Noord
INC-20437	3	Meter feed delay Delft

$headers = @{ Authorization = "Bearer $env:KESTREL_API_TOKEN" }
$response = Invoke-RestMethod -Uri 'https://api.kestrel.example/v1/tickets?status=open' `
    -Headers $headers -TimeoutSec 30 -MaximumRetryCount 3 -RetryIntervalSec 5
$response.items | Select-Object ticket, priority, summary

ticket    priority summary
------    -------- -------
INC-20419        2 Storm damage Utrecht Noord
INC-20437        3 Meter feed delay Delft

What the curl options do, and their PowerShell counterparts:

Sending data works the same way in reverse: build the body with a JSON tool and send it with the right content type. With -d @-, curl reads the body from standard input, so the JSON never appears on the command line:

jq -n --arg ticket INC-20419 --arg note 'Crew on site 14:20' '{ticket: $ticket, note: $note}' |
  curl -fsS --max-time 30 -X POST \
    -H "Authorization: Bearer $KESTREL_API_TOKEN" \
    -H 'Content-Type: application/json' \
    -d @- 'https://api.kestrel.example/v1/notes'

$body = [ordered]@{ ticket = 'INC-20419'; note = 'Crew on site 14:20' } | ConvertTo-Json -Depth 5
try {
    Invoke-RestMethod -Method Post -Uri 'https://api.kestrel.example/v1/notes' `
        -Headers $headers -ContentType 'application/json' -Body $body -ErrorAction Stop
} catch {
    Write-Error "Adding the note failed: $($_.Exception.Message)"
    exit 1
}

Working from a recorded response

Develop and test the processing part against a saved response instead of the live API: you avoid rate limits, you can test edge cases (an empty list, a missing field) by editing the file, and the script's logic can be checked without a network. Here is the processing half of Tomasz's script in both shells, running on a recorded response that also contains a ticket without a priority:

cat > open-tickets.json <<'EOF'
{"items":[
  {"ticket":"INC-20419","priority":2,"summary":"Storm damage Utrecht Noord"},
  {"ticket":"INC-20437","priority":3,"summary":"Meter feed delay Delft"},
  {"ticket":"INC-20440","summary":"Portal login errors"}
],"next":null}
EOF
jq -r '.items[] | [.ticket, (.priority // "none"), .summary] | @tsv' open-tickets.json |
  while IFS=$'\t' read -r ticket priority summary; do
    printf '%-10s P%-5s %s\n' "$ticket" "$priority" "$summary"
  done
echo "Urgent (priority 1 or 2): $(jq '[.items[] | select((.priority // 9) <= 2)] | length' open-tickets.json)"

INC-20419  P2     Storm damage Utrecht Noord
INC-20437  P3     Meter feed delay Delft
INC-20440  Pnone  Portal login errors
Urgent (priority 1 or 2): 1

@'
{"items":[
  {"ticket":"INC-20419","priority":2,"summary":"Storm damage Utrecht Noord"},
  {"ticket":"INC-20437","priority":3,"summary":"Meter feed delay Delft"},
  {"ticket":"INC-20440","summary":"Portal login errors"}
],"next":null}
'@ | Set-Content open-tickets.json

$response = Get-Content -Path open-tickets.json -Raw | ConvertFrom-Json
foreach ($item in $response.items) {
    '{0,-10} P{1,-5} {2}' -f $item.ticket, ($item.priority ?? 'none'), $item.summary
}
"Urgent (priority 1 or 2): $(@($response.items | Where-Object { ($_.priority ?? 9) -le 2 }).Count)"

INC-20419  P2     Storm damage Utrecht Noord
INC-20437  P3     Meter feed delay Delft
INC-20440  Pnone  Portal login errors
Urgent (priority 1 or 2): 1

The "next":null field hints at pagination: many APIs return results in pages and give a link or cursor to the next one. A script that reads only the first page silently misses data, so loop until next is empty.

Calling APIs safely

• Keep secrets out of the script. Read the token from an environment variable, a file only your account can read, or a secrets manager (see keeping credentials safe and Writing Robust and Safe Scripts). Never commit it to Git, and never put it in a URL, where it ends up in proxy and server logs.
• Mind the process list. Command-line arguments, such as a token after curl -H, can be visible to other users on the same machine while the command runs. On shared servers, let curl read headers from a protected file (-H @headers.txt) or a config file (-K).
• Never disable certificate checks with curl -k or -SkipCertificateCheck outside a lab. Fix the trust store instead; a script that ignores certificates will also talk to an attacker in the middle.
• Fail loudly. Use curl -f or check the status code, and wrap Invoke-RestMethod in try/catch with -ErrorAction Stop (see Error Handling). A script that writes an HTML error page into a JSON file causes confusing failures later.
• Be gentle. Set timeouts, retry only transient errors with a pause, respect rate limits (HTTP 429), and only retry requests that are safe to repeat (idempotence and retries).
• Do not log secrets or personal data. When you log a request for troubleshooting, log the URL path and status, not the headers or full body.

Side by side