Search and Replace

How substitution works

A substitution finds the matches of a pattern in a text and builds a new text in which each match is replaced. The engine scans from left to right: after a match it continues after that match, so replaced text is never searched again. These functions never change the original string; they always return a new one.

A typical task: before a log goes to a supplier, the IP addresses in it must be hidden. In Python, re.sub() replaces every match unless you pass count; re.subn() also tells you how many replacements it made.

import re

log = ("09:14:07 login failed from 192.0.2.17\n"
       "09:14:09 login failed from 198.51.100.4\n"
       "09:15:30 login ok from 192.0.2.17")
ip = r"\b\d{1,3}(?:\.\d{1,3}){3}\b"

print(re.sub(ip, "[ip]", log))
print(re.sub(ip, "[ip]", log, count=1).splitlines()[1])
print(re.subn(ip, "[ip]", log)[1], "replacements")

09:14:07 login failed from [ip]
09:14:09 login failed from [ip]
09:15:30 login ok from [ip]
09:14:09 login failed from 198.51.100.4
3 replacements

JavaScript decides by the global flag: replace() with a regex without g replaces only the first match. replaceAll() with a regex demands the g flag, so it cannot be forgotten there. A plain string as the first argument is not a pattern at all: the dot below is a literal dot.

const line = 'failed from 192.0.2.17, then 198.51.100.4';
const ip = /\b\d{1,3}(?:\.\d{1,3}){3}\b/;

console.log(line.replace(ip, '[ip]'));
console.log(line.replace(new RegExp(ip.source, 'g'), '[ip]'));
console.log('192.0.2.17'.replace('.', '-'));
console.log('192.0.2.17'.replaceAll('.', '-'));

failed from [ip], then 198.51.100.4
failed from [ip], then [ip]
192-0.2.17
192-0-2-17

PHP's preg_replace() replaces all matches by default. The fourth argument limits the number of replacements, the fifth receives the count:

<?php
$line = 'failed from 192.0.2.17, then 198.51.100.4';
$ip = '/\b\d{1,3}(?:\.\d{1,3}){3}\b/';

echo preg_replace($ip, '[ip]', $line), "\n";
echo preg_replace($ip, '[ip]', $line, 1), "\n";
preg_replace($ip, '[ip]', $line, -1, $count);
echo $count, " replacements\n";

failed from [ip], then [ip]
failed from [ip], then 198.51.100.4
2 replacements

Replacement strings and group references

The second argument is a replacement string: a template for the new text. Most of it is copied literally, but it can refer to what the pattern captured in its capture groups. That is what makes regex replacement more than "find and replace": you can reorder, wrap or reuse parts of each match.

Turning Dutch-style dates (day-month-year) from a meter export into ISO 8601 dates looks like this in the three languages:

import re

text = "MTR-0042 read on 02-03-2026, MTR-0107 on 28-02-2026"
date = r"(\d\d)-(\d\d)-(\d{4})"

print(re.sub(date, r"\3-\2-\1", text))
print(re.sub(r"(?P<d>\d\d)-(?P<m>\d\d)-(?P<y>\d{4})", r"\g<y>-\g<m>-\g<d>", text))
print(re.sub(r"MTR-(\d+)", r"[\g<0>]", text))

MTR-0042 read on 2026-03-02, MTR-0107 on 2026-02-28
MTR-0042 read on 2026-03-02, MTR-0107 on 2026-02-28
[MTR-0042] read on 02-03-2026, [MTR-0107] on 28-02-2026

const text = 'MTR-0042 read on 02-03-2026, MTR-0107 on 28-02-2026';

console.log(text.replace(/(\d\d)-(\d\d)-(\d{4})/g, '$3-$2-$1'));
console.log(text.replace(/(?<d>\d\d)-(?<m>\d\d)-(?<y>\d{4})/g, '$<y>-$<m>-$<d>'));
console.log(text.replace(/MTR-\d+/g, '[$&]'));

MTR-0042 read on 2026-03-02, MTR-0107 on 2026-02-28
MTR-0042 read on 2026-03-02, MTR-0107 on 2026-02-28
[MTR-0042] read on 02-03-2026, [MTR-0107] on 28-02-2026

<?php
$text = 'MTR-0042 read on 02-03-2026, MTR-0107 on 28-02-2026';

echo preg_replace('/(\d\d)-(\d\d)-(\d{4})/', '$3-$2-$1', $text), "\n";
echo preg_replace('/(\d\d)-(\d\d)-(\d{4})/', '\3-\2-\1', $text), "\n";
echo preg_replace('/MTR-\d+/', '[$0]', $text), "\n";

MTR-0042 read on 2026-03-02, MTR-0107 on 2026-02-28
MTR-0042 read on 2026-03-02, MTR-0107 on 2026-02-28
[MTR-0042] read on 02-03-2026, [MTR-0107] on 28-02-2026

Two traps hide in this table. First, each language ignores the other languages' syntax: in Python $1 is just the text $1, and in JavaScript \1 is not a group reference. A pattern copied from a JavaScript answer into Python code fails quietly. Second, a group number followed by a digit is ambiguous: does \10 mean group 10 or group 1 followed by 0? Python's \g<1>0 and PHP's ${1}0 make it unambiguous.

import re

print(re.sub(r"(\d+) kWh", r"$1 kilowatt-hours", "12 kWh"))
print(re.sub(r"(\d+)", r"\g<1>0", "tariff step 5"))

$1 kilowatt-hours
tariff step 50

When the replacement comes from data

Because $ and \ are special in replacement strings, inserting text that comes from somewhere else (a customer name, a file path, a price) can go wrong. The text is interpreted instead of copied:

import re

path = r"C:\new\quarterly"
print(re.sub(r"FOLDER", path, "Export to FOLDER"))

re.error: bad escape \q at position 6

<?php
$price = 'costs $5';
echo preg_replace('/PRICE/', $price, 'Upgrade PRICE per month'), "\n";

Upgrade costs  per month

Python stops with an error because \q is not a valid escape in a replacement (and \n would silently have become a newline). PHP is worse: $5 refers to a group that does not exist, so it is replaced by nothing, without any warning. JavaScript would turn $& in the data into the matched text.

The robust fix in every language is to pass a function that returns the text: what a function returns is used as is, with no special characters. The next section shows how. If you must use a replacement string, escape it first: s.replace("\\", "\\\\") in Python, s.replaceAll('$', '$$$$') in JavaScript (four dollar signs, because the replacement string of replaceAll itself gives $$ a meaning).

Replacement callbacks

A replacement callback is a function that the engine calls for every match; whatever it returns becomes the replacement. That lets you compute the new text: convert units, look values up in a table, change case, or decide per match whether to change anything at all. It is the same idea as a callback function anywhere else.

The example converts meter readings from watt-hours to kilowatt-hours. In Python the function receives a Match object:

import re

readings = "MTR-0042 15320 Wh, MTR-0107 980 Wh, MTR-0311 2045500 Wh"

def to_kwh(m):
    kwh = int(m.group(1)) / 1000
    return f"{kwh:g} kWh"

print(re.sub(r"(\d+) Wh\b", to_kwh, readings))
print(re.sub(r"MTR-\d+", lambda m: m.group().lower(), readings))

MTR-0042 15.32 kWh, MTR-0107 0.98 kWh, MTR-0311 2045.5 kWh
mtr-0042 15320 Wh, mtr-0107 980 Wh, mtr-0311 2045500 Wh

In JavaScript the function receives the whole match, then each group as a separate argument, then the position and the full string, and finally an object with the named groups (when the pattern has any):

const readings = 'MTR-0042 15320 Wh, MTR-0107 980 Wh, MTR-0311 2045500 Wh';

const kwh = readings.replace(/(\d+) Wh\b/g, (match, wh) => `${wh / 1000} kWh`);
console.log(kwh);

const tagged = readings.replace(/MTR-(?<id>\d+)/g, (...args) => {
  const groups = args.at(-1);
  return `meter#${Number(groups.id)}`;
});
console.log(tagged);

MTR-0042 15.32 kWh, MTR-0107 0.98 kWh, MTR-0311 2045.5 kWh
meter#42 15320 Wh, meter#107 980 Wh, meter#311 2045500 Wh

PHP has a separate function, preg_replace_callback(). The callback gets the same array that preg_match() fills, including named groups:

<?php
$readings = 'MTR-0042 15320 Wh, MTR-0107 980 Wh, MTR-0311 2045500 Wh';

echo preg_replace_callback(
    '/(?<wh>\d+) Wh\b/',
    fn($m) => ($m['wh'] / 1000) . ' kWh',
    $readings
), "\n";

$price = 'costs $5';
echo preg_replace_callback('/PRICE/', fn($m) => $price, 'Upgrade PRICE per month'), "\n";

MTR-0042 15.32 kWh, MTR-0107 0.98 kWh, MTR-0311 2045.5 kWh
Upgrade costs $5 per month

The last line shows the fix from the previous section: returned text is never interpreted, so $5 survives.

Splitting text with a regex

A regex split cuts a string into pieces wherever the pattern matches, and returns the pieces in between. Plain split(",") needs one exact separator; a regex copes with real-world input where people type commas, semicolons and random spaces in the same list.

import re

tickets = "INC-31877, INC-31902;INC-31910 ;  CHG-0042"

print(re.split(r"\s*[,;]\s*", tickets))
print(re.split(r"\s*([,;])\s*", tickets))
print(re.split(r",", ",a,,b,"))
print(re.split(r"\s*[,;]\s*", tickets, maxsplit=1))

['INC-31877', 'INC-31902', 'INC-31910', 'CHG-0042']
['INC-31877', ',', 'INC-31902', ';', 'INC-31910', ';', 'CHG-0042']
['', 'a', '', 'b', '']
['INC-31877', 'INC-31902;INC-31910 ;  CHG-0042']

Three details matter in every language. A capture group in the separator pattern puts the separators into the result, which is useful when you need to know which one was used. Separators at the start, at the end or next to each other produce empty strings. And the limit argument behaves differently per language:

const tickets = 'INC-31877, INC-31902;INC-31910 ;  CHG-0042';

console.log(tickets.split(/\s*[,;]\s*/));
console.log(tickets.split(/\s*([,;])\s*/));
console.log(',a,,b,'.split(/,/));
console.log(tickets.split(/\s*[,;]\s*/, 1));

[ 'INC-31877', 'INC-31902', 'INC-31910', 'CHG-0042' ]
[ 'INC-31877', ',', 'INC-31902', ';', 'INC-31910', ';', 'CHG-0042' ]
[ '', 'a', '', 'b', '' ]
[ 'INC-31877' ]

<?php
$tickets = 'INC-31877, INC-31902;INC-31910 ;  CHG-0042';

echo json_encode(preg_split('/\s*[,;]\s*/', $tickets)), "\n";
echo json_encode(preg_split('/,/', ',a,,b,', -1, PREG_SPLIT_NO_EMPTY)), "\n";
echo json_encode(preg_split('/\s*[,;]\s*/', $tickets, 2)), "\n";

["INC-31877","INC-31902","INC-31910","CHG-0042"]
["a","b"]
["INC-31877","INC-31902;INC-31910 ;  CHG-0042"]

Empty matches and inserting text

A pattern that can match zero characters matches between characters too. That surprises people when they replace with x* or \s*: every position gets a replacement. All three languages give the same result here:

import re

print(re.sub(r"x*", "-", "abc"))
print(re.sub(r"\s*", "_", "a b"))

-a-b-c-
_a__b_

The fix is usually to require at least one character: x+ or \s+. Used on purpose, empty matches are a way to insert text at a position. Lookarounds match positions only, so this inserts thousands separators:

const kwh = '2045500';
console.log(kwh.replace(/(?<=\d)(?=(?:\d{3})+$)/g, ','));

2,045,500

For numbers shown to people, a formatting function (f"{n:,}" in Python, toLocaleString() in JavaScript) is clearer; the regex version is useful when the number is already part of a larger text.

Replacing on the command line with sed

On a server you often edit configuration files and logs without writing a program. sed's s command is the classic tool: s/pattern/replacement/flags. The replacement uses \1 for groups and & for the whole match; the g flag replaces every match on a line instead of the first. With -E, sed uses extended syntax, so groups are written ( ) instead of \( \). The full story, including basic and extended syntax, is on Regex on the Command Line.

printf '%s\n' 'MTR-0042;02-03-2026;15320' 'MTR-0107;28-02-2026;980' > readings.csv

sed -E 's/([0-9]{2})-([0-9]{2})-([0-9]{4})/\3-\2-\1/' readings.csv
sed -E 's/;([0-9]+)$/;\1 Wh/' readings.csv
sed 's/[0-9]/#/' readings.csv
sed 's/[0-9]/#/g' readings.csv

MTR-0042;2026-03-02;15320
MTR-0107;2026-02-28;980
MTR-0042;02-03-2026;15320 Wh
MTR-0107;28-02-2026;980 Wh
MTR-#042;02-03-2026;15320
MTR-#107;28-02-2026;980
MTR-####;##-##-####;#####
MTR-####;##-##-####;###

Without -i, sed only prints the result and leaves the file alone. That is the safe default: look first, then write.

Doing bulk edits safely

A regex replacement across many lines or many files is powerful and fast, and so are its mistakes. Kestrel Energy's NOC renames switch noc-sw3 to noc-sw03 in its monitoring configuration. The obvious command also hits noc-sw31, an overmatch that a preview catches before it does damage:

printf '%s\n' 'host noc-sw3 192.0.2.3' 'host noc-sw31 192.0.2.31' \
  'parent noc-sw3' > hosts.cfg

# 1. How many lines will change?
grep -c 'noc-sw3' hosts.cfg

# 2. Preview the change as a diff (the file is not touched)
sed 's/noc-sw3/noc-sw03/' hosts.cfg | diff -u --label before --label after hosts.cfg -
echo "---"

# 3. Fix the pattern: the name must end there
sed -E 's/noc-sw3\b/noc-sw03/' hosts.cfg | diff -u --label before --label after hosts.cfg -
echo "---"

# 4. Apply it, keeping a backup, and check
sed -E -i.bak 's/noc-sw3\b/noc-sw03/' hosts.cfg
grep -c 'noc-sw03' hosts.cfg
ls

3
--- before
+++ after
@@ -1,3 +1,3 @@
-host noc-sw3 192.0.2.3
-host noc-sw31 192.0.2.31
-parent noc-sw3
+host noc-sw03 192.0.2.3
+host noc-sw031 192.0.2.31
+parent noc-sw03
---
--- before
+++ after
@@ -1,3 +1,3 @@
-host noc-sw3 192.0.2.3
+host noc-sw03 192.0.2.3
 host noc-sw31 192.0.2.31
-parent noc-sw3
+parent noc-sw03
---
2
hosts.cfg
hosts.cfg.bak

The first count is already a warning: there are only two references to noc-sw3, but three lines match. The diff shows the third, and \b (a word boundary, a GNU sed extension here) fixes it.

A checklist for every bulk edit:

1. Search before you replace. Run the pattern with grep -n (or grep -rn for a folder) and read every hit. The count tells you whether the pattern is too broad or too narrow.
2. Make the pattern exact. Use anchors and word boundaries, escape the dots in host names and IP addresses (192\.0\.2\.3), and avoid .* where you can name what should be there.
3. Preview. Print the result without -i and compare with diff, or use your editor's replace preview.
4. Keep a way back. sed -i.bak keeps a copy; in a Git repository, git diff shows exactly what changed and git restore undoes it (see Staging and Committing). For production configuration, treat it as a change with a back-out plan.
5. Check it is idempotent. Running the same edit twice should change nothing the second time. Adding a domain with s/noc-sw03/noc-sw03.example.com/ is not: a second run makes noc-sw03.example.com.example.com. Anchor the pattern (noc-sw03\b(?!\.) in a flavor with lookahead) or make sure the edit runs once.