Running Java in Production

The JVM process

Whatever framework a Java application uses, on the server it is a process running the java launcher of a JDK or runtime (see What Is Java?). Its command line tells you most of what you need: which Java, which options, which application.

ps -o pid,rss,args -C java

    PID     RSS COMMAND
   4711  912344 /opt/java/jdk-25/bin/java -Xms768m -Xmx768m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/billing -Dspring.profiles.active=prod -jar /opt/billing/billing-service.jar

RSS (resident set size, in KB) is the memory the process really uses: about 890 MB here, more than the 768 MB heap, for reasons explained below. The JDK's own tool jps -l lists the Java processes of the current user with their main class or JAR.

JVM options

A JVM option is a command-line argument placed before the main class or -jar that configures the Java Virtual Machine itself rather than the application, such as the heap size, the garbage collector or what to do on errors. There are three families:

Options end up in different places: a systemd unit file, a start script such as setenv.sh for Tomcat, a container's environment, or one of two environment variables. JDK_JAVA_OPTIONS is added to the command line of the java launcher; JAVA_TOOL_OPTIONS is read by the JVM itself, also when it is started some other way. Both announce themselves on standard error, which is worth knowing when an option "comes from nowhere":

NOTE: Picked up JDK_JAVA_OPTIONS: -Xmx200m
Picked up JAVA_TOOL_OPTIONS: -Xmx300m

A typo in an option stops the JVM before the application starts, with exit code 1. Sizes take a unit of one letter (k, m, g), and -XX names are case-sensitive:

$ java -Xmx2gb -jar billing-service.jar
Invalid maximum heap size: -Xmx2gb
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

$ java -XX:MaxRamPercentage=75 -jar billing-service.jar
Unrecognized VM option 'MaxRamPercentage=75'
Did you mean 'MaxRAMPercentage=<value>'?
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

To see the values the JVM actually uses, including the ones it chose itself, run java -XX:+PrintFlagsFinal -version on the same machine with the same options, or ask a running process with jcmd <pid> VM.flags (see jcmd below).

Starting, stopping and exit codes

A service manager stops a Java process with the signal SIGTERM. The JVM then runs its shutdown hooks, code the application registered to close connections and finish work, and exits. SIGKILL (kill -9) ends the process at once, without any of that:

void main() {
    Runtime.getRuntime().addShutdownHook(Thread.ofPlatform().name("shutdown")
            .unstarted(() -> IO.println("shutdown hook: closing database connections")));
    IO.println("billing run finished");
}

billing run finished
shutdown hook: closing database connections

The exit code tells you how a Java process ended:

The heap and memory limits

Java objects live on the heap, and a garbage collector frees the ones that are no longer reachable. The maximum heap size is the upper limit on the memory the JVM may use for Java objects, set with -Xmx (the same as -XX:MaxHeapSize) or as a percentage of the available memory with -XX:MaxRAMPercentage; when live objects need more, the application gets an OutOfMemoryError. -Xms sets the initial size; for servers the two are often set to the same value, so the heap never has to grow.

Without these options the JVM chooses for itself: at most 25% of the memory it sees (MaxRAMPercentage defaults to 25), with a higher share only for very small machines. On a server with 16 GB, that is a heap of about 4 GB, whatever the application needs.

Containers

Inside a container the JVM reads the container's memory and CPU limits instead of the host's: this container support is on by default since JDK 10 (and was backported to Java 8 update 191). So a JVM in a container with a 512 MB limit gets a default heap of only 128 MB, and the remaining 75% mostly sits unused. That is why containerized Java services usually set a percentage:

default:                    size_t MaxHeapSize = 134217728      (128 MB)
-XX:MaxRAMPercentage=75:    size_t MaxHeapSize = 402653184      (384 MB)

Do not give the heap everything. A JVM also needs memory outside the heap: metaspace for class metadata, a stack for every platform thread, the code cache for compiled code, direct buffers for network and file I/O, and the garbage collector's own bookkeeping. Common advice is a heap of about 50 to 75% of the container limit; the right value depends on the application, so watch the real memory use (RSS) under load.

Garbage collection

The garbage collector (GC) finds objects that nothing refers to any more and reuses their memory. Most objects die young (a request's temporary strings and lists), a few live long (caches, sessions), so HotSpot's collectors are generational: they collect the young generation often and cheaply, and the old generation rarely.

The G1 garbage collector (Garbage-First) is HotSpot's default collector: it divides the heap into many equal regions, collects the regions with the most garbage first, and does most of its marking work while the application keeps running, aiming at short, predictable pauses (200 ms by default) on heaps of any size. The JDK has several collectors; the JVM picks one at start-up unless an option chooses:

GC logs

The GC log is the first thing to look at for memory and pause problems, and it is cheap enough to leave on in production. -Xlog:gc prints one line per collection; this is the start of a real log of a program with a 128 MB heap:

[0.002s][info][gc] Using G1
[0.041s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 29M->3M(128M) 2.438ms
[0.075s][info][gc] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 58M->9M(128M) 14.615ms
[0.107s][info][gc] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 77M->6M(128M) 4.329ms

Read 58M->9M(128M) 14.615ms as: heap use was 58 MB before the collection and 9 MB after it, the heap is 128 MB, and the application was paused for 14.6 milliseconds. The number to watch is the value after collections: if it keeps rising over hours and never comes down, objects are being kept that should not be, a memory leak. Lines with Pause Full mean the collector had to stop everything to clean the whole heap, usually a sign that it is nearly full. In production, write the log to rotating files:

java -Xlog:gc*:file=/var/log/billing/gc.log:time,uptime,level,tags:filecount=5,filesize=10m -jar billing-service.jar

OutOfMemoryError

An OutOfMemoryError is the Java error thrown when the JVM cannot satisfy a memory request even after garbage collection, most often because the heap is full of live objects; its message names the area that ran out, such as Java heap space. It is an Error, not an Exception (see the exception hierarchy): applications are not expected to catch it, and after one the state of the application is uncertain. Here the program asks for a single array of about 17 GB, which is more than the maximum heap of the machine that checked this page, so the JVM refuses at once:

void main() {
    IO.println("loading tariff table");
    long[] readings = new long[Integer.MAX_VALUE - 8];   // about 17 GB in one piece
    IO.println("loaded " + readings.length);
}

loading tariff table
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

In real applications the heap usually fills up gradually: a cache without a size limit, a list that collects every request, a huge query result read into memory at once. The message tells you which memory ran out:

The array limit, for example, is independent of the heap size: even a JVM with a huge heap refuses an int array with Integer.MAX_VALUE elements:

void main() {
    int[] readings = new int[Integer.MAX_VALUE];
}

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit

The first and fifth messages were reproduced on JDK 25 for this page; the wording of the others comes from the HotSpot JVM and may vary slightly between versions.

Heap dumps

A heap dump is a snapshot of all objects on the Java heap at one moment, written to a file (HPROF format, usually as large as the used heap) and opened with an analysis tool such as Eclipse Memory Analyzer (MAT) to find which objects use the memory and what keeps them alive. The JVM can write one automatically at the first OutOfMemoryError, which is the most useful moment:

java -Xmx768m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/billing/ -jar billing-service.jar

java.lang.OutOfMemoryError: Java heap space
Dumping heap to dumps/java_pid1908584.hprof ...
Heap dump file created [59792152 bytes in 0.116 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at Fill.main(Fill.java:5)

The directory needs enough free disk space for a file as large as the heap. Many teams add -XX:+ExitOnOutOfMemoryError as well: the JVM then prints Terminating due to java.lang.OutOfMemoryError: Java heap space and exits with code 3, so the service manager or Kubernetes starts a fresh instance instead of leaving a damaged one running. A running JVM can also be asked for a dump at any time with jcmd <pid> GC.heap_dump /var/log/billing/now.hprof.

For a first look without a full dump, jcmd <pid> GC.class_histogram counts the objects per class. A class from the application near the top, with hundreds of thousands of instances, is a strong hint. This is a real histogram of a program that keeps every meter reading in a static map:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:        301013        9632416  java.util.HashMap$Node (java.base@25.0.4.1)
   2:        308766        7590776  [B (java.base@25.0.4.1)
   3:        308688        7408512  java.lang.String (java.base@25.0.4.1)
   4:        300000        7200000  Leak$Reading
   5:        300134        4802144  java.lang.Integer (java.base@25.0.4.1)

[B is the JVM's name for byte[], the storage inside strings. 300,000 Reading objects, each with a string and a map entry, are the leak.

Thread dumps

When an application hangs, responds slowly or uses all CPUs, the question is: what are its threads doing right now? A thread dump is a text snapshot of every platform thread in a JVM (virtual threads only appear with jcmd Thread.dump_to_file) at one moment, with its name, state and current stack trace, and the locks it holds or waits for; it is taken from outside, without stopping the application. Three ways to get one:

jcmd 4711 Thread.print > /tmp/dump-1.txt     # recommended
jstack 4711 > /tmp/dump-1.txt                # older tool, same output
kill -3 4711                                 # SIGQUIT: the JVM prints the dump to its own standard output (often its log)

Run jcmd and jstack as the same user as the Java process. This is part of a real dump of a small billing service:

Full thread dump OpenJDK 64-Bit Server VM (25.0.4.1+1-LTS mixed mode, sharing):

"meter-poller" #23 [1908112] prio=5 os_prio=0 cpu=0.41ms elapsed=3.80s tid=0x00007fff7c11d4a0 nid=1908112 waiting on condition  [0x00007fff557fe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleepNanos0(java.base@25.0.4.1/Native Method)
	at java.lang.Thread.sleepNanos(java.base@25.0.4.1/Thread.java:509)
	at java.lang.Thread.sleep(java.base@25.0.4.1/Thread.java:540)
	at BillingService.lambda$main$0(BillingService.java:10)

"tariff-updater" #24 [1908113] prio=5 os_prio=0 cpu=0.16ms elapsed=3.79s tid=0x00007fff7c11e850 nid=1908113 waiting for monitor entry  [0x00007fff555fe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at BillingService.lambda$main$1(BillingService.java:16)
	- waiting to lock <0x00000000ffd17ea8> (a java.lang.Object)
	- locked <0x00000000ffd17e98> (a java.lang.Object)

"invoice-worker" #25 [1908114] prio=5 os_prio=0 cpu=0.16ms elapsed=3.79s tid=0x00007fff7c11fc50 nid=1908114 waiting for monitor entry  [0x00007fff553fe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at BillingService.lambda$main$2(BillingService.java:22)
	- waiting to lock <0x00000000ffd17e98> (a java.lang.Object)
	- locked <0x00000000ffd17ea8> (a java.lang.Object)

How to read one thread entry:

• The name in quotes, here "meter-poller", tells you which part of the application it is. Thread pools of servers have recognizable names (for Tomcat, for example, http-nio-8080-exec-12); anonymous names such as pool-3-thread-7 are why developers should name their threads.
• cpu= is the CPU time the thread has used so far; compare two dumps to see which thread is burning CPU.
• nid= is the operating system's thread ID, in decimal in current JDKs (older JDKs printed it in hexadecimal, nid=0x…). It is the same number that top -H -p <pid> shows in its PID column, which is how you link a thread at 100% CPU to its stack trace.
• The state (see thread states): RUNNABLE threads are working (or waiting in a network read), BLOCKED ones wait for a lock, WAITING and TIMED_WAITING ones wait for something else, often idle pool threads waiting for work.
• The stack reads from top (where the thread is now) to bottom (where it started). Look for the first lines from the application's own packages, such as com.kestrel….

The two BLOCKED threads above each hold one lock and wait for the other's: a deadlock. They will wait forever. The JVM detects this kind of deadlock and reports it at the end of the dump:

Found one Java-level deadlock:
=============================
"tariff-updater":
  waiting to lock monitor 0x00007fff18000ed0 (object 0x00000000ffd17ea8, a java.lang.Object),
  which is held by "invoice-worker"

"invoice-worker":
  waiting to lock monitor 0x00007fff0c000ed0 (object 0x00000000ffd17e98, a java.lang.Object),
  which is held by "tariff-updater"

Found 1 deadlock.

You can watch a thread enter the BLOCKED state in a small program: the main thread holds a lock, the worker waits for it, and gets it as soon as main lets go:

void main() throws InterruptedException {
    Object tariffLock = new Object();
    Thread invoicer = Thread.ofPlatform().name("invoice-worker").unstarted(() -> {
        synchronized (tariffLock) {
            IO.println("invoice-worker got the lock");
        }
    });

    synchronized (tariffLock) {          // main holds the lock ...
        invoicer.start();
        while (invoicer.getState() != Thread.State.BLOCKED) {
            Thread.onSpinWait();         // ... until the worker is stuck waiting for it
        }
        IO.println("invoice-worker is " + invoicer.getState());
    }
    invoicer.join();
    IO.println("invoice-worker is " + invoicer.getState());
}

invoice-worker is BLOCKED
invoice-worker got the lock
invoice-worker is TERMINATED

jstack and Thread.print show only platform threads. An application that uses virtual threads can have thousands of them that never appear there. They are included in the newer dump format, written to a file, in plain text or JSON:

jcmd 4711 Thread.dump_to_file /tmp/threads.txt
jcmd 4711 Thread.dump_to_file -format=json /tmp/threads.json

In a test with 1,000 sleeping virtual threads, jstack listed 17 threads, while the file from Thread.dump_to_file contained all 1,000 virtual threads as well.

One more thread problem is easy to miss: an exception that is not caught in a thread other than main ends only that thread. The JVM prints a stack trace to standard error, starting with Exception in thread "meter-poller", and the rest of the application keeps running without, for example, its meter poller. Applications should install a handler that logs such deaths:

void main() throws InterruptedException {
    Thread.setDefaultUncaughtExceptionHandler((thread, error) ->
            IO.println("ERROR thread " + thread.getName() + " died: " + error));

    Thread poller = Thread.ofPlatform().name("meter-poller").start(() -> {
        throw new IllegalStateException("meter gateway timed out");
    });
    poller.join();
    IO.println("main thread still running; poller state: " + poller.getState());
}

ERROR thread meter-poller died: java.lang.IllegalStateException: meter gateway timed out
main thread still running; poller state: TERMINATED

jcmd and Java Flight Recorder

jcmd is the JDK's command-line tool for sending diagnostic commands to a running JVM, identified by its process ID: thread dumps, heap information and dumps, the active JVM options and system properties, and Java Flight Recorder recordings. It replaces several older tools (jstack, jmap, jinfo). jcmd without arguments lists the Java processes; jcmd <pid> help lists the commands that JVM supports. It comes with the JDK, so a server or container image with only a minimal runtime may not have it.

$ jcmd 1908091 VM.version
1908091:
OpenJDK 64-Bit Server VM version 25.0.4.1+1-LTS
JDK 25.0.4.1

$ jcmd 1908091 GC.heap_info
1908091:
garbage-first heap   total reserved 262144K, committed 262144K, used 4750K [0x00000000f0000000, 0x0000000100000000)
 region size 1024K, 4 young (4096K), 0 survivors (0K)

$ jcmd 1908091 VM.flags
1908091:
… -XX:InitialHeapSize=268435456 … -XX:MaxHeapSize=268435456 … -XX:+UseCompressedOops -XX:+UseG1GC

Java Flight Recorder

Java Flight Recorder (JFR) is built into the JVM and records events: garbage collections, lock contention, slow I/O, CPU samples of the busiest methods, exceptions. It is designed to be cheap enough to run in production, so you can record the problem while it happens instead of trying to reproduce it.

jcmd 4711 JFR.start duration=2m filename=/var/log/billing/slow.jfr
jfr summary /var/log/billing/slow.jfr
jfr print --events jdk.GarbageCollection /var/log/billing/slow.jfr

Started recording 1. The result will be written to:

/var/log/billing/slow.jfr

The jfr tool summarizes or prints a recording; JDK Mission Control (JMC), a separate free download, shows it graphically. On JDK 25 a recording also contains the command-line arguments, environment variables and system properties of the process, verbatim, which may include passwords and tokens. JDK 27 redacts many of these by default before they are written (JEP 536). Handle recordings with the same care as logs.

Logs

Java applications log through a logging library, not with System.out. The common combinations: the SLF4J API with Logback behind it (the default in Spring Boot), Log4j 2, or java.util.logging, which is part of the JDK. They all use levels (ERROR, WARN, INFO, DEBUG, TRACE), are configured in a file outside the code (logback.xml, log4j2.xml, application.properties), and write to standard output or to files that they rotate themselves. Under systemd, standard output ends up in the journal (see journalctl); in containers, it is what docker logs or kubectl logs show.

An error in a Java log usually comes with a stack trace. Read it as explained in Exceptions and Error Handling: the first line gives the exception and its message, and the last Caused by: section is usually the root cause, for example a refused database connection underneath a generic "could not process request". Copy the whole trace, not just the first line.

Logging libraries are dependencies like any other and need patching: Log4Shell, a critical vulnerability in Log4j 2 in December 2021, let attackers run code through text that ended up in a log message. See Keeping dependencies safe. For an operations team the practical lesson is to know which applications contain which library versions, which updating the JDK alone does not change.

Common failures and first checks

Two of these, shown for real. A class file compiled by a newer JDK than the one that runs it fails before the application starts. The class file version is the Java version plus 44: 65 is Java 21, 69 is Java 25, 70 is Java 26:

Error: LinkageError occurred while loading main class com.kestrel.billing.Main
	java.lang.UnsupportedClassVersionError: com/kestrel/billing/Main has been compiled by a more recent version of the Java Runtime (class file version 70.0), this version of the Java Runtime only recognizes class file versions up to 69.0

And a second instance of a service that tries to listen on a port the first one already uses:

void main() throws Exception {
    try (ServerSocket portal = new ServerSocket(0)) {           // port 0: any free port
        int port = portal.getLocalPort();
        IO.println("portal listening");
        try (ServerSocket second = new ServerSocket(port)) {    // a second instance, same port
            IO.println("second instance listening");
        }
    }
}

portal listening
Exception in thread "main" java.net.BindException: Address already in use

What to put in a ticket

A Java incident is solved much faster when the first ticket note already contains what the developers or the next line will ask for. See Writing Tickets and Technical Documentation for ticket notes in general; for Java applications, collect:

• Which application and instance: host or pod, PID, and the time of the problem with time zone.
• Java version: java -version of the runtime the service really uses, or jcmd <pid> VM.version.
• JVM options and limits: the command line (ps) or jcmd <pid> VM.flags, and the memory and CPU limits of the container or VM.
• The error itself: the first error line, the full stack trace with all Caused by: sections, and the log lines just before it. Copy text, not screenshots.
• Evidence for the type of problem: for a hang, three thread dumps ten seconds apart; for memory, the GC log and a class histogram, and the location of a heap dump (not the dump itself); for a vanished process, the exit code and the kernel log.
• What changed: a deployment, a configuration or JDK update, more users or data than usual.
• What you did: a restart (after taking the dumps, because a restart destroys the evidence), a rollback, a workaround.

A note from Kestrel Energy, where billing-service is the Java service behind KBS's billing-api:

7 Oct 2026 06:42 CEST — Mei Chen (service desk)

Symptom: billing-service on bill-app-02 stopped answering at 06:10; portal
shows "invoice temporarily unavailable". bill-app-01 is fine.

Findings:
- Log 06:09:57: java.lang.OutOfMemoryError: Java heap space (full trace attached
  as text, no Caused by). Heap dump written to
  /var/log/billing/java_pid4711.hprof (2.9 GB, NOT attached, contains customer
  data; access via Joost Bakker).
- Java: OpenJDK 25.0.4.1 (jcmd VM.version). Options: -Xms3g -Xmx3g
  -XX:+HeapDumpOnOutOfMemoryError. Host: 8 GB RAM, no container limit.
- GC log: heap after GC rose from 0.8 GB (Mon) to 2.9 GB (today), steady
  climb, no drops -> looks like a leak, not a peak.
- jcmd GC.class_histogram (06:35, before restart): 4.1 million
  com.kestrel.billing.TariffCacheEntry at the top. Output attached.
- Change: release 4.12.0 of billing-service deployed Mon 5 Oct (CHG-2293).

Action: restarted billing-service on bill-app-02 at 06:40 after collecting the
above; invoices work again. Expect a repeat in about two days.
Assigned to Billing (Joost Bakker) for analysis of the tariff cache in 4.12.0.

Nothing in this note requires programming. It requires knowing which tools exist, which evidence disappears on a restart, and which files must not travel by ticket or email.