Thread Pools & ExecutorService

What Are Thread Pools and Why Do They Exist?

Creating a platform thread is expensive: the JVM allocates a stack (~512KB–1MB), the OS allocates a kernel thread structure, and context-switching has measurable overhead. Creating a new thread per task at scale means thousands of threads competing for CPU time — most doing nothing but sleeping or waiting.

A thread pool solves this by maintaining a fixed number of reusable worker threads. Tasks are submitted to a queue; idle workers pick them up. Thread creation happens once at startup, not per task.

// ❌ Thread-per-task: 10,000 requests = 10,000 threads = OOM or thrashing
for (Request req : requests) {
    new Thread(() -> handle(req)).start();  // unbounded, uncontrolled
}

// ✅ Thread pool: 10,000 requests handled by N reusable workers
ExecutorService pool = Executors.newFixedThreadPool(20);
for (Request req : requests) {
    pool.submit(() -> handle(req));  // excess tasks queue, don't spawn threads
}

/*
 *  Internal flow:
 *
 *  submit(task) ──► [Task Queue] ──► [Worker Thread 1]
 *                                 ──► [Worker Thread 2]
 *                                 ──► [Worker Thread N]
 *                   (tasks wait)       (pick up next task when free)
 */

ExecutorService API

submit vs execute

ExecutorService pool = Executors.newFixedThreadPool(4);

// execute(Runnable) — fire and forget, no result, exceptions swallowed silently
pool.execute(() -> processEvent(event));

// submit(Callable) — returns Future, exceptions held until future.get()
Future<Report> future = pool.submit(() -> generateReport(params));

// Always use get() with timeout — never block indefinitely
try {
    Report report = future.get(30, TimeUnit.SECONDS);
} catch (TimeoutException e) {
    future.cancel(true);   // interrupt the task
    throw new ServiceException("Report generation timed out");
} catch (ExecutionException e) {
    throw new ServiceException("Report generation failed", e.getCause());
}

invokeAll and invokeAny — batch operations

List<Callable<Price>> tasks = providers.stream()
    .map(p -> (Callable<Price>) () -> p.fetchPrice(sku))
    .toList();

// invokeAll: wait for ALL to complete (or timeout) — returns in submission order
List<Future<Price>> results = pool.invokeAll(tasks, 5, TimeUnit.SECONDS);
for (Future<Price> f : results) {
    if (!f.isCancelled()) prices.add(f.get());  // timed-out tasks are cancelled
}

// invokeAny: return FIRST successful result, cancel the rest
// Perfect for "try multiple sources, take fastest" pattern
Price best = pool.invokeAny(tasks, 3, TimeUnit.SECONDS);

Shutdown — always, always shut down

// ✅ Graceful shutdown pattern
void shutdownGracefully(ExecutorService pool) {
    pool.shutdown();  // reject new tasks, let queued tasks finish
    try {
        if (!pool.awaitTermination(30, TimeUnit.SECONDS)) {
            pool.shutdownNow();  // interrupt running tasks
            if (!pool.awaitTermination(10, TimeUnit.SECONDS)) {
                log.error("Pool did not terminate");
            }
        }
    } catch (InterruptedException e) {
        pool.shutdownNow();
        Thread.currentThread().interrupt();  // restore interrupt status
    }
}

// ✅ Java 19+: ExecutorService implements AutoCloseable
try (ExecutorService pool = Executors.newFixedThreadPool(4)) {
    pool.submit(task1);
    pool.submit(task2);
}  // calls shutdown() + awaitTermination() automatically

ThreadPoolExecutor — Under the Hood

All Executors factory methods return a ThreadPoolExecutor (or subclass). Understanding its parameters is essential for production tuning — the defaults are rarely correct for your workload.

new ThreadPoolExecutor(
    4,                              // corePoolSize: threads kept alive even when idle
    16,                             // maximumPoolSize: max threads when queue is full
    60L, TimeUnit.SECONDS,          // keepAliveTime: idle extra threads terminated after
    new LinkedBlockingQueue<>(1000), // workQueue: bounded — critical for production
    new ThreadFactory() { ... },   // threadFactory: name threads, set daemon status
    new CallerRunsPolicy()           // rejectionHandler: what to do when queue is full
);

/*
 *  Task submission flow:
 *
 *  submit(task)
 *    │
 *    ├─ threads < corePoolSize?     → create new core thread
 *    │
 *    ├─ queue not full?             → add to queue (existing threads will pick up)
 *    │
 *    ├─ threads < maximumPoolSize?  → create non-core thread (temporary)
 *    │
 *    └─ queue full AND at max?      → RejectedExecutionHandler fires
 */

Named thread factory — non-negotiable in production

// Anonymous threads in stack traces = debugging nightmare
// "pool-3-thread-7 threw an exception" tells you nothing

ThreadFactory factory = new ThreadFactory() {
    private final AtomicInteger count = new AtomicInteger();
    @Override
    public Thread newThread(Runnable r) {
        Thread t = new Thread(r, "order-processor-" + count.incrementAndGet());
        t.setDaemon(true);  // JVM can exit even if task is running
        return t;
    }
};

// Or with Guava (cleaner):
ThreadFactory factory = ThreadFactoryBuilder.newBuilder()
    .setNameFormat("payment-worker-%d")
    .setDaemon(true)
    .build();

Rejection policies

/*
 *  AbortPolicy (default)  — throws RejectedExecutionException
 *                           Caller must catch it or the task is lost silently
 *
 *  CallerRunsPolicy       — task runs on the submitting thread
 *                           Natural backpressure: slows down producers
 *                           ✅ Recommended for most services
 *
 *  DiscardPolicy          — task silently dropped
 *                           ❌ Almost never correct — you lose work
 *
 *  DiscardOldestPolicy    — drops the oldest queued task to make room
 *                           ❌ Also rarely correct
 */

// Custom policy: log + metric + controlled fail
new RejectedExecutionHandler() {
    @Override
    public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
        metrics.increment("pool.rejections");
        log.error("Task rejected — queue={}, active={}",
                  executor.getQueue().size(), executor.getActiveCount());
        throw new RejectedExecutionException("Service overloaded");
    }
}

Pool Sizing — The Science

Wrong pool size is one of the most common performance bugs in Java services. Too small: throughput limited. Too large: context-switching overhead, memory pressure, I/O contention.

/*
 *  Little's Law applied to thread pools:
 *
 *  N = λ × W
 *
 *  N  = number of threads needed
 *  λ  = task arrival rate (tasks/sec)
 *  W  = average task duration (seconds)
 *
 *  Example: 500 req/s, each takes 200ms average
 *  N = 500 × 0.2 = 100 threads minimum for full throughput
 *
 *  ─────────────────────────────────────────────────────
 *  CPU-BOUND tasks (image processing, crypto, computation):
 *
 *  threads = N_cpu + 1
 *
 *  +1 for when a thread stalls on GC or page fault.
 *  More threads just means context-switching overhead.
 *
 *  ─────────────────────────────────────────────────────
 *  I/O-BOUND tasks (DB, HTTP, files):
 *
 *  threads = N_cpu × (1 + wait_time / compute_time)
 *
 *  Example: 8 cores, task spends 90% waiting on DB
 *  threads = 8 × (1 + 0.9 / 0.1) = 8 × 10 = 80
 *
 *  Java 21 virtual threads remove this calculation for I/O-bound work.
 */

int cores = Runtime.getRuntime().availableProcessors();

// CPU-bound pool
ExecutorService cpuPool = Executors.newFixedThreadPool(cores + 1);

// I/O-bound pool (rule of thumb: 2–10x cores, measure your wait ratio)
ExecutorService ioPool  = Executors.newFixedThreadPool(cores * 4);

// Or for I/O-bound on Java 21: virtual threads eliminate the guessing
ExecutorService vtPool  = Executors.newVirtualThreadPerTaskExecutor();

// Runtime.getRuntime().availableProcessors() returns HOST CPU count, not container limit.
// A container limited to 2 CPUs on a 64-core host returns 64. Pool of 65 threads
// on 2 CPUs = severe context-switching overhead.

// ✅ Fix 1: JVM flags (Java 8u191+ / Java 10+)
// -XX:ActiveProcessorCount=2  ← override what availableProcessors() returns

// ✅ Fix 2: Read CPU limit from cgroup (Java 11+ does this automatically with
//    -XX:+UseContainerSupport, which is ON by default since Java 10)

// ✅ Fix 3: Read the limit programmatically for logging/sizing decisions
int effectiveCpus = Runtime.getRuntime().availableProcessors();
log.info("Sizing pool for {} CPUs", effectiveCpus);

ScheduledExecutorService

The modern replacement for Timer. Timer uses a single thread for all tasks — one slow task delays all others, and an uncaught exception kills the timer permanently. ScheduledExecutorService has none of these problems.

ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2,
    r -> { Thread t = new Thread(r, "scheduler"); t.setDaemon(true); return t; });

// schedule: run once after delay
scheduler.schedule(() -> retryFailedPayments(), 30, TimeUnit.MINUTES);

// scheduleAtFixedRate: next run starts N units after PREVIOUS START
// If task takes longer than the period, runs back-to-back (no overlap)
scheduler.scheduleAtFixedRate(() -> {
    metrics.report();
}, 0, 1, TimeUnit.MINUTES);

// scheduleWithFixedDelay: next run starts N units after PREVIOUS END
// Guarantees a gap between runs — use for tasks of variable duration
scheduler.scheduleWithFixedDelay(() -> {
    processPendingOrders();  // takes unpredictable time
}, 0, 5, TimeUnit.SECONDS);  // always 5s gap after completion

// ⚠️ Uncaught exceptions in scheduled tasks SILENTLY STOP the schedule
// The future completes exceptionally — no more executions, no log output
ScheduledFuture<?> handle = scheduler.scheduleAtFixedRate(() -> {
    try {
        doWork();
    } catch (Exception e) {
        log.error("Scheduled task failed — will retry next cycle", e);
        // Don't rethrow: that would cancel the schedule
    }
}, 0, 1, TimeUnit.MINUTES);

ForkJoinPool — Divide and Conquer

ForkJoinPool is designed for recursive, CPU-bound work that can be split into subtasks. It uses work stealing: idle threads steal tasks from busy threads' queues, maximising CPU utilisation. It is the engine behind parallel streams and CompletableFuture defaults.

// Direct use: RecursiveTask returns a result
class SumTask extends RecursiveTask<Long> {
    private static final int THRESHOLD = 10_000;
    private final long[] array;
    private final int from, to;

    SumTask(long[] array, int from, int to) {
        this.array = array; this.from = from; this.to = to;
    }

    @Override
    protected Long compute() {
        int size = to - from;
        if (size <= THRESHOLD) {
            // Base case: compute directly
            long sum = 0;
            for (int i = from; i < to; i++) sum += array[i];
            return sum;
        }
        // Recursive case: split and fork
        int mid = from + size / 2;
        SumTask left  = new SumTask(array, from, mid);
        SumTask right = new SumTask(array, mid, to);
        left.fork();                     // schedule left asynchronously
        long rightResult = right.compute(); // compute right on current thread
        return left.join() + rightResult;   // wait for left, combine
    }
}

long[] data = /* large array */ new long[10_000_000];
ForkJoinPool pool = ForkJoinPool.commonPool();  // shared across JVM
long total = pool.invoke(new SumTask(data, 0, data.length));

// For custom parallelism level (avoid monopolising common pool):
ForkJoinPool custom = new ForkJoinPool(4);
long result = custom.submit(new SumTask(data, 0, data.length)).get();

Monitoring Pool Health

A thread pool that silently saturates is one of the hardest production issues to diagnose. Expose metrics from the start.

ThreadPoolExecutor tpe = (ThreadPoolExecutor) pool;

// Key metrics to expose (Micrometer, Prometheus, etc.)
tpe.getPoolSize();           // current number of threads
tpe.getActiveCount();        // threads actively executing tasks
tpe.getCorePoolSize();       // core thread count
tpe.getMaximumPoolSize();    // max thread count
tpe.getQueue().size();       // tasks waiting — KEY signal for saturation
tpe.getQueue().remainingCapacity(); // 0 = about to reject
tpe.getCompletedTaskCount(); // total tasks completed since pool creation
tpe.getTaskCount();          // total submitted

// Hook into lifecycle for custom metrics
ExecutorService monitored = new ThreadPoolExecutor(4, 16, 60L, TimeUnit.SECONDS,
        new LinkedBlockingQueue<>(1000)) {
    @Override
    protected void beforeExecute(Thread t, Runnable r) {
        metrics.gauge("pool.queue.size", getQueue().size());
    }
    @Override
    protected void afterExecute(Runnable r, Throwable ex) {
        if (ex != null) metrics.increment("pool.task.failures");
    }
};

Interview Questions