threadmill-core
The framework-agnostic core of Threadmill: the job model, the state machine, the
JobStore SPI, the JSON serializer, the processing engine, the scheduling API,
and the interceptor surface. Every other module depends on this one. No storage
implementation, no UI, and no framework code (Spring, CDI, etc.) lives here.
Quick map
Section titled “Quick map”- The model.
Jobis the live aggregate handed to engine code.JobSnapshotis the immutable view the serializer walks.JobSpec+JobArgumentdescribe the work;JobPayloadis the user-facing typed wrapper.JobStateis the centralised state machine — illegal transitions throw. - The engine.
ProcessingNodeis one per JVM. It runs one or moreDispatchers (one perQueueLaneorQueueFamily), theJobRunnerthat executes user handlers, theMaintenanceCyclethat promotes scheduled jobs and reclaims orphans, and theNodeRegistrythat fights for the master lease. - The SPI.
JobStoreis the persistence boundary, expressed in operations and guarantees, not SQL. Every backend extendsAbstractJobStoreContractTestinthreadmill-test-supportand is held to the same shared contract suite. - The scheduler.
Scheduleris the user-facing API for enqueue / schedule / recurring. It needs only aJobStoreand aJobSerializer— no running engine — so submission-only nodes work.
Lifecycle of a job
Section titled “Lifecycle of a job” enqueue claim run saveENQUEUED ────► claimReady ────► PROCESSING ───► handler.run ───► SUCCEEDED|FAILED|… │ └─ heartbeat refresh ◄─── owner-heartbeat thread (every node) └─ orphan reclaim ─────► MaintenanceCycle on the master, if heartbeat expiresEvery save funnels through store.saveAtomic(job, expectedVersion). Every
failure funnels through JobRunner.recordFailure(...). Both invariants
are load-bearing — see AGENTS.md §6.
Key invariants
Section titled “Key invariants”- Optimistic-lock
versionis persisted state. The in-memory job’s version is only advanced after a successful save, viaJob.adoptVersion(long). A failed save (StaleJobException,OversizedJobException, anything) leaves the job reusable. - Snapshot-on-serialize.
Job.snapshot()copies user-touchable areas under the job’s monitor before serialization. Concurrent mutation during serialization is impossible by construction. - One state machine, centralised and tested.
JobStateMachine.requireLegalthrows on illegal transitions.PROCESSING → ENQUEUEDis explicitly illegal: orphan recovery routes throughFAILED. - One failure code path.
JobRunner.recordFailure(...)is the only place “a job failed” is expressed. Exceptions, per-job timeouts, orphan reclaim, and quarantine all flow through it. - Scoped values, not
ThreadLocal.JobExecutionContexts.CURRENTis bound aroundhandler.run(...). The binding is inherited by structured-concurrency forks (aStructuredTaskScopeopened in the handler), but not by virtual threads the handler spawns directly via an executor — useEngineScopedValues.capturing(...)to carry it across that boundary.JobExecutionContext.current()is the handler-facing accessor for the same binding. - The deadline rule lives on the context.
ExecutionContext.watchdogDeadline()is the one formula read by both the timeout watchdog and the handler-facingdeadline()/remaining(), so the two can never disagree;deadline()additionally caps it at the node’s shutdown deadline onceclose()begins.cancellation()is recorded before the interrupt is sent, and the failure path classifiesTIMEOUT/SHUTDOWNfrom that record, not from the exception type.
Virtual threads
Section titled “Virtual threads”Every worker is a virtual thread (Executors.newVirtualThreadPerTaskExecutor).
The Dispatcher’s polling loop runs on a platform thread. Capacity is enforced
by a Semaphore per QueueLane — that’s the engine’s defence against
starvation. A flood of jobs on one queue cannot occupy capacity reserved for
another.
The interceptor surface
Section titled “The interceptor surface”JobInterceptor has four hooks:
onProcessingStarting (before handler.run)onProcessingSucceeded (after normal return)onProcessingFailed (single failure path: EXCEPTION, TIMEOUT, ORPHAN_RECLAIM, QUARANTINE)onStateChange (every persisted transition)RetryInterceptor and WorkflowInterceptor are auto-attached by
ProcessingNode. User interceptors come via ProcessingNode.builder().interceptor(...).
Retry policy precedence: per-job metadata > per-exception-type > global default.
Serialization
Section titled “Serialization”JSON via Jackson is the default (JsonJobSerializer). The wire form is
explicit and typed; supplementary-plane Unicode round-trips losslessly. Two
serializeJob overloads:
serializeJob(snapshot, long maxBytes)— strict size cap; no truncation.serializeJob(snapshot, JobStoreCapabilities)— applies log truncation (FIFO from the head) and FAILED/QUARANTINED message capping with a sentinel before the size check. Stores use this overload so aPROCESSING → FAILEDsave can never be blocked by an oversized exception trace.
Concurrency primitives
Section titled “Concurrency primitives”concurrencyKey + ConcurrencyMode {EXCLUSIVE, SHARED} on Job plus
workflow_root_id for workflow inheritance. Enforcement is at claim time,
not enqueue time — JobStore.claimReady consults per-key bookkeeping (counts +
pending) before moving a job to PROCESSING. The lock is held continuously
from the workflow root’s claim to the last descendant’s terminal save.
See docs/concurrency.md for worked examples.
Public API surface
Section titled “Public API surface”Read AGENTS.md §5 for the canonical vocabulary table — every
type listed there is part of the public API and has Javadoc on every public
method.
./gradlew :threadmill-core:testThe shared contract tests run as part of every store module’s suite; module-local tests cover the state machine, the JSON serializer, JobLog bounds, queue families, the wake signal, and cron expressions.