The Job Script Grew Up. Here Is How It Works Now.

I started job-scripts to find internships and put useful postings in a spreadsheet. As I used it for my own search, I added filters, application tracking, and checks for closed postings. Sharing the results brought more requirements: public boards needed their own criteria, my application notes had to stay private, and overlapping filters were paying to read some of the same descriptions.

The project now has a shared job catalog, a PostgreSQL-backed worker fleet, and separate views for personal tracking and public job lists. I host the backend through my homelab and maintain the public boards so other people can use the work that helped with my own search. It remains a personal, community-oriented project.

In the original post, I described getting out of the copy-paste loop. This follow-up covers the problems that came with running the pipeline continuously: recovering interrupted work, reusing decisions safely, collecting delayed results, and publishing a board while its configuration is changing. I'll follow a posting through those stages and explain the data and transaction boundaries that keep them connected.

First, the map

The frontend lives in my personal portfolio on Vercel. It talks to a Python API in my self-hosted infrastructure, where background workers fetch sources and process postings. PostgreSQL holds the shared catalog, task queue, review decisions, and board membership. The API and workers deploy separately, so fetching a slow careers page or waiting for a model response can happen independently of serving a board.

The full map below groups the code by responsibility: requests and authorization in blue, acquisition in amber, background execution in green, user workflows in rose, and operations in purple. Use the zoom controls and drag to explore a group, or reset to see the whole system. Nodes labeled with source files link to their implementations; the smaller diagrams later in the post follow individual execution paths.

1. Bringing different sources into one catalog

Consider a company publishing a backend engineering role. Depending on its hiring platform, the collector might receive a structured response containing the description, a listing with a URL to fetch, or a page that needs source-specific parsing. Each source adapter converts that response into the common fields the pipeline uses, including title, company, URL, and location.

This is the adapter pattern: code at an integration boundary translates an external interface into the application's own representation. A filter can then operate on posting fields without knowing the response format of every hiring platform. Supporting another platform requires acquisition logic for that platform, while the downstream review and board code can continue consuming the same representation. The adapter still has to preserve useful source information and handle missing fields; normalization alone cannot establish that a posting is complete or current.

Once a listing is normalized, catalog admission decides whether it enters the population available for further processing. Filtering here helps control volume, but discarding unmatched listings would make the filter hard to evaluate. If a title pattern excludes a useful role before it is recorded, I lose the evidence needed to identify the mistake. The system therefore records observed listings, including titles outside the source's pattern, before applying admission rules.

For admitted postings, ingestion caches the description supplied by the source or fetches the posting page when needed. This gives later checks a stored input to work with and avoids an extra page fetch when the adapter already has the text.

Ingestion stops at acquisition and storage. Later stages decide which checks an admitted posting needs, giving them a chance to apply eligibility rules and cheaper checks before submitting model requests. The stored listing evidence also gives me a starting point when a posting is missing from a board. I can trace whether it was observed, admitted, fetched, and reviewed, then investigate the stage where it stopped advancing.

These records have retention limits, and some catalog state is updated in place. They provide evidence about acquisition without preserving every historical version indefinitely. Reconciliation also needs to account for the source's behavior: a failed fetch or unexpected empty response is insufficient evidence to declare all of a company's postings closed.

2. Separating evidence, decisions, and views

As more filters began using the same catalog, a property such as passed needed a more precise identity. A role could pass my filter and fail an internship board's filter. Its page could be open while its location made it ineligible. A later prompt or description edit could change any of those answers. Storing a single result on the job would lose the question and input that produced it.

I organize the data around four responsibilities:

For a model review, the stored input and returned answer document what the review did. The answer remains an interpretation that another review or a person may correct. Keeping that interpretation associated with its input makes it possible to explain a decision after the posting or criteria change.

This separation lets each kind of change reach the appropriate data. A description edit updates the available posting evidence. A filter revision changes the criteria used to derive a selection. An application or note records a person's action. A public view can consume the resulting selection without copying the person's history or introducing another ingestion path.

The read side follows a CQRS-style separation: processing computes results, and board queries use a representation prepared for reading. Here that means ordinary tables and workers within the same application, with PostgreSQL holding both sides. There is no requirement for a separate service or database. Materializing the read model moves computation out of page requests, at the cost of refresh work and a period when the view can lag its inputs.

Parts of the system also use append-only records. The newer review-decision records survive the tasks that created them, so later investigation can refer to the recorded policy and evidence. The whole application is not event-sourced: mutable catalog rows and retention-limited processing records still exist, and an arbitrary past database state cannot be reconstructed from a complete event stream.

The schema preserves incomplete and failed outcomes as well. A missing compatible profile leaves a candidate on the detailed-review path; an unparsable provider response records a failed review. Preserving these states prevents later projections from treating unavailable evidence as a rejection.

3. Reducing repeated review work

Several filters can select the same posting, which makes repeated model calls a significant design concern. Some of that work is necessary: my search and a public internship board may ask different questions about the same description. Other work can be shared, such as previously derived evidence about the role. The review path needs enough context to distinguish a reusable result from one produced for a different input or policy.

Candidate selection begins with the eligibility rules for each kind of work. Freshness, location, verification, and filter fit have their own checks, and personal filters and managed boards select their respective populations. Candidates entering detailed filter processing then reach a review gate. For an enabled scope, the gate examines title rules and compatible profile evidence to determine whether a detailed call can be skipped. Missing or ambiguous evidence leaves the candidate on the detailed path.

For a technical-role filter, an unmistakably unrelated occupation can support a title exclusion. Broad words such as “Analyst” or “Operations” provide less evidence because they also appear in product and engineering roles. Rules are therefore scoped to specific filter revisions, with uncertainty sent to detailed review. The skip record retains the rule and policy responsible for the decision, separately from model verdicts.

Establishing when a result can be reused

Suppose a profile says a role requires several years of experience. An early-career filter could use that evidence to exclude the posting. If the company later edits the description while keeping the URL, however, the profile may describe an older requirement. A URL match alone would allow the old interpretation to control the new selection.

The profile reuse path in review_gate.py checks the content, title included in the original input, classifier version, model, instructions, and retained response that produced the profile. Older records may lack some input fields, so the saved batch request and consumed result must establish their provenance. If that evidence is unavailable, or the bounded lookup cannot produce a compatible profile, the candidate continues to detailed review.

This is a form of versioned derivation reuse. The identity of a derived result includes the input and the procedure that interpreted it. Changing either can invalidate its use. The additional validation limits reuse, particularly for older records, but prevents an input edit from silently inheriting an obsolete interpretation.

The existing verdict cache uses a different identity: URL, prompt hash, and model. Filter execution checks that scope before buying another answer. That cache and profile reuse have separate validity rules, so they remain separate mechanisms in the code. A cache key should describe the guarantee its reader relies on; the presence of a result for a URL is insufficient to generalize its use to every filter or current input.

Comparing a gate with the detailed path

The gate has three operating modes:

• Off: the gate does not change admission.
• Shadow: record the gate's proposed decision while still running detailed review.
• Enforce: allow an enabled exclusion to skip detailed review.

In shadow mode, a candidate the title gate would exclude still receives a paid review. If that review accepts the candidate, the linked outcomes expose a disagreement to investigate before enforcement. This shadow-evaluation pattern compares a proposed optimization with the existing path on the same inputs, avoiding the workload differences that complicate comparisons between separate weeks. The experiment incurs review cost while gathering that evidence.

A detailed model verdict is a comparison target rather than human-labeled ground truth. Candidates without a linked outcome stay in a separate “cannot compare” population. Including them as agreements would let missing or delayed results improve the reported quality of the gate. These limits constrain what the comparison can establish, even when the observed disagreement rate is low.

Before submitting paid requests, the runtime writes a durable admission decision containing the policy and evidence used. A later outcome links back to that decision, and a retry can resume from the saved plan. This preserves the basis of the comparison even if an administrator changes the gate settings while work is in flight.

4. PostgreSQL is also the work queue

PostgreSQL also coordinates the worker fleet. Every worker can notice that a source is due for ingestion, so scheduling has to tolerate several machines trying to create the same work. The scheduler derives a deduplication key for the logical cycle, and the queue writer uses a uniqueness constraint to accept one insertion:

INSERT INTO tasks (kind, payload, dedupe_key, parent_id)
VALUES (%s, %s, %s, %s)
ON CONFLICT (dedupe_key) DO NOTHING
RETURNING id

This is idempotent scheduling: competing attempts to enqueue the same key converge on one task row. It lets the fleet schedule work without a dedicated scheduler leader. A task may still run more than once through retries, so execution needs its own ownership and replay rules.

Workers claim eligible tasks using row locking and SKIP LOCKED, allowing each claimant to move past rows another worker has locked. Selection and the transition to running happen in one SQL statement. The query also checks not_before, the applicable host budget, and the task kinds supported by that worker's release. These conditions keep delayed work and unfamiliar task kinds out of a worker's current claim.

The fleet follows the competing-consumers pattern: workers independently claim from a shared queue. Each worker runs one task at a time, though a handler can perform concurrent operations internally. Larger filter runs can create child chunks for different workers to claim. Adding consumers increases available execution capacity, while the database remains responsible for coordinating claims and storing progress.

Recovering ownership after a stalled worker

Consider worker A claiming a task and then stopping its heartbeats. The reaper eventually returns the task to the queue, and worker B claims it. If A was delayed rather than terminated, it may later resume and try to finish the same task. At that point status = 'running' still matches the database row, even though B owns the current execution.

A claim therefore contains the task ID, worker name, and attempt number. Each claim increments the attempt, and guarded lifecycle writes must match that generation. When A resumes with its old attempt number, its completion update affects no row. B's update can succeed because it carries the current claim.

The guard in the task lifecycle code adds these predicates to lifecycle writes:

AND worker = %(_claim_worker)s
AND attempts = %(_claim_attempts)s

The attempt number acts as a generation token for ownership. A re-claim invalidates earlier generations, including earlier work on the same host. This protects lifecycle updates that use the guard. An external request already sent by A remains outside that protection; the batch receipt mechanism described below handles replay of saved results.

Alive and making progress are different signals

A heartbeat tied to completed operations would stop during a slow provider response, making a live worker look dead. Running the heartbeat as a coroutine on the handler's event loop has a similar failure mode: a blocking handler can prevent it from being scheduled. The worker therefore uses a separate thread for its liveness heartbeat, with updates guarded by the current claim. If the claim is lost, that thread stops updating the row.

Progress has a separate timestamp, progress_at, which advances when the saved progress value changes. Repeatedly reporting the same count keeps no appearance of forward progress. Together, these signals distinguish an execution that is alive but stalled from one whose worker has disappeared.

Choosing a recovery action

The worker's state machine records the recovery action in the task row. A host-budget delay returns the task to pending with a future not_before and restores its attempt allowance. Transient errors and expired heartbeats can requeue within the attempt budget; terminal errors or exhausted attempts produce failed. A submitted batch moves to awaiting_batch, freeing the worker while preserving outstanding work. These explicit states let another worker continue from the stored condition.

The claim query and recovery branches are in worker.py. The conditions on those transitions are as important as the status names: they establish when work becomes claimable again and which execution may update it.

Recording recovery and notifying the interface

The reaper reads expired heartbeats directly from PostgreSQL and updates the affected task rows. It emits tasks_reaped when tasks are requeued and tasks_lost when retries are exhausted; the requeue metric also records the recovered count. On the worker's exception path, retryable failures emit task_requeued, while terminal failures emit task_failed and an exception capture. These events provide operational context around the durable state changes.

The interface receives a separate task-state message through the realtime publisher. Its delivery is best effort, so recovery continues if the realtime service is unavailable. The database row remains the state another worker uses to claim or resume work.

The publisher in events.py reads the task's current status, attempts, progress, error, and subject identifiers before sending a type: task message. A configured admin channel receives it, and a task with a user ID also goes to that person's channel. Subject identifiers allow a page to recognize work about a particular job after a reload or when another tab initiated it.

These messages contain current-state snapshots. Delivery can miss intermediate transitions, and the reaper's telemetry summarizes counts, so neither is a complete per-task history. Investigating a run requires the persisted task state and its associated records as well as any available notifications and telemetry.

5. Waiting for a batch should not occupy a worker

With a live request, the worker sends input and waits for the response. A batch can spend much longer outside my system before it returns. Keeping a worker occupied for that interval would reduce the fleet's capacity even though the machine has no useful computation to perform. Instead, the task submits its requests, saves the provider's batch identity, and enters awaiting_batch. The worker can then claim another task.

This is a durable continuation: the information needed to continue lives in the database, so the original process can exit. A later worker loads that state and resumes collection. The explicit parked state also tells the rest of the runtime that the task is unfinished but does not currently need an execution slot. Parent tasks can wait for it without treating the handler's return as completion.

The continuation also needs an immutable request snapshot. Suppose I edit a filter while its batch is running. When the response arrives, the collector needs the original input and filter context to interpret it. Loading the latest settings would attach the answer to a question the provider never received.

The snapshot writer preserves the first saved request for a task and custom ID. A retry asking to save a replacement gets the original snapshot back. The conflict clause deliberately returns the existing value:

ON CONFLICT (task_id, custom_id)
DO UPDATE SET snapshot = batch_requests.snapshot
RETURNING snapshot

That makes retries refer to the same question even if the surrounding configuration has changed.

Recovery starts by looking for work already purchased

When the task runs again, the batch runtime checks for existing batch work before considering submission. If saved IDs or checkpointed results exist, it collects or consumes those. Starting over with fresh requests is not the default recovery strategy.

Returned responses are checkpointed into receipt rows before finished batch IDs are removed from the task. Both changes occur in one transaction. The order matters: clearing the IDs first would erase the route back to a response that had not yet been saved.

Some batches can finish while others are still pending. The runtime keeps the unfinished IDs, consumes available receipts, and parks again when necessary. Completion of one part does not imply completion of the whole run.

One failure window remains: the provider can accept a batch just before the process dies, before its ID reaches the database. PostgreSQL cannot roll back the provider's acceptance, so submission across this boundary is not exactly once. Persisting accepted identities promptly limits the window; recovery can reliably reuse the work whose identity was saved.

This lifecycle also determines where spending controls belong. Disabling new submissions should leave collection available for batches already purchased. Their cost is committed, and their results may still be useful. Separating submission from collection lets me stop new spending while draining outstanding work.

6. Applying a result and its accounting together

Once a response is stored, the next failure to handle is a worker dying halfway through applying it. Writing the verdict and then crashing before recording completion could cause a retry to write the verdict again. Recording completion first could lose the verdict instead. Usage accounting has the same problem: replaying a response should not count its tokens twice.

The consumer uses an idempotent consumer pattern. Each receipt is identified by provider batch ID and custom request ID, with ownership checked against the task. Re-downloading the same output reaches the same receipt. Consumption locks that row with FOR UPDATE, checks whether it has been acknowledged, and commits the verdict, usage, linked decision outcome, and acknowledgment together. A concurrent consumer waits for the lock, then sees the committed acknowledgment and skips the work.

Here is the shape of the caller, condensed to show the transaction boundary rather than every parsing and accounting argument:

with consume_result(task_id, result) as receipt:
    if not receipt.pending:
        continue

    query_id = record_verdict(result)
    record_usage(result.usage)
    link_decision_outcome(decision_id, query_id)
    receipt.outcome = "written"

The shortened function names above are illustrative. The actual caller is in filter_execution.py, and the context manager is in batch_results.py.

The with block defines a unit of work. The context manager owns the transaction and receipt lifecycle; the caller owns the domain result and accounting. Keeping those responsibilities separate lets the receipt mechanism coordinate different result handlers while still committing their related writes together.

If the worker dies before commit, the database rolls back those writes. If it dies after commit, the receipt is already marked as consumed. The application does not need to guess whether the verdict write “probably happened” based on a progress count.

This also explains why the usage write belongs inside the boundary. A duplicated verdict is a data-quality problem. A duplicated usage record is an accounting problem. Moving only one of them into an idempotent path would leave the other broken.

Failures are outcomes too. An unparsable provider response should not become a passing verdict, but its returned usage can still be recorded. The consumer can acknowledge a failed parse instead of repeatedly treating the same unusable response as brand-new work.

The recovery contract ends at this database transaction. It covers application of a saved response, including its accounting, while the earlier network acceptance window remains a separate submission concern.

7. A board is a view, not another copy of the pipeline

Imagine you find a role, apply, and leave a note about the recruiter. A week later, you narrow your search to a different location. The role no longer qualifies as a recommendation, but you still need it in your application history to prepare for the interview and remember the conversation. Recomputing the search must preserve that history.

A single “jobs on my board” flag cannot express both meanings very well. One meaning is computed by the system: this is worth showing under the current criteria. The other comes from you: this is now part of my search history. Letting the first overwrite the second would make changing a filter destructive.

So the personal board combines two kinds of state. A worker materializes computed membership as job IDs. Your actions live separately, and the visibility rules preserve explicit reasons such as an application status or notes. A filter can stop recommending a posting without pretending you never interacted with it.

The computed membership is a materialized view, maintained by application workers. Evaluating which postings qualify happens in the background; a page request reads the saved IDs and combines them with personal state. This moves expensive computation out of the read path and lets multiple reads reuse one refresh. The tradeoff is freshness: after a setting changes, the saved membership can lag until recomputation finishes.

The next question was sharing. I wanted to send useful jobs to people who did not have an account. But sharing my tracker would share the wrong thing. The role's title and posting text are useful to them. My recruiter notes and application status are not.

This is where the projection idea pays off a second time. A public board publishes a selection of posting data, not a serialized personal board with a few private columns hidden in the browser. The public response has its own boundary. Private fields never need to arrive at the public client.

There are two ways to produce that selection. A managed board can evaluate its own criteria, such as an early-career search. Or it can reuse a sponsor filter's selection when the goal is simply to share those results. The latter does not need another model call to answer a question the personal filter already answered.

The schema makes the sharing boundary visible. All three membership or state tables below refer to the same catalog job, but they have different owners and lifecycles. board_visible holds computed personal membership. user_jobs holds personal fields. managed_board_jobs belongs to a managed board and records the revision that published each member.

For example, two people can each have a user_jobs row for the same job_id, with independent notes and application dates. A public board can include that job through its own membership row without reading either person's notes. The sponsor relationship identifies who owns the board configuration; it does not make the sponsor's application fields part of the public response. That response still needs an explicit posting-only shape, even though the underlying tables are separated.

The slowest run should not get the final say

Sharing adds another failure case. Suppose a managed board starts reviewing candidates under revision A. While its batches are running, I update the criteria to revision B. B's run finishes first. Then A returns with a perfectly valid set of answers to an outdated question.

If publishing just means “replace the list with whatever finished most recently,” the older criteria win. Nothing crashed. Every model request may have succeeded. The board is still wrong.

The publication step therefore carries the revision of the configuration that started the run. Inside a database transaction, it locks the matching board row. No match means the configuration changed, so the old run cannot replace the list.

The key query in the managed-board publisher is:

SELECT id
FROM managed_boards
WHERE id = %s AND revision = %s
FOR UPDATE

This is optimistic revision validation followed by a short locked publication transaction. The run performs its expensive work without holding the board lock. At publication, the revision predicate checks whether its inputs are still current, and the row lock keeps that answer stable while the transaction replaces membership.

Deletion of the old membership and insertion of the replacement happen in that same transaction. If insertion fails, the deletion rolls back too. Readers do not observe the intermediate “we deleted everything and are still inserting” state.

The run also waits for its batch work before replacing the projection. Together, the completion check, revision check, and transaction protect three separate properties: the selection is ready to publish, it belongs to the current configuration, and readers see a committed replacement. Classification accuracy still depends on the review criteria and evidence; publication correctness can be enforced independently.

8. The application keeps going after you click apply

A job-search tool could stop at discovery. Mine became less useful if it did. The interesting part of an application often happens somewhere else: a form on a hiring platform, an acknowledgement in Gmail, a recruiter asking for availability.

Those integrations bring the same identity problem back in a different form. A message naming a company is not automatically proof of an application. Two roles at the same employer are not interchangeable. And an outreach email is not the same thing as a response to something I submitted.

The browser extension handles the application-side interaction using the tracker's profile, drafts, and remembered answers. It helps fill the form; the person reviews and submits it. The backend remains the place that owns the application data rather than turning each hiring website into another independent tracker.

Mail has a separate acquisition path. Sync gets messages into storage. Later classification and matching can interpret them. These are separate, configurable stages, so ingesting mail is not a promise to run every available model-backed feature on it.

Suppose I have already recorded an application when its acknowledgment email arrives. Creating an application directly from that email would leave two records for the same activity and make later matching harder. The handler therefore seeds applications from explicit tracker activity first, then matches messages against that history. Only afterward does it consider deriving applications from qualifying unmatched mail. Those candidates still need supporting evidence, including an eligible message kind and a role; recruiter outreach alone does not establish that I applied.

When matching needs a person, the resolution API supplies the evidence and available choices, including whether an action is eligible and whether it needs a target. This keeps the frontend's explanation aligned with the backend's policy. The command handler checks ownership and state again when the person acts, since a client can send a request independently of the buttons it displays.

There is also a smaller version of the stale-worker problem here: I can leave a review page open while the underlying match changes. Confirming an attachment that has already been superseded should not silently approve the old one. The match-resolution handler checks that the requested attachment is current and returns a stale-state conflict if it is not.

This repeats the revision-checking approach used in board publication. The browser can hold a useful snapshot for display, but the server validates the action against current state before committing it. A stale-state conflict tells the client to refresh its evidence instead of silently applying an outdated choice.

The same care applies to reading. Being signed in does not entitle someone to any job ID they can guess. Per-job access uses the board visibility predicate and the current user. A public projection has its own response shape. Authorization has to follow the object through the workflow, not just protect the navigation menu.

Implementation references: mail matching, resolution commands, and per-job access.

9. “Why did this happen?” is now a feature

Early investigations started with task payloads. As postings accumulated retries, reviews for multiple filters, and publications under different revisions, reconstructing their history became harder. Task records can expire, and current configuration values cannot establish which settings a worker used yesterday. The explanation needs its own stored identity and context.

The newer review records keep the decision, the policy, the input identity, and any reusable evidence. Paid outcomes link back to those records. Admin configuration saves keep a separate history with the actor and the before-and-after values.

The frontend presents those records through named filter selection, decision timelines, configuration history, and a review-gate funnel with links to the decisions behind its counts. Missing evidence has explicit labels such as “unknown,” “no linked outcome,” and “earlier history unavailable.” These distinguish an incomplete record from a successful check or a request known to have cost nothing.

The limits matter, too. Configuration history covers changes through the admin writer, not arbitrary direct database edits. The funnel covers recorded review-gate decisions, not an invented accounting of every optimization. And historical task-only decisions are not reconstructed from whatever the settings happen to be now.

Follow one unexpected posting backward

Suppose a role appears on a public early-career board and I am not convinced it belongs there. The first useful question is not “which prompt should I tweak?” It is “which decision actually admitted this posting?”

I want to work backward from the projection to the run, then to the relevant filter scope, input, and verdict. Did the detailed review accept it? Did reusable evidence avoid a call? Was the result evaluated under the criteria I am looking at now, or an earlier revision? Is there an outcome at all, or only a record that the candidate was scheduled for review?

Those answers lead to different fixes. A wrong judgment suggests reviewing the criteria and examples. An outdated projection suggests a publication or refresh problem. Missing evidence suggests an observability gap. Running the model again before distinguishing them could spend more money without addressing the actual failure.

The durable review records make that investigation a supported path rather than a one-off log search. They also make it possible to separate admission counts from paid outcomes. A decision to review is not itself a completed model request, just as a scheduled task is not completed work.

A cheaper day is not automatically a better pipeline

Cost reporting has the same problem of mismatched meanings. If today's bill is lower, the optimization might be working. Or fewer postings arrived. Or a source failed. Or a batch is still waiting to return. An aggregate alone cannot distinguish those explanations.

So I keep recorded usage, attribution, and hypothetical avoided cost separate. Recorded token usage can be priced and grouped by purpose. A gate's skipped candidates can support an avoided-work estimate. But a skipped request did not return actual token usage, so the exact price of the request that never happened is unknowable.

A useful comparison keeps the workload definition fixed: the same candidate population, the same policy scope, and a clear account of missing outcomes. Otherwise I could “improve” cost per job just by counting more cheap candidates in the denominator while leaving expensive reviews unchanged.

This is why the interface needs words such as “recorded estimate.” The local usage ledger is not the provider's invoice. It can contain unpriced rows or incomplete attribution. A zero known subtotal with unknown usage is different from proof of zero spend.

The engineering lesson is that observability belongs in the execution contract. If each new handler invents its own cost row, progress counter, and explanation format, the dashboard eventually becomes a collection of incompatible numbers. Capturing identity, purpose, and outcomes where the work happens gives the views something consistent to report.

10. The homelab is part of the design

The backend runs across machines that do not all have the same hardware or availability. That is another reason to keep the queue and progress outside an individual worker.

Deployment follows the same idea: identify the exact release, rather than assume a machine running a container called “latest” is actually current.

Schema changes need to tolerate a rolling deployment because old and new workers can briefly coexist. Tests cover the database behavior, not just mocked function calls. Runtime monitoring then answers a different question: is the deployed system actually making progress?

During that overlap, one worker can know a new task kind while another still runs the previous image. The older worker must leave that task alone, and both versions must be able to use the schema. Additive schema changes allow the new representation to arrive before every consumer depends on it.

The task claim query's known-handler restriction is one small answer to that problem. Versioned image identities are another. A merged commit tells me what should be running. The reported worker revision and container digest tell me what is running. Those checks are related, but they are not interchangeable.

For the same reason, I do not treat a healthy HTTP endpoint as proof that the whole pipeline is healthy. An API can serve pages while source acquisition is stalled, or while a queue contains work no online worker can handle. Process availability, task liveness, progress, and data freshness answer different operational questions.

I have a separate homelab architecture post for the broader infrastructure. For job-scripts, the main point is simpler: a worker can come and go without becoming the only place that knows what work exists.

The central database remains a shared availability dependency. Workers can resume each other's saved work, but they all need PostgreSQL to claim tasks and persist results. A database outage therefore requires database recovery before distributing more workers will help.

There are costs to these choices. Materialized views need refreshes. Durable evidence needs retention rules. A database queue competes for database resources. Strict provenance can reject reuse that might have been correct. More specialized infrastructure could change some of those tradeoffs, but it would also add more systems whose failures I would need to understand.

For a self-hosted project, I prefer a design whose recovery I can explain from the stored state. That does not make it the right queue for every workload. It makes the operational model fit the person who has to maintain it.

The part I would keep if I started over

The original script saved me from copying rows into a spreadsheet. Maintaining this version has pushed me to make the path behind each row inspectable: which source supplied it, which input was reviewed, which configuration produced the decision, and which run published it. Keeping evidence, decisions, projections, and personal actions separate gives those questions specific places to look.

The same boundaries make recovery manageable. A worker resumes from saved batch identities, a receipt keeps replay from duplicating local effects, and a revision check stops an outdated run from replacing a current board. These are small, explicit contracts I can inspect when something fails. If I started again, I would establish those identities and transaction boundaries early, alongside the first working pipeline.

The code is in job-scripts, and the story starts with the spreadsheet that got me here.