Kubernetes CronJobs and Overlapping Runs

A CronJob looks like crontab in YAML, and for a job that takes 30 seconds every hour it behaves like crontab too. The trouble starts when a run takes longer than the gap between schedules. Then two copies of the same work can run at once, and whether that’s harmless or a disaster depends on a field most people leave at its default.

This guide uses Kubernetes 1.30 and later. The manifests are illustrative and the timelines are reasoned from the documented controller behaviour rather than captured from a cluster.

The scenario: a 15-minute schedule and a 22-minute run

An invoice-export CronJob runs every 15 minutes. It reads invoices created since the last export, writes them to a CSV and uploads it to the finance team’s SFTP server. Normally a run takes 4 minutes. At month end the volume triples and a run takes 22 minutes.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: invoice-export
  namespace: finance
spec:
  schedule: "*/15 * * * *"
  timeZone: "Europe/London"
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: export
              image: registry.example.com/invoice-export:2.8.1
              args: ["--since-last-export"]

There’s no concurrencyPolicy, so it’s Allow. At 10:00 the first Job starts. At 10:15 it’s still running, and the controller starts a second Job because nothing told it not to. Both read “invoices since the last export” from the same checkpoint, because the first run hasn’t written its checkpoint yet. Both upload a file. Finance imports both files, and every invoice between 09:45 and 10:15 is booked twice.

Nothing in Kubernetes failed here. The controller did exactly what Allow means. The design flaw is that the job assumes it runs alone without anything enforcing that.

The three concurrency policies

spec.concurrencyPolicy decides what the controller does when it’s time for a new run and a previous Job is still active.

• Allow (default): start the new Job anyway. Fine for independent work, such as a probe that pings an endpoint, where overlapping copies don’t share state.
• Forbid: don’t start the new Job while the old one runs. The schedule is treated as missed.
• Replace: delete the active Job (and its Pods) and start the new one. Use this when only the latest run matters, like refreshing a cache from the current state of a database, and a half-finished older run is worthless.

For invoice-export, Replace would be bad: it would kill the 10:00 run at 10:15, after it may already have uploaded part of a file, then the 10:15 run would hit the same volume and be killed at 10:30. At month end, no run would ever finish. Forbid is the right choice:

spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300

Now the timeline at month end is: 10:00 run starts; 10:15 schedule arrives, a Job is active, so no new Job; 10:22 run finishes. The controller doesn’t queue every missed run. On its next pass it may start the most recent missed schedule (10:15) late, but only if that’s still within startingDeadlineSeconds. At 10:22 the 10:15 run is 7 minutes late, beyond the 300-second deadline, so it’s skipped, and the next run happens at 10:30. Because the job exports “since the last checkpoint”, skipping 10:15 loses nothing; the 10:30 run picks up the extra invoices.

That last sentence matters. Forbid is only safe when the work is cumulative or catch-up based. If each run handled exactly “the 15 minutes ending at the schedule time”, skipped runs would leave gaps.

startingDeadlineSeconds and the 100-missed-runs trap

startingDeadlineSeconds is the answer to “how late may a run start and still be worth running?” It also prevents a nasty failure mode.

If the CronJob controller can’t create Jobs for a while (the control plane was down, or the CronJob was suspended and then resumed), it counts how many schedules it missed. With no startingDeadlineSeconds, it counts from the last scheduled time; if that count exceeds 100, it doesn’t start the Job and logs an error along the lines of “too many missed start times”. A CronJob running every minute that’s been suspended for two hours would hit this. With startingDeadlineSeconds: 300, the controller only counts misses in the last 300 seconds, so the count stays small and the CronJob carries on.

Keep the deadline at least 10 seconds; the controller checks roughly every 10 seconds, so a smaller value can cause runs to be skipped even when nothing is wrong.

Forbid is not a lock

Forbid coordinates Jobs that one CronJob object creates. It knows nothing about:

• Jobs created by hand from the same template: kubectl create job manual-export --from=cronjob/invoice-export -n finance starts a Job that runs alongside the scheduled one.
• A second CronJob in another namespace or cluster that runs the same image against the same database.
• The rare case the documentation calls out: the controller may create two Jobs for one schedule, or none. It’s “approximately once”, not exactly once.

So the job itself needs protection. The usual pattern for an export like this is a database row that acts as both a lock and a checkpoint:

-- Claim the window; fails with a unique-constraint error if another run already took it.
INSERT INTO export_runs (window_end, started_at, status)
VALUES (date_trunc('minute', now()), now(), 'running');

Combined with an idempotent upload (write invoices-2026-10-07T10-00.csv, a deterministic name, so a repeat overwrites rather than adds), a duplicate Job becomes a no-op instead of double-booked invoices. An advisory lock (pg_try_advisory_lock) or a lease object also works; the point is that correctness lives in the application, and concurrencyPolicy reduces how often you depend on it.

Interview tip

Set activeDeadlineSeconds on the Job template as well. A run that hangs on a dead SFTP connection would otherwise stay active forever, and with Forbid that blocks every later run silently.

Other fields worth knowing

• timeZone (stable since 1.27) takes an IANA name. Without it the schedule uses the kube-controller-manager’s time zone, usually UTC. Daylight-saving changes can make a run happen twice or not at all on the switch night if you schedule inside the skipped or repeated hour.
• successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1) control how many finished Jobs, and therefore Pods and logs, you can still inspect.
• suspend: true stops new runs without deleting the CronJob. Combine with startingDeadlineSeconds so resuming doesn’t trigger a burst or the 100-miss refusal.

To see what’s happening during an incident:

kubectl get cronjob invoice-export -n finance
kubectl get jobs -n finance --sort-by=.metadata.creationTimestamp
kubectl describe cronjob invoice-export -n finance   # events show skipped and created Jobs

The LAST SCHEDULE and ACTIVE columns from the first command tell you whether a run is still holding the slot.

Try it yourself: an interview question

“An hourly CronJob sends a ‘daily summary so far’ email to customers. Occasionally customers get two identical emails a few seconds apart. The job takes about 2 minutes. What’s happening and how would you fix it?”

Answer

A 2-minute job on an hourly schedule shouldn’t overlap with the next run, so concurrencyPolicy probably isn’t the cause. Two emails seconds apart suggests two Jobs (or two Pods) started for the same schedule. The likely culprits are: the CronJob controller occasionally creating two Jobs, which the docs explicitly allow; a Pod failing after sending emails and the Job’s backoffLimit retrying it, so the retry re-sends; or someone running the job manually with --from=cronjob.

I’d check kubectl get jobs and the Pods’ restart and failure history around the incident times to tell those apart. Whatever the cause, the fix is to make sending idempotent: record “summary for customer X, hour H” with a unique key before or atomically with sending, and skip if it exists. Many email providers also accept an idempotency key. I’d still set concurrencyPolicy: Forbid and startingDeadlineSeconds as good hygiene, but I wouldn’t claim they fix duplicates on their own.

Related reading

• Kubernetes interview questions
• Kubernetes MCQs
• Jobs, retries and repeatable batch work
• Official docs: CronJob