Kubernetes Jobs and Repeatable Batch Work

A Job’s promise is simple: run Pods until a given number of them succeed. The interesting part is what happens when they don’t. Kubernetes will retry, and by default it retries a lot, which is good for a flaky network call and wasteful for a bug that fails the same way every time. Getting batch work right means deciding which failures deserve a retry, how long the whole thing may take, and making sure a retry can’t corrupt what a previous attempt already did.

Examples target Kubernetes 1.30 and later; features with a newer minimum version are flagged. Manifests and status output are illustrative.

The retry loop you get by default

Start with the smallest useful Job: a database migration.

apiVersion: batch/v1
kind: Job
metadata:
  name: orders-migrate-58
  namespace: shop
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/orders-migrate:58
          args: ["up"]

If the container exits non-zero, the Pod fails. The Job controller creates a new Pod after a delay, then another, with the delay doubling: 10 seconds, 20, 40, and so on, capped at 6 minutes. It stops after backoffLimit failures, which defaults to 6. The Job then gets a Failed condition with reason BackoffLimitExceeded and the message “Job has reached the specified backoff limit”.

Seven attempts and several minutes of waiting is reasonable for “the database wasn’t reachable yet”. It’s pointless for “migration 58 has a syntax error”, which fails identically each time.

restartPolicy must be Never or OnFailure for a Job (Always is rejected). The two behave differently:

• Never: each failure produces a new Pod. Failed Pods stay around, so you can run kubectl logs on every attempt. Easier to debug.
• OnFailure: the kubelet restarts the container inside the same Pod. Fewer Pods, but the logs of earlier attempts are mostly gone (you get one previous run via --previous), and the Pod shows CrashLoopBackOff between restarts, which confuses people who think something else is wrong. Restarts still count toward backoffLimit.

For anything you’ll need to debug, Never is usually the better choice. It’s also required for podFailurePolicy.

Bounding time as well as attempts

backoffLimit counts failures; it doesn’t bound time. A Pod that hangs on a lock never fails, so it never triggers a retry, and the Job runs until someone notices. Add a deadline:

spec:
  backoffLimit: 3
  activeDeadlineSeconds: 900
  ttlSecondsAfterFinished: 86400

activeDeadlineSeconds counts from when the Job starts, across all attempts. When it passes, the controller terminates running Pods and marks the Job failed with reason DeadlineExceeded. It takes precedence over backoffLimit: a Job with retries left still stops at the deadline. Note there’s also an activeDeadlineSeconds on the Pod spec; they’re different fields, and the Job-level one is the one you usually want.

ttlSecondsAfterFinished deletes the Job (and its Pods) a day after it finishes, success or failure. Without it, finished Jobs pile up unless something else cleans them; CronJobs have their own history limits instead.

Telling bugs from bad luck with podFailurePolicy

podFailurePolicy (stable since 1.31) lets you decide per failure whether it should count, fail the Job immediately, or be ignored. It matches on container exit codes or on Pod conditions.

Imagine a thumbnail job that processes a batch of uploaded images. The program exits with code 42 when the input batch is malformed: a retry will never help. It exits 1 on transient errors such as an S3 timeout, which a retry often fixes. And Pods sometimes die because a node was drained for maintenance, which isn’t the job’s fault at all.

apiVersion: batch/v1
kind: Job
metadata:
  name: thumbnails-2026-10-07
  namespace: media
spec:
  backoffLimit: 4
  activeDeadlineSeconds: 3600
  podFailurePolicy:
    rules:
      - action: FailJob
        onExitCodes:
          containerName: resize
          operator: In
          values: [42]
      - action: Ignore
        onPodConditions:
          - type: DisruptionTarget
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: resize
          image: registry.example.com/thumbnails:3.1.0

Walk through three failures:

1. A Pod exits 1 after an S3 timeout. No rule matches, so it counts toward backoffLimit, and the controller retries with backoff.
2. A node is drained. The evicted Pod gets the DisruptionTarget condition, the Ignore rule matches, and the controller replaces the Pod without spending a retry. Without this rule, a busy maintenance window could exhaust the backoff budget on failures the code didn’t cause.
3. A Pod exits 42. FailJob matches, the controller terminates any other running Pods and fails the Job straight away with reason PodFailurePolicy. You find out in seconds instead of after several minutes of doomed retries.

There’s also a Count action (the default behaviour, stated explicitly) and FailIndex for Indexed Jobs.

Parallel work and per-index retries

For larger batches, split the work. completionMode: Indexed with completions: 10 and parallelism: 5 runs ten Pods, five at a time, each with a JOB_COMPLETION_INDEX environment variable from 0 to 9. Shard 3 processes batch-03, and so on.

In plain Indexed mode, backoffLimit is shared: one stubborn shard can use up retries for all of them. backoffLimitPerIndex (stable in recent releases; check your cluster version) gives each index its own budget, and maxFailedIndexes decides how many indexes may fail before the whole Job does. With that, shard 7 failing on a corrupt file doesn’t stop the other nine.

Retries mean work runs twice

Here’s the part interviewers care most about. A retry restarts the whole Pod, not the step that failed. If the thumbnail job had resized 600 of 1,000 images when S3 timed out, the next attempt starts from image 1. Any side effect from the first attempt happens again.

Kubernetes can also run a Job’s Pod more than once even when nothing failed from the app’s point of view: a node can become unreachable, the controller creates a replacement, and the original was actually still running. For a short window, two copies can be active. The docs say plainly that your program should cope with this.

So the work needs to be idempotent:

• Write output to deterministic names (thumbs/<image-id>-320.webp) so a repeat overwrites the same object rather than adding a new one.
• Skip items already done: check whether the output exists, or keep a checkpoint per shard.
• For database writes, use upserts or a unique key on the business identifier (ON CONFLICT DO NOTHING), not blind inserts.
• For external calls that can’t be repeated (charging a card, sending an email), pass an idempotency key the provider de-duplicates on.

The migration Job from the first example is a good case: most migration tools record applied versions in a table, so re-running up after a partial failure applies only what’s missing, provided each migration is transactional.

Note

To see why a Job failed, read its conditions and events first: kubectl describe job thumbnails-2026-10-07 -n media. Then list its Pods with kubectl get pods -n media -l job-name=thumbnails-2026-10-07 and read the logs of the failed ones.

Try it yourself: an interview question

“A nightly Job imports a CSV of 200,000 customer records into Postgres. Last night it failed at row 140,000 when the database restarted, retried, and finished. This morning there are 340,000 rows. Explain what happened and how you’d change the Job and the code.”

Answer

The database restart killed the first Pod (or made it exit non-zero), and the Job controller retried it as designed. The retry started from the first row because a Job retries the whole Pod, not the failed step. The import used plain inserts, so the 140,000 rows from the first attempt were inserted again, plus all 200,000 from the second attempt: 340,000.

The real fix is in the code: make the import idempotent. Use the customer’s natural key with a unique constraint and INSERT ... ON CONFLICT (customer_id) DO UPDATE, or load into a staging table and swap it in one transaction at the end, so a partial attempt leaves no visible rows. Committing in batches and recording the last committed row would also let a retry resume rather than restart.

On the Job, I’d keep retries for transient failures like this one, add activeDeadlineSeconds so a hung import can’t run into the business day, and use podFailurePolicy to fail immediately on an exit code that means “the CSV is invalid”, since retrying bad input wastes the window. restartPolicy: Never keeps each attempt’s logs for the investigation.

See also

• Kubernetes interview questions
• Kubernetes MCQs
• CronJobs and overlapping runs
• Pod lifecycle
• Official docs: Jobs