How To Create Kubernetes Jobs/Cron Jobs – Getting Started Guide

Kubernetes jobs are primarily meant for short-lived and batch workloads. It runs for completion as opposed to other objects like replicasets, replication controllers and DaemonSets which runs continuously. This tutorial explains the process of creating kubernetes jobs and cron jobs along with a few tips and tricks.

Jobs run until the tasks specified in the job is completed. This means, if the pods give exit code 0, the job will exit. Whereas in normal Kubernetes deployments, irrespective of the exit codes, the deployment object will create new pods when it terminates or throws an error to keep the desired stated of the deployment.

Also, during a Job run, if the node hosting the pod fails, the job pod will get automatically rescheduled to another node.

Kubernetes Jobs Use Case

The best use case for Kubernetes jobs are,

1. Batch processing: Let’s say you want to run a batch task one time a day or during a specific schedule. It could be something like reading files from storage or a database and feed it to a service to process the files.
2. Operations/ad-hoc tasks: Let’s say you want to run a script/code which runs a database cleanup activity.

Creating a Kubernetes Job

In this example, we will use an Ubuntu container that runs a for loop and echoes a message based on the argument you pass to the container. An argument is a number that decides how many times the loop should run.

You Might Like: Get 16% Off + $100 Off On Kubernetes Certification & Course Bundle

For example, if you pass 100 as an argument, it will echo the message 100 times and the container will exit.

You can view the Dockerfile and script from here -> kube-job-example Docker configs

Let’s get started with a job with a simple setup.

1. Create a job.yaml file with our custom Docker image with 100 as a command argument. The value 100 will be passed to the docker ENTRYPOINT script as an argument.

   apiVersion: batch/v1 
   kind: Job metadata:   
   name: kubernetes-job-example   
   labels:     
     jobgroup: jobexample 
   spec:   
     template:     
       metadata:       
         name: kubejob       
         labels:         
           jobgroup: jobexample     
       spec:       
         containers:       
         - name: c         
           image: devopscube/kubernetes-job-demo:latest         
           args: ["100"]       
        restartPolicy: OnFailure
   

2. Let’s create a job using kubectl with the job.yaml file.

   kubectl apply -f job.yaml

3. Check the status on the job using kubectl.

   kubectl get jobs

4. You can get the list of pods using kubectl.

   kubectl get po

5. You can get the job pod logs using kubectl.

   kubectl logs kubernetes-job-example-bc7s9 -f

Multiple Job Pods and Parallelism

When a job is deployed you can make it run on multiple pods with parallelism.

For example, if you want to run 6 pods and run 2 pods in parallel, you need to add the following two parameters to your job manifest.

completions: 5
parallelism: 2

Here is the manifest file with those parameters.

apiVersion: batch/v1
kind: Job
metadata:
  name: kubernetes-parallel-job
  labels:
    jobgroup: jobexample
spec:
  completions: 5
  parallelism: 2
  template:
    metadata:
      name: kubernetes-parallel-job
      labels:
        jobgroup: jobexample
    spec:
      containers:
      - name: c
        image: devopscube/kubernetes-job-demo:latest
        args: ["100"]
      restartPolicy: OnFailure

Generate Random Job Name

You cannot have a single job manifest file and create multiple jobs from it. Kubernetes will throw an error stating that a job with the same name exists.

To circumvent this problem, you can add the generateName name parameter to the metadata.

For example,

apiVersion: batch/v1
kind: Job
metadata:
  generateName: kube-job-
  labels:
    jobgroup: jobexample

In the above example, every time you run the manifest, a job will get created with kube-job- as a prefix followed by a random string.

Creating a Kubernetes CronJob

What if you want to run a batch job on specific schedules, for example, every 2 hours. You can create a Kubernetes cronjob with a cron expression. The job will automatically kick in as per the schedule you mention in the job.

Here is how we specify a cron schedule. You can use the crontab generator to generate your own schedule.

schedule: "0,15,30,45 * * * *"

If we were to run our previous job as a cron job every 15 minutes, here is how the manifest looks. Create a file named cron-job.yaml and copy the following manifest.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
    name: kubernetes-cron-job
spec:
  schedule: "0,15,30,45 * * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: cron-batch-job
        spec:
          restartPolicy: OnFailure
          containers:
          - name: kube-cron-job
            image: devopscube/kubernetes-job-demo:latest
            args: ["100"]

Let’s deploy the cronjob using kubectl.

kubectl create -f cron-job.yaml

List the cronjobs

kubectl get cronjobs

Trigger a CronJob Manually

There are situations where you might want to execute the cronjob in an ad-hoc manner. You can do this by creating a job from an existing cron job.

For example, if you want a cronjob to be triggered manually, here is what we should do.

kubectl create job --from=cronjob/kubernetes-cron-job manual-cron-job

--from=cronjob/kubernetes-cron-job will copy the cronjob template and creates a job named manual-cron-job

Few Key Parameters

There are a few more key parameters you can use with kubernetes jobs/cronjobs based on your needs. Let’s have a look at each.

1. failedJobHistoryLimit & successfulJobsHistoryLimit: Deletes the failed and successful job history based on the retention number you provide. This is super useful to trim down all failed entries when you try to list the jobs. For example,

   failedJobHistoryLimit: 5 
   successfulJobsHistoryLimit: 10

2. backoffLimit: Total number of retries if your pod fails.