Kubernetes jobs are primarily meant for short-lived and batch workloads. It runs for completion as opposed to other objects like replicasets, replication controllers and DaemonSets which runs continuously. This tutorial explains the process of creating kubernetes jobs and cron jobs along with a few tips and tricks.
Jobs run until the tasks specified in the job is completed. This means, if the pods give exit code 0, the job will exit. Whereas in normal Kubernetes deployments, irrespective of the exit codes, the deployment object will create new pods when it terminates or throws an error to keep the desired stated of the deployment.
Also, during a Job run, if the node hosting the pod fails, the job pod will get automatically rescheduled to another node.
Kubernetes Jobs Use Case
The best use case for Kubernetes jobs are,
- Batch processing: Let’s say you want to run a batch task one time a day or during a specific schedule. It could be something like reading files from storage or a database and feed it to a service to process the files.
- Operations/ad-hoc tasks: Let’s say you want to run a script/code which runs a database cleanup activity.
Creating a Kubernetes Job
In this example, we will use an Ubuntu container that runs a for loop and echoes a message based on the argument you pass to the container. An argument is a number that decides how many times the loop should run.
For example, if you pass 100 as an argument, it will echo the message 100 times and the container will exit.
You can view the Dockerfile and script from here ->
Let’s get started with a job with a simple setup.
- Create a
job.yamlfile with our custom Docker image with 100 as a command argument. The value 100 will be passed to the docker ENTRYPOINT script as an argument.
apiVersion: batch/v1 kind: Job metadata: name: kubernetes-job-example labels: jobgroup: jobexample spec: template: metadata: name: kubejob labels: jobgroup: jobexample spec: containers: - name: c image: devopscube/kubernetes-job-demo:latest args: ["100"] restartPolicy: OnFailure
- Let’s create a job using kubectl with the
kubectl apply -f job.yaml
- Check the status on the job using kubectl.
kubectl get jobs
- You can get the list of pods using kubectl.
kubectl get po
- You can get the job pod logs using kubectl.
kubectl logs kubernetes-job-example-bc7s9 -f
Multiple Job Pods and Parallelism
When a job is deployed you can make it run on multiple pods with parallelism.
For example, if you want to run 6 pods and run 2 pods in parallel, you need to add the following two parameters to your job manifest.
completions: 5 parallelism: 2
Here is the manifest file with those parameters.
apiVersion: batch/v1 kind: Job metadata: name: kubernetes-parallel-job labels: jobgroup: jobexample spec: completions: 5 parallelism: 2 template: metadata: name: kubernetes-parallel-job labels: jobgroup: jobexample spec: containers: - name: c image: devopscube/kubernetes-job-demo:latest args: ["100"] restartPolicy: OnFailure
Generate Random Job Name
You cannot have a single job manifest file and create multiple jobs from it. Kubernetes will throw an error stating that a job with the same name exists.
To circumvent this problem, you can add the
generateName name parameter to the metadata.
apiVersion: batch/v1 kind: Job metadata: generateName: kube-job- labels: jobgroup: jobexample
In the above example, every time you run the manifest, a job will get created with
kube-job- as a prefix followed by a random string.
Creating a Kubernetes CronJob
What if you want to run a batch job on specific schedules, for example, every 2 hours. You can create a Kubernetes cronjob with a cron expression. The job will automatically kick in as per the schedule you mention in the job.
Here is how we specify a cron schedule. You can use the crontab generator to generate your own schedule.
schedule: "0,15,30,45 * * * *"
If we were to run our previous job as a cron job every 15 minutes, here is how the manifest looks. Create a file named
cron-job.yaml and copy the following manifest.
apiVersion: batch/v1beta1 kind: CronJob metadata: name: kubernetes-cron-job spec: schedule: "0,15,30,45 * * * *" jobTemplate: spec: template: metadata: labels: app: cron-batch-job spec: restartPolicy: OnFailure containers: - name: kube-cron-job image: devopscube/kubernetes-job-demo:latest args: ["100"]
Let’s deploy the cronjob using kubectl.
kubectl create -f cron-job.yaml
List the cronjobs
kubectl get cronjobs
Trigger a CronJob Manually
There are situations where you might want to execute the cronjob in an ad-hoc manner. You can do this by creating a job from an existing cron job.
For example, if you want a cronjob to be triggered manually, here is what we should do.
kubectl create job --from=cronjob/kubernetes-cron-job manual-cron-job
--from=cronjob/kubernetes-cron-job will copy the cronjob template and creates a job named
Few Key Parameters
There are a few more key parameters you can use with kubernetes jobs/cronjobs based on your needs. Let’s have a look at each.
- failedJobHistoryLimit & successfulJobsHistoryLimit: Deletes the failed and successful job history based on the retention number you provide. This is super useful to trim down all failed entries when you try to list the jobs. For example,
failedJobHistoryLimit: 5 successfulJobsHistoryLimit: 10
- backoffLimit: Total number of retries if your pod fails.