How To Create Kubernetes Jobs/Cron Jobs – Getting Started Guide
- Last Updated On: February 18, 2021
- By: devopscube
Kubernetes jobs are primarily meant for short-lived and batch workloads. It runs for completion instead of other objects like deployment, replicasets, replication controllers, and DaemonSets, which continuously runs.
This tutorial explains creating kubernetes jobs and cronjobs, along with a few tips and tricks.
Kubernetes Jobs run until the tasks specified in the job are completed. Meaning, if the pods give exit code 0, the job will exit. Whereas in normal Kubernetes deployments, irrespective of the exit codes, the deployment object will create new pods when it terminates or throws an error to keep the deployment’s desired state.
During a Job run, if the node hosting the pod fails, the job pod will get automatically rescheduled to another node.
Kubernetes Jobs Use Case
The best use case for Kubernetes jobs are,
- Batch processing: Let’s say you want to run a batch task once a day or during a specific schedule. It could be something like reading files from storage or a database and feed them to a service to process the files.
- Operations/ad-hoc tasks: Let’s say you want to run a script/code which runs a database cleanup activity or even backup a kubernetes cluster itself.
How to Create a Kubernetes Job
In this example, we will use an Ubuntu container that runs a shell script with for loop and echoes a message based on the argument you pass to the container. This argument is a number that decides how many times the shell script loop should run.
You Might Like: CKA, CKAD & CKS Exam Coupon Codes
For example, if you pass 100 as an argument, the shell script will echo the message 100 times and the container will exit.
You can view the Dockerfile and the shell script from here ->
Let’s get started with a job with a simple setup.
Step 1: Create a job.yaml
file with our custom Docker image with 100 as a command argument. The value 100 will be passed to the docker ENTRYPOINT script as an argument.
apiVersion: batch/v1
kind: Job
metadata:
name: kubernetes-job-example
labels:
jobgroup: jobexample
spec:
template:
metadata:
name: kubejob
labels:
jobgroup: jobexample
spec:
containers:
- name: c
image: devopscube/kubernetes-job-demo:latest
args: ["100"]
restartPolicy: OnFailure
Step 2: Let’s create a job using kubectl with the job.yaml
file.
kubectl apply -f job.yaml
Step 3: Check the status of the job using kubectl.
kubectl get jobs
Step 4: You can get the list of pods using kubectl.
kubectl get po
Step 5: You can get the job pod logs using kubectl. Replace the pod name with the pod name you see in the output.
kubectl logs kubernetes-job-example-bc7s9 -f
You should see an output as shown below.
Multiple Job Pods and Parallelism
When a job is deployed you can make it run on multiple pods with parallelism.
For example, in a job if you want to run 6 pods and run 2 pods in parallel, you need to add the following two parameters to your job manifest.
completions: 6
parallelism: 2
Here is the manifest file with those parameters.
apiVersion: batch/v1
kind: Job
metadata:
name: kubernetes-parallel-job
labels:
jobgroup: jobexample
spec:
completions: 5
parallelism: 2
template:
metadata:
name: kubernetes-parallel-job
labels:
jobgroup: jobexample
spec:
containers:
- name: c
image: devopscube/kubernetes-job-demo:latest
args: ["100"]
restartPolicy: OnFailure
Generate Random Name for Kubernetes Job
You cannot have a single job manifest file and create multiple jobs from it. Kubernetes will throw an error stating that a job with the same name exists.
To circumvent this problem, you can add the generateName
name parameter to the metadata.
For example,
apiVersion: batch/v1
kind: Job
metadata:
generateName: kube-job-
labels:
jobgroup: jobexample
In the above example, every time you run the manifest, a job will get created with kube-job-
as a prefix followed by a random string.
How to Create a Kubernetes CronJob
What if you want to run a batch job on specific schedules, for example, every 2 hours. You can create a Kubernetes cronjob with a cron expression. The job will automatically kick in as per the schedule you mention in the job.
Here is how we specify a cron schedule. You can use the crontab generator to generate your own schedule.
schedule: "0,15,30,45 * * * *"
The following image shows the kubernetes cronjob scheduling syntax.
If we were to run our previous job as a cronjob every 15 minutes, here is how the manifest looks. Create a file named cron-job.yaml
and copy the following manifest.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: kubernetes-cron-job
spec:
schedule: "0,15,30,45 * * * *"
jobTemplate:
spec:
template:
metadata:
labels:
app: cron-batch-job
spec:
restartPolicy: OnFailure
containers:
- name: kube-cron-job
image: devopscube/kubernetes-job-demo:latest
args: ["100"]
Let’s deploy the cronjob using kubectl.
kubectl create -f cron-job.yaml
List the cronjobs
kubectl get cronjobs
To check Cronjob logs, you can list down the cronjob pod and get the logs from the pods in running state or from the finished pods.
Run a Kubernetes CronJob Manually
There are situations where you might want to execute the cronjob in an ad-hoc manner. You can do this by creating a job from an existing cronjob.
For example, if you want a cronjob to be triggered manually, here is what we should do.
kubectl create job --from=cronjob/kubernetes-cron-job manual-cron-job
--from=cronjob/kubernetes-cron-job
will copy the cronjob template and creates a job named manual-cron-job
Few Key Kubernetes Job Parameters
There are a few more key parameters you can use with kubernetes jobs/cronjobs based on your needs. Let’s have a look at each.
- failedJobHistoryLimit & successfulJobsHistoryLimit: Deletes the failed and successful job history based on the retention number you provide. This is super useful to trim down all failed entries when you try to list the jobs. For example,
failedJobHistoryLimit: 5 successfulJobsHistoryLimit: 10
- backoffLimit: Total number of retries if your pod fails.
- activeDeadlineSeconds: You can use this parameter if you want to specify a hard limit on how the time the cronjob runs. For example, if you want to run your cronjob only for one minute, you can set this to 60.
devopscube
Other Interesting Blogs
Helm Tutorial: How To Install and Configure Helm
This post explains how to install helm 3 on kubernetes and configure components for managing and deploying applications on the Kubernetes cluster.
List of the best programming languages to learn in 2017
Programming Languages… AH! There are so many!! And it is really confusing to figure out which language you should learn. Should you
Kubernetes Certification Coupon: 15% Off + $100 Off On Kubernetes Course bundle
The cloud-native foundation has two Kubernetes certifications. It is one of the most preferred certifications for organizations due to the wide adoption of
Comments
I think your job.yaml file is not proper. There’s a syntax error
“`
apiVersion: batch/v1
kind: Job metadata:
name: kubernetes-job-example
“`
Thanks for the update. We have corrected the YAML.