How to Learn Linux Shell Scripting for DevOps?

For aspiring DevOps engineers, it is essential to know shell Scripting or bash scripting. In this shell scripting for DevOps guide, I will share my tips & resources to learn Linux shell scripting the right way.

I have spoken about the importance of shell scripting in my becoming a DevOps engineers guide as well. This guide is for anyone who wants to learn shell scripting the right way that can help you with your day-to-day work and DevOps interviews. It’s very important to have patience and discipline in the learning process.

If you are someone who wants to just get things done, then Google & Stackoverflow is more than enough 🙂

Here is what we are going to look at in this blog.

Shell Scripting for DevOps

The first question you might have is, how important is shell scripting for a DevOps engineer? I have got this question from freshers as well experienced IT folks.

The answer is, it’s very important. The following image shows the Stackoverflow 2021 survey, where 27% of the respondents said they use shell scripting.

But most of the time, I get asked, saying that there are many automation tools out there and do we still need to learn and write shell scripts. The answer is yes. We no more create shell scripts for full-fledged automation but we use scripting along with automation tools and for ad-hoc tasks.

For example, if you use the AWS user data option, there is a chance you will use a shell script inside it. Another example is, while creating AMI images using a packer, you will end up using a shell script for the AMI configuration. Also, I have had instances where I had to use shell scripts with configuration management tools, containers, etc. We will talk more about this in the following sections.

Also, shell script comes in handy for repeatable development tasks. For example, it could be spinning up a Vagrant VM with the essential software or setting up the development environment itself.

Most importantly, hands-on scripting and programming are becoming mandatory in preliminary interview rounds in DevOps interviews. So it is another crucial reason why you should learn shell scripting.

How to Get Started With Shell Scripting?

The prerequisite to getting started with shell scripting is hands-on experience with Linux. Therefore, before getting started with shell scripting, you should be comfortable working with Linux commands.

If you are totally new to the Linux world, you could start by spinning up a few Linux servers locally using Vagrant or on cloud platforms like AWS, Google Cloud, etc. Every platform offers free credits for beginners. You can check out the free $100 credit from the digital ocean as well. It’s effortless to get started.

If you think you are comfortable with Linux, you can get started with Shell scripting basic tutorials. Also, I want you to do the following.

Create a Github repo, create folders for each concept you learn, and commit all your work scripts. It doesn’t matter if the script is available online. Have your own documentation. You can add references and credit websites/blog links in the README section. Once your learning is over, you can share it on Linkedin. Trust me; it will help you retain the knowledge and will help others as well.

What are the Best Free Shell Scripting Learning Resources?

You don’t need to pay for any expensive courses to learn Linux shell scripting. There is too much marketing around courses that promises to make you a DevOps Shell scripting ninja. However, I don’t have anything against courses; go for it if you feel comfortable learning via a paid course. I will add those resources as well.

I have listed down the best free shell scripting resources for you to get started. Go through all the resources once, pick one resource you feel comfortable with, and ensure you go through the material from start to finish. As I said earlier, mastering technology requires patience and discipline.

Follow the free resources. It has web-based content, web-based interactive tutorials, free courses, and pdf materials.

1. Linux Shell Scripting Tutorial [Web]
2. Free interactive shell scripting tutorials [Web]
3. Shell Scripting tutorial [Web]
4. Bash Guide [web]
5. Shell Scripting Free Course [Udemy]
6. Advanced Bash Scripting Guide [PDF]
7. Bash Academy [Web]
8. Bash Notes for Professionals [PDF]
9. Bash Reference manual [PDF]
10. The Linux command line [PDF]

But if you say, hey, I am a person who likes guided courses, here are some of my recommendations.

1. The Complete Guide to Bash Programming – Educative
2. Shell Scripts for Beginners – KodeKloud
3. Shell scripting with Bash – Pluralsight
4. Operationalizing Bash and Z Shell Scripts – Pluralsight
5. Linux Shell Scripting: A Project-Based Approach to Learning

Real-world Shell Scripting Use Cases

Assuming you have learned all the shell scripting concepts and probably written and executed shell scrips for learning purposes. The next question anyone would have is, how to learn real-world shell scripting use cases?

If you work in a company, you can find project documentation and infrastructure automation codes where existing projects use shell scripts.

If you don’t have access to project documents and code, here is what you can do.

Find the GitHub repo of the most common docker base images, For example, the Nginx base image. 

There you will find an entrypoint folder with multiple scripts or an entrypoint.sh script file. Here is an nginx example

Try to understand the shell script and its functionality. In this process, you will figure out many concepts and logic that can be used while writing a shell script. (functions, conditionals, switch cases, and many more)

It is not just limited to Docker repositories, you can find scripts in many open-source community repositories.

Shell Scripting Real-Time Scenarios

It is always better to learn with some real-time scenarios in terms of DevOps engineer interviews. So I have come up with some real-world scenarios you can try out. I will keep updating this list.

1. Write a shell script function to find and kill all the zombie processes.
2. Find the first 10 biggest files in the file system and write the output to a file.
3. Shell script to gracefully unmount a disk.
4. shell script to send email
5. Shell script to monitor CPU, Memory, and Disk usage and send the output to a file in table format and send an alert if either of them exceeds a certain threshold.
6. Shell script to find the files created and their sizes. It should accept the number of days as input. Or a from and to date format as inputs.
7. Write a shell script to automate the process of creating new user accounts on a Linux server and setting up their permissions and SSH access.
8. Write a shell script to the list of users logged in by date and write it to an output file.
9. Shell script to Copy files recursively to remote hosts
10. Shell script that displays the number of failed login attempts by IP address and location.
11. Shell script parses a log file and forwards a specific value with a timestamp to an output file.
12. Write a shell script to automate the process of rotating log files and compressing old files to save disk space.
13. Write a shell script to check the status of a list of URLs and send an email notification if any of them are down.
14. Write a shell script to automate the process of updating a list of servers with the latest security patches.

Ensure you include the following concepts in the shell scripts you are writing.

1. variable definitions
2. Use cut, awk, and grep
3. Handling Input/output /error redirections.
4. Conditions/If else statements
5. Case statement scripts
6. Loops (For/Do-While)
7. Exist status
8. Command line arguments

Shell Scripting Languages

First of all, there are no specific shell scripting languages. When it comes to Shell scripting, it refers to sh, bash, csh, tcsh. Here we are primarily talking about bash shell scripting. Bash is an interpreted scripting language for Unix-based systems.

A shell script is a computer program designed to be run by the Unix shell, a command-line interpreter.

Wikipedia

But scripting languages, in general, means multiple languages. The following are the common scripting languages.

1. Powershell
2. Python
3. Pearl
4. Groovy

Shell Scripting DevOps Interview Questions

Shell scripting DevOps interview questions differ from company to company.

For example, a Service based company would be just interested in your basic Linux and shell scripting knowledge. However, a product-based company might expect a good level of knowledge of Linux command line and shell scripting.

Nowadays, most companies would provide platforms like Hackerank to test your scripting skills. So it is necessary to practice shell scripting to solve problems.

Following are some of the DevOps interview shell script questions.

1. Can you explain how shell scripting fits into a larger DevOps workflow?
2. What is the need for shell scripts when there are automation tools to do the work?
3. In what scenarios would you opt for shell script rather than programming languages like Python or Golang?
4. How would you approach writing a shell script to automate a particular task?
5. How to do static analysis for Shell script?
6. How do you ensure that shell scripts are error-free in your CI/CD pipeline?
7. How can you handle errors and exceptions in shell scripts?
8. Shell script to reverse a string
9. Parse a log in the desired format and find all occurrences of a word
10. Shell script to send the first ten lines of a file to another file.
11. Merge(concatenate) two files and write the output to the third file.
12. Find all duplicate strings in a file and replace them with another string
13. Find all the IP addresses from a log file and write it to another file.
14. How would you debug a shell script that isn’t working correctly?
15. What is the difference between a “for” loop and a “while” loop in shell scripting?

I will be updating the list with more generic interview questions soon.

Shell Scripting FAQs

Following are some of the frequently asked shell scripting questions by aspiring DevOps engineers.

Conclusion

I hope you enjoyed this guide on learning Shell scripting for DevOps engineers.

Now I’d like to hear from you: Which method or resource from today’s guide are you going to try first?

Are you going to start with the Linux basics? Or start focusing on interview-based questions

Let me know by leaving a quick comment below right now.