Consul is a cluster management tool from Hashicorp and it is very useful for creating advanced micro-services architecture. Consul is a distributed configuration system which provides high availability, multi-data center, service discovery and High fault tolerance. So managing micro-services with Consul is pretty easy and simple.
Current micro-service architecture based infrastructure has following challenges
Uncertain service locations
Load balancing between multiple data-centers
Since Consul is distributed and agent-based, It could solve all the above challenges easily.
Consul Technology and Architecture
Consul is an agent-based tool, which means it should be installed in each and every node of the cluster with servers and a client agent nodes. Hashicorp provides an open source binary package for installing Consul, and it can be downloaded from this (https://www.consul.io/downloads.html).
To install Consul to all the nodes, we need to download the binary file and keep it in the bin folder (/etc/local/bin), so that we can run this from anywhere within the node.
Consul needs to be started as a process and it will continuously shares information. For this, we should start the agent on all the nodes and connect each other for communicating each other.
Communication between nodes will be done through gossip protocol, Which means each node will send some data to other nodes like a virus and eventually to others.
Before going to the demonstration, I would like to explain about the architecture of this tool. So basically agent will be started as servers within the nodes where services are running. and a client agent can be used for UI and query the information about the cluster of server. It is not necessary that client can not have the service within it.
To start the agent as a server, we need to mention server as a parameter.
Consul exposes the information about the instance through API and because of this consul can be used for other infrastructural application, example dashboard, Monitoring tool or our own event management system.
Service discovery is an another great feature of consul. For our infrastructure services, we need to create separate service configuration file in JSON format for consul. service configuration file should be kept inside the consul.d configuration folder for getting identified by consul agent. so we need to create consul.d inside /etc/ folder.
Later when we are starting our agent, we can see our services which are mentioned inside consul.d folder also synced with the consul agent.
[email protected]:~$ consul agent -server -config-dir /etc/consul.d
==> Starting Consul agent...
[INFO] agent: Synced service 'nginx'
Which mean the service can be communicating with consul agent. So the availability of a node and health of the node can be shared across the cluster.
We can query the service using either DNS or HTTP API. If we are using DNS, then we need to use dig for query and DNS name will be like NODE_NAME.service.consul. If we are having multiple services with the same application, we can separate it with tags. And its DNS will be like TAG.NODE_NAME.service.consul. Since we have an internal DNS name within the cluster, we can manage DNS issue which usually occurs while load balancer fails.
[email protected]:~$ dig @127.0.0.1 -p 8600 web.service.consul
;; QUESTION SECTION:
;web.service.consul. IN A
;; ANSWER SECTION:
web.service.consul. 0 IN A 172.20.20.11
If we use HTTP API for querying the service then it will be like
So here we could see how it can be helpful for service discovery right..?
Just like Service discovery, Health checking of nodes is also taken care by this consul. Consul could expose the status of the node so that we could easily find the solution for failure detection among the nodes. For this example, i am manually crashing the server
Converting your application into microservices is not a big deal. Making it as a scalable application is always a challenging thing. This challenges can be solved if we can able to combine tools like Consul, serf, messaging queue tools together. This will make your microservices scalable, Fault tolerant and highly available for Zero downtime application.
GlusterFS is one of the best open source distributed file systems. If you want a highly available distributed file system for your applications, GlusterFs is one of the good options.
Now: AWS offers a managed scalable file storage called Elastic File System. If you don’t want the administrative overhead of glusterFS clusters, you can give EFS a try.
GlusterFS cluster on AWS ec2
This guide covers all the necessary steps to setup a GlusterFs cluster using ec2 instances and extra EBS volumes. Here we are setting up a two node cluster, however, you can increase the node count based on your needs.
1. Create two instances with extra EBS volumes.
2. Make sure the instances can talk to each other.
3. Login to the servers and set the hostnames as node1 and node2 using the following commands.
The data will get replicated only if you are writing from a GlusterFS client. You can mount the GlusterFS volume to any number of clients. We highly recommend you to map the gluster nodes to a domain name and use it with the clients for mounting.
Note A client machine is not part of the glusterFS cluster. It is the machine in which you want to mount the replicated volume.
1. Create a client mount directory.
sudo mkdir /gfdata
2. Mount gfdata directory to the glusterFS replicated volume.
sudo mount -t glusterfs node1:/gdata /gfdata
1. You can view all the logs of gluserFS server on the following directory.
To monitor logs in real time you can use tail -f along with the path to log file.
2. All glusterfs client logs are saved in the following with the volume name.
In our last blog post, we have covered Linux CLI Productivity tips. When you work on Linux systems, vi editor is something which we use very often for editing files. It is not like using a GUI editor and people using it for the first time gets intimidated by it as you have to use various keystrokes for controlling the editing. However with a little bit of practice and using the vi editor Shortcuts in our day to day activities will save your time, increases the Productivity and you will start loving the VI editor. Moreover, it is a very powerful test editor in the Linux ecosystem.
VI Editor Shortcuts For Beginners
In this article, we will cover the necessary shortcuts that you could use in day to day Linux activities that include the vi editor.
Note: Most of the commands explained in this tutorial works in normal mode. (Press ESC to make sure you are in Normal mode before executing commands)
Setting up VI environment
Before diving into shortcuts and commands, you must understand the vi editor settings. You can set all the necessary vi editor parameters in the ~/.vimrc file that will be loaded by default. If you don’t have that file, create one using touch ~/.vimrc
Following are the common parameters that you might need in the vimrc file.
The set number parameter will set line number for your vi editor. You can unset this temporarily by running :set nonumber from the editor.
Cut Copy Paste
1. press ESC and click v and move the cursor to select the string you want to copy. Use capital V to select the whole line.
2. press y to copy.
3. Place the cursor in the desired location and press p to paste.
1. Place the cursor on the line you want to delete and press dd to delete that line
2. To delete a specific number of lines, you can use d10d. This will delete 10 lines starting from the cursor. You can give any number in place of 10 based on your needs.
3. To select and delete specific line, use Shift + V and then use up and down arrows for the selection. Once you selected the lines, press d for deleting all the selected lines.
In insert mode, you can do the following.
1. Ctrl + w will delete the last word where the cursor is in.
3. Ctrl + u will delete the all the words which are immediate to left of the cursor.
Searching and Replacing Texts
You can search through your files by pressing /.
1. For example, if you want to search a keyword data, you should do the following.
To find next occurrence, just press n
Note: Searches are case sensitive. If you want case-insensitive search, you should set :set ignorecase in the editor.
2. Use the following syntax for replacing a pattern.
3. Use the following syntax for replacing every occurrence with a prompt. It will highlight all the occurrences.
Convert to Upper and Lowercase
For any case conversions, keep the cursor on that line and use the following shortcuts.
1. To convert a line to uppercase use gUU
2. To convert a line to Lowercase, use guu
Copying Contents From Another File
This not something that you do very often. However, if you need to add the contents of another file to the file you are editing, place the cursor on the desired line and you can use the following. The contents of the specified file will be copied from the next line of the cursor.
Fo example, if you to copy CPU info to an existing file, you would use the following command.
Executing/Copying Contents From Command
1. To execute commands from the editor, you can use :! <command>. For example,
2. If you want the output of a command to be copied to the editing file, you can use :r! <command>. For example,to get and copy eth0 IP address, you could do the following.
At any point in time, if you need help on the vi editor, you can run vimtutor command to open the command line tutorial in your terminal.
You can use all the normal vi commands to browse through the help document.
VI is a very powerful editor. Understanding the full functionality takes time and constant practice. We have explained some commands, tips and vi editor shortcuts that will save some time while working with the vi editor. If you think you have some tip, please share it with us in the comment section. It could help others.