GlusterFS is one of the best open source distributed file systems. If you want a highly available distributed file system for your applications, GlusterFs is one of the good options.
Now: AWS offers a managed scalable file storage called Elastic File System. If you don’t want the administrative overhead of glusterFS clusters, you can give EFS a try.
GlusterFS cluster on AWS ec2
This guide covers all the necessary steps to setup a GlusterFs cluster using ec2 instances and extra EBS volumes. Here we are setting up a two node cluster, however, you can increase the node count based on your needs.
Instance Configurations
1. Create two instances with extra EBS volumes.
2. Make sure the instances can talk to each other.
3. Login to the servers and set the hostnames as node1 and node2 using the following commands.
hostnamectl set-hostname node1 hostnamectl set-hostname node2
Run bash on the terminal for the hostname to show up.
4. Make an entry in the /etc/hosts file with the IP and hostname of both servers as shown below. The IP’s should be resolvable using the hostnames. Change the IP in the following to your server IP’s.
172.31.21.201 node1 172.31.21.202 node2
Instance Port Configuration
You need to open the following ports int he ec2 security groups as well the server firewall if it is enabled.
111 24007 -GlusterFS Daemon.24008 -GlusterFS Management38465 to 38467 -GlusterFS NFS service49152 to n -Depends on number of bricks.
Create Mount Points for EBS Volumes
You need to do the following in both the ec2 instances. 1. Format the volume to xfs.
sudo mkfs -t xfs /dev/xvdb
xvdb is the name of the EBS volume. You can list the available devices using lsblk command.
2. Create a mount directory named /gshare and mount the formatted volume.
sudo mkdir /gshare sudo mount /share /dev/xvdb
3. Add the mount to /etc/fstab
/dev/xvdb /gshare xfs defaults,nofail 0
GlusterFS Installation
Perform the following steps on both the servers.
1. Create a GlusterFs repo.
sudo vi /etc/yum.repos.d/Gluster.repo
Copy the following to the repo file.
[gluster38] name=Gluster 3.8 baseurl=http://mirror.centos.org/centos/7/storage/$basearch/gluster-3.8/ gpgcheck=0 enabled=1
2. Install GlusterFS server.
sudo yum install glusterfs-server -y
3. Start and verify the glusterd service.
sudo systemctl start glusterd sudo systemctl status glusterd
GlusterFs Configuration
1. From node1 execute the following command to create a trusted storage pool with node2.
sudo gluster peer probe node2
After successful execution, you would get peer probe: success. as output.
2. Check the peer status using the following command.
[ec2-user@node1 ~]$ sudo gluster peer status Number of Peers: 1 Hostname: node2 Uuid: 47ee8304-36ea-4b95-9214-4854bc98b737 State: Peer in Cluster (Connected) [ec2-user@node1 ~]$
3. Create a data directory on gshare mount on both the servers.
sudo mkdir /gshare/data
4. Create a GlusterFS HA shared volume.
sudo gluster volume create gdata replica 2 node1:/gshare/data node2:/gshare/data
5. Start the gdata volume.
sudo gluster volume start gdata
6. By default NFS is disabled. If you want NFS functionality for glusterFs volume, you can enable it using the following command.
sudo gluster volume set gdata nfs.disable off
7. Set the volume permissions for gdata volume for client access. Here I am using 172.* CIDR. You need to replace it based on your network range.
sudo gluster volume set gdata auth.allow "172.*"
7. To get all the info about the volume, execute the following command.
sudo gluster volume info gdata
GlusterFS Client Setup
1. Enable fuse kernel module.
sudo modprobe fuse
2. Install all the glusterFS client dependencies.
sudo yum install fuse fuse-libs openib libibverbs -y
3. Install the GlusterFS client.
sudo yum install glusterfs-client -y
GlusterFS Client Configuration
The data will get replicated only if you are writing from a GlusterFS client. You can mount the GlusterFS volume to any number of clients. We highly recommend you to map the gluster nodes to a domain name and use it with the clients for mounting.
Note A client machine is not part of the glusterFS cluster. It is the machine in which you want to mount the replicated volume.
1. Create a client mount directory.
sudo mkdir /gfdata
2. Mount gfdata directory to the glusterFS replicated volume.
sudo mount -t glusterfs node1:/gdata /gfdata
Troubleshooting GlusterFs
1. You can view all the logs of gluserFS server on the following directory.
/var/log/glusterfs
To monitor logs in real time you can use tail -f along with the path to log file.
2. All glusterfs client logs are saved in the following with the volume name.
/var/log/glusterfs/
