How to Setup a Replicated GlusterFS Cluster on AWS EC2

GlusterFS cluster on AWS ec2

GlusterFS  is one of the best open source distributed file systems. If you want a highly available distributed file system for your applications, GlusterFs is one of the good options.

Now: AWS offers a managed scalable file storage called Elastic File System. If you don’t want the administrative overhead of glusterFS clusters, you can give EFS a try.

GlusterFS cluster on AWS ec2

This guide covers all the necessary steps to setup a GlusterFs cluster using ec2 instances and extra EBS volumes. Here we are setting up a two node cluster, however, you can increase the node count based on your needs.

Instance Configurations

1. Create two instances with extra EBS volumes.

2. Make sure the instances can talk to each other.

3. Login to the servers and set the hostnames as node1 and node2 using the following commands.

hostnamectl set-hostname node1
hostnamectl set-hostname node2

Run bash on the terminal for the hostname to show up.

4. Make an entry in the /etc/hosts file with the IP and hostname of both servers as shown below. The IP’s should be resolvable using the hostnames. Change the IP in the following to your server IP’s.

172.31.21.201 node1
172.31.21.202 node2

Instance Port Configuration

You need to open the following ports int he ec2 security groups as well the server firewall if it is enabled.

111
24007 - GlusterFS Daemon.
24008 - GlusterFS Management
38465 to 38467 - GlusterFS NFS service
49152 to  n  - Depends on number of bricks.

Create Mount Points for EBS Volumes

You need to do the following in both the ec2 instances. 1. Format the volume to xfs.

sudo mkfs -t xfs /dev/xvdb

xvdb is the name of the EBS volume. You can list the available devices using lsblk command.

2. Create a mount directory named /gshare and mount the formated volume.

sudo mkdir /gshare
sudo mount /share /dev/xvdb

3. Add the mount to /etc/fstab

/dev/xvdb    /gshare   xfs   defaults,nofail     0

GlusterFS Installation

Perform the following steps on both the servers.

1. Create a GlusterFs repo.

sudo vi /etc/yum.repos.d/Gluster.repo

Copy the following to the repo file.

[gluster38]
name=Gluster 3.8
baseurl=http://mirror.centos.org/centos/7/storage/$basearch/gluster-3.8/
gpgcheck=0
enabled=1

2. Install GlusterFS server.

sudo yum install glusterfs-server -y

3. Start and verify the glusterd service.

sudo systemctl  start glusterd
sudo systemctl  status glusterd

GlusterFs Configuration

1. Form node1 execute the following command to create a trusted storage pool with node2.

sudo gluster peer probe node2

After successful execution, you would get peer probe: success. as output.

2. Check the peer status using the following command.

[[email protected] ~]$ sudo gluster peer status
Number of Peers: 1

Hostname: node2
Uuid: 47ee8304-36ea-4b95-9214-4854bc98b737
State: Peer in Cluster (Connected)
[[email protected] ~]$

3. Create a data directory on gshare mount on both the servers.

sudo mkdir /gshare/data

4. Create a GlusterFS HA shared volume.

sudo gluster volume create gdata replica 2 node1:/gshare/data node2:/gshare/data

5. Start the gdata volume.

sudo gluster volume start gdata

6. By default NFS is disabled. If you want NFS functionality for glusterFs volume, you can enable it using the following command.

sudo gluster volume set gdata nfs.disable off

7. Set the volume permissions for gdata volume for client access. Here I am using 172.* CIDR. You need to replace it based on your network range.

sudo gluster volume set gdata auth.allow "172.*"

7. To get all the info about the volume, execute the following command.

sudo gluster volume info gdata

GlusterFS Client Setup

1. Enable fuse kernel module.

sudo modprobe fuse

2. Install all the glusterFS client dependencies.

sudo yum install fuse fuse-libs openib libibverbs -y

3. Install the GlusterFS client.

sudo yum install  glusterfs-client -y

GlusterFS Client Configuration

The data will get replicated only if you are writing from a GlusterFS client. You can mount the GlusterFS volume to any number of clients. We highly recommend you to map the gluster nodes to a domain name and use it with the clients for mounting.

Note A client machine is not part of the glusterFS cluster. It is the machine in which you want to mount the replicated volume.

1. Create a client mount directory.

sudo mkdir /gfdata

2. Mount gfdata directory to the glusterFS replicated volume.

sudo mount -t glusterfs node1:/gdata /gfdata

Troubleshooting GlusterFs

1. You can view all the logs of gluserFS server on the following directory.

/var/log/glusterfs

To monitor logs in real time you can use tail -f along with the path to log file.
2. All glusterfs client logs are saved in the following with the volume name.

/var/log/glusterfs/

Free DevOps Resources

Get DevOps news, tutorials and resources in your inbox. A perfect way If you want to get started with devops. Like you, we dont like spam.

Leave a Reply

Your email address will not be published. Required fields are marked *