GlusterFS

Installation

1	`aptitude install glusterfs-server`

1	`aptitude install glusterfs-client`

Start the GlusterFS daemon.

1	`service glusterd start`

Firewall

Before starting to set up GlusterFS, access should be allowed among all participating systems where the GlusterFS server is running¹.

Peer

A peer is a server used by GlusterFS.

Add peer

1	`gluster peer probe [server ip or domain]`

Hint

The command can be executed on any server who runs GlusterFS.

Show peer status

Shows the current status of all peers.

1	`gluster peer status`

Remove peer

1	`gluster peer detach [server ip or domain]`

List peers

1	`gluster pool list`

Volume

A volume is a virtual collection of multiple bricks.

Add volume

1	`gluster volume create [volume name] replica [number of peers] [server IP or FQDN]:[server directory path] force`

Examples

1	`gluster volume create my-volume-1 replica 2 server-1.com:/path/to/my-volume-1 server-2.com:/path/to/my-volume-1 force`

1	`gluster volume create my-volume-1 replica 4 server-1.com:/path/to/my-volume-2 server-2.com:/path/to/my-volume-2 server-3.com:/path/to/my-volume-3 server-4.com:/path/to/my-volume-4 force`

Number of peers

The number of peers have to be lower or equal like the amount of servers, but must be higher than 1 to provide redundancy.

Start volume

After a volume is created, it must be started before it can be used.

1	`gluster volume start [volume name]`

Show volume status

Shows the current status of all volumes.

1	`gluster volume status`

Shows the current status the given volume.

1	`gluster volume status [volume name]`

Show volume info

1	`gluster volume info`

Volume details

To get more detailed information about a specific volume run the following two commands.

1	`gluster volume profile [volume name] start`

1	`gluster volume profile [volume name] info`

List volumes

1	`gluster volume list`

Access

Allow the volume access only for a specific IP or FQDN.

1	`gluster volume set [volume name] auth.allow [server IP or FQDN]`

Allow the volume access for multiple specific IP's or FQDN's.

1	`gluster volume set [volume name] auth.allow [server-1 IP or FQDN],[server-2 IP or FQDN]`

Allow the access for everyone.

1	`gluster volume set [volume name] auth.allow *`

Allow the access for everyone on encrypted connections.

1	`gluster volume set [volume name] auth.ssl-allow *`

Volume restart

A volume restart may be required for the changes to take effect.

Delete volume

The volume must be stopped first.

1	`gluster volume stop [volume name]`

1	`gluster volume delete [volume name]`

Brick

A brick is a directory where the volume data is stored.

Add brick

1	`gluster volume add-brick [volume name] replica [number of peers] [server IP or FQDN]:[server directory path] force`

Add brick to an existing volume.

1	`gluster volume add-brick [volume name] [server IP or FQDN]:[server directory path] force`

Remove brick

1	`gluster volume remove-brick [volume name] replica [number of peers] [server IP or FQDN]:[server directory path] force`

Adjust replicas

To adjust the replicas when expanding or reducing the cluster, the new number of peers can be specified when adding or removing the Brick.

Encryption

Between server

To use encryption between the servers, a private key and certificate are needed. The private key must be stored like /etc/ssl/glusterfs.key and the certificate must be stored like /etc/ssl/glusterfs.pem.

Private key and certificate

See TLS to create a private key and certificate.

Additional a third file is needed to combine the certificates of all servers and must be stored like /etc/ssl/glusterfs.ca. The order of the certificates does not matter.

To enable encryption between the servers the following file must be created, and the daemon restarted.

1	`touch /var/lib/glusterd/secure-access`

1	`service glusterd restart`

To clients

Server instructions

To encrypt the client communication all instructions for the servers must be executed too.

Enable client and server side encryption for a volume. Both commands must be executed, because they are only working together.

1	`gluster volume set [volume name] client.ssl on`

1	`gluster volume set [volume name] server.ssl on`

Volume restart

The volume must be stopped and started afterwards, otherwise it cannot be mounted².

Heal

Volume

Enter the following command to get healing information about the volume.

1	`gluster volume heal [volume name] info`

If everything is OK, an output similar to this one will be displayed.

Brick [server-1 IP or FQDN]:/path/to/brick
Status: Connected
Number of entries: 0

Brick [server-2 IP or FQDN]:/path/to/brick
Status: Connected
Number of entries: 0

A possible problem would look like this³.

1 2	`[volume name]: Not able to fetch volfile from glusterd Volume heal failed.`

Log

The logfiles are located at /var/log/glusterfs.

Name	File	Description
Glusterd	`glusterd.log`	One glusterd log file per server. This log file also contains the snapshot and user logs.
Gluster cli command	`cli.log`	Gluster commands executed on a node in a GlusterFS.
Bricks	`bricks/[brick path].log`	One log file per brick on the server.
Rebalance	`[volume]-rebalance.log`	One log file per volume on the server.
Self heal deamon	`glustershd.log`	One log file per server.
Quota	`quotad.log`	The quota daemons running on each node.

Client

Mount

1	`mount -t glusterfs server:/path/to/volume /path/to/mount-point`

The port 24007 for the GlusterFS daemon, as well as for each created volume, must be released a port ascending from 49152. The first volume uses port 49152, the second volume uses port 49153, the third volume 49154, and so on. ↩
The volume heal command could be display an error like at the volume section under Heal. ↩
This problem could mean that encryption was enabled, but the volume was not restarted. See Client. ↩