Disk encryption

Implement the previous work done in HekaFS into Gluster. This allows a volume (or per-tenant part of a volume) to be encrypted “at rest” on the server using keys only available on the client. [Note: We encrypt only content of regular files. File names are not encrypted! Also, encryption does not work in NFS mounts.]

Setting

# gluster volume info
Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2

# gluster volume set vol2 encryption on
volume set: success

# gluster volume info
Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
Options Reconfigured:
features.encryption: on

# for x in quick-read write-behind open-behind; do gluster volume set vol2 performance.$x off; done
volume set: success
volume set: success
volume set: success

# gluster volume set vol2 encryption.master-key /var/lib/glusterd/vols/vol2/encryption.master-key
volume set: success

# openssl rand -hex 32 > /var/lib/glusterd/vols/vol2/encryption.master-key

# gluster volume set vol2 encryption.data-key-size 512
volume set: success

# gluster volume info
Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
Options Reconfigured:
encryption.data-key-size: 512
encryption.master-key: /var/lib/glusterd/vols/vol2/encryption.master-key
performance.open-behind: off
performance.write-behind: off
performance.quick-read: off
features.encryption: on

# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/encryption.master-key localhost:/vol2 /mnt/glusterfs/vol2

Encryption

# echo "test" > /mnt/glusterfs/vol2/test.txt
# cat /mnt/glusterfs/vol2/test.txt
test

[eins]# cat /mnt/lv2/vol2/test.txt
Zd??]K!q??tuv
[zwei]# cat /mnt/lv2/vol2/test.txt
Zd??]K!q??tuv

# dd if=/dev/zero of=/mnt/glusterfs/vol1/test.dat bs=1 count=32
# dd if=/dev/zero of=/mnt/glusterfs/vol2/test.dat bs=1 count=32

[eins]# dd if=/dev/zero of=/tmp/test.dat bs=1 count=32
[eins]# diff /tmp/test.dat /mnt/lv2/vol2/test.dat
Binary files /tmp/test.dat and /mnt/lv2/vol2/test.dat differ

[eins]# diff /tmp/test.dat /mnt/lv1/vol1/test.dat
#

Performance

# dd if=/dev/zero of=/mnt/glusterfs/vol1/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 18.4542 s, 58.2 MB/s
# dd if=/dev/zero of=/mnt/glusterfs/vol2/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 263.633 s, 4.1 MB/s

Decryption

# dd if=/dev/zero of=/tmp/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.61505 s, 297 MB/s

# diff3 /tmp/1gb.dat /mnt/glusterfs/vol1/1gb.dat /mnt/glusterfs/vol2/1gb.dat
#

Packet

# tcpdump -i eth0 -XX

NFS

# mount -t nfs -o vers=3,hard,intr,nosuid localhost:/vol2 /mnt/nfs/vol2
mount.nfs: Connection timed out

Attack

the same master-key

# cp /var/lib/glusterd/vols/vol2/encryption.master-key /tmp
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/tmp/encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt
# diff /mnt/glusterfs/vol-crypt/test.txt /tmp/test.txt
#

make the MK invalid when mounting

# openssl rand -hex 32 > /tmp/encryption.master-key
# diff /mnt/glusterfs/vol-crypt/test.txt /tmp/test.txt
#

invalid master-key

# umount /mnt/glusterfs/vol-crypt
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/tmp/encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt
# diff /mnt/glusterfs/vol-crypt/test.txt /tmp/test.txt
diff: /mnt/glusterfs/vol-crypt/test.txt: Invalid argument

# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root   32 May 18 22:57 test.dat
-rw-r--r-- 1 root root    5 May 18 22:55 test.txt

# cp /mnt/glusterfs/vol-crypt/test.txt ~/
cp: reading `/mnt/glusterfs/vol-crypt/test.txt': Invalid argument

# ls -l ~/test.txt
-rw-r--r-- 1 root root 0 May 19 00:38 /root/test.txt

# echo "test2" > /mnt/glusterfs/vol-crypt/test2.txt
# cat /mnt/glusterfs/vol-crypt/test2.txt
test2

# diff /mnt/glusterfs/vol-crypt/test2.txt /tmp/test2.txt
#


# \rm /mnt/glusterfs/vol-crypt/test.txt
mv: reading `/mnt/glusterfs/vol-crypt/test.txt': Invalid argument
# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root    6 May 19 00:39 test2.txt
-rw-r--r-- 1 root root   32 May 18 22:57 test.dat
-rw-r--r-- 1 root root    5 May 18 22:55 test.txt
# \rm /mnt/glusterfs/vol-crypt/test2.txt
# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root   32 May 18 22:57 test.dat
-rw-r--r-- 1 root root    5 May 18 22:55 test.txt


# mv /mnt/glusterfs/vol-crypt/test.txt /mnt/glusterfs/vol-crypt/test2.txt
mv: cannot move `/mnt/glusterfs/vol-crypt/test.txt' to a subdirectory of itself, `/mnt/glusterfs/vol-crypt/test2.txt'


[2014-05-19 0:44:25]# umount /mnt/glusterfs/vol-crypt
[2014-05-19 0:44:31]# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt
[2014-05-19 0:44:58]# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root    6 May 19 00:44 test2.txt
-rw-r--r-- 1 root root   32 May 18 22:57 test.dat
-rw-r--r-- 1 root root    5 May 18 22:55 test.txt
[2014-05-19 0:45:12]# cat /mnt/glusterfs/vol-crypt/test2.txt
cat: /mnt/glusterfs/vol-crypt/test2.txt: Invalid argument
[2014-05-19 0:45:22]# rm /mnt/glusterfs/vol-crypt/test2.txt
mv: reading `/mnt/glusterfs/vol-crypt/test2.txt': Invalid argument
[2014-05-19 0:45:31]# \rm /mnt/glusterfs/vol-crypt/test2.txt
rm: cannot remove `/mnt/glusterfs/vol-crypt/test2.txt': Invalid argument

volume reset

# gluster volume info vol2

Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
Options Reconfigured:
encryption.data-key-size: 512
encryption.master-key: /var/lib/glusterd/vols/vol2/encryption.master-key
performance.open-behind: off
performance.write-behind: off
performance.quick-read: off
features.encryption: on

# gluster volume reset vol2
volume reset: success: reset volume successful

# gluster volume info vol2

Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2

# cat /mnt/glusterfs/vol-crypt/test2.txt
U�%U?0��x^-�bO
# cat /mnt/glusterfs/vol-crypt/test.txt
Zd��]K!q�tuv

reconfigure

# gluster volume set vol2 encryption.master-key /var/lib/glusterd/vols/vol2/encryption.master-key
volume set: success
# gluster volume set vol2 encryption.data-key-size 512
volume set: success
# for x in quick-read write-behind open-behind; do gluster volume set vol2 performance.$x off; done
volume set: success
volume set: success
volume set: success

# cat /mnt/glusterfs/vol-crypt/test.txt
Zd��]K!q�tuv
# cat /mnt/glusterfs/vol-crypt/test2.txt
U�%U?0��x^-�bO

# umount /mnt/glusterfs/vol-crypt
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt

# cat /mnt/glusterfs/vol-crypt/test2.txt
U�%U?0��x^-�bO
# cat /mnt/glusterfs/vol-crypt/test.txt
Zd��]K!q�tuv

# echo "test" > /mnt/glusterfs/vol-crypt/test.txt
# cat /mnt/glusterfs/vol-crypt/test.txt
test
[eins]# cat /mnt/lv2/vol2/test.txt
test


[2014-05-19 0:55:24]# gluster volume stop vol2
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: vol2: success
[2014-05-19 0:56:30]# gluster volume start vol2
volume start: vol2: success
[2014-05-19 0:56:39]# gluster volume info vol2

Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
[2014-05-19 0:56:42]# umount /mnt/glusterfs/vol-crypt
[2014-05-19 0:56:54]# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt
[2014-05-19 0:56:57]# cat /mnt/glusterfs/vol-crypt/test.txt
test

recreate

[2014-05-19 1:00:53]# gluster volume stop vol2
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: vol2: success
[2014-05-19 1:01:07]# gluster volume delete vol2
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: vol2: success
[2014-05-19 1:01:18]# brick="/mnt/lv2/vol2"; gluster volume create vol2 replica 2 eins:$brick zwei:$brick
volume create: vol2: failed: Staging failed on zwei. Error: /mnt/lv2/vol2 is already part of a volume
Staging failed on eins. Error: /mnt/lv2/vol2 is already part of a volume

[eins]# \rm -rf /mnt/lv2/vol2
[zwei]# \rm -rf /mnt/lv2/vol2
# brick="/mnt/lv2/vol2"; gluster volume create vol2 replica 2 eins:$brick zwei:$brick root@sieben:/root
volume create: vol2: success: please start the volume to access data


[2014-05-19 1:07:27]# gluster volume stop vol2
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: vol2: success
[2014-05-19 1:07:48]# gluster volume start vol2
volume start: vol2: success
[2014-05-19 1:07:56]# gluster volume info vol2

[2014-05-19 1:10:19]# gluster volume info vol2

Volume Name: vol2
Type: Replicate
Volume ID: a698e3bd-a9ea-46ff-9560-c9f91e84155d
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
Options Reconfigured:
features.encryption: on
encryption.data-key-size: 512
encryption.master-key: /var/lib/glusterd/vols/vol2/encryption.master-key
performance.open-behind: off
performance.write-behind: off
performance.quick-read: off



[2014-05-19 1:14:24]# gluster volume stop vol2
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: vol2: success
[2014-05-19 1:15:37]# gluster volume delete vol2
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: vol2: success
[2014-05-19 1:15:43]# brick="/mnt/lv2/vol2"; gluster volume create vol2 replica 2 eins:$brick zwei:$brick root@sieben:/root
volume create: vol2: success: please start the volume to access data
[2014-05-19 1:16:44]# gluster volume start vol2
volume start: vol2: success
[2014-05-19 1:17:04]# gluster volume set vol2 encryption on
volume set: success
[2014-05-19 1:17:12]# for x in quick-read write-behind open-behind; do gluster volume set vol2 performance.$x off; done
volume set: success
volume set: success
volume set: success
[2014-05-19 1:17:25]# gluster volume set vol2 encryption.master-key /var/lib/glusterd/vols/vol2/encryption.master-key
volume set: success
[2014-05-19 1:17:32]# gluster volume set vol2 encryption.data-key-size 512
volume set: success
[2014-05-19 1:17:47]# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt
Mount failed. Please check the log file for more details.
[2014-05-19 1:17:59]# gluster volume info vol2

Volume Name: vol2
Type: Replicate
Volume ID: d0517697-5372-44a1-960f-6db0d988f3b2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
Options Reconfigured:
encryption.data-key-size: 512
encryption.master-key: /var/lib/glusterd/vols/vol2/encryption.master-key
performance.open-behind: off
performance.write-behind: off
performance.quick-read: off
features.encryption: on
[2014-05-19 1:18:35]# openssl rand -hex 32 > /var/lib/glusterd/vols/vol2/encryption.master-key
[2014-05-19 1:19:17]# gluster volume stop vol2; gluster volume start vol2
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: vol2: success
volume start: vol2: success
[2014-05-19 1:19:35]# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt