Posts tagged with: gateway

installing a NFS gateway on Sandbox

NFS gateway is a neat way to access HDFS without a HDFS client, HDFS would then be appears mounted on the local filesystem as any directory.

We have to start by saying NFS user to be able to inpersonate users which will access our cluster, so let’s add in HDFS/configs/custom core-site.xml hadoop.proxyuser.nfsserver.groups and hadoop.proxyuser.nfsserver.hosts

NFS proxyuser

 

Then we add on custom hdfs-site.xml our Kerberos credentials (of course, your Sandbox is kerberized, is it?)

NFS: Kerberos credentials

 

in the same custom hdfs-site.xml, add the following properties which will respectively indicates a spool temporary directory (to re-order sequential writes before writing to HDFS) and the access control policy (here anyone can read/write but you could use another policy represented by MACHINE_NAME RW_POLICY, the latest could be rw (read&write) or ro (read-only))

NFS: mount points

Of course we have to add principal and get keytab for our NFS gateway.
Notice I had to use dfs.nfs.keytab.file and dfs.nfs.kerberos.principal for nfs3 gateway to launch.

We have to launch portmap and nfs3 :

[root@sandbox ~]# hadoop-daemon.sh start portmap

[root@sandbox ~]# hadoop-daemon.sh start nfs3

and mount a new directory as the new mount point for accessing HDFS :

[root@sandbox ~]# mkdir -p /media/hdfs

[root@sandbox ~]# mount -t nfs -o vers=3,proto=tcp,nolock 10.0.2.15:/ /media/hdfs/

We can check NFS is functional :

[root@sandbox ~]# ls -l /media/hdfs/
total 5
drwxrwxrwx 3 yarn hadoop 96 2015-12-03 14:42 app-logs
drwxr-xr-x 5 hdfs hdfs 160 2015-04-24 15:11 apps
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 15:56 demo
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 14:53 hdp
drwxr-xr-x 3 mapred hdfs 96 2015-04-24 14:52 mapred
drwxrwxrwx 4 hdfs hdfs 128 2015-04-24 14:52 mr-history
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 15:41 ranger
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 14:57 system
drwxrwxrwx 14 hdfs hdfs 448 2015-12-14 15:24 tmp
drwxr-xr-x 11 hdfs hdfs 352 2015-04-24 15:33 user

[root@sandbox ~]# cp ./test01 /media/hdfs/tmp/
[root@sandbox ~]# ls -l /media/hdfs/tmp/
total 16
drwx------ 3 ambari-qa hdfs 96 2015-12-03 14:44 ambari-qa
drwx-wx-wx 6 ambari-qa hdfs 192 2015-04-24 15:32 hive
-rw-r--r-- 1 root hdfs 87 2015-12-14 16:24 test01
drwxrwxrwx 8 hdfs hdfs 256 2015-04-24 15:31 udfs
drwx------ 3 ambari-qa hdfs 96 2015-12-03 14:44 yarn

Perfect ! :)