Posts tagged with: mount

installing a NFS gateway on Sandbox

NFS gateway is a neat way to access HDFS without a HDFS client, HDFS would then be appears mounted on the local filesystem as any directory.

We have to start by saying NFS user to be able to inpersonate users which will access our cluster, so let’s add in HDFS/configs/custom core-site.xml hadoop.proxyuser.nfsserver.groups and hadoop.proxyuser.nfsserver.hosts

NFS proxyuser


Then we add on custom hdfs-site.xml our Kerberos credentials (of course, your Sandbox is kerberized, is it?)

NFS: Kerberos credentials


in the same custom hdfs-site.xml, add the following properties which will respectively indicates a spool temporary directory (to re-order sequential writes before writing to HDFS) and the access control policy (here anyone can read/write but you could use another policy represented by MACHINE_NAME RW_POLICY, the latest could be rw (read&write) or ro (read-only))

NFS: mount points

Of course we have to add principal and get keytab for our NFS gateway.
Notice I had to use dfs.nfs.keytab.file and dfs.nfs.kerberos.principal for nfs3 gateway to launch.

We have to launch portmap and nfs3 :

[root@sandbox ~]# start portmap

[root@sandbox ~]# start nfs3

and mount a new directory as the new mount point for accessing HDFS :

[root@sandbox ~]# mkdir -p /media/hdfs

[root@sandbox ~]# mount -t nfs -o vers=3,proto=tcp,nolock /media/hdfs/

We can check NFS is functional :

[root@sandbox ~]# ls -l /media/hdfs/
total 5
drwxrwxrwx 3 yarn hadoop 96 2015-12-03 14:42 app-logs
drwxr-xr-x 5 hdfs hdfs 160 2015-04-24 15:11 apps
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 15:56 demo
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 14:53 hdp
drwxr-xr-x 3 mapred hdfs 96 2015-04-24 14:52 mapred
drwxrwxrwx 4 hdfs hdfs 128 2015-04-24 14:52 mr-history
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 15:41 ranger
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 14:57 system
drwxrwxrwx 14 hdfs hdfs 448 2015-12-14 15:24 tmp
drwxr-xr-x 11 hdfs hdfs 352 2015-04-24 15:33 user

[root@sandbox ~]# cp ./test01 /media/hdfs/tmp/
[root@sandbox ~]# ls -l /media/hdfs/tmp/
total 16
drwx------ 3 ambari-qa hdfs 96 2015-12-03 14:44 ambari-qa
drwx-wx-wx 6 ambari-qa hdfs 192 2015-04-24 15:32 hive
-rw-r--r-- 1 root hdfs 87 2015-12-14 16:24 test01
drwxrwxrwx 8 hdfs hdfs 256 2015-04-24 15:31 udfs
drwx------ 3 ambari-qa hdfs 96 2015-12-03 14:44 yarn

Perfect ! :)

remount without noexec attribute

CentOS /var mount point is usually with noexec attribute. This is annoying when executing scripts on that mount point, like Ambari scripts !

SO if you have these “permission denied” on executing a script, simply remount your mount point :

$ sudo mount -o remount,exec /var

Don’t forget to modify accordingly your /etc/fstab file so that modification will be permanent and not loose at each reboot.