Monthly Archives: June 2015

make Hive console verbose

If you have some failing request in your Hive CLI with as much details as FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask then you may want to add some verbosity… Simply launch hive with redirecting logger to console :

hive -hiveconf hive.root.logger=INFO,console

HDFS permissions swept

I’ve got this strange behaviour in my HDP 2.2 preview sandbox :

[root@sandbox ~]# kinit guest

[root@sandbox ~]# hdfs dfs -ls -d /user/laurent
drwx------ - laurent readers 0 2015-06-18 13:47 /user/laurent

[root@sandbox ~]# hdfs dfs -touchz /user/laurent/guest02

[root@sandbox ~]# hdfs dfs -ls /user/laurent
-rw-r--r-- 1 guest readers 0 2015-06-18 13:47 /user/laurent/guest02

Ok. Weird, I can create a file in a directory theorically unreadable.

When suddenly:

[root@sandbox ~]# /etc/init.d/xapolicymgr stop
XAPolicyManager has been stopped.

[root@sandbox ~]# # restart HDFS service through Ambari as XA-Secure is a wrapper around HDFS processes

[root@sandbox ~]# hdfs dfs -touchz /user/laurent/guest03
touchz: Permission denied: user=guest, access=EXECUTE, inode="/user/laurent":laurent:readers:drwx------

Let the party begin !

So if you’re running into that case, please check xasecure (or argus, or ranger) are not active and then bypassing HDFS rights, for example /etc/init.d/xapolicymgr stop and /etc/init.d/argus-usersync stop

sticky bit

in Linux, the sticky bit is used on 2 levels : directories and files.


Set the sticky bit so that a user can only delete files which belongs to him.
For example, /tmp have 777 permissions, but you maybe don’t want users to be able to delete all the files in that folder (which is why /tmp has the sticky bit, by the way)

So let’s have a look : first we have a 777 directory and files with different users

[root@sandbox ~]# ls -ld /test
drwxrwxrwx 2 777 root 4096 Jun 11 08:31 /test
[root@sandbox ~]# ls -l /test
-rw-rw-r-- 1 guest guest 0 Jun 11 08:32 guest
-rw-r--r-- 1 hdfs hadoop 0 Jun 11 08:32 hdfs
-rw-r--r-- 1 hdfs hadoop 0 Jun 11 08:32 hdfs2
-rw-rw-r-- 1 hue hue 0 Jun 11 08:32 hue

As expected, hue user can delete hdfs file :

[root@sandbox ~]# su - hue -c "rm -f /test/hdfs2"

Now let’s set the sticky bit to the directory and retry to delete another user file

[root@sandbox ~]# chmod +t /test
[root@sandbox ~]# su - hue -c "rm -f /test/hdfs"
rm: cannot remove `/test/hdfs': Operation not permitted

You’ll notice the “t” in the permissions list :

[root@sandbox ~]# ls -ld /test
drwxrwxrwt 2 777 root 4096 Jun 11 08:51 /test


The sticky bit itself isn’t used (or in very specific cases / os). We rather talk about setuid, which is generally called sticky bit as a misnomer.

When a file belongs to a user, it can also be set as setuid, meaning it’ll be executed with file owner rights, not the launcher.
As an example, passwd is setuid root (it is setuid and belongs to root): that means if launch it under “guest“, it will be executed as root so that it can write in /etc/passwd (which is writable only by root)

You can setuid with

chmod +s /etc/passwd