Browsing posts in: hdfs

HDFS ls and Out of Memory (GC Overhead limit)

If you have an error when doing a ls like

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringBuffer.toString(StringBuffer.java:561)
at java.net.URI.toString(URI.java:1926)
at java.net.URI.<init>(URI.java:749)
at org.apache.hadoop.fs.Path.makeQualified(Path.java:467)
...

You might increase the client memory :

HADOOP_CLIENT_OPTS="-Xmx4g" hdfs dfs -ls -R /

installing a NFS gateway on Sandbox

NFS gateway is a neat way to access HDFS without a HDFS client, HDFS would then be appears mounted on the local filesystem as any directory.

We have to start by saying NFS user to be able to inpersonate users which will access our cluster, so let’s add in HDFS/configs/custom core-site.xml hadoop.proxyuser.nfsserver.groups and hadoop.proxyuser.nfsserver.hosts

NFS proxyuser

 

Then we add on custom hdfs-site.xml our Kerberos credentials (of course, your Sandbox is kerberized, is it?)

NFS: Kerberos credentials

 

in the same custom hdfs-site.xml, add the following properties which will respectively indicates a spool temporary directory (to re-order sequential writes before writing to HDFS) and the access control policy (here anyone can read/write but you could use another policy represented by MACHINE_NAME RW_POLICY, the latest could be rw (read&write) or ro (read-only))

NFS: mount points

Of course we have to add principal and get keytab for our NFS gateway.
Notice I had to use dfs.nfs.keytab.file and dfs.nfs.kerberos.principal for nfs3 gateway to launch.

We have to launch portmap and nfs3 :

[root@sandbox ~]# hadoop-daemon.sh start portmap

[root@sandbox ~]# hadoop-daemon.sh start nfs3

and mount a new directory as the new mount point for accessing HDFS :

[root@sandbox ~]# mkdir -p /media/hdfs

[root@sandbox ~]# mount -t nfs -o vers=3,proto=tcp,nolock 10.0.2.15:/ /media/hdfs/

We can check NFS is functional :

[root@sandbox ~]# ls -l /media/hdfs/
total 5
drwxrwxrwx 3 yarn hadoop 96 2015-12-03 14:42 app-logs
drwxr-xr-x 5 hdfs hdfs 160 2015-04-24 15:11 apps
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 15:56 demo
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 14:53 hdp
drwxr-xr-x 3 mapred hdfs 96 2015-04-24 14:52 mapred
drwxrwxrwx 4 hdfs hdfs 128 2015-04-24 14:52 mr-history
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 15:41 ranger
drwxr-xr-x 3 hdfs hdfs 96 2015-04-24 14:57 system
drwxrwxrwx 14 hdfs hdfs 448 2015-12-14 15:24 tmp
drwxr-xr-x 11 hdfs hdfs 352 2015-04-24 15:33 user

[root@sandbox ~]# cp ./test01 /media/hdfs/tmp/
[root@sandbox ~]# ls -l /media/hdfs/tmp/
total 16
drwx------ 3 ambari-qa hdfs 96 2015-12-03 14:44 ambari-qa
drwx-wx-wx 6 ambari-qa hdfs 192 2015-04-24 15:32 hive
-rw-r--r-- 1 root hdfs 87 2015-12-14 16:24 test01
drwxrwxrwx 8 hdfs hdfs 256 2015-04-24 15:31 udfs
drwx------ 3 ambari-qa hdfs 96 2015-12-03 14:44 yarn

Perfect ! :)


hdfs disk usage for humans

hdfs du is a powerful command, but could be not very handsome…

Here is a trick to have your subdirectories, sorted by size, in human-readable format

[root ~]# hdfs dfs -du -s -h "/*" | awk '{print $1 $2 " " $3}' | sort -h
39.8G /mr-history
216.9G /backup
362.5G /app-logs
20.0T /user
76.0T /tmp
138.6T /apps


HDFS permissions swept

I’ve got this strange behaviour in my HDP 2.2 preview sandbox :

[root@sandbox ~]# kinit guest

[root@sandbox ~]# hdfs dfs -ls -d /user/laurent
drwx------ - laurent readers 0 2015-06-18 13:47 /user/laurent

[root@sandbox ~]# hdfs dfs -touchz /user/laurent/guest02

[root@sandbox ~]# hdfs dfs -ls /user/laurent
-rw-r--r-- 1 guest readers 0 2015-06-18 13:47 /user/laurent/guest02

Ok. Weird, I can create a file in a directory theorically unreadable.

When suddenly:

[root@sandbox ~]# /etc/init.d/xapolicymgr stop
XAPolicyManager has been stopped.

[root@sandbox ~]# # restart HDFS service through Ambari as XA-Secure is a wrapper around HDFS processes

[root@sandbox ~]# hdfs dfs -touchz /user/laurent/guest03
touchz: Permission denied: user=guest, access=EXECUTE, inode="/user/laurent":laurent:readers:drwx------

Let the party begin !

So if you’re running into that case, please check xasecure (or argus, or ranger) are not active and then bypassing HDFS rights, for example /etc/init.d/xapolicymgr stop and /etc/init.d/argus-usersync stop



Failures : what I learned

in my HA cluster, Namenodes failed to start with the following :

2015-03-16 15:11:44,724 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(198)) - caught exception initializing http://gw.example.com:8480/getJournal?jid=mycluster&segmentTxId=88798&storageInfo=-56%3A567324971%3A0%3ACID-7958b480-2d52-49dc-8d71-e0d14429dbce
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$LogHeaderCorruptException: Unexpected version of the file system log file: -620756992. Current version = -56.
{...}
015-03-16 15:11:45,057 FATAL namenode.NameNode (NameNode.java:main(1400)) - Exception in namenode join
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 88798

at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:193)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:480)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:536)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:695)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:680)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1329)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1395)
Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 88618; expected file to go up to 14352384
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:180)
... 12 more
2015-03-16 15:11:45,066 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2015-03-16 15:11:45,079 INFO namenode.NameNode (StringUtils.java:run(640)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at dn1.example.com/240.0.0.12
************************************************************/

procedure is to use the recover mode :

[root@dn1 ~]# kinit -kt /etc/security/keytabs/nn.service.keytab nn/dn1.example.com

[root@dn1 ~]# /usr/bin/hadoop namenode -recover

{...}
15/03/16 15:33:01 ERROR namenode.MetaRecoveryContext: We failed to read txId 88798
15/03/16 15:33:01 INFO namenode.MetaRecoveryContext:
Enter 'c' to continue, skipping the bad section in the log
Enter 's' to stop reading the edit log here, abandoning any later edits
Enter 'q' to quit without saving
Enter 'a' to always select the first choice in the future without prompting. (c/s/q/a)

c
15/03/16 15:33:13 INFO namenode.MetaRecoveryContext: Continuing
15/03/16 15:33:13 INFO namenode.FSImage: Edits file http://nn.example.com:8480/getJournal?jid=mycluster&segmentTxId=88798&storageInfo=-56%3A567324971%3A0%3ACID-7958b480-2d52-49dc-8d71-e0d14429dbce of size 1048576 edits # 0 loaded in 11 seconds
15/03/16 15:33:13 INFO namenode.FSNamesystem: Need to save fs image? false (staleImage=true, haEnabled=true, isRollingUpgrade=false)
15/03/16 15:33:13 INFO namenode.NameCache: initialized with 9 entries 212 lookups
15/03/16 15:33:13 INFO namenode.FSNamesystem: Finished loading FSImage in 18865 msecs
15/03/16 15:33:13 INFO namenode.FSImage: Save namespace ...
15/03/16 15:33:14 INFO namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 88618
15/03/16 15:33:14 INFO namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/hadoop/hdfs/namenode/current/fsimage_0000000000000088386, cpktTxId=0000000000000088386)
15/03/16 15:33:15 INFO namenode.MetaRecoveryContext: RECOVERY COMPLETE
15/03/16 15:33:15 INFO namenode.FSNamesystem: Stopping services started for active state
15/03/16 15:33:15 INFO namenode.FSNamesystem: Stopping services started for standby state
15/03/16 15:33:15 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at dn1.example.com/240.0.0.12
************************************************************/

Then we just have to start NameNode (you can expect some missing blocks though)


upload a file with WebHDFS

By default, WebHDFS is enabled on your cluster, allowing you to make any HDFS operation through this REST API.

If you want to upload a file to HDFS, this has to be done in 2 steps :
1. create the resource

[hdfs@gw vagrant]$ curl -i --negotiate -u : -X PUT "http://nn.example.com:50070/webhdfs/v1/tmp/testfile?op=CREATE&overwrite=true"
HTTP/1.1 401 Authentication required
Date: Fri, 13 Feb 2015 11:29:54 GMT
Pragma: no-cache
Date: Fri, 13 Feb 2015 11:29:54 GMT
Pragma: no-cache
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Fri, 13 Feb 2015 11:29:54 GMT
Date: Fri, 13 Feb 2015 11:29:54 GMT
Pragma: no-cache
Expires: Fri, 13 Feb 2015 11:29:54 GMT
Date: Fri, 13 Feb 2015 11:29:54 GMT
Pragma: no-cache
Content-Type: application/octet-stream
Set-Cookie: hadoop.auth=u=hdfs&p=hdfs@EXAMPLE.COM&t=kerberos&e=1423862994233&s=+gAWB/1q0QOKjK9Wf6W4Bl2B6BY=; Path=/; Expires=Fri, 13-Feb-2015 21:29:54 GMT; HttpOnly
Location: http://gw.example.com:1022/webhdfs/v1/tmp/testfile?op=CREATE&delegation=HAAEaGRmcwRoZGZzAIoBS4KzxDuKAUumwEg7CQgUs7isYeQ5F6u4cV-oSig--MQFgU8SV0VCSERGUyBkZWxlZ2F0aW9uDzI0MC4wLjAuMTE6ODAyMA&namenoderpcaddress=mycluster&overwrite=true
Content-Length: 0
Server: Jetty(6.1.26)

2. upload the file in that resource
Notice that we obtained a location in the last request result, with the datanode where the resource will be created.
Now we upload our file to that URL.

[hdfs@gw vagrant]$ curl -i -X PUT -T MY_LOCAL_FILE "http://gw.example.com:1022/webhdfs/v1/tmp/testfile?op=CREATE&delegation=HAAEaGRmcwRoZGZzAIoBS4KzxDuKAUumwEg7CQgUs7isYeQ5F6u4cV-oSig--MQFgU8SV0VCSERGUyBkZWxlZ2F0aW9uDzI0MC4wLjAuMTE6ODAyMA&namenoderpcaddress=mycluster&overwrite=true"
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Cache-Control: no-cache
Expires: Fri, 13 Feb 2015 11:30:20 GMT
Date: Fri, 13 Feb 2015 11:30:20 GMT
Pragma: no-cache
Expires: Fri, 13 Feb 2015 11:30:20 GMT
Date: Fri, 13 Feb 2015 11:30:20 GMT
Pragma: no-cache
Content-Type: application/octet-stream
Location: webhdfs://mycluster/tmp/testfile
Content-Length: 0
Server: Jetty(6.1.26)

Test :

[hdfs@gw vagrant]$ hdfs dfs -cat /tmp/testfile
This is a test

more info on the Hadoop WebHDFS page