Browsing posts in: webhdfs

upload a file with WebHDFS

By default, WebHDFS is enabled on your cluster, allowing you to make any HDFS operation through this REST API.

If you want to upload a file to HDFS, this has to be done in 2 steps :
1. create the resource

[hdfs@gw vagrant]$ curl -i --negotiate -u : -X PUT "http://nn.example.com:50070/webhdfs/v1/tmp/testfile?op=CREATE&overwrite=true"
HTTP/1.1 401 Authentication required
Date: Fri, 13 Feb 2015 11:29:54 GMT
Pragma: no-cache
Date: Fri, 13 Feb 2015 11:29:54 GMT
Pragma: no-cache
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Fri, 13 Feb 2015 11:29:54 GMT
Date: Fri, 13 Feb 2015 11:29:54 GMT
Pragma: no-cache
Expires: Fri, 13 Feb 2015 11:29:54 GMT
Date: Fri, 13 Feb 2015 11:29:54 GMT
Pragma: no-cache
Content-Type: application/octet-stream
Set-Cookie: hadoop.auth=u=hdfs&p=hdfs@EXAMPLE.COM&t=kerberos&e=1423862994233&s=+gAWB/1q0QOKjK9Wf6W4Bl2B6BY=; Path=/; Expires=Fri, 13-Feb-2015 21:29:54 GMT; HttpOnly
Location: http://gw.example.com:1022/webhdfs/v1/tmp/testfile?op=CREATE&delegation=HAAEaGRmcwRoZGZzAIoBS4KzxDuKAUumwEg7CQgUs7isYeQ5F6u4cV-oSig--MQFgU8SV0VCSERGUyBkZWxlZ2F0aW9uDzI0MC4wLjAuMTE6ODAyMA&namenoderpcaddress=mycluster&overwrite=true
Content-Length: 0
Server: Jetty(6.1.26)

2. upload the file in that resource
Notice that we obtained a location in the last request result, with the datanode where the resource will be created.
Now we upload our file to that URL.

[hdfs@gw vagrant]$ curl -i -X PUT -T MY_LOCAL_FILE "http://gw.example.com:1022/webhdfs/v1/tmp/testfile?op=CREATE&delegation=HAAEaGRmcwRoZGZzAIoBS4KzxDuKAUumwEg7CQgUs7isYeQ5F6u4cV-oSig--MQFgU8SV0VCSERGUyBkZWxlZ2F0aW9uDzI0MC4wLjAuMTE6ODAyMA&namenoderpcaddress=mycluster&overwrite=true"
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Cache-Control: no-cache
Expires: Fri, 13 Feb 2015 11:30:20 GMT
Date: Fri, 13 Feb 2015 11:30:20 GMT
Pragma: no-cache
Expires: Fri, 13 Feb 2015 11:30:20 GMT
Date: Fri, 13 Feb 2015 11:30:20 GMT
Pragma: no-cache
Content-Type: application/octet-stream
Location: webhdfs://mycluster/tmp/testfile
Content-Length: 0
Server: Jetty(6.1.26)

Test :

[hdfs@gw vagrant]$ hdfs dfs -cat /tmp/testfile
This is a test

more info on the Hadoop WebHDFS page