Posts tagged with: kerberos

Kerberos client howto

If you need debugging on client side, Kerberos doesn’t do a lot of things for you.

You can then position the KRB5_TRACE environment variable, standard system out shall be enough for your needs !

[root@dn01 ~]# env KRB5_TRACE=/dev/stdout kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase
[21232] 1444825930.913224: Getting initial credentials for hbase@REALM
[21232] 1444825930.914101: Looked up etypes in keytab: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac
[21232] 1444825930.914173: Sending request (198 bytes) to REALM
[21232] 1444825930.914599: Sending initial UDP request to dgram
[21232] 1444825930.918945: Received answer from dgram
[21232] 1444825930.919086: Response was from master KDC
[21232] 1444825930.919142: Received error from KDC: -1765328359/Additional pre-authentication required
[21232] 1444825930.919236: Processing preauth types: 136, 19, 2, 133
[21232] 1444825930.919257: Selected etype info: etype aes256-cts, salt "(null)", params ""
[21232] 1444825930.919265: Received cookie: MIT
[21232] 1444825930.919598: Retrieving hbase@REALM from FILE:/etc/security/keytabs/hbase.headless.keytab (vno 0, enctype aes256-cts) with result: 0/Success
[21232] 1444825930.919650: AS key obtained for encrypted timestamp: aes256-cts/0B3B
[21232] 1444825930.919754: Encrypted timestamp (for 1444825930.919656): plain 301AA011180F32303135313031343132333231305AA10502030E0868, encrypted DD77AE6EF5A9EFFA1A546BC34E964986BAFF339C5695B68A70689B84707503DB3FF2ECA23A30BFB5C4306E81EFFD445284E6328E9757501D
[21232] 1444825930.919778: Preauth module encrypted_timestamp (2) (flags=1) returned: 0/Success
[21232] 1444825930.919787: Produced preauth for next request: 133, 2
[21232] 1444825930.919817: Sending request (293 bytes) to REALM (master)
[21232] 1444825930.919977: Sending initial UDP request to dgram
[21232] 1444825930.927790: Received answer from dgram
[21232] 1444825930.927858: Processing preauth types: 19
[21232] 1444825930.927871: Selected etype info: etype aes256-cts, salt "(null)", params ""
[21232] 1444825930.927879: Produced preauth for next request: (empty)
[21232] 1444825930.927888: Salt derived from principal: REALMhbase
[21232] 1444825930.927903: AS key determined by preauth: aes256-cts/0B3B
[21232] 1444825930.928019: Decrypted AS reply; session key is: aes256-cts/4F13
[21232] 1444825930.928054: FAST negotiation: available
[21232] 1444825930.928099: Initializing FILE:/tmp/krb5cc_0 with default princ hbase@REALM
[21232] 1444825930.928497: Removing hbase@REALM -> krbtgt/REALM@REALM from FILE:/tmp/krb5cc_0
[21232] 1444825930.928516: Storing hbase@REALM -> krbtgt/REALM@REALM in FILE:/tmp/krb5cc_0
[21232] 1444825930.928687: Storing config in FILE:/tmp/krb5cc_0 for krbtgt/REALM@REALM: fast_avail: yes
[21232] 1444825930.928732: Removing hbase@REALM -> krb5_ccache_conf_data/fast_avail/krbtgt\/REALM\@REALM@X-CACHECONF: from FILE:/tmp/krb5cc_0
[21232] 1444825930.928748: Storing hbase@REALM -> krb5_ccache_conf_data/fast_avail/krbtgt\/REALM\@REALM@X-CACHECONF: in FILE:/tmp/krb5cc_0
[root@dn01 ~]#

enabling Application Timeline Server on a kerberized cluster

When you enable security on your HDP cluster, the wizard deletes the ATS (App Timeline Server) which is useful to follow YARN applications history.

Re-enabling it is not very difficult :

1. Installation through Ambari API

[vagrant@gw ~]# curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST

HTTP/1.1 201 Created

2. Component activation

[vagrant@gw ~]# curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context": "Install AppTimelineServer via REST","query":"HostRoles/'APP_TIMELINE_SERVER')"}, "Body":{"HostRoles": {"state": "INSTALLED"}}}'

HTTP/1.1 202 Accepted
Set-Cookie: AMBARISESSIONID=1dkx65ox0k0vx1vigxw7r0i4aw;Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 170
Server: Jetty(7.6.7.v20120910)

"href" : "",
"Requests" : {
"id" : 260,
"status" : "InProgress"

3. Keytab for ATS creation

[vagrant@gw ~]# ipa service-add ats/

4. Copy

[vagrant@gw ~]# ipa-getkeytab -s -p ats/ -k /etc/security/keytabs/ats.service.keytab

Keytab successfully retrieved and stored in: /etc/security/keytabs/ats.service.keytab
[vagrant@gw ~]# chown yarn:hadoop /etc/security/keytabs/ats.service.keytab && chmod 400 /etc/security/keytabs/ats.service.keytab

5. Add parameters in YARN

yarn.timeline-service.principal = ats/_HOST@EXAMPLE.COM
yarn.timeline-service.keytab = /etc/security/keytabs/ats.service.keytab
yarn.timeline-service.http-authentication.type = kerberos
yarn.timeline-service.http-authentication.kerberos.principal = HTTP/_HOST@EXAMPLE.COM
yarn.timeline-service.http-authentication.kerberos.keytab = /etc/security/keytabs/spnego.service.keytab

6. Restart YARN + ATS

The UI is enabled by default on port 8188 :

Kerberizing my cluster

In a customer perspective, security is not an option.

You can hear about cryptography, firewalling, etc. but no security will take place without the 2 main components of security : authentication and authorization.
The former takes place for insuring that you are the person you pretend to be, the latter that you have the right to access a resource.

In this post we’ll talk about authentication, and the best and only existing way to implement that is Kerberos, a 30+ years protocol from MIT.

First let’s talk about glossary :

– Client has its secret key
– Server has its secret key
– TGS (Ticket Granting System) has its secret key and knows the Server key
– KDC (Key Distribution Center) knows secret keys of Client and TGS

Steps are :

– Client identifies itself with KDC, which gives back a ticket authorizing him to request the TGS
– Client asks TGS for a ticket
– Client get his ticket and send his id with the ticket, sever checks the ticket validity and authorize the access.


Now, let’s get to work :)

On my 3 machines (vagrant powered) cluster, first let’s install free-ipa server, which is a really great/simple/robust way to have Kerberos on your system.
This cluster is under HDP 2.1.3 and Ambari 1.6.1.

The ipa-server package (provided by RedHat) should be in your repos.

On the KDC machine :

[vagrant@gw ~]$ sudo yum install -y ipa-server
[vagrant@gw ~]$ sudo ipa-server-install

Since my cluster is virtual machines with not “real” IP addresses and FQDN, I updated the /usr/lib/python2.6/site-packages/ipapython/ to get rid of the IANA Ip address check.

So you can now add users & groups in IPA, after doing a kinit admin. Now on every other hosts, install the client IPA package :

$ sudo yum install -y ipa-client
$ sudo ipa-client-install --enable-dns-updates

Now let’s kerberize our cluster ! In Ambari, let’s go to Security
Enable Security

Let’s get started.
step 01

Click Next and provide the information requested. Basically, the only information you need to provide is the realm.
step 02

Now Ambari proposes a smart way to generate all keytabs : download a CSV file that will be used with a script, which will take care of all that stuff.
step 03

Download that CSV file and put it on your KDC machine. For using with IPA we have to make a slight modification into Hortonwork’s script : add the -x ipa-setup-override-restrictions parameter after the kadmin.local -q “addprinc -randkey $principal” command.
Now let’s make all that keytabs.

Please note that on this version there is no rm (ResourceManager) keytab generated, so add the following line (assuming host is in before executing it :
kadmin.local -q "addprinc -randkey rm/" -x ipa-setup-override-restrictions

[vagrant@gw ~]$ /var/lib/ambari-server/resources/scripts/ /vagrant/host-principal-keytab-list.csv > ./
[vagrant@gw ~]$ chmod +x ./
[vagrant@gw ~]$ sudo su -
[root@gw ~]$ cd /home/vagrant
[root@gw ~]$ kinit admin
[root@gw ~]$ ./

This will generate all keytabs in a tar archive we copy to each host.

Then on each machine, extract & copy the keytabs

[vagrant@gw ~]$ sudo tar xpvf --strip=1
[vagrant@dn1 ~]$ sudo tar xpvf --strip=1
[vagrant@nn ~]$ sudo tar xpvf --strip=1

Please note the p option to preserve ownership, and sudo to make that option works. The –strip=1 is to avoid the ./ extraction which will make the current directory unbrowsable. Now we have all keytabs in /etc/security/keytabs directory.

It’s now time to activate our kerberized cluster by clicking the Apply button in Ambari interface.

Note : Generating a keytab will invalidate all that related keytabs on the realm !
As an example, if you re-run the script, this will ask new keytabs so you’ll got to copy it on all the servers.
Note 2 : If you want to enable HA on your cluster, you’ll need new keytabs because of the new components. The easiest way is to redownload the csv and regenerate all the keytabs.