Browsing posts in: kerberos

Kerberos client howto

If you need debugging on client side, Kerberos doesn’t do a lot of things for you.

You can then position the KRB5_TRACE environment variable, standard system out shall be enough for your needs !

[root@dn01 ~]# env KRB5_TRACE=/dev/stdout kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase
[21232] 1444825930.913224: Getting initial credentials for hbase@REALM
[21232] 1444825930.914101: Looked up etypes in keytab: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac
[21232] 1444825930.914173: Sending request (198 bytes) to REALM
[21232] 1444825930.914599: Sending initial UDP request to dgram 192.168.1.88:88
[21232] 1444825930.918945: Received answer from dgram 192.168.1.88:88
[21232] 1444825930.919086: Response was from master KDC
[21232] 1444825930.919142: Received error from KDC: -1765328359/Additional pre-authentication required
[21232] 1444825930.919236: Processing preauth types: 136, 19, 2, 133
[21232] 1444825930.919257: Selected etype info: etype aes256-cts, salt "(null)", params ""
[21232] 1444825930.919265: Received cookie: MIT
[21232] 1444825930.919598: Retrieving hbase@REALM from FILE:/etc/security/keytabs/hbase.headless.keytab (vno 0, enctype aes256-cts) with result: 0/Success
[21232] 1444825930.919650: AS key obtained for encrypted timestamp: aes256-cts/0B3B
[21232] 1444825930.919754: Encrypted timestamp (for 1444825930.919656): plain 301AA011180F32303135313031343132333231305AA10502030E0868, encrypted DD77AE6EF5A9EFFA1A546BC34E964986BAFF339C5695B68A70689B84707503DB3FF2ECA23A30BFB5C4306E81EFFD445284E6328E9757501D
[21232] 1444825930.919778: Preauth module encrypted_timestamp (2) (flags=1) returned: 0/Success
[21232] 1444825930.919787: Produced preauth for next request: 133, 2
[21232] 1444825930.919817: Sending request (293 bytes) to REALM (master)
[21232] 1444825930.919977: Sending initial UDP request to dgram 192.168.1.88:88
[21232] 1444825930.927790: Received answer from dgram 192.168.1.88:88
[21232] 1444825930.927858: Processing preauth types: 19
[21232] 1444825930.927871: Selected etype info: etype aes256-cts, salt "(null)", params ""
[21232] 1444825930.927879: Produced preauth for next request: (empty)
[21232] 1444825930.927888: Salt derived from principal: REALMhbase
[21232] 1444825930.927903: AS key determined by preauth: aes256-cts/0B3B
[21232] 1444825930.928019: Decrypted AS reply; session key is: aes256-cts/4F13
[21232] 1444825930.928054: FAST negotiation: available
[21232] 1444825930.928099: Initializing FILE:/tmp/krb5cc_0 with default princ hbase@REALM
[21232] 1444825930.928497: Removing hbase@REALM -> krbtgt/REALM@REALM from FILE:/tmp/krb5cc_0
[21232] 1444825930.928516: Storing hbase@REALM -> krbtgt/REALM@REALM in FILE:/tmp/krb5cc_0
[21232] 1444825930.928687: Storing config in FILE:/tmp/krb5cc_0 for krbtgt/REALM@REALM: fast_avail: yes
[21232] 1444825930.928732: Removing hbase@REALM -> krb5_ccache_conf_data/fast_avail/krbtgt\/REALM\@REALM@X-CACHECONF: from FILE:/tmp/krb5cc_0
[21232] 1444825930.928748: Storing hbase@REALM -> krb5_ccache_conf_data/fast_avail/krbtgt\/REALM\@REALM@X-CACHECONF: in FILE:/tmp/krb5cc_0
[root@dn01 ~]#

Kerberos Tips & Tricks

Read a keytab to see principals :

[root@gw ~]# ktutil
ktutil: read_kt /etc/security/keytabs/nn.service.keytab
ktutil: list
slot KVNO Principal

---- ---- ---------------------------------------------------------------------
1 3 nn/gw.example.com@EXAMPLE.COM
2 3 nn/gw.example.com@EXAMPLE.COM
3 3 nn/gw.example.com@EXAMPLE.COM
4 3 nn/gw.example.com@EXAMPLE.COM
5 3 nn/nn.example.com@EXAMPLE.COM
6 3 nn/nn.example.com@EXAMPLE.COM
7 3 nn/nn.example.com@EXAMPLE.COM
8 3 nn/nn.example.com@EXAMPLE.COM
ktutil: quit
[root@gw ~]#

Service keytabs are for a service, so added for a specific machine.
Therefore, if you want to add an existing service to another node, you must create that service for that additional node.

[root@ ~]# ipa service-add zookeeper/newnode@MY_CLUSTER
[root@~]# ipa-getkeytab -s IPASERVER -p zookeeper/newnode@MY_CLUSTER -k zk.service.keytab.newnode
[root@~]# chmod 400 zk.service.keytab.newnode
[root@~]# scp zk.service.keytab.newnode NEWNODE:/etc/security/keytabs/.
[root@NEWNODE ~]# mv /etc/security/keytabs/zk.service.keytab{.newnode,}
[root@NEWNODE ~]# chown zookeeper:hadoop /etc/security/keytabs/zk.service.keytab

If you do the ipa-getkeytab on an existing keytab, it will add the service in the keytab, not replace it.

 

If for some reason IPA doesn’t work :

// adding principal
[root@gw ~]# kadmin.local -q "addprinc -randkey hbase/nn.example.com@EXAMPLE.COM" -x ipa-setup-override-restrictions
// then get the keytab
[root@gw ~]# kadmin.local -q "xst -k /home/vagrant/tmp_keytabs/hbase.service.keytab.nn hbase/nn.example.com@EXAMPLE.COM"

 


enabling Application Timeline Server on a kerberized cluster

When you enable security on your HDP cluster, the wizard deletes the ATS (App Timeline Server) which is useful to follow YARN applications history.

Re-enabling it is not very difficult :

1. Installation through Ambari API

[vagrant@gw ~]# curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST http://gw.example.com:8080/api/v1/clusters/hdp-cluster/hosts/gw.example.com/host_components/APP_TIMELINE_SERVER

HTTP/1.1 201 Created

2. Component activation

[vagrant@gw ~]# curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context": "Install AppTimelineServer via REST","query":"HostRoles/component_name.in('APP_TIMELINE_SERVER')"}, "Body":{"HostRoles": {"state": "INSTALLED"}}}' http://gw.example.com:8080/api/v1/clusters/hdp-cluster/hosts/gw.example.com/host_components

HTTP/1.1 202 Accepted
Set-Cookie: AMBARISESSIONID=1dkx65ox0k0vx1vigxw7r0i4aw;Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 170
Server: Jetty(7.6.7.v20120910)

{
"href" : "http://gw.example.com:8080/api/v1/clusters/hdp-cluster/requests/260",
"Requests" : {
"id" : 260,
"status" : "InProgress"
}
}

3. Keytab for ATS creation

[vagrant@gw ~]# ipa service-add ats/gw.example.com@EXAMPLE.COM

4. Copy

[vagrant@gw ~]# ipa-getkeytab -s nn.example.com -p ats/gw.example.com@EXAMPLE.COM -k /etc/security/keytabs/ats.service.keytab

Keytab successfully retrieved and stored in: /etc/security/keytabs/ats.service.keytab
[vagrant@gw ~]# chown yarn:hadoop /etc/security/keytabs/ats.service.keytab && chmod 400 /etc/security/keytabs/ats.service.keytab

5. Add parameters in YARN

yarn.timeline-service.principal = ats/_HOST@EXAMPLE.COM
yarn.timeline-service.keytab = /etc/security/keytabs/ats.service.keytab
yarn.timeline-service.http-authentication.type = kerberos
yarn.timeline-service.http-authentication.kerberos.principal = HTTP/_HOST@EXAMPLE.COM
yarn.timeline-service.http-authentication.kerberos.keytab = /etc/security/keytabs/spnego.service.keytab

6. Restart YARN + ATS

The UI is enabled by default on port 8188 : http://gw.example.com:8188/applicationhistory