one-shot backup all config files with Ambari API

Before performing an upgrade for example, or in a more general manner, it’s a good thing to get backups of your config files.

In an upgrade situation, this allows you to perform a quick diff to see if there are parameters which had been reset, or new parameters to take care of.

Ambari API allows you to get all config files, and we’ll use the amazing configs.sh script provided by Hortonworks in HDP to perform backups.

First, config types : when looking at the output of http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/, we can find them on Clusters / desired_config.

So let’s dig into that part : go to http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs

Now let’s get only file names :

[root@sandbox ~]# curl -s -u admin:admin http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2
ams-env
ams-hbase-env
ams-hbase-log4j
ams-hbase-policy
ams-hbase-security-site
ams-hbase-site
ams-log4j
ams-site
capacity-scheduler
cluster-env
core-site
...

Now for each file we can get a backup with the following command :

[root@sandbox ~]# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {'

where CONFIG_TYPE = core-site for example.

So we can now have a complete backup:

#!/bin/bash
AMBARI_HOST=sandbox.hortonworks.com
CLUSTER_NAME=Sandbox
AMBARI_USER=admin
AMBARI_PASSWORD=admin
AMBARI_PORT=8080
timeNow=`date +%Y%m%d_%H%M%S`
RESULT_DIR=/root/migrationHDP/configs.sh/$timeNow
mkdir -p $RESULT_DIR
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2`; do
echo "backuping $CONFIG_TYPE"
/var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {' > $RESULT_DIR/$CONFIG_TYPE.conf
done

Note than you can also output to a single file to make the diff easier, adding the CONFIG_TYPE to have a better view :

#!/bin/bash
AMBARI_HOST=sandbox.hortonworks.com
CLUSTER_NAME=Sandbox
AMBARI_USER=admin
AMBARI_PASSWORD=admin
AMBARI_PORT=8080
timeNow=`date +%Y%m%d_%H%M%S`
RESULT_DIR=/root/migrationHDP/configs.sh/$timeNow
mkdir -p $RESULT_DIR
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2`; do
echo "backuping $CONFIG_TYPE"
/var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {' | sed "1i ##### $config_type #####" >> $RESULT_DIR/all.conf
done

6 Comments

Reply sree | September 29th, 2016

Thank you for helpfull script.
Reply Mirotvorec | October 21st, 2016

Super!!! Thank You
Do you have a script which can compare different files and save only differences
- Reply administrator | October 21st, 2016
  
  that depends of what you’re really trying to do : if you want to really see differences then you’ll use the diff command
  
  diff backup1 backup2 >res
Reply Mirotvorec | October 25th, 2016

Here is my example of script.

#!/bin/bash
AMBARI_HOST=xxxxxxx.com
CLUSTER_NAME=xxxxxx
AMBARI_USER=xxxxx
AMBARI_PASSWORD=xxxxx
AMBARI_PORT=xxxxx
timeNow=`date +%Y%m%d_%H%M%S`
BACKUP_DIR=/root/xxxxxx/
COMPARE_DIR=/root/xxxxx/compare
RESULT_DIR=/root/xxxxxx/$timeNow
mkdir -p $RESULT_DIR
mkdir -p $COMPARE_DIR/$timeNow
#### Grep configuration files from AMBARI. One file for each service #####################
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep ‘” : {‘ | grep -v Clusters | grep -v desired_configs | cut -d'”‘ -f2`; do
echo “backuping $CONFIG_TYPE”
ssh root@$AMBARI_HOST /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep ‘^”‘ | grep -v ‘^”properties” : {‘ > $RESULT_DIR/$CONFIG_TYPE.conf
done
###### Identify previous backup directory #########
PREVIOUS_DIR=$(find $BACKUP_DIR -mmin +1 -type d -printf ‘%f\n’ | sort -r | sed -n 3p)
##### Comparison files with previous backup #######
for i in $(ls -1 $RESULT_DIR); do
sdiff -s $BACKUP_DIR$PREVIOUS_DIR/$i $RESULT_DIR/$i > $COMPARE_DIR/$timeNow/dif-$i
done
echo “Comparison done”
##### Delete empty files and directories ###############
find $COMPARE_DIR/$timeNow/ -size 0 -print0 |xargs -0 rm
find $COMPARE_DIR -type d -empty
Reply David | July 8th, 2017

Hello. Thanks for this script and the explanation.
One question, in case you want to restore a configuration for any of the Ambari components, which would be the step to follow ?
- Reply administrator | July 11th, 2017
  
  In this case I would probably manually extract the part of the backup file I’d like to restore and put it with the configs.py (or configs.sh depending of your Ambari version)
  
  /var/lib/ambari-server/resources/scripts/configs.py --help Usage: configs.py [options] Options: -h, --help show this help message and exit -t PORT, --port=PORT Optional port number for Ambari server. Default is '8080'. Provide empty string to not use port. -s PROTOCOL, --protocol=PROTOCOL Optional support of SSL. Default protocol is 'http' -a ACTION, --action=ACTION Script action: , , -l HOST, --host=HOST Server external host name -n CLUSTER, --cluster=CLUSTER Name given to cluster. Ex: 'c1' -c CONFIG_TYPE, --config-type=CONFIG_TYPE One of the various configuration types in Ambari. Ex: core-site, hdfs-site, mapred-queue-acls, etc. To specify credentials please use "-e" OR "-u" and "-p'": -u USER, --user=USER Optional user ID to use for authentication. Default is 'admin' -p PASSWORD, --password=PASSWORD Optional password to use for authentication. Default is 'admin' -e CREDENTIALS_FILE, --credentials-file=CREDENTIALS_FILE Optional file with user credentials separated by new line. To specify property(s) please use "-f" OR "-k" and "-v'": -f FILE, --file=FILE File where entire configurations are saved to, or read from. Supported extensions (.xml, .json>) -k KEY, --key=KEY Key that has to be set or deleted. Not necessary for 'get' action. -v VALUE, --value=VALUE Optional value to be set. Not necessary for 'get' or 'delete' actions.

Le Cluster de Laurent Just another Hadoop blog

one-shot backup all config files with Ambari API

6 Comments

So, what do you think ?