Before performing an upgrade for example, or in a more general manner, it’s a good thing to get backups of your config files.
In an upgrade situation, this allows you to perform a quick diff to see if there are parameters which had been reset, or new parameters to take care of.
Ambari API allows you to get all config files, and we’ll use the amazing configs.sh script provided by Hortonworks in HDP to perform backups.
First, config types : when looking at the output of http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/, we can find them on Clusters / desired_config.
So let’s dig into that part : go to http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs
Now let’s get only file names :
Now for each file we can get a backup with the following command :
where CONFIG_TYPE = core-site for example. So we can now have a complete backup: Note than you can also output to a single file to make the diff easier, adding the CONFIG_TYPE to have a better view :[root@sandbox ~]# curl -s -u admin:admin http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2
ams-env
ams-hbase-env
ams-hbase-log4j
ams-hbase-policy
ams-hbase-security-site
ams-hbase-site
ams-log4j
ams-site
capacity-scheduler
cluster-env
core-site
...[root@sandbox ~]# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {'#!/bin/bash
AMBARI_HOST=sandbox.hortonworks.com
CLUSTER_NAME=Sandbox
AMBARI_USER=admin
AMBARI_PASSWORD=admin
AMBARI_PORT=8080
timeNow=`date +%Y%m%d_%H%M%S`
RESULT_DIR=/root/migrationHDP/configs.sh/$timeNow
mkdir -p $RESULT_DIR
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2`; do
echo "backuping $CONFIG_TYPE"
/var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {' > $RESULT_DIR/$CONFIG_TYPE.conf
done#!/bin/bash
AMBARI_HOST=sandbox.hortonworks.com
CLUSTER_NAME=Sandbox
AMBARI_USER=admin
AMBARI_PASSWORD=admin
AMBARI_PORT=8080
timeNow=`date +%Y%m%d_%H%M%S`
RESULT_DIR=/root/migrationHDP/configs.sh/$timeNow
mkdir -p $RESULT_DIR
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2`; do
echo "backuping $CONFIG_TYPE"
/var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {' | sed "1i ##### $config_type #####" >> $RESULT_DIR/all.conf
done
Thank you for helpfull script.
Super!!! Thank You
Do you have a script which can compare different files and save only differences
that depends of what you’re really trying to do : if you want to really see differences then you’ll use the diff command
diff backup1 backup2 >resHere is my example of script.
#!/bin/bash
AMBARI_HOST=xxxxxxx.com
CLUSTER_NAME=xxxxxx
AMBARI_USER=xxxxx
AMBARI_PASSWORD=xxxxx
AMBARI_PORT=xxxxx
timeNow=`date +%Y%m%d_%H%M%S`
BACKUP_DIR=/root/xxxxxx/
COMPARE_DIR=/root/xxxxx/compare
RESULT_DIR=/root/xxxxxx/$timeNow
mkdir -p $RESULT_DIR
mkdir -p $COMPARE_DIR/$timeNow
#### Grep configuration files from AMBARI. One file for each service #####################
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep ‘” : {‘ | grep -v Clusters | grep -v desired_configs | cut -d'”‘ -f2`; do
echo “backuping $CONFIG_TYPE”
ssh root@$AMBARI_HOST /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep ‘^”‘ | grep -v ‘^”properties” : {‘ > $RESULT_DIR/$CONFIG_TYPE.conf
done
###### Identify previous backup directory #########
PREVIOUS_DIR=$(find $BACKUP_DIR -mmin +1 -type d -printf ‘%f\n’ | sort -r | sed -n 3p)
##### Comparison files with previous backup #######
for i in $(ls -1 $RESULT_DIR); do
sdiff -s $BACKUP_DIR$PREVIOUS_DIR/$i $RESULT_DIR/$i > $COMPARE_DIR/$timeNow/dif-$i
done
echo “Comparison done”
##### Delete empty files and directories ###############
find $COMPARE_DIR/$timeNow/ -size 0 -print0 |xargs -0 rm
find $COMPARE_DIR -type d -empty
Hello. Thanks for this script and the explanation.
One question, in case you want to restore a configuration for any of the Ambari components, which would be the step to follow ?
In this case I would probably manually extract the part of the backup file I’d like to restore and put it with the configs.py (or configs.sh depending of your Ambari version)
/var/lib/ambari-server/resources/scripts/configs.py --help, , Usage: configs.py [options]
Options:
-h, --help show this help message and exit
-t PORT, --port=PORT Optional port number for Ambari server. Default is
'8080'. Provide empty string to not use port.
-s PROTOCOL, --protocol=PROTOCOL
Optional support of SSL. Default protocol is 'http'
-a ACTION, --action=ACTION
Script action:
-l HOST, --host=HOST Server external host name
-n CLUSTER, --cluster=CLUSTER
Name given to cluster. Ex: 'c1'
-c CONFIG_TYPE, --config-type=CONFIG_TYPE
One of the various configuration types in Ambari. Ex:
core-site, hdfs-site, mapred-queue-acls, etc.
To specify credentials please use "-e" OR "-u" and "-p'":
-u USER, --user=USER
Optional user ID to use for authentication. Default is
'admin'
-p PASSWORD, --password=PASSWORD
Optional password to use for authentication. Default
is 'admin'
-e CREDENTIALS_FILE, --credentials-file=CREDENTIALS_FILE
Optional file with user credentials separated by new
line.
To specify property(s) please use "-f" OR "-k" and "-v'":
-f FILE, --file=FILE
File where entire configurations are saved to, or read
from. Supported extensions (.xml, .json>)
-k KEY, --key=KEY Key that has to be set or deleted. Not necessary for
'get' action.
-v VALUE, --value=VALUE
Optional value to be set. Not necessary for 'get' or
'delete' actions.