Before performing an upgrade for example, or in a more general manner, it’s a good thing to get backups of your config files.
In an upgrade situation, this allows you to perform a quick diff to see if there are parameters which had been reset, or new parameters to take care of.
Ambari API allows you to get all config files, and we’ll use the amazing configs.sh script provided by Hortonworks in HDP to perform backups.
First, config types : when looking at the output of http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/, we can find them on Clusters / desired_config.
So let’s dig into that part : go to http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs
Now let’s get only file names :
Now for each file we can get a backup with the following command :
where CONFIG_TYPE = core-site for example. So we can now have a complete backup: Note than you can also output to a single file to make the diff easier, adding the CONFIG_TYPE to have a better view :[root@sandbox ~]# curl -s -u admin:admin http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2
ams-env
ams-hbase-env
ams-hbase-log4j
ams-hbase-policy
ams-hbase-security-site
ams-hbase-site
ams-log4j
ams-site
capacity-scheduler
cluster-env
core-site
...[root@sandbox ~]# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {'#!/bin/bash
AMBARI_HOST=sandbox.hortonworks.com
CLUSTER_NAME=Sandbox
AMBARI_USER=admin
AMBARI_PASSWORD=admin
AMBARI_PORT=8080
timeNow=`date +%Y%m%d_%H%M%S`
RESULT_DIR=/root/migrationHDP/configs.sh/$timeNow
mkdir -p $RESULT_DIR
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2`; do
echo "backuping $CONFIG_TYPE"
/var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {' > $RESULT_DIR/$CONFIG_TYPE.conf
done#!/bin/bash
AMBARI_HOST=sandbox.hortonworks.com
CLUSTER_NAME=Sandbox
AMBARI_USER=admin
AMBARI_PASSWORD=admin
AMBARI_PORT=8080
timeNow=`date +%Y%m%d_%H%M%S`
RESULT_DIR=/root/migrationHDP/configs.sh/$timeNow
mkdir -p $RESULT_DIR
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2`; do
echo "backuping $CONFIG_TYPE"
/var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {' | sed "1i ##### $config_type #####" >> $RESULT_DIR/all.conf
done
Thank you for helpfull script.
Super!!! Thank You
Do you have a script which can compare different files and save only differences
that depends of what you’re really trying to do : if you want to really see differences then you’ll use the diff command
diff backup1 backup2 >resHere is my example of script.
#!/bin/bash
AMBARI_HOST=xxxxxxx.com
CLUSTER_NAME=xxxxxx
AMBARI_USER=xxxxx
AMBARI_PASSWORD=xxxxx
AMBARI_PORT=xxxxx
timeNow=`date +%Y%m%d_%H%M%S`
BACKUP_DIR=/root/xxxxxx/
COMPARE_DIR=/root/xxxxx/compare
RESULT_DIR=/root/xxxxxx/$timeNow
mkdir -p $RESULT_DIR
mkdir -p $COMPARE_DIR/$timeNow
#### Grep configuration files from AMBARI. One file for each service #####################
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep ‘” : {‘ | grep -v Clusters | grep -v desired_configs | cut -d'”‘ -f2`; do
echo “backuping $CONFIG_TYPE”
ssh root@$AMBARI_HOST /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep ‘^”‘ | grep -v ‘^”properties” : {‘ > $RESULT_DIR/$CONFIG_TYPE.conf
done
###### Identify previous backup directory #########
PREVIOUS_DIR=$(find $BACKUP_DIR -mmin +1 -type d -printf ‘%f\n’ | sort -r | sed -n 3p)
##### Comparison files with previous backup #######
for i in $(ls -1 $RESULT_DIR); do
sdiff -s $BACKUP_DIR$PREVIOUS_DIR/$i $RESULT_DIR/$i > $COMPARE_DIR/$timeNow/dif-$i
done
echo “Comparison done”
##### Delete empty files and directories ###############
find $COMPARE_DIR/$timeNow/ -size 0 -print0 |xargs -0 rm
find $COMPARE_DIR -type d -empty