one-shot backup all config files with Ambari API

Before performing an upgrade for example, or in a more general manner, it’s a good thing to get backups of your config files.

In an upgrade situation, this allows you to perform a quick diff to see if there are parameters which had been reset, or new parameters to take care of.

Ambari API allows you to get all config files, and we’ll use the amazing configs.sh script provided by Hortonworks in HDP to perform backups.

First, config types : when looking at the output of http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/, we can find them on Clusters / desired_config.

Ambari API result

So let’s dig into that part : go to http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs

Ambari API desired_configs

Now let’s get only file names :

[root@sandbox ~]# curl -s -u admin:admin http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2
ams-env
ams-hbase-env
ams-hbase-log4j
ams-hbase-policy
ams-hbase-security-site
ams-hbase-site
ams-log4j
ams-site
capacity-scheduler
cluster-env
core-site
...

Now for each file we can get a backup with the following command :

[root@sandbox ~]# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {'

where CONFIG_TYPE = core-site for example.

So we can now have a complete backup:

#!/bin/bash
AMBARI_HOST=sandbox.hortonworks.com
CLUSTER_NAME=Sandbox
AMBARI_USER=admin
AMBARI_PASSWORD=admin
AMBARI_PORT=8080
timeNow=`date +%Y%m%d_%H%M%S`
RESULT_DIR=/root/migrationHDP/configs.sh/$timeNow
mkdir -p $RESULT_DIR
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2`; do
echo "backuping $CONFIG_TYPE"
/var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {' > $RESULT_DIR/$CONFIG_TYPE.conf
done

 

Note than you can also output to a single file to make the diff easier, adding the CONFIG_TYPE to have a better view :

#!/bin/bash
AMBARI_HOST=sandbox.hortonworks.com
CLUSTER_NAME=Sandbox
AMBARI_USER=admin
AMBARI_PASSWORD=admin
AMBARI_PORT=8080
timeNow=`date +%Y%m%d_%H%M%S`
RESULT_DIR=/root/migrationHDP/configs.sh/$timeNow
mkdir -p $RESULT_DIR
for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2`; do
echo "backuping $CONFIG_TYPE"
/var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep '^"' | grep -v '^"properties" : {' | sed "1i ##### $config_type #####" >> $RESULT_DIR/all.conf
done

6 Comments

  • Reply Mirotvorec |

    Super!!! Thank You
    Do you have a script which can compare different files and save only differences

    • Reply administrator |

      that depends of what you’re really trying to do : if you want to really see differences then you’ll use the diff command

      diff backup1 backup2 >res

  • Reply Mirotvorec |

    Here is my example of script.

    #!/bin/bash
    AMBARI_HOST=xxxxxxx.com
    CLUSTER_NAME=xxxxxx
    AMBARI_USER=xxxxx
    AMBARI_PASSWORD=xxxxx
    AMBARI_PORT=xxxxx
    timeNow=`date +%Y%m%d_%H%M%S`
    BACKUP_DIR=/root/xxxxxx/
    COMPARE_DIR=/root/xxxxx/compare
    RESULT_DIR=/root/xxxxxx/$timeNow
    mkdir -p $RESULT_DIR
    mkdir -p $COMPARE_DIR/$timeNow
    #### Grep configuration files from AMBARI. One file for each service #####################
    for CONFIG_TYPE in `curl -s -u $AMBARI_USER:$AMBARI_PASSWORD http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/?fields=Clusters/desired_configs | grep ‘” : {‘ | grep -v Clusters | grep -v desired_configs | cut -d'”‘ -f2`; do
    echo “backuping $CONFIG_TYPE”
    ssh root@$AMBARI_HOST /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PASSWORD -port $AMBARI_PORT get $AMBARI_HOST $CLUSTER_NAME $CONFIG_TYPE | grep ‘^”‘ | grep -v ‘^”properties” : {‘ > $RESULT_DIR/$CONFIG_TYPE.conf
    done
    ###### Identify previous backup directory #########
    PREVIOUS_DIR=$(find $BACKUP_DIR -mmin +1 -type d -printf ‘%f\n’ | sort -r | sed -n 3p)
    ##### Comparison files with previous backup #######
    for i in $(ls -1 $RESULT_DIR); do
    sdiff -s $BACKUP_DIR$PREVIOUS_DIR/$i $RESULT_DIR/$i > $COMPARE_DIR/$timeNow/dif-$i
    done
    echo “Comparison done”
    ##### Delete empty files and directories ###############
    find $COMPARE_DIR/$timeNow/ -size 0 -print0 |xargs -0 rm
    find $COMPARE_DIR -type d -empty

  • Reply David |

    Hello. Thanks for this script and the explanation.
    One question, in case you want to restore a configuration for any of the Ambari components, which would be the step to follow ?

    • Reply administrator |

      In this case I would probably manually extract the part of the backup file I’d like to restore and put it with the configs.py (or configs.sh depending of your Ambari version)

      /var/lib/ambari-server/resources/scripts/configs.py --help
      Usage: configs.py [options]
      Options:
      -h, --help show this help message and exit
      -t PORT, --port=PORT Optional port number for Ambari server. Default is
      '8080'. Provide empty string to not use port.
      -s PROTOCOL, --protocol=PROTOCOL
      Optional support of SSL. Default protocol is 'http'
      -a ACTION, --action=ACTION
      Script action: , ,
      -l HOST, --host=HOST Server external host name
      -n CLUSTER, --cluster=CLUSTER
      Name given to cluster. Ex: 'c1'
      -c CONFIG_TYPE, --config-type=CONFIG_TYPE
      One of the various configuration types in Ambari. Ex:
      core-site, hdfs-site, mapred-queue-acls, etc.
      To specify credentials please use "-e" OR "-u" and "-p'":
      -u USER, --user=USER
      Optional user ID to use for authentication. Default is
      'admin'
      -p PASSWORD, --password=PASSWORD
      Optional password to use for authentication. Default
      is 'admin'
      -e CREDENTIALS_FILE, --credentials-file=CREDENTIALS_FILE
      Optional file with user credentials separated by new
      line.
      To specify property(s) please use "-f" OR "-k" and "-v'":
      -f FILE, --file=FILE
      File where entire configurations are saved to, or read
      from. Supported extensions (.xml, .json>)
      -k KEY, --key=KEY Key that has to be set or deleted. Not necessary for
      'get' action.
      -v VALUE, --value=VALUE
      Optional value to be set. Not necessary for 'get' or
      'delete' actions.

So, what do you think ?