General arc.conf python configuration parser

Parsing configuration

Initial configuration parsing

The latest version of arcconfig-parser is designed to operate with the defaults file that holds the default values for all possible parameters.

You find the arcconfig-parser in your $ARC_LOCATION/libexec/arc folder which depends on your OS and installation options, but an example is /usr/libexec/arc/.

At the time of initial parsing the following chain is executed:

  • All blocks and options are parsed from arc.conf

  • All blocks and options are parsed from arc.conf.d and merged with arc.conf

  • For all blocks defined in the merged arc.conf, missing options are added from the defaults file

  • Special constructs in values are substituted (see Special constructs can be used as values)

Parameters that are optional by design and do not have a default value (specified with not set value) are not included to the parsed configuration.

Runtime configuration

Configuration that includes both arc.conf and defaults config is called the runtime configuration.

In some cases it is useful to save and load the runtime configuration:

  • To supply C++ services (a-rex, gridftpd) with configuration that includes defaults from common place

  • For repetitive operations on config to eliminate full-chain processing of each invocation

To save the runtime configuration to the default location (/var/run/arc/):

arcconfig-parser --save

To save the runtime configuration to a specified location:

arcconfig-parser --save -r /var/run/arc/arex.arc.conf

To load the runtime configuration instead of the full-chain processing and e.g. get the value of the x509_host_key in the [common] block:

arcconfig-parser --load -b common -o x509_host_key

Special constructs can be used as values

When defaults include references to another config part to be consistent with the implied arc.conf structure.

The most obvious example is if x509_host_key is not found in e.g. the [arex/jura] block, then it should be taken from the [common] block.

The config parser is following this logic (especially in respect to defaults) and use special constructs to accomplish this behaviour.

Command substitutions

Configuration option values can contain the construct $EXEC{<command>} that substituted to the stdout of <command>.

For example:

hostname=$EXEC{hostname -f}

Option values substitutions

The construct $VAR{[block]option} can be used to substitute the values of another option value.

If the option is in the same block as the referencing option, the block name can be omitted - $VAR{option}.

For example:

x509_host_key=$VAR{[common]x509_host_key}
bdii_update_cmd=$VAR{bdii_location}/sbin/bdii-update

Evaluation of simple code

For limited number of cases the arc.conf default values relies on arithmetic operations. For this purpose the $EVAL{string} special construct has been introduced.

For example:

bdii_read_timeout=$EVAL{$VAR{bdii_provider_timeout} + $VAR{[arex]infoproviders_timelimit} + $VAR{[arex]wakeupperiod}}

Getting the configuration values

If the --option argument is passed to arcconfig-parser, it returns the value of the specified option to stdout.

Without --option arcconfig-parser can be used to operate with configuration blocks:

  • check blocks existence (exit code used to indicate the status of the check)

  • return the list of subblocks

With the --export option arcconfig-parser allows to export config in the following formats:

  • json - returns entire configuration or subset of blocks as-is in JSON to stdout

  • bash - for [common] block or specified configuration subset returns CONFIG_option_name=value pairs to stdout. Block names ARE NOT included in the exports and option values precedence will be used in the order of passed blocks. If automatic subblocks expansion used with bash export, for every block in sequence - it’s subblocks are processed first (in arc.conf defined order). It is possible to filter the options that will be exported with additional --filter option that can be specified several times.

Common configuration parsing sequence

digraph { node [shape=Rectangle]; forcelabels=true; reference [label="arc.conf.reference", xlabel="Developers entry-point to put info"]; subst [label="Substitution syntax", style=filled]; reference -> subst [dir=back]; {rank = same; subst; reference;} reference -> "arc.parser.defaults" [ label="buildtime", fontcolor=red ]; subgraph cluster_0 { style = "dashed"; color = "red"; label = "Binary Distribution"; doc [ label="/usr/share/doc" ]; arcconf [ label="/etc/arc.conf"]; defconf [ label="/usr/share/arc/parser.defaults" ]; # hack rank arcconf -> defconf [style=invis]; } reference -> doc; "arc.parser.defaults" -> defconf; subst -> "arc.parser.defaults" subst -> defconf; parser [ label="arcconfig-parser" ]; arcconf -> parser [label="1. parse, get defined blocks"] defconf -> parser [label="2. add defaults for defined blocks"] runconfig [ label="runtime configuration", shape=oval, color=red ] parser -> runconfig [ label="3. evaluate substitutions" ] json [label="export JSON"] bash [label="export BASH"] check [label="get value"] runconfig -> json runconfig -> bash runconfig -> check subgraph cluster_1 { label = "Startup scripts" runconf [ label="/var/run/arc/arc.conf" ]; define [ label="define ENV variables" ] arex [ label="start a-rex" ] runconf -> define -> arex } runconfig -> runconf [ label="dump config" ] }

Options reference

The full list of available options of arcconfig-parser is available instantly using --help option. An online version is available here.

Examples

Get value of an option in a block:

# arcconfig-parser --block infosys --option logfile
/var/log/arc/infoprovider.log

Get value of an option in blocks in order they are specified (e.g. if not found in [gridftpd] look in the [common] block [1]):

# arcconfig-parser --block gridftpd --block common --option x509_host_key
/etc/grid-security/hostkey.pem

Export entire configuration to JSON [2]:

# arcconfig-parser --export json

Export [infosys] block options to JSON (for Perl):

# arcconfig-parser --block infosys --export json
{"infosys": {"loglevel": "5"},...

Export [infosys] block and all their subblocks options to JSON:

# arcconfig-parser --block infosys --subblocks --export json
{"infosys/glue2/ldap": {"showactivities": "no"},...

Export for BASH (compatible with config representation in shell-based LRMS backends):

# arcconfig-parser --block infosys --block arex --block common --export bash
CONFIG_controldir="/var/spool/arc/jobstatus"
CONFIG_defaultttl="1210000"
CONFIG_delegationdb="sqlite"
CONFIG_hostname="ce01.example.org"
CONFIG_maaxrerun="5"
CONFIG_maxjobs="10000 -1"
CONFIG_runtimedir="/home/grid/arc/runtime"
CONFIG_sessiondir="__array__" # <= NEW define for multivalued values that indicate indexed vars
CONFIG_sessiondir_0="/mnt/scratch/grid/arc/session"
CONFIG_sessiondir_1="/home/grid/arc/session drain"
...

Export for BASH with exported options filtering:

# arcconfig-parser -b common -f hostname -f x509_cert_dir -e bash
CONFIG_hostname="ce01.example.org"
CONFIG_x509_cert_dir="/etc/grid-security/certificates"

Using BASH export:

# eval "$( arcconfig-parser --block infosys --block arex --block common --export bash )"
# echo "$CONFIG_gridmap"

Check block(s) exists ([common/perflog] does not exists in the example):

# arcconfig-parser --block common/perflog --block arex
# echo $?
1

List block subblocks:

# arcconfig-parser --block infosys --subblocks
infosys
infosys/ldap
infosys/nordugrid
infosys/glue2
infosys/glue2/ldap
infosys/cluster

Using the parser as a Python module:

from arc.utils import config

# initial parsing with defaults
config.parse_arc_conf('/etc/arc.conf', '/usr/share/arc/arc.parser.defaults')

# get parsed dictionary and list of blocks in the arc.conf order
>>> confdict = config.get_config_dict()
>>> confblocks = config.get_config_blocks()

# get list of all [queue] subblocks sorted by name
>>> sb = config.get_subblocks(['queue'], is_sorted=True)
>>> sb
['queue:grid', 'queue:grid_rt']

# get value of 'x509_host_key' from [arex] block and than from [common] if not found in [arex]
>>> a = config.get_value('x509_host_key', ['arex', 'common'])
>>> a
'/etc/grid-security/hostkey.pem'

# get value of 'allowactivedata' option from [gridftpd] block
>>> b = config.get_value('allowactivedata', 'gridftpd')
>>> b
'yes'

# get value of 'allowactivedata' option from [gridftpd] block (always return list)
>>> c = config.get_value('allowactivedata', 'gridftpd', force_list=True)
>>> c
['yes']

# get value of 'allowactivedata' option from [gridftpd] block (return boolean value)
>>> d = config.get_value('allowactivedata', 'gridftpd', bool_yesno=True)
>>> d
True