Chapter 5. Configuration

Table of Contents

5.1. PoD user defaults configuration
5.2. User's environment on workers
5.3. XROOTD/XPROOFD

As it was mentioned above PoD consists of several modules, each module respects PoD user defaults settings. PoD is shipped with predefined configuration values, which should work in most use cases. However by changing PoD user defaults values, you can fine-tune PoD for a specific environment and needs. Recommended only for advanced users.

5.1. PoD user defaults configuration

Since PoD v2.1.1 a user defaults configuration file is supported. This is the configuration entry point of PoD. All modules configure themselves according to that file. It must be located either in the $HOME/.PoD/ directory on in the $POD_LOCATION/etc/ directory and be called PoD.cfg.

[Tip]Tip

Every time users sources PoD's environment script, the script checks whether the configuration file is exists and creates it if it's missing. Also the pod-user-defaults(1) command can be used to create the default configuration file.

The PoD.cfg is a simple INI-like configuration file. Configuration file syntax is line based:

  • A line in the form:

    key_name=value
    

    gives a value to an option.

  • A line in the form:

    [section name]
    

    introduces a new section in the configuration file.

  • The # character introduces a comment that spans until the end of the line.

    The option names are relative to the section names, so the following configuration file part:

    [gui.accessibility]
    visual_bell=yes
    

    is equivalent to

    gui.accessibility.visual_bell=yes
    

Table 5.1. PoD server configuration

keyvalueDescription
server.work_dirstring (default: $HOME/.PoD)PoD's working directory. Used by PoD modules to store temporary files, like pid files, for example. A string of the value will be evaluated before it is used, it therefore can contain environment variables.
server.logfile_dirstring (default: $HOME/.PoD/log)A path for PoD's log files. By the defined path PoD modules will place log files.
server.logfile_overwriteyes/no (default: yes)Defines whether PoD should overwrite its log files when starting a new session (PoD server's start/restart cycle)
server.log_levelnumeric (default: 1)Defines the level of the log. There are following numeric values are allowed:

  • 0: Fault/Critical

  • 1: Fault/Critical/Info

  • 2: Fault/Critical/Info/Warning

  • 3: Fault/Critical/Info/Warning/Debug

server.agent_shutdown_if_idle_for_secnumeric (default: 1800)Shut down a server if its idle time is higher than the defined value in seconds.
server.agent_local_client_port_(min/max)numeric (default: 20000/25000)

Recommended for advanced users only.

The following range is used by PoD agent locally on the server host, when in the packet-forwarding mode. Each PROOF client gets its proxy redirected vie the ports from that range.
server.xproof_ports_range_(min/max)numeric (default: 21001/22000)

Recommended for advanced users only.

PoD's automatic port mapping algorithms use this range to dynamically assign ports to xproof plug-in of xrootd when restarting a PoD server. In multi-user/core environment, when there are many PoD processes on the same physical machine, the automatic port mapping prevents different PoD process of different users to disturb each other.
server.agent_ports_range_(min/max)numeric (default: 22001/23000)

Recommended for advanced users only.

PoD's automatic port mapping algorithms use this range to dynamically assign ports to pod-agent when restarting a PoD server. In multi-user/core environment, when there are many PoD processes on the same physical machine, the automatic port mapping prevents different PoD process of different users to disturb each other.
server.agent_threadsnumeric (default: 5) A number of threads in a thread pool. The thread pool is used by the pod-agent to distribute tasks of a proxy, when in the packet-forwarding mode.
server.agent_node_readbuffernumeric (default: 5000)A buffer size, used by the packet-forwarding algorithms (in bytes). It will be allocated for each PoD worker.
server.packet_forwarding yes/no/auto (default: auto)If workers are behind a firewall than PoD will use its packet-forwarding (PF) algorithms to maintain the PROOF traffic between server and workers. By setting this key to "yes" you force PoD to use PF in any case. If "auto" is set than PoD will decide on the fly whether to use PF for each worker individually based on the possibility to directly connect to worker.


Table 5.2. PoD worker configuration

keyvalueDescription
worker.work_dirstring (default: $POD_LOCATION)PoD's working directory. Used by PoD modules to store temporary files, like pid files, for example. A string of the value will be evaluated before it is used, it therefore can contain environment variables.
worker.logfile_dirstring (default: $POD_LOCATION)A path for PoD's log files. By the defined path PoD modules will place log files.
worker.logfile_overwriteyes/no (default: yes)Defines whether PoD should overwrite its log files when starting a new session (PoD worker's start/restart cycle)
worker.log_levelnumeric (default: 1)Defines the level of the log. There are following numeric values are allowed:

  • 0: Fault/Critical

  • 1: Fault/Critical/Info

  • 2: Fault/Critical/Info/Warning

  • 3: Fault/Critical/Info/Warning/Debug

worker.set_my_rootsysyes/no (default: yes)Whether to use user's ROOTSYS on workers. If set to "yes", than the value of the worker.my_rootsys key, will be exported to the workers. See worker.my_rootsys for more details. If set to "no", PoD will download a default, pre-compiled version of ROOT according to WN's environment.
worker.my_rootsysstring (default: $ROOTSYS)User's ROOTSYS to use on workers. If set_my_rootsys is set to "yes", than PoD will export bin and library locations of this ROOT version on the worker nodes. This is especially useful if you use shared home file system on the nodes where PoD workers are started or you know for sure the location of the ROOT installation on the worker nodes. A string of the value will be evaluated before it is used, it therefore can contain environment variables.
worker.agent_shutdown_if_idle_for_secnumeric (default: 1800)Shut down a worker if its idle time is higher than the defined value in seconds.
worker.xproof_ports_range_(min/max)numeric (default: 21001/22000)

Recommended for advanced users only.

PoD's automatic port mapping algorithms use this range to dynamically assign ports to xproof plug-in of xrootd when starting a PoD worker. In multi-user/core environment, when there are many PoD processes on the same physical machine, the automatic port mapping prevents different PoD process of different users to disturb each other.
worker.agent_node_readbuffernumeric (default: 5000)A buffer size, used by the packet-forwarding algorithms (in bytes). It will be allocated for each PoD worker.


Table 5.3. LSF plug-in configuration

keyvalueDescription
lsf_plugin.email_job_outputyes/no (default: no)The parameter specifies whether job's output is sent to the user by mail. if "no" is set, output will be delivered to the log directory in std_[INDEX].err and std_[INDEX].out files
lsf_plugin.upload_job_logyes/no (default: no)The parameter specifies whether to upload jobs log files from workers when PoD jobs are completed. Jobs log files include a full log of PROOF, XROOTD and pod-agent's log files.


Table 5.4. PBS plug-in configuration

keyvalueDescription
pbs_plugin.upload_job_logyes/no (default: no)The parameter specifies whether to upload jobs log files from workers when PoD jobs are completed. Jobs log files include a full log of PROOF, XROOTD and pod-agent's log files.
pbs_plugin.options_filestring (default: $POD_LOCATION/etc/Job.pbs.option) This file can be used to provide addirtional PBS (qsub) options. Just create a file and set its path in pbs_plugin.options_file. Write valid qsub options in one line, like if you would write them in a command line when calling qsub. PoD will automatically use it (if exists) while submitting PBS jobs. Be advised, that the following options are reserved and are set by PoD, if you want to adjust them in anyway, then, please, contact PoD support and we will find a way. The reserved options are: -N, -q, -j, -V, -v.


Table 5.5. Grid Engine plug-in configuration

keyvalueDescription
ge_plugin.upload_job_logyes/no (default: no) The parameter specifies whether to upload jobs log files from workers when PoD jobs are completed. Jobs log files include a full log of PROOF, XROOTD and pod-agent's log files.
ge_plugin.options_filestring (default: $POD_LOCATION/etc/Job.ge.option) PoD also supports an GE option file. If you want to provide some additional Grid Engine options to your PoD jobs submitted to OE cluster, to select some specific resource or something like that, than PoD gives you this possibility via an GE option file. Just create a file and set its path in ge_plugin.options_file. Write valid GE options in it. PoD will automatically use it (if exists) while submitting GE jobs. See qsub man page of GE for more information on the option file (search for the "-@" option in the man page).


Table 5.6. Condor plug-in configuration

keyvalueDescription
condor_plugin.upload_job_logyes/no (default: no)The parameter specifies whether to upload jobs log files from workers when PoD jobs are completed. Jobs log files include a full log of PROOF, XROOTD and pod-agent's log files.
condor_plugin.options_filestring (default: $POD_LOCATION/etc/Job.condor.option) PoD is shipped with a default Condor job description file, which is used to submit PoD jobs. If users need to use additional settings or requirements, in order to tune PoD job submission, these settings can be provided via a file specified by the condor_plugin.options_file option. Settings from this file will be added to the default PoD job description file. The options file should in the format of standard condor description files.


[Important]Important
All port ranges in the PoD configuration must not have intersections.

5.2. User's environment on workers

PoD provides a possibility for users to execute a custom environment script on workers before PoD processes start.

Users need to create a shell script file with the $POD_LOCATION/etc/user_worker_env.sh or $HOME/.PoD/user_worker_env.sh name and to code there all variables and commands to export to the workers. PoD will automatically transfer the script to each worker node and source it there.

For example, If I need to set the path to my ROOT installation on workers. I would create the following file.

#! /usr/bin/env bash

source /usr/local/pub/debian4.0/x86_64/gcc411-21/526-00/bin/thisroot.sh

export LD_LIBRARY_PATH=$MYLIBS/lib:$LD_LIBRARY_PATH
export MYVAR="some vallue :)"

# I need also my special profile there
source /etc/profile_extr

Be advised, that you need to recreate PoD worker package every time, when you modify the user script or if you removed it. To recreate the package, just call: pod-prep-worker(1)

The SSH plug-in has it's own machinery to setup custom environment on worker nodes. Please check the pod-ssh(1) documentation for more information.

5.3. XROOTD/XPROOFD

There is a default XROOTD configuration file, $HOME/.PoD/etc/xpd.cf. The file is generated from the template ($POD_LOCATION/etc/xpd.cf.in) each time PoD server is started. PoD uses this file to configure both local server and remote workers.

[Tip]Tip

In XROOTD documentation you can find details of fine tuning of xrootd. But it is only recommended for advanced users.

The default xrootd configuration, which comes with PoD should be sufficient for basic operations. In most of use cases it is not needed to modify the configuration.

If you need additional xpd configuration settings, you can add custom xpd configuration files. PoD will scan for $HOME/.PoD/user_xpd.cf* and for $POD_LOCATION/etc/user_xpd.cf* and append the found files to the main xpd.cf. The star symbol in the file names can be change to any other symbol. For example, the following files will be appended to the main xpd.cf: $POD_LOCATION/etc/user_xpd.cf0, $POD_LOCATION/etc/user_xpd.cf1, $POD_LOCATION/etc/user_xpd.cf2.

PoD is only meant to help to setup a PROOF cluster on the fly using remote worker nodes. A data access is not a part of its responsibility.