Configuration of the CREAM CE

From Gridkaschool


Now get your site-info.def from your site BDII or start with the example file /opt/glite/yaim/examples/siteinfo/site-info.def and save it somewhere you like, e.g. /root/yaim/ or /opt/glite/yaim/ and make sure that it is only readable by root.

mkdir /root/yaim
cd /root/yaim
scp gks-X-XYZ:/root/yaim/site-info.def /root/yaim
chmod 600 /root/yaim


For every supported virtual organization you need to define users in the users.conf file. For details please go to users.conf.


For every group of a virtual organization you are supporting you need to define it in the groups.conf file. For details please refer to groups.conf.


For the TORQUE and the CREAM Compute Element you need a file where all your worker nodes are listed. By default the file should be stored at /opt/glite/yaim/etc/wn-list.conf. In this course you have only one host available for a worker node, so create a file /opt/glite/yaim/etc/wn-list.conf and add its full hostname.


As for most grid services you need a host certificate for the CREAM Compute Element with the right permissions. For more details please refer to Host certificates.


Before the configuration script can be run the site-info.def has to be adapted. A good overview on the parameters available can be found in the LCG Wiki at CERN. Required Variables are:





#  # use this only if you still run a (depreceated) mon host
APEL_MYSQL_HOST="gks-X-XYZ"  # point to APEL host

CE_OS_ARCH=x86_64           # "uname -i" executed on the WN
CE_OS=ScientificSL          # "lsb_release -i | cut -f2" executed on the WN
CE_OS_RELEASE=5.4           # "lsb_release -r | cut -f2" executed on the WN
CE_OS_VERSION=Boron         # "lsb_release -c | cut -f2" executed on the WN

CE_CPU_MODEL=Opteron        # "grep 'model name' /proc/cpuinfo" executed on the WN - just take the main description (e.g. Opteron or Xeon)
CE_CPU_VENDOR=AMD           # "grep vendor /proc/cpuinfo" executed on the WN - just take the company name (e.g. AMD or Intel)
CE_CPU_SPEED=1800           # "grep MHz /proc/cpuinfo" executed on the WN
CE_MINPHYSMEM=1024          # "grep MemTotal /proc/meminfo" executed on the WN - attention: value is shown in kB, CE_MINPHYSMEM must be in MB!
CE_MINVIRTMEM=2048          # "grep SwapTotal /proc/meminfo" executed on the WN - attention: value is shown in kB, CE_MINVIRTMEM must be in MB!
CE_OUTBOUNDIP=TRUE          # do the WNs have permission for direct outbound connectivity?
CE_INBOUNDIP=FALSE          # do the WNs have permission for inbound connectivity?
CE_RUNTIMEENV="             # list of supported middleware/software
CE_CAPABILITY="none"        # if VO fairshares are defined in the batch system, they need to be reflected here, otherwise this variable must be none
CE_OTHERDESCR="Cores=4"     # number of cores per CPU for a typical WN ("grep -c "physical id.*: 0" /proc/cpuinfo")
CE_PHYSCPU=2                # total number of CPUs in the batch system ("grep -c "core id.*: 0" /proc/cpuinfo" * #WN)
CE_LOGCPU=1                 # number of logical CPUs or job slots in the batch cluster ("grep -c processor /proc/cpuinfo" * #WN)
CE_SMPSIZE=1                # number logical CPUs per Node ("grep -c processor /proc/cpuinfo")
CE_SI00=1592                # SpecInt value (see [1] for example values)
CE_SF00=1927                # SpecFP value (see [2] for example values)
CE_OTHER_DESCR="Cores=1, Benchmark=1.11-HEP-SPEC06"   # number of cores per CPU and Version of SPEC Benchmark

SE_LIST=""     # List of CloseSEs
SE_MOUNT_INFO_LIST="none"   # Mount info of the SE. If not supported by SE put "none" here.

VOS="dech"                  # supported VOs
QUEUES="test"               # The name of the queues defined in the Batch System
TEST_GROUP_ENABLE="dech"    # format of this parameter is: $QUEUE_GROUP_ENABLE e.g. for queue "test" and VO "dech" put TEST_GROUP_ENABLE="dech" here

ACCESS_BY_DOMAIN=false      # allow access to CREAM DB from other computers?
CREAM_DB_USER=creamdb       # Cream DB user name 
CREAM_DB_PASSWORD="secretPassword" # CREAM DB password 
BLPARSER_HOST=$CE_HOST      # Fully qualified name of machine hosting the BLAH blparser
BLP_PORT=33333              # Port where BLAH Blparser listens to 
CREAM_PORT=56565            # Port to access CREAM CE
CEMON_HOST=$CE_HOST         # Fully qualified name of CEMon host (do not use localhost !)

In addition some information for the supported virtual organizations have to be set. See VO configuration.

Configuration with yaim

NOTE: The function lists for the yaim configuration are located in /opt/glite/yaim/node-info.d/. You can have a look at the lists on the machines or in the wiki under category "Function lists". For the CREAM Compute Element configuration without a batch server you need the lists for glite-creamCE and glite-TORQUE_utils:

Functions used for the configuration of the CREAM Compute Element: Function list CREAM CE
Functions used for the configuration of the Torque utils: Function list TORQUE_utils

The accounting service running on the CREAM compute element will periodically check for new data in the directory /var/spool/pbs/server_priv/accounting. In our setup, this directory does not exist on the CREAM CE, but at the batch system server and is also filled by it. To allow the proper operation of the accounting service on the CREAM CE, we need to export this directory from the batch system server to the compute element.

  • Create the accounting directory on the CE and mount it from the batch system
mkdir -p /var/spool/pbs/server_priv/accounting
mount 141.52.174.XYZ:/var/spool/pbs/server_priv/accounting /var/spool/pbs/server_priv/accounting
  • If mounting the directory works, it needs to be included in the fstab. Add the following line to /etc/fstab
141.52.174.XYZ:/var/spool/pbs/server_priv/accounting /var/spool/pbs/server_priv/accounting nfs rw,soft 0 0

Similar steps are required for another service located at the CREAM compute element. The BLParser service is responsible for parsing the batch system server log files. By parsing the log files, the BLParser gains information about the status (queued, running, done, aborted,...) of the various grid jobs in the batch system. For proper operation you need to mount the log files from the batch system server to the compute element.

mount 141.52.174.XYZ:/var/spool/pbs/server_logs /var/spool/pbs/server_logs # on the CE command line

or alternatively

141.52.174.XYZ:/var/spool/pbs/server_logs /var/spool/pbs/server_logs nfs rw,soft 0 0 # in the CE /etc/fstab
mount -a # on the command line on the CE

Now call yaim for the node type creamCE and TORQUE_utils:

/opt/glite/yaim/bin/yaim -c -s /root/yaim/site-info.def -n glite-creamCE -n glite-TORQUE_utils

The output of yaim is listed below. Your output should look similar to it.

  INFO: Using site configuration file: /root/yaim/site-info.def
        .             /'.-. ')
        .     yA,-"-,( ,m,:/ )   .oo.     oo    o      ooo  o.     .oo
        .    /      .-Y a  a Y-.     8. .8'    8'8.     8    8b   d'8
        .   /           ~ ~ /         8'    .8oo88.     8    8  8'  8
        . (_/         '===='          8    .8'     8.   8    8  Y   8
        .   Y,--,Yy,-.,/           o8o  o8o    o88o  o8o  o8o    o8o
        .    I_))_) I_))_)
        current working directory: /root/yaim
        site-info.def date: Sep 1 12:19 /root/yaim/site-info.def
        yaim command: -c -s /root/yaim/site-info.def -n glite-creamCE -n glite-TORQUE_utils
        log file: /opt/glite/yaim/bin/../log/yaimlog
        Wed Sep  1 12:19:58 CEST 2010 : /opt/glite/yaim/bin/yaim
        Installed YAIM versions:
        glite-yaim-core 4.0.12-1
        glite-yaim-cream-ce 4.1.0-14
        glite-yaim-torque-utils 4.0.4-1
  INFO: The default location of the grid-env.(c)sh files will be: /opt/glite/etc/profile.d
  INFO: Sourcing the utilities in /opt/glite/yaim/functions/utils
  INFO: Detecting environment
  INFO: Executing function: config_cream_stop_check 
  INFO: Executing function: config_cream_clean_check 
  INFO: Executing function: config_cream_db_check 
  INFO: Executing function: config_add_pool_env_check 
  INFO: Executing function: config_host_certs_check 
  INFO: Executing function: config_vomsdir_check 
  INFO: Executing function: config_vomses_check 
  INFO: Executing function: config_users_check 
  INFO: Executing function: config_edgusers_check 
  INFO: Executing function: config_cream_glexec_user_check 
  INFO: Executing function: config_cream_sudoers_check 
  INFO: Executing function: config_secure_tomcat_check 
  INFO: Executing function: config_vomsmap_check 
  INFO: Executing function: config_globus_clients_check 
  INFO: Executing function: config_rgma_client_check 
  INFO: Executing function: config_lcas_lcmaps_gt4_check 
  INFO: Executing function: config_globus_gridftp_check 
  INFO: Executing function: config_cream_glexec_check 
  INFO: Executing function: config_cream_blah_check 
  INFO: Executing function: config_cream_ce_check 
  INFO: Executing function: config_cream_logrotation_check 
  INFO: Executing function: config_cream_gip_check 
  INFO: Executing function: config_gip_scheduler_plugin_check 
  INFO: Executing function: config_gip_vo_tag_check 
  INFO: Executing function: config_info_service_cream_ce_check 
  INFO: Executing function: config_info_service_cemon_check 
  INFO: Executing function: config_cream_cemon_check 
  INFO: Executing function: config_cream_gliteservices_check 
  INFO: Executing function: config_cream_locallogger_check 
  INFO: Executing function: config_glite_locallogger_check 
  INFO: Executing function: config_maui_cfg_check 
  INFO: Executing function: config_apel_pbs_check 
  INFO: Executing function: config_gip_sched_plugin_pbs_check 
  INFO: Executing function: config_torque_submitter_ssh_check 
  INFO: Executing function: config_bdii_only_check 
  INFO: Executing function: config_cream_stop_setenv 
  INFO: Executing function: config_cream_stop 
  INFO: blah not running
  INFO: tomcat not running
  INFO: blah not running
  INFO: lb processes not running
  INFO: Executing function: config_cream_clean_setenv 
  INFO: Executing function: config_cream_clean 
  INFO: Executing function: config_cream_db_setenv 
  INFO: Executing function: config_cream_db 
Initializing MySQL database:  Installing MySQL system tables...
Filling help tables...

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h gks-1-126 password 'new-password'

Alternatively you can run:

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with
cd mysql-test ; perl

Please report any problems with the /usr/bin/mysqlbug script!

The latest information about MySQL is available on the web at
Support MySQL by buying support/licenses at
                                                          [  OK  ]
Starting MySQL:                                            [  OK  ]
The database version requested by cream service is 2.4
Impossible to retrieve the version of creamdb database. Database will be created from scratch.
Creating/Updating creamdb database...
creamdb database created!
The database version requested by cream service is 2.4
Impossible to retrieve the version of delegationdb database. Database will be created from scratch.
Creating/Updating delegationdb database...
delegationdb database created!
  INFO: Executing function: config_add_pool_env_setenv 
  INFO: Executing function: config_add_pool_env 
  INFO: Executing function: config_ldconf 
  INFO: Executing function: config_sysconfig_edg 
  INFO: Executing function: config_host_certs 
  INFO: Executing function: config_crl 
  INFO: Now updating the CRLs - this may take a few minutes...
  WARNING: /opt/glite/libexec/ didn't finish succesfully
  WARNING: CRLs may not be updated, please have a look !
  INFO: Executing function: config_vomsdir_setenv 
  INFO: Executing function: config_vomsdir 
  INFO: Executing function: config_vomses 
  INFO: Executing function: config_users 
  INFO: Executing function: config_edgusers 
  INFO: Executing function: config_cream_glexec_user_setenv 
  INFO: Executing function: config_cream_glexec_user 
  INFO: CONFIG_USERS is set to yes
  INFO: Executing function: config_cream_sudoers_setenv 
  INFO: Executing function: config_cream_sudoers 
  INFO: Executing function: config_secure_tomcat_setenv 
/usr/bin/rebuild-jar-repository: error: Could not find xml-commons-apis Java extension for this JVM
/usr/bin/rebuild-jar-repository: error: Some detected jars were not found for this jvm
  INFO: Executing function: config_secure_tomcat 
  INFO: Check that java is installed
  INFO: Check that tomcat is installed
  INFO: Stop tomcat in case it's running
Stopping tomcat5:    INFO: Copying hostcert to /etc/grid-security/tomcat-cert.pem for tomcat:root......
  INFO: Copying hostkey to /etc/grid-security/tomcat-key.pem for tomcat:root...
  INFO: Configuring /etc/tomcat5/server.xml...
  INFO: Copying trustmanager deps to tomcat server lib directory..
  INFO: Defining JAVA_HOME in the Tomcat configuration file
  INFO: Starting Tomcat
Starting tomcat5:                                          [  OK  ]
  INFO: Executing function: config_vomsmap_setenv 
  INFO: Executing function: config_vomsmap 
  INFO: Creating grid-map directory in /etc/grid-security/gridmapdir
  INFO: Creating voms grid-map file in /etc/grid-security/voms-grid-mapfile
  INFO: Creating voms groupmap file in /etc/grid-security/groupmapfile
  INFO: Copying the /etc/grid-security/voms-grid-mapfile in the standard location /etc/grid-security/grid-mapfile
  INFO: Executing function: config_globus_clients_setenv 
  INFO: Executing function: config_globus_clients 
  INFO: Configure the globus service
setup-tmpdirs: creating ./config.status
config.status: creating globus-script-initializer
config.status: creating
creating globus-script-initializer
creating Globus::Core::Paths
checking globus-hostname
  INFO: Executing function: config_rgma_client_setenv 
  INFO: Executing function: config_rgma_client 
  INFO: YAIM has detected the OS is SL5. The rgma client is no longer configured in SL5.
  INFO: Executing function: config_lcas_lcmaps_gt4_setenv 
  INFO: Executing function: config_lcas_lcmaps_gt4 
  INFO: Creating LCAS_DB_FILE in /opt/glite/etc/lcas/lcas.db
  INFO: Creating LCMAPS_DB_FILE in /opt/glite/etc/lcmaps/lcmaps.db
  INFO: Executing function: config_globus_gridftp_setenv 
  INFO: Executing function: config_globus_gridftp 
  INFO: Starting gridftp service :
Shutting down globus-gridftp-server: [FAILED]
Starting globus-gridftp-server[  OK  ]
  INFO: Executing function: config_cream_glexec_setenv 
  INFO: Executing function: config_cream_glexec 
  INFO: Executing function: config_cream_blah_setenv 
  INFO: Executing function: config_cream_blah 
  INFO: Executing function: config_cream_ce_setenv 
  INFO: Executing function: config_cream_ce 
Starting tomcat5: tomcat5 process already running
  INFO: Executing function: config_cream_logrotation_setenv 
  INFO: Executing function: config_cream_logrotation 
  INFO: Executing function: config_gip_only 
  INFO: Executing function: config_cream_gip_setenv 
  INFO: Executing function: config_cream_gip 
  INFO: Executing function: config_gip_scheduler_plugin_setenv 
  INFO: Executing function: config_gip_scheduler_plugin 
  INFO: Executing function: config_cream_gip_software_plugin 
  INFO: Executing function: config_gip_vo_tag 
  INFO: Executing function: config_gip_service_release 
  INFO: Executing function: config_info_service_cream_ce_setenv 
  INFO: Executing function: config_info_service_cream_ce 
  INFO: Executing function: config_info_service_cemon_setenv 
glite-lb-interlogd: no process killed
[26574] Initializing...
[26574] Parse messages for correctness... [yes]
[26574] Send messages also to inter-logger... [yes]
[26574] Messages will be stored with the filename prefix "/var/glite/log/dglogd.log".
[26574] Server running with certificate: /C=DE/O=GermanGrid/OU=dech-school/
[26574] Listening on port 9002
[26574] Running as daemon... [yes]
  INFO: Executing function: config_info_service_cemon 
  INFO: Executing function: config_cream_cemon_setenv 
  INFO: Executing function: config_cream_cemon 
  INFO: Executing function: config_cream_gliteservices_setenv 
  INFO: Executing function: config_cream_gliteservices 
  INFO: Executing function: config_cream_locallogger_setenv 
  INFO: Executing function: config_cream_locallogger 
  INFO: Executing function: config_glite_locallogger_setenv 
  INFO: Executing function: config_glite_locallogger 
  INFO: Applying the workaround for bug 22389...
Stopping glite-lb-logd ... not running
Stopping glite-lb-interlogd ... not running
Starting glite-lb-logd ...This is LocalLogger, part of Workload Management System in EU DataGrid & EGEE.
Starting glite-lb-interlogd ... done
  INFO: Executing function: config_glite_initd 
  INFO: Executing function: config_maui_cfg_setenv 
  INFO: Executing function: config_maui_cfg 
  INFO: configuring maui ...
  INFO: Executing function: config_apel_pbs_setenv 
  INFO: Executing function: config_apel_pbs 
  INFO: Executing function: config_gip_sched_plugin_pbs_setenv 
  INFO: Executing function: config_gip_sched_plugin_pbs 
  INFO: Executing function: config_torque_submitter_ssh 
Reloading sshd: [  OK  ]
  INFO: Executing function: config_bdii_only 
Stopping BDII: BDII Already stopped
Starting SLAPD: [  OK  ]

Starting update process: [  OK  ]

  INFO: Configuration Complete.                                               [  OK  ]
  INFO: YAIM terminated succesfully 

As you may have noticed yaim gave a warning:

WARNING: /opt/glite/libexec/ didn't finish succesfully
WARNING: CRLs may not be updated, please have a look !

If you also receive this warning, please try to find out the cause for this warning. You could e.g. have a look at the contents of the /opt/glite/libexec/ script and execute them manually.

Sometimes also another error shows up:

  ERROR: Error during the execution of function: config_bdii_only
  ERROR: Error during the configuration.Exiting.                              [FAILED]
  ERROR: One of the functions returned with error without specifying it's nature !

In this case restart the bdii service manually by

sh -x /etc/init.d/bdii start

and check if it is properly starting.

/etc/init.d/bdii status

The last step is to configure the BLParser by running yaim again with the options as shown below

/opt/glite/yaim/bin/yaim -f -s /root/yaim/site-info.def -f config_cream_blparser

Go to CREAM Compute Element Testing

Go back to gLite Administration Course, CREAM CE, Installation of a CREAM CE