Sandbox Frequently Asked Questions¶

• Table of contents
• Sandbox Frequently Asked Questions
  • Installation of applications and software
    • How do I install external libraries (e.g. HDF5)?
    • Several jobs of my workflow use the same software, where is it installed?
    • Am I a sudoer ?
  • Parameters and inputs
    • How do I manage the inputs to a job?
  • Jobs
    • What environmental variables can I use in my jobs?
    • How do I test a single job of a workflow?
  • Workflows
    • How do I test a workflow?
    • How do I access the details of my workflow run?
    • How do I access the results of my workflow?

Installation of applications and software¶

How do I install external libraries (e.g. HDF5)?¶

Libraries and associated binaries (e.g. h5dump) shall be made available via yum:

[user@fb ~] sudo yum search hdf

This will list all packages related with HDF.
We will install the hdf5 libraries and binaries with:

[user@sb ~]sudo yum install hdf5.x86_64

Several jobs of my workflow use the same software, where is it installed?¶

When several jobs use the same software (e.g. NEST toolbox or CDAT) the location to install these packages is:

/application/share/<software name>

instead of installing it several times under

/application/job1/
/application/job2/

Am I a sudoer ?¶

Yes, you are, but with the command listed below:

• yum - yum allows you to install packages on your sandbox

Parameters and inputs¶

How do I manage the inputs to a job?¶

There are several ways to pass inputs to a job:

• Local inputs - local files will use the file:// protocol and are defined in the workflow as follows:

<workflow id="somename">                            
    <workflowVersion>1.0</workflowVersion>
    <node id="somenodeid">
        <job id="ceda-collect"></job>
        <sources>
            <source refid="file:urls" >/application/input.urls</source>
        </sources>
    </node>
</workflow>

and the file input.urls contains the references to the local files:

[ user@sb ~] cat /application/input.urls
file:///home/user/somefile1
file:///home/user/somefile2
file:///home/user/somefile3

Then the job executable can use ciop-copy to copy the files if needed.

while read inputfile
do
    echo $inputfile | ciop-copy -o ./ - 
done 

• Values

Passing values to a job follows the same approach as above.

<workflow id="somename">                            
    <workflowVersion>1.0</workflowVersion>
    <node id="somenodeid">
        <job id="ceda-collect"></job>
        <sources>
            <source refid="file:urls" >/application/inputparams</source>
        </sources>
    </node>
</workflow>

and the file inputparams contains the list of values:

[ user@sb ~] cat /application/inputparams
-10,-10,10,10
10,10,20,20

In the example above, the executable manages the parameters (bounding boxes) with:

while read bbox
do
    echo "processing bounding box $bbox" 
done 

• Products available in the Sandbox internal catalogue

During the sandbox definition and creation you may have selected a list of data products, the references to these products are available in the sandbox internal catalogue.
The workflow is defined as follows:

<workflow id="testVomir">
    <workflowVersion>1.0</workflowVersion>
    <node id="Vimage">                            <!-- workflow node unique id -->
    <job id="imager"></job>                    <!-- job defined above -->
    <sources>
        <source refid="cas:serie" >ATS_TOA_1P</source>
    </sources>
    <parameters>                            <!-- parameters of the job -->
        <parameter id="volcano_db"></parameter>
    </parameters>
    </node>

As an example, the job executable would contain the lines below to copy the data products locally:

while read product
do
    echo $product | ciop-copy -o ./ -
done

Jobs¶

What environmental variables can I use in my jobs?¶

CCBox provides the environmental variables:

• _CIOP_APPLICATION_PATH is the path to the application.xml files and all other underlying folders. Its value is /application

Note: do not use its value in the executable scripts, always use $_CIOP_APPLICATION_PATH

• _JOB_DIR
• TMPDIR is temporary directory for the task.
• _JOB_ID contains the job id
• _JOB_LOCAL_DIR is the job specific shared scratch space
• _TASK_ID is the task id
• _TASK_LOCAL_DIR is the task specific scratch space
• _TASK_NUM contains the number of tasks
• _TASK_INDEX

The best way to get acquainted to the values of the environmental variables is to have them logged in a job with:

ciop-log "DEBUG" "TMPDIR                  = $TMPDIR"                 
ciop-log "DEBUG" "_JOB_ID                 = ${_JOB_ID}"              
ciop-log "DEBUG" "_JOB_LOCAL_DIR          = ${_JOB_LOCAL_DIR}"          
ciop-log "DEBUG" "_TASK_ID                = ${_TASK_ID}"             
ciop-log "DEBUG" "_TASK_LOCAL_DIR         = ${_TASK_LOCAL_DIR}"      
ciop-log "DEBUG" "_TASK_NUM               = ${_TASK_NUM}"            
ciop-log "DEBUG" "_TASK_INDEX             = ${_TASK_INDEX}" 

How do I test a single job of a workflow?¶

For that you have to know the nodeid of the job in the workflow.

<workflow id="testVomir">
    <workflowVersion>1.0</workflowVersion>
    <node id="Vimage">
    ...
    </node>

With that value simply do:

[user@sb ~] ciop-simjob -f Vimage

Workflows¶

How do I test a workflow?¶

Simply run the command:

[ user@sb ~] ciop-simwf

How do I access the details of my workflow run?¶

When you run the ciop-simwf you'll see on your terminal window the image below. The link to the details is highlighted.
Copy and paste the URL on your browser and navigate through the pages to find details about the workflow execution.

How do I access the results of my workflow?¶

After a successful run of your workflow, your results including logs can be found in the folder:

/share/tmp/sandbox/<workflow name>