Frequently Asked Questions¶

• Table of contents
• Frequently Asked Questions
  • Installation of application and software
    • How do I install external libraries (e.g. HDF5)?
    • Several jobs of my workflow use the same software, where is it installed?
    • Am I a sudoer?
  • Parameters and inputs
    • How do I manage the inputs to a job?
  • Job
    • What environmental variables can I use in my jobs?
    • How do I test a single job of a workflow?
  • Workflow
    • How do I test a workflow?
    • How do I access the details of my workflow run?
    • How do I access the results of my workflow run?

Installation of application and software¶

How do I install external libraries (e.g. HDF5)?¶

HDF5 libraries and associated binaries (e.g. h5dump) are available via yum:

[user@fb ~] sudo yum search hdf

This will list all packages related with HDF.
We will install the hdf5 libraries and binaries with:

[user@sb ~] sudo yum install hdf5.x86_64

Several jobs of my workflow use the same software, where is it installed?¶

When several jobs use the same software (e.g. NEST toolbox or CDAT) the location to install these packages is:

/application/share/<software name>

instead of installing it several times under

/application/job1/
/application/job2/

Am I a sudoer?¶

Yes, you are but with limited power, you can do:

• yum - yum allows you to install packages on your sandbox

Parameters and inputs¶

How do I manage the inputs to a job?¶

There are several ways to pass inputs to a job:

• Local inputs - local files will use the file:// protocol and are defined in the workflow as follows:

<workflow id="somename">                            
    <workflowVersion>1.0</workflowVersion>
    <node id="somenodeid">
        <job id="ceda-collect"></job>
        <sources>
            <source refid="file:urls" >/application/input.urls</source>
        </sources>
    </node>
</workflow>

and the file input.urls contains the references to the local files:

[ user@sb ~] cat /application/input.urls
file:///home/user/somefile1
file:///home/user/somefile2
file:///home/user/somefile3

Then the job executable can use ciop-copy to copy the files if needed.

while read inputfile
do
    echo $inputfile | ciop-copy -o ./ - 
done 

Tip: check the [[ciop-copy] CLI reference

• Values

Passing values to a job follows the same approach as above.

<workflow id="somename">                            
    <workflowVersion>1.0</workflowVersion>
    <node id="somenodeid">
        <job id="ceda-collect"></job>
        <sources>
            <source refid="file:urls" >/application/inputparams</source>
        </sources>
    </node>
</workflow>

and the file inputparams contains the list of values:

[ user@sb ~] cat /application/inputparams
-10,-10,10,10
10,10,20,20

In the example above, the executable manages the parameters (bounding boxes) with:

while read bbox
do
    echo "processing bounding box $bbox" 
done 

• Products available in the sandbox internal catalogue

During the sandbox definition and creation you may have selected a list of EO products, the references to these products are available in the sandbox internal catalogue.
The workflow is defined as follows:

<workflow id="testVomir">
    <workflowVersion>1.0</workflowVersion>
    <node id="Vimage">                            <!-- workflow node unique id -->
    <job id="imager"></job>                    <!-- job defined above -->
    <sources>
        <source refid="cas:serie" >ATS_TOA_1P</source>
    </sources>
    <parameters>                            <!-- parameters of the job -->
        <parameter id="volcano_db"></parameter>
    </parameters>
    </node>

As an example, the job executable would contain the lines below to copy the EO products locally:

while read product
do
    echo $product | ciop-copy -o ./ -
done

Tip: check the [[ciop-copy] CLI reference

Job¶

What environmental variables can I use in my jobs?¶

CIOP provides the environmental variables:

• _CIOP_APPLICATION_PATH is the path to the application.xml files and all other underlying folders. Its value is /application

Note: do not use its value in the executable scripts, always use $_CIOP_APPLICATION_PATH

• _JOB_DIR
• TMPDIR is temporary directory for the task.
• _JOB_ID contains the job id
• _JOB_LOCAL_DIR is the job specific shared scratch space
• _TASK_ID is the task id
• _TASK_LOCAL_DIR is the task specific scratch space
• _TASK_NUM contains the number of tasks
• _TASK_INDEX

The best way to get acquainted to the values of the environmental variables is to have them logged in a job with:

ciop-log "DEBUG" "TMPDIR                  = $TMPDIR"                 
ciop-log "DEBUG" "_JOB_ID                 = ${_JOB_ID}"              
ciop-log "DEBUG" "_JOB_LOCAL_DIR          = ${_JOB_LOCAL_DIR}"          
ciop-log "DEBUG" "_TASK_ID                = ${_TASK_ID}"             
ciop-log "DEBUG" "_TASK_LOCAL_DIR         = ${_TASK_LOCAL_DIR}"      
ciop-log "DEBUG" "_TASK_NUM               = ${_TASK_NUM}"            
ciop-log "DEBUG" "_TASK_INDEX             = ${_TASK_INDEX}" 

How do I test a single job of a workflow?¶

For that you have to know the nodeid of the job in the workflow. Use the command below to list the nodes of the application workflow:

[user@sb ~] ciop-simjob -n 

Or check the application.xml file and look for the node name

<workflow id="testVomir">
    <workflowVersion>1.0</workflowVersion>
    <node id="Vimage">
    ...
    </node>

With that value simply do:

[user@sb ~] ciop-simjob -f Vimage

Tip: check the [[ciop-simjob] CLI reference

Workflow¶

How do I test a workflow?¶

Simply run the command:

[ user@sb ~] ciop-simwf

Tip: check the ciop-simwf CLI reference

How do I access the details of my workflow run?¶

When you run the ciop-simwf you'll see on your terminal window the image below. The link to the details is highlighted.
Copy and paste the URL on your browser and navigate through the pages to find details about the workflow execution.

How do I access the results of my workflow run?¶

After a successful run of your workflow, your results including logs can be found in the folder:

/share/tmp/sandbox/<workflow name>

You can list the run identifiers with:

[ user@sb ~] ciop-simwf -l