Developer Cloud Sandboxes solution

h1. Frequently Asked Questions

{{toc}}

h2. Installation of application and software

h3. How do I install external libraries (e.g. HDF5)?

HDF5 libraries and associated binaries (e.g. h5dump) are available via yum:

<pre>

[user@fb ~] sudo yum search hdf

</pre>

This will list all packages related with HDF. 

We will install the hdf5 libraries and binaries with:

<pre>

[user@sb ~] sudo yum install hdf5.x86_64

</pre>

h3. Several jobs of my workflow use the same software, where is it installed?

When several jobs use the same software (e.g. NEST toolbox or CDAT) the location to install these packages is:

<pre>

/application/share/<software name>

</pre>

instead of installing it several times under 

<pre>

/application/job1/

/application/job2/

</pre>

h3. Am I a sudoer?

Yes, you are but with limited power, you can do:

* yum - yum allows you to install packages on your sandbox

h2. Parameters and inputs

h3. How do I manage the inputs to a job?

There are several ways to pass inputs to a job:

* Local inputs - local files will use the file:// protocol and are defined in the workflow as follows:

<pre><code class="xml">

<workflow id="somename">							

	<workflowVersion>1.0</workflowVersion>

	<node id="somenodeid">

		<job id="ceda-collect"></job>

		<sources>

			<source refid="file:urls" >/application/input.urls</source>

		</sources>

	</node>

</workflow>

</code></pre>

and the file _input.urls_ contains the references to the local files:

<pre><code class="ruby">

[ user@sb ~] cat /application/input.urls

file:///home/user/somefile1

file:///home/user/somefile2

file:///home/user/somefile3

</code></pre>

Then the job executable can use ciop-copy to copy the files if needed.

<pre><code class="c">

while read inputfile

do

	echo $inputfile | ciop-copy -o ./ - 

done 

</code></pre>

> Tip: check the [[ciop-copy] CLI reference

* Values

Passing values to a job follows the same approach as above. 

<pre><code class="xml">

<workflow id="somename">							

	<workflowVersion>1.0</workflowVersion>

	<node id="somenodeid">

		<job id="ceda-collect"></job>

		<sources>

			<source refid="file:urls" >/application/inputparams</source>

		</sources>

	</node>

</workflow>

</code></pre>

and the file _inputparams_ contains the list of values:

<pre><code class="ruby">

[ user@sb ~] cat /application/inputparams

-10,-10,10,10

10,10,20,20

</code></pre>

In the example above, the executable manages the parameters (bounding boxes) with:

<pre><code class="ruby">

while read bbox

do

	echo "processing bounding box $bbox"

done 

</code></pre>

* Products available in the sandbox internal catalogue

During the sandbox definition and creation you may have selected a list of EO products, the references to these products are available in the sandbox internal catalogue.

The workflow is defined as follows:

<pre><code class="xml">

<workflow id="testVomir">

	<workflowVersion>1.0</workflowVersion>

	<node id="Vimage">							<!-- workflow node unique id -->

	<job id="imager"></job>					<!-- job defined above -->

	<sources>

		<source refid="cas:serie" >ATS_TOA_1P</source>

	</sources>

	<parameters>							<!-- parameters of the job -->

		<parameter id="volcano_db"></parameter>

	</parameters>

	</node>

</code></pre>

As an example, the job executable would contain the lines below to copy the EO products locally: 

<pre><code class="ruby">

while read product

do

	echo $product | ciop-copy -o ./ -

done

</code></pre>

> Tip: check the [[ciop-copy] CLI reference

h2. Job 

h3. What environmental variables can I use in my jobs?

CIOP provides the environmental variables:

* _CIOP_APPLICATION_PATH is the path to the application.xml files and all other underlying folders. Its value is /application

> Note: do not use its value in the executable scripts, always use $_CIOP_APPLICATION_PATH  

* _JOB_DIR 

* TMPDIR is temporary directory for the task.

* _JOB_ID contains the job id

* _JOB_LOCAL_DIR is the job specific shared scratch space 

* _TASK_ID is the task id

* _TASK_LOCAL_DIR is the task specific scratch space

* _TASK_NUM contains the number of tasks

* _TASK_INDEX 

The best way to get acquainted to the values of the environmental variables is to have them logged in a job with:

<pre>

ciop-log "DEBUG" "TMPDIR                  = $TMPDIR"                 

ciop-log "DEBUG" "_JOB_ID                 = ${_JOB_ID}"              

ciop-log "DEBUG" "_JOB_LOCAL_DIR          = ${_JOB_LOCAL_DIR}"          

ciop-log "DEBUG" "_TASK_ID                = ${_TASK_ID}"             

ciop-log "DEBUG" "_TASK_LOCAL_DIR         = ${_TASK_LOCAL_DIR}"      

ciop-log "DEBUG" "_TASK_NUM               = ${_TASK_NUM}"            

ciop-log "DEBUG" "_TASK_INDEX             = ${_TASK_INDEX}"

</pre>

h3. How do I test a single job of a workflow?

For that you have to know the nodeid of the job in the workflow. Use the command below to list the nodes of the application workflow:

<pre>

[user@sb ~] ciop-simjob -n 

</pre>

Or check the *application.xml* file and look for the node name

<pre><code class="xml">

<workflow id="testVomir">

	<workflowVersion>1.0</workflowVersion>

	<node id="Vimage">

	...

	</node>

</code></pre>

With that value simply do:

<pre>

[user@sb ~] ciop-simjob -f Vimage

</pre>

> Tip: check the [[ciop-simjob] CLI reference

h2. Workflow

h3. How do I test a workflow?

Simply run the command:

<pre><code class="ruby">

[ user@sb ~] ciop-simwf

</code></pre>

> Tip: check the [[ciop-simwf]] CLI reference

h3. How do I access the details of my workflow run?

When you run the _ciop-simwf_ you'll see on your terminal window the image below. The link to the details is highlighted.

Copy and paste the URL on your browser and navigate through the pages to find details about the workflow execution.

!workflow_url.png!

h3. How do I access the results of my workflow run?

After a successful run of your workflow, your results including logs can be found in the folder:

<pre>

/share/tmp/sandbox/<workflow name>

</pre>

You can list the run identifiers with:

<pre><code class="ruby">

[ user@sb ~] ciop-simwf -l

</code></pre>