Developer Cloud Sandboxes solution

h1. GMTSAR tutorial

{{>toc}}

GMTSAR: Generic Mapping Tools (GMT) for Synthetic Aperture Radar

h2. Sandbox per-requisites

*We assume that you have already a Sandbox ready and the following items are completed:*

* *You have accessed your Sandbox as described in the Getting Started guide*

> It is not mandatory but strongly recommended to follow the [[Getting started]] tutorial before starting this one.

h2. 1. Application concepts and terminology

In order to ease the execution of the tutorial, it is important to understand the concept of an application and its terminology. This section describes the example application used all along this guide. When a word is put in *+underlined bold+*, it is an terminology keyword always designating the same concept.

h3. 1.1 The application workflow

Our example in this tutorial is an interferometry application that processes SAR data (Envisat ASAR Image Mode level 0) to generate interferogram between a master and one or more slave products. It is composed of 6 steps that run independently but in a specific order and produce results that are inputs for the other remaining steps.

The following figure illustrates the *+Workflow+* of our application as a directed acyclic graph (DAG). This is also how the CIOP framework handles the execution of processes in terms of parallel computing and orchestration of the processing steps.

!! _<<insert here the DAG that represent the workflow >>_

Each box represents a *+Job+* which is a step of our application process. The arrows represents the data flow between the *+jobs+*. When a *+job+* is connected to another, it means that the *+output+* of this *+job+* is passed as *+input+* for the other.

It is important to keep in mind that in CIOP framework, *+input+* and *+ouput+* are text references (e.g. to data). Indeed, when a *+job+* process *+input+*, it actually reads *line by line* the reference workflow as described in the next figure

!https://ciop.eo.esa.int/attachments/40/focus-align.png!

It is therefore important to define precisely the inter- *+job+* references.

h3. 1.2 The job

Each *+job+* has a set of basic characteristics:

* a unique *+Job name+* in the workflow. (e.g. 'PreProc')

* zero, 1 or several *+sources+* that define the *+jobs+* interdependency. In the example, the *+job+* 'Interfere' has 2 dependencies: 'AlignSlave' and 'MakeTropo'.

* a maximum number of simultaneous *+tasks+* in which it can be forked. This is further explained in section [[Sandbox Application Integration Tutorial#1.3 The processing task|1.3 The processing task]].

* a *+processing trigger*+ which is a software executable of the *+job template+* that handles *+input+*/*+output+* streaming process. Practically, the executable that reads the *+input+* lines and writes the *+output+* lines.

The job characteristics above are mandatory in the *+workflow+* definition. 

If incomplete, the CIOP framework reports error in the workflow.

For out tutorial example, here are the characteristics for 'PreProc', 'AlignSlave' and 'Interfere' *+jobs+*:

* *PreProc*

PreProc is the first job in the workflow. It takes both SAR products, master and slave, and pre-processess them:

> It is a job with a single task, the _defaultJobconf_ _property_ (a CIOP property, not an application property) _*ciop.job.max.tasks*_ is set to *1*

> Its executable is located is /application/preproc/run

> It has a number of application parameters to run: SAT, master, num_patches, near_range, earth_radius, fd1, stop_on_error. The parameter values are not set in this job template.

<pre><code class="xml">

<jobTemplate id="preproc">

	<streamingExecutable>/application/preproc/run</streamingExecutable>	<!-- processing trigger -->

	<defaultParameters>	<!-- default parameters of the job -->

		<!-- Default values are specified here -->

		<parameter id="SAT"></parameter>	<!-- no default value -->

		<parameter id="master"></parameter>

		<parameter id="num_patches"></parameter>	<!-- no default value -->

		<parameter id="near_range"></parameter>	<!-- no default value -->

		<parameter id="earth_radius"></parameter>	<!-- no default value -->

		<parameter id="fd1">1</parameter>	<!-- no default value -->

		<parameter id="stop_on_error">false</parameter> 	<!-- don't stop on error by default -->

	</defaultParameters>

	<defaultJobconf>

		<property id="ciop.job.max.tasks">1</property>	<!-- Maximum number of parallel tasks -->

	</defaultJobconf>

</jobTemplate>

</code></pre>

* *+Job name+*: *'AlignSlave'*

* *+sources+*: 'PreProc'

* maximum number of simultaneous *+tasks+*: unlimited

* *+processing trigger*+: /application/align/run

* *+Job name+*: *'Interfere'*

* *+sources+*:  'AlignSlave' and 'MakeTropo'

* maximum number of simultaneous *+tasks+*: 1

* *+processing trigger*+: /application/interfere/run

h3. 1.3 The processing task

To exploit the parallelism offered by the CIOP framework, a *+job+* may process its *+input+* in several *+tasks+*. In principle, the CIOP framework will run those *+tasks+* in parallel. This is an important and sometimes complex paradigm that can be addressed in different ways. 

The following questions & answers describe the parallelism paradigm of the CIOP framework.

* Is it *+task+* parallelism or *+job+* parallelism?

> In this section, we definitely speak about task parallelism. Job parallelism is at level !+1. In the example, 'MakeTropo' and 'AlignSlave' are two *+jobs+* that run in parallel besides the number of *+tasks+* each of them may trigger.

> 'AlignSlave' may be forked into an unlimited number of *+tasks+*. Practically the framework calculates automatically the number *n* of available processing slots in the computing resource and start *n* times the *+processing trigger+* (based on the number of *+inputs+*).

* How to divide a *+job+* into *+tasks+*?

> It is actually the application developer who chooses the granularity of the *+job+* division. The computing framework will simply divide the *+input+* flow (*k* lines) in to the *n* *+tasks+*. In the example provided in this tutorial, if the *+job+* 'PreProc' produces *+output+* of 11 lines and the computing resources divide the *+job+* 'AlignSlave' into 4 *+tasks+* and the following division is done:

!https://ciop.eo.esa.int/attachments/41/parallelism.png!

Where stands the processing loop?

> The processing loop stands in the +*processing trigger*+. As shown in this tutorial with the example, the *+processing triggers+* implements a loop that reads *line by line* the *+task*+ input.

h2. 2. Home Directory

The *+home+* directory of your Sandbox is attached to a separate savable disk and thus persistent. This disk is mounted on the /home/{USERNAME} folder. In this folder you may upload, compile, test manually, etc. all your data. This is a free space where you can do anything BUT:

> *+Be careful to never link any element (e.g. executable, auxiliary data) from the application directory to the home directory which is critical for the application+*. Indeed, the *+home+* directory is not present when using CIOP in Runtime Environment mode and therefore any linked elements won't be available and cause the processing phase to fail.

h2. 3. Application Directory

The *+application directory+* of your Sandbox is attached to a separate savable disk and thus persistent. This disk is mounted on the /application folder.

The application directory is the place where integrated application resides. 

It should be a clean environment and thus +*SHOULD NOT*+ contain any temporary files or used to do compilations and/or manual testing. Instead, it is used for the simulation the application *+jobs+* and +*workflow*+.

At the instantiation of your Sandbox, the *+application directory+* contains this sample application example unless you configured the Sandbox with one of your application disk previously saved in your application library.

In the next sections the elements of the application directory are described.

h3. 3.1 Files and folders structure

The application directory follows some best practices in its folders and files structure to ease the subsequent deployment of the application to the CIOP Runtime Environment.

The folder structure of the application example with the description of each item is shown below:

!https://ciop.eo.esa.int/attachments/44/application_folder.png!

> Even if the names are quite similar in the our tutorial example, *+job+* and +*job template*+ are not the same concept. _A *job* is an instance of a +*job template*+ in a given +*workflow*+. This paradigm allows to have several *+jobs+* in a +*workflow*+ that point to the same *+job template+*. This is explained in more detail in the next section.

h3. 3.2 The Application XML definition file

The Application XML definition file is the reference of your application for the CIOP computing framework. It contains all the characteristics of the *+job templates+* and the *+workflows+*.

The Application XML definition file is is described in the page [[Application XML definition file]].

h2. 4 -- Using sample datasets -- (section under revision)

This section guides you through the tutorial example to introduce the data manipulation tools. 

There are mainly two command line tools to discover and access data previously selected in your Sandbox ([[Getting_Started#2.1 Sandbox EO Data Services|see here]]):

* *ciop-catquery* to query the Sandbox catalogue containing all the metadata of the selected sample dataset

* *ciop-copy* to copy the data from logical of physical location to a local directory of the Sandbox

Use

<pre>ciop-<command> -h</pre>

to display the CLI reference

These commands can be used in the processing triggers of a +*job*+.

h3. 4.1 Query Sandbox catalogue

For the tutorial purpose, the first test is to ensure the test dataset needed for the Application integration and testing is complete.

(to be updated)

h3. 4.2 Copy data

To copy data from a reference link as displayed in the previous section, just use the following command:

<pre><code class="ruby">[user@sb ~]$ ciop-copy http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1/rdf</code></pre>

output:

<pre>

[INFO   ][ciop-copy][starting] url 'http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1/rdf' > local '/application/'

[INFO   ][ciop-copy][success] got URIs 'https://eo-virtual-archive4.esa.int/supersites/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1 '

[INFO   ][ciop-copy][starting] url 'https://eo-virtual-archive4.esa.int/supersites/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1' > local '/application/'

[INFO   ][ciop-copy][success] url 'https://eo-virtual-archive4.esa.int/supersites/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1' > local '/application/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1'

/home/user/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1

</pre>

The command displays information on the _stderr_ by default and returns on _stdout_ the path of the copied data.

Many other data schemas are supported by the ciop-copy CLI such as http, https, hdfs, etc. 

There are also many other options to specify the output directory or to unpack compressed data. 

The complete reference is available here [[ciop-copy CLI reference|ciop-copy usage]] or by using the inline help typing:

<pre><code class="ruby">[user@sb ~]$ciop-copy -h</code></pre>

h3. 4.3 Using other sources of data in a job

So far we have introduced two types of data sources:

* data coming from a catalogue series

* data coming from a previous job in the workflow.

In the first case, we define the workflow for the job imager:

<pre><code class="xml"><workflow id="testVomir">							<!-- Sample workflow -->

		<workflowVersion>1.0</workflowVersion>

		<node id="Vimage">							<!-- workflow node unique id -->

			<job id="imager"></job>					<!-- job defined above -->

			<sources>

				<source refid="cas:serie" >ATS_TOA_1P</source>

			</sources>

			<parameters>							<!-- parameters of the job -->

				<parameter id="volcano_db"></parameter>

			</parameters>

		</node>

		<node id="Quarc">

			<job id="quarcXML"/>

			<sources>

				<source refid="wf:node" >Vimage</source>

			</sources>

		</node></code></pre>

In the second case, we define the workflow for the Quarc job:

<pre><code class="xml"><workflow id="testVomir">							<!-- Sample workflow -->

		<workflowVersion>1.0</workflowVersion>

		<node id="Vimage">							<!-- workflow node unique id -->

			<job id="imager"></job>					<!-- job defined above -->

			<sources>

				<source refid="cas:serie" >ATS_TOA_1P</source>

			</sources>

			<parameters>							<!-- parameters of the job -->

				<parameter id="volcano_db"></parameter>

			</parameters>

		</node>

		<node id="Quarc">

			<job id="quarcXML"/>

			<sources>

				<source refid="wf:node" >Vimage</source>

			</sources>

		</node></code></pre>

It may be the case where the input data does not come from EO catalogues and thus there is the need to define another source of data.

<pre><code class="xml"><workflow id="someworkflow">							<!-- Sample workflow -->

		<workflowVersion>1.0</workflowVersion>

		<node id="somenode">							<!-- workflow node unique id -->

			<job id="somejobid"></job>					<!-- job defined above -->

			<sources>

				<source refid="file:urls" >/application/test.urls</source>

			</sources>

		</node>

</code></pre>

where the file test.urls contains the input lines that will be pipped to the processing trigger executable

h2. 5. Job integration

In this section, the +*job template+* 'align' and its instance the *+job+* 'AlignSlave' is integrated using the tools previously introduced in this tutorial.

h3. 5.1 Installation and configuration of the GMTSAR toolbox on the Sandbox

The steps below install the GMTSAR toolbox on the Sandbox. These are specific but it shows the common approach to follow when installing software on the Sandbox.

> The steps below are done in the +*home directory*+

* Step - Download the GMTSAR

GMTSAR software is available on the University of California web server:

<pre><code class="ruby">

[user@sb ~]$ wget http://topex.ucsd.edu/gmtsar/tar/GMTSAR.tar

</code></pre>

Then the GMTSAR.tar archive is unpacked

<pre><code class="ruby">

[user@sb ~]$ tar xvf GMTSAR.tar 

</code></pre>

GMTSAR relies on GMT with the dependencies netCDF, GMT is installed via yum:

<pre><code class="ruby">

[user@sb ~]$ cd GMTSAR

[user@sb GMTSAR]$ sudo yum search gmt

[user@sb GMTSAR]$ sudo yum install GMT-devel

[user@sb GMTSAR]$ sudo yum install netcdf-devel

[user@sb GMTSAR]$ make

</code></pre>

The steps above compile GMTSAR in the +*home directory*+. The required files (binaries, libraries, etc.) are copied to the /Application environment (remember that the +*home directory*+ is only available in the CIOP Sandbox mode and not in the Runtime Environment).

h3. 5.2 Job template definition in the application.xml

The application.xml file has two main blocks: the job template section and the workflow template section.

The first part is to define the *+job templates+* in the workflow XML application definition file.

Each processing block of the GMTSAR workflow needs a *+job template+*.

Here is the *+job template+* for the 'align'

<pre><code class="xml">

<jobTemplate id="preproc">

	<streamingExecutable>/application/preproc/run</streamingExecutable> <!-- processing trigger -->		

	<defaultParameters> <!-- default parameters of the job -->

		<!-- Default values are specified here -->

		<parameter id="SAT"></parameter>	<!-- no default value -->

		<parameter id="master"></parameter>

		<parameter id="num_patches"></parameter>	<!-- no default value -->

		<parameter id="near_range"></parameter>	<!-- no default value -->

		<parameter id="earth_radius"></parameter>	<!-- no default value -->

		<parameter id="fd1">1</parameter>	<!-- no default value -->

		<parameter id="stop_on_error">false</parameter>	<!-- dont stop on error by default -->

	</defaultParameters>

	<defaultJobconf>

		<property id="ciop.job.max.tasks">1</property>	<!-- Maximum number of parallel tasks -->

		</defaultJobconf>

</jobTemplate>

</code></pre>

To test this +*job*+ with the +*ciop-symjob*+, we need to fill the second part of the +*application.xml*+ to add a *+node+*:

<pre><code class="xml">

<node id="PreProc">	<!-- workflow node unique id -->

	<job id="preproc"></job> <!-- job template defined before -->

	<sources>

		<!-- Source is the series of data selection -->

		<source refid="cas:serie">ASA_IM__0P</source>

	</sources>

	<parameters>	<!-- parameters of the job -->

		<parameter id="SAT">ENV</parameter>

		<parameter id="master">http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1/rdf</parameter>

		<parameter id="near">978992.922</parameter>

		<parameter id="radius">6378000</parameter>

		<parameter id="stop_on_error">true</parameter>	<!-- during integration, preferably stop on error -->	

	</parameters>

</node>

</code></pre>

> The complete application.xml is available here: TBW

> The application.xml and all its elements are described in details in the page [[Application XML definition file]].

Since this processing is the first in the *+workflow+* chain, it has a special source which is the serie 'ASA_IM__0P'. Practically, it means that when the *+job+* is submitted for execution, the computing framework will query the Sandbox catalogue for the data ASA_IM__0P registered as sample dataset and will prepare a list of data reference as *+input+* for the job 'PreProc'. In our example, the resulting list is:

<pre>

http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPDE20090412_092436_000000162078_00079_37207_1556.N1/rdf

http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPAM20080427_092430_000000172068_00079_32197_3368.N1/rdf

</pre>

h3. 5.3 Processing trigger script 

In section 1.2, we have seen that each +*job*+ must have a processing trigger which is specified in <streamingExecutable> element of the +*job template*+. In our example, this executable shall be a shell script:

<pre><code class="ruby">

# FIRST OF ALL, LOAD CIOP INCLUDES

source ${ciop_job_include}

# If you want to have a complete debug information during implementation

ciop-enable-debug

# All return codes are predefined

SUCCESS=0

ERR_BADARG=2

ERR_MISSING_PREPROC_BIN=3

ERR_MISSING_NEAR_PARAM=4

ERR_MISSING_RADIUS_PARAM=5

ERR_MISSING_MASTER_PARAM=6

ERR_MISSING_SAT_PARAM=7

ERR_MISSING_FD1_PARAM=8

ERR_MISSING_NUMPATCH_PARAM=9

ERR_INPUT_DATA_COPY=18

ERR_PREPROC_ERROR=19

ERR_NOOUTPUT=20

DEBUG_EXIT=66

# This functions handle the exit of the executable

# with the corresponding error codes and will return a short message

# with the termination reason. It is important to have a synthetic and brief

# message because it will be raised to many upper level of the computing framework

# up to the user interface

function cleanExit ()

{

   local retval=$?

   local msg=""

   case "$retval" in

		$SUCCESS)

    		msg="Processing successfully concluded";;

		$ERR_BADARG)

    		msg="function checklibs called with non-directory parameter, returning $res";;

		$ERR_MISSING_PREPROC_BIN)

    		msg="binary 'pre_proc' not found in path, returning $res";;

		$ERR_MISSING_NEAR_PARAM)

    		msg="parameter 'near_range' missing or empty, returning $res";;

		$ERR_MISSING_RADIUS_PARAM)

    		msg="parameter 'earth_radius' missing or empty, returning $res";;

        $ERR_MISSING_MASTER_PARAM)

            msg="parameter 'master' missing or empty, returning $res";;

        $ERR_MISSING_FD1_PARAM)

            msg="parameter 'fd1' missing or empty, returning $res";;

        $ERR_MISSING_SAT_PARAM)

            msg="parameter 'sat' missing or empty, returning $res";;

        $ERR_MISSING_NUMPATCH_PARAM)

            msg="parameter 'num_patch' missing or empty, returning $res";;

        $ERR_INPUT_DATA_COPY)

            msg="Unable to retrieve an input file";;

        $ERR_PREPROC_ERROR)

            msg="Error during processing, aborting task [$res]";;

        $ERR_NOOUTPUT)

            msg="No output results";;

		$DEBUG_EXIT)

    		msg="Breaking at debug exit";;

   *)

      msg="Unknown error";;

   esac

   [ "$retval" != 0 ] && ciop-log "ERROR" "Error $retval - $msg, processing aborted" || ciop-log "INFO" "$msg"

   exit "$retval"

}

# trap an exit signal to exit properly

trap cleanExit EXIT

# Use ciop-log to log message at different level : INFO, WARN, DEBUG

ciop-log "DEBUG" '##########################################################'

ciop-log "DEBUG" '# Set of useful environment variables                    #'

ciop-log "DEBUG" '##########################################################'

ciop-log "DEBUG" "TMPDIR                  = $TMPDIR"                  # The temporary directory for the task.

ciop-log "DEBUG" "_JOB_ID                 = ${_JOB_ID}"               # The job id

ciop-log "DEBUG" "_JOB_LOCAL_DIR          = ${_JOB_LOCAL_DIR}" 		  # The job specific shared scratch space 

ciop-log "DEBUG" "_TASK_ID                = ${_TASK_ID}"              # The task id

ciop-log "DEBUG" "_TASK_LOCAL_DIR         = ${_TASK_LOCAL_DIR}"       # The task specific scratch space

ciop-log "DEBUG" "_TASK_NUM               = ${_TASK_NUM}"             # The number of tasks

ciop-log "DEBUG" "_TASK_INDEX             = ${_TASK_INDEX}"           # The id of the task within the job

# Get the processing trigger directory to link binaries and libraries

PREPROC_BASE_DIR=`dirname $0`

export PATH=$PREPROC_BASE_DIR/bin:$PATH

export LD_LIBRARY_PATH=$PREPROC_BASE_DIR/lib:$LD_LIBRARY_PATH

${_CIOP_APPLICATION_PATH}/GMTSAR/gmtsar_config

# Test that all my necessary binaries are accessible

# if not, exit with the corresponding error.

PREPROC_BIN=`which pre_proc_batch.csh`

[ -z "$PREPROC_BIN" ] && exit $ERR_MISSING_PREPROC_BIN

# Processor Environment

# definition and creation of input/output directory

OUTPUTDIR="$_TASK_LOCAL_DIR/output"		    # results directory

INPUTDIR="$_TASK_LOCAL_DIR/input"		    # data input directory

MASTERDIR="$_TASK_LOCAL_DIR/master"

mkdir -p $OUTPUTDIR $INPUTDIR

# Processing Variables

# Retrieve the near variable

NUMPATCH=`ciop-getparam num_patches`

[ $? != 0 ] && exit $ERR_MISSING_NUMPATCH_PARAM

NEAR=`ciop-getparam near_range`

[ $? != 0 ] && exit $ERR_MISSING_NEAR_PARAM

RADIUS=`ciop-getparam earth_radius`

[ $? != 0 ] && exit $ERR_MISSING_RADIUS_PARAM

FD1=`ciop-getparam fd1`

[ $? != 0 ] && exit $ERR_MISSING_FD1_PARAM

SAT=`ciop-getparam SAT`

[ $? != 0 ] && exit $ERR_MISSING_SAT_PARAM

MASTER=`ciop-getparam master`

[ $? != 0 ] && exit $ERR_MISSING_MASTER_PARAM

STOPONERROR=`ciop-getparam stop_on_error`

[ $? != 0 ] && STOPONERROR=false

# Create the batch.config parameter file

cat >${_TASK_LOCAL_DIR}/batch.config << EOF

num_patches = $NUMPATCH

earth_radius = $RADIUS

near_range = $NEAR

fd1 = $FD1

EOF

# the parameter 'master' is a reference to a data file

# we need to copy it for the rest of our processing

# This parameter is at job level so if another prallel task on the same

# computing resource has already copied it, we save a useless copy

masterFile=`ciop-copy -c -o "$MASTERDIR" -r 10 "$MASTER"`

[[ -s $masterFile ]] || { 

	ciop-log "ERROR" "Unable to retrieve master input at $url" ; exit $ERR_INPUT_DATA_COPY ; 

}

ciop-log "INFO" "Retrieved master input at $masterFile"

echo $masterFile >${_TASK_LOCAL_DIR}/data.in

# Begin processing loop

# Read line by line the input in url variable

while read url

do

		# First we copy the data in the INPUT dire

		ciop-log "INFO" "Copying data $url" "preproc"

		# ciop-copy $url in $INPUTDIR and retry 10 times in case of failure

		# local path of the copied file is returned in the $tmpFile variable

        tmpFile=`ciop-copy -o "$INPUTDIR" -r 10 "$url"`

        [[ -s $tmpFile ]] || { 

			ciop-log "ERROR" "Unable to retrieve inputfile at $url" ; 

			[[ $STOPONERROR == true ]] && exit $ERR_INPUT_DATA_COPY ; 

		}

        ciop-log "INFO" "Retrieved inputfile $tmpFile"

		echo $tmpFile >${_TASK_LOCAL_DIR}/data.in

done

# here we start the processing of the stack of data

ciop-log "INFO" "Processing stack of data" "preproc"

ciop-log "DEBUG" "$PREPROC_BIN $SAT data.in batch.config"

$PREPROC_BIN $SAT ${_TASK_LOCAL_DIR}/data.in ${_TASK_LOCAL_DIR}/batch.config >$OUTPUTDIR/preproc.log 2>&1

rcpp=$?

if [ "$rcpp" != 0 ]; then

	ciop-log "ERROR" "$PREPROC_BIN failed to process, return code $?" "preproc"

	cat $OUTPUTDIR/preproc.log >&2

	exit $ERR_PREPROC_ERROR

fi

ciop-log "INFO" "Processing complete" "preproc"

# The the results are "published" for next job

# Practically, the output shall be published to a job shared space

# and the directory referenced as an url for next job

ciop-publish $OUTPUTDIR/

exit 0

</code></pre>

> /\ !!! Keep in mind that the execution shall take place in a non-interactive environment so +error catching+ and +logging+ are very important. They enforce the robustness of your application and avoid loosing time later in debugging !!!

Here is a summary of the framework tools used in this script with their eventual online help:

* *source ${ciop_job_include}* --> include library for many functions such as ciop-log, ciop-enable-debug and ciop-getparam

* *ciop-enable-debug* --> this enable the DEBUG level for logging system, otherwise just INFO and WARN message are displayed

* *ciop-log* --> log message both in interactive computing framework and in processing stdout/err files. [[ciop-log CLI reference|ciop-log usage]]

* *ciop-getparam* --> retrieve job parameter. [[ciop-getparam CLI reference|ciop-getparam usage]]

* *ciop-catquery* --> query the EO Catalogue of the Sandbox. [[ciop-catquery CLI reference|ciop-catquery usage]]

* *ciop-copy* --> copy remote file to local directory. [[ciop-copy CLI reference|ciop-copy usage]]

* *ciop-publish* --> copy *+task+* result files in +*workflow*+ shared space. [[ciop-publish CLI reference|ciop-publish usage]]

h3. 5.4 Simulating a single job of the workflow

*ciop-simjob* --> simulates the execution of one processing +*job*+ of the +*work=flow*+. [[ciop-simjob CLI reference|ciop-simjob usage]]

We will use it to test the first processing block of GMTSAR:

<pre><code class="ruby">ciop-simjob -f PreProc</code></pre>

This will output to _stdout_ the URL of the Hadoop Map/Reduce job. Open the link to check if the processing is correctly executed. 

The command will show the progress messages:

<pre>

Deleted hdfs://sb-10-10-14-24.lab14.sandbox.ciop.int:8020/tmp/sandbox/sample/input.0

rmr: cannot remove /tmp/sandbox/sample/PreProc/logs: No such file or directory.

mkdir: cannot create directory /tmp/sandbox/sample/PreProc: File exists

Deleted hdfs://sb-10-10-14-24.lab14.sandbox.ciop.int:8020/tmp/sandbox/sample/workflow-params.xml

Submitting job 25764 ...

12/11/21 12:26:56 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.

packageJobJar: [/var/lib/hadoop-0.20/cache/emathot/hadoop-unjar5187515757952179540/] [] /tmp/streamjob7738227981987732817.jar tmpDir=null

12/11/21 12:26:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

12/11/21 12:26:58 WARN snappy.LoadSnappy: Snappy native library not loaded

12/11/21 12:26:58 INFO mapred.FileInputFormat: Total input paths to process : 1

12/11/21 12:26:58 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop-0.20/cache/emathot/mapred/local]

12/11/21 12:26:58 INFO streaming.StreamJob: Running job: job_201211101342_0045

12/11/21 12:26:58 INFO streaming.StreamJob: To kill this job, run:

12/11/21 12:26:58 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=sb-10-10-14-24.lab14.sandbox.ciop.int:8021 -kill job_201211101342_0045

12/11/21 12:26:58 INFO streaming.StreamJob: Tracking URL: http://sb-10-10-14-24.lab14.sandbox.ciop.int:50030/jobdetails.jsp?jobid=job_201211101342_0045

12/11/21 12:26:59 INFO streaming.StreamJob:  map 0%  reduce 0%

12/11/21 12:27:06 INFO streaming.StreamJob:  map 17%  reduce 0%

12/11/21 12:27:07 INFO streaming.StreamJob:  map 33%  reduce 0%

12/11/21 12:27:13 INFO streaming.StreamJob:  map 67%  reduce 0%

12/11/21 12:27:18 INFO streaming.StreamJob:  map 83%  reduce 0%

12/11/21 12:27:19 INFO streaming.StreamJob:  map 100%  reduce 0%

12/11/21 12:27:24 INFO streaming.StreamJob:  map 100%  reduce 33%

12/11/21 12:27:27 INFO streaming.StreamJob:  map 100%  reduce 100%

^@12/11/21 12:28:02 INFO streaming.StreamJob:  map 100%  reduce 0%

12/11/21 12:28:05 INFO streaming.StreamJob:  map 100%  reduce 100%

12/11/21 12:28:05 INFO streaming.StreamJob: To kill this job, run:

12/11/21 12:28:05 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=sb-10-10-14-24.lab14.sandbox.ciop.int:8021 -kill job_201211101342_0045

12/11/21 12:28:05 INFO streaming.StreamJob: Tracking URL: http://sb-10-10-14-24.lab14.sandbox.ciop.int:50030/jobdetails.jsp?jobid=job_201211101342_0045

12/11/21 12:28:05 ERROR streaming.StreamJob: Job not successful. Error: NA

12/11/21 12:28:05 INFO streaming.StreamJob: killJob...

Streaming Command Failed!

[INFO   ][log] All data, output and logs available at /share//tmp/sandbox/sample/PreProc

</pre>

At this point you can use your browser to display the URL of the _*Traking URL*_

!https://ciop.eo.esa.int/attachments/46/single_job_debug_1.png!

This is a single thread job, see the reduce in the table

You can click on the kill job in the reduce line. The page shows the task attempt, usually one.

!https://ciop.eo.esa.int/attachments/47/single_job_debug_2.png!

In this case, the job ended with exit code 19

Click on the task link, the same info as before is shown but there are more details (e.g. log in the last column)

You can click on _*all*_

_stout_ and _stderr_ appears, you can debug the processing job with this information.

h3. 5.6 -- Simulating a complete workflow -- (Section under revision)

h2. 6. -- Application deployment -- (section under revision)

This section describes the procedure to deploy your application once ready and successfully integrated.

h3. 6.1 Deploy as a service

h3. 6.2 Test application in pre-operation

h3. 6.3 Plan the production