Project

General

Profile

Actions
History

GMTSAR tutorial

GMTSAR: Generic Mapping Tools (GMT) for Synthetic Aperture Radar

Sandbox per-requisites

We assume that you have already a Sandbox ready and the following items are completed:
  • You have accessed your Sandbox as described in the Getting Started guide

It is not mandatory but strongly recommended to follow the Getting started tutorial before starting this one.

1. Application concepts and terminology

In order to ease the execution of the tutorial, it is important to understand the concept of an application and its terminology. This section describes the example application used all along this guide. When a word is put in underlined bold, it is an terminology keyword always designating the same concept.

1.1 The application workflow

Our example in this tutorial is an interferometry application that processes SAR data (Envisat ASAR Image Mode level 0) to generate interferogram between a master and one or more slave products. It is composed of 6 steps that run independently but in a specific order and produce results that are inputs for the other remaining steps.
The following figure illustrates the Workflow of our application as a directed acyclic graph (DAG). This is also how the CIOP framework handles the execution of processes in terms of parallel computing and orchestration of the processing steps.

!! <<insert here the DAG that represent the workflow >>

Each box represents a Job which is a step of our application process. The arrows represents the data flow between the jobs. When a job is connected to another, it means that the output of this job is passed as input for the other.

It is important to keep in mind that in CIOP framework, input and ouput are text references (e.g. to data). Indeed, when a job process input, it actually reads line by line the reference workflow as described in the next figure

It is therefore important to define precisely the inter- job references.

1.2 The job

Each job has a set of basic characteristics:

  • a unique Job name in the workflow. (e.g. 'PreProc')
  • zero, 1 or several sources that define the jobs interdependency. In the example, the job 'Interfere' has 2 dependencies: 'AlignSlave' and 'MakeTropo'.
  • a maximum number of simultaneous tasks in which it can be forked. This is further explained in section 1.3 The processing task.
  • a processing trigger which is a software executable of the job template that handles input/*+output+* streaming process. Practically, the executable that reads the input lines and writes the output lines.

The job characteristics above are mandatory in the workflow definition.
If incomplete, the CIOP framework reports error in the workflow.

For out tutorial example, here are the characteristics for 'PreProc', 'AlignSlave' and 'Interfere' jobs:

  • PreProc

PreProc is the first job in the workflow. It takes both SAR products, master and slave, and pre-processess them:

It is a job with a single task, the defaultJobconf property (a CIOP property, not an application property) ciop.job.max.tasks is set to 1
Its executable is located is /application/preproc/run
It has a number of application parameters to run: SAT, master, num_patches, near_range, earth_radius, fd1, stop_on_error. The parameter values are not set in this job template.

<jobTemplate id="preproc">
    <streamingExecutable>/application/preproc/run</streamingExecutable>    <!-- processing trigger -->
    <defaultParameters>    <!-- default parameters of the job -->
        <!-- Default values are specified here -->
        <parameter id="SAT"></parameter>    <!-- no default value -->
        <parameter id="master"></parameter>
        <parameter id="num_patches"></parameter>    <!-- no default value -->
        <parameter id="near_range"></parameter>    <!-- no default value -->
        <parameter id="earth_radius"></parameter>    <!-- no default value -->
        <parameter id="fd1">1</parameter>    <!-- no default value -->
        <parameter id="stop_on_error">false</parameter>     <!-- don't stop on error by default -->
    </defaultParameters>
    <defaultJobconf>
        <property id="ciop.job.max.tasks">1</property>    <!-- Maximum number of parallel tasks -->
    </defaultJobconf>
</jobTemplate>
  • Job name: 'AlignSlave'
  • sources: 'PreProc'
  • maximum number of simultaneous tasks: unlimited
  • processing trigger: /application/align/run
  • Job name: 'Interfere'
  • sources: 'AlignSlave' and 'MakeTropo'
  • maximum number of simultaneous tasks: 1
  • processing trigger: /application/interfere/run

1.3 The processing task

To exploit the parallelism offered by the CIOP framework, a job may process its input in several tasks. In principle, the CIOP framework will run those tasks in parallel. This is an important and sometimes complex paradigm that can be addressed in different ways.
The following questions & answers describe the parallelism paradigm of the CIOP framework.

  • Is it task parallelism or job parallelism?

In this section, we definitely speak about task parallelism. Job parallelism is at level !+1. In the example, 'MakeTropo' and 'AlignSlave' are two jobs that run in parallel besides the number of tasks each of them may trigger.
'AlignSlave' may be forked into an unlimited number of tasks. Practically the framework calculates automatically the number n of available processing slots in the computing resource and start n times the processing trigger (based on the number of inputs).

  • How to divide a job into tasks?

It is actually the application developer who chooses the granularity of the job division. The computing framework will simply divide the input flow (k lines) in to the n tasks. In the example provided in this tutorial, if the job 'PreProc' produces output of 11 lines and the computing resources divide the job 'AlignSlave' into 4 tasks and the following division is done:

Where stands the processing loop?

The processing loop stands in the processing trigger. As shown in this tutorial with the example, the processing triggers implements a loop that reads line by line the task input.

2. Home Directory

The home directory of your Sandbox is attached to a separate savable disk and thus persistent. This disk is mounted on the /home/{USERNAME} folder. In this folder you may upload, compile, test manually, etc. all your data. This is a free space where you can do anything BUT:

Be careful to never link any element (e.g. executable, auxiliary data) from the application directory to the home directory which is critical for the application. Indeed, the home directory is not present when using CIOP in Runtime Environment mode and therefore any linked elements won't be available and cause the processing phase to fail.

3. Application Directory

The application directory of your Sandbox is attached to a separate savable disk and thus persistent. This disk is mounted on the /application folder.
The application directory is the place where integrated application resides.
It should be a clean environment and thus SHOULD NOT contain any temporary files or used to do compilations and/or manual testing. Instead, it is used for the simulation the application jobs and workflow.

At the instantiation of your Sandbox, the application directory contains this sample application example unless you configured the Sandbox with one of your application disk previously saved in your application library.
In the next sections the elements of the application directory are described.

3.1 Files and folders structure

The application directory follows some best practices in its folders and files structure to ease the subsequent deployment of the application to the CIOP Runtime Environment.
The folder structure of the application example with the description of each item is shown below:

Even if the names are quite similar in the our tutorial example, job and job template are not the same concept. _A job is an instance of a job template in a given workflow. This paradigm allows to have several jobs in a workflow that point to the same job template. This is explained in more detail in the next section.

3.2 The Application XML definition file

The Application XML definition file is the reference of your application for the CIOP computing framework. It contains all the characteristics of the job templates and the workflows.

The Application XML definition file is is described in the page Application XML definition file.

4 -- Using sample datasets -- (section under revision)

This section guides you through the tutorial example to introduce the data manipulation tools.

There are mainly two command line tools to discover and access data previously selected in your Sandbox (see here):
  • ciop-catquery to query the Sandbox catalogue containing all the metadata of the selected sample dataset
  • ciop-copy to copy the data from logical of physical location to a local directory of the Sandbox

Use

ciop-<command> -h

to display the CLI reference

These commands can be used in the processing triggers of a job.

4.1 Query Sandbox catalogue

For the tutorial purpose, the first test is to ensure the test dataset needed for the Application integration and testing is complete.

(to be updated)

4.2 Copy data

To copy data from a reference link as displayed in the previous section, just use the following command:

[user@sb ~]$ ciop-copy http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1/rdf

output:

[INFO   ][ciop-copy][starting] url 'http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1/rdf' > local '/application/'
[INFO   ][ciop-copy][success] got URIs 'https://eo-virtual-archive4.esa.int/supersites/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1 '
[INFO   ][ciop-copy][starting] url 'https://eo-virtual-archive4.esa.int/supersites/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1' > local '/application/'
[INFO   ][ciop-copy][success] url 'https://eo-virtual-archive4.esa.int/supersites/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1' > local '/application/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1'
/home/user/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1

The command displays information on the stderr by default and returns on stdout the path of the copied data.

Many other data schemas are supported by the ciop-copy CLI such as http, https, hdfs, etc.
There are also many other options to specify the output directory or to unpack compressed data.
The complete reference is available here ciop-copy usage or by using the inline help typing:

[user@sb ~]$ciop-copy -h

4.3 Using other sources of data in a job

So far we have introduced two types of data sources:
  • data coming from a catalogue series
  • data coming from a previous job in the workflow.

In the first case, we define the workflow for the job imager:

<workflow id="testVomir">                            <!-- Sample workflow -->
        <workflowVersion>1.0</workflowVersion>
        <node id="Vimage">                            <!-- workflow node unique id -->
            <job id="imager"></job>                    <!-- job defined above -->
            <sources>
                <source refid="cas:serie" >ATS_TOA_1P</source>
            </sources>
            <parameters>                            <!-- parameters of the job -->
                <parameter id="volcano_db"></parameter>
            </parameters>
        </node>
        <node id="Quarc">
            <job id="quarcXML"/>
            <sources>
                <source refid="wf:node" >Vimage</source>
            </sources>
        </node>

In the second case, we define the workflow for the Quarc job:

<workflow id="testVomir">                            <!-- Sample workflow -->
        <workflowVersion>1.0</workflowVersion>
        <node id="Vimage">                            <!-- workflow node unique id -->
            <job id="imager"></job>                    <!-- job defined above -->
            <sources>
                <source refid="cas:serie" >ATS_TOA_1P</source>
            </sources>
            <parameters>                            <!-- parameters of the job -->
                <parameter id="volcano_db"></parameter>
            </parameters>
        </node>
        <node id="Quarc">
            <job id="quarcXML"/>
            <sources>
                <source refid="wf:node" >Vimage</source>
            </sources>
        </node>

It may be the case where the input data does not come from EO catalogues and thus there is the need to define another source of data.

<workflow id="someworkflow">                            <!-- Sample workflow -->
        <workflowVersion>1.0</workflowVersion>
        <node id="somenode">                            <!-- workflow node unique id -->
            <job id="somejobid"></job>                    <!-- job defined above -->
            <sources>
                <source refid="file:urls" >/application/test.urls</source>
            </sources>
        </node>

where the file test.urls contains the input lines that will be pipped to the processing trigger executable

5. Job integration

In this section, the job template 'align' and its instance the job 'AlignSlave' is integrated using the tools previously introduced in this tutorial.

5.1 Installation and configuration of the GMTSAR toolbox on the Sandbox

The steps below install the GMTSAR toolbox on the Sandbox. These are specific but it shows the common approach to follow when installing software on the Sandbox.

The steps below are done in the home directory

  • Step - Download the GMTSAR

GMTSAR software is available on the University of California web server:

[user@sb ~]$ wget http://topex.ucsd.edu/gmtsar/tar/GMTSAR.tar

Then the GMTSAR.tar archive is unpacked

[user@sb ~]$ tar xvf GMTSAR.tar

GMTSAR relies on GMT with the dependencies netCDF, GMT is installed via yum:

[user@sb ~]$ cd GMTSAR
[user@sb GMTSAR]$ sudo yum search gmt
[user@sb GMTSAR]$ sudo yum install GMT-devel
[user@sb GMTSAR]$ sudo yum install netcdf-devel
[user@sb GMTSAR]$ make

The steps above compile GMTSAR in the home directory. The required files (binaries, libraries, etc.) are copied to the /Application environment (remember that the home directory is only available in the CIOP Sandbox mode and not in the Runtime Environment).

5.2 Job template definition in the application.xml

The application.xml file has two main blocks: the job template section and the workflow template section.

The first part is to define the job templates in the workflow XML application definition file.
Each processing block of the GMTSAR workflow needs a job template.

Here is the job template for the 'align'

<jobTemplate id="preproc">
    <streamingExecutable>/application/preproc/run</streamingExecutable> <!-- processing trigger -->        
    <defaultParameters> <!-- default parameters of the job -->
        <!-- Default values are specified here -->
        <parameter id="SAT"></parameter>    <!-- no default value -->
        <parameter id="master"></parameter>
        <parameter id="num_patches"></parameter>    <!-- no default value -->
        <parameter id="near_range"></parameter>    <!-- no default value -->
        <parameter id="earth_radius"></parameter>    <!-- no default value -->
        <parameter id="fd1">1</parameter>    <!-- no default value -->
        <parameter id="stop_on_error">false</parameter>    <!-- dont stop on error by default -->
    </defaultParameters>
    <defaultJobconf>
        <property id="ciop.job.max.tasks">1</property>    <!-- Maximum number of parallel tasks -->
        </defaultJobconf>
</jobTemplate>

To test this job with the ciop-symjob, we need to fill the second part of the application.xml to add a node:

<node id="PreProc">    <!-- workflow node unique id -->
    <job id="preproc"></job> <!-- job template defined before -->
    <sources>
        <!-- Source is the series of data selection -->
        <source refid="cas:serie">ASA_IM__0P</source>
    </sources>
    <parameters>    <!-- parameters of the job -->
        <parameter id="SAT">ENV</parameter>
        <parameter id="master">http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPDE20040602_091147_000000152027_00222_11799_1335.N1/rdf</parameter>
        <parameter id="near">978992.922</parameter>
        <parameter id="radius">6378000</parameter>
        <parameter id="stop_on_error">true</parameter>    <!-- during integration, preferably stop on error -->    
    </parameters>
</node>

The complete application.xml is available here: TBW
The application.xml and all its elements are described in details in the page Application XML definition file.

Since this processing is the first in the workflow chain, it has a special source which is the serie 'ASA_IM__0P'. Practically, it means that when the job is submitted for execution, the computing framework will query the Sandbox catalogue for the data ASA_IM__0P registered as sample dataset and will prepare a list of data reference as input for the job 'PreProc'. In our example, the resulting list is:

http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPDE20090412_092436_000000162078_00079_37207_1556.N1/rdf
http://localhost/catalogue/sandbox/ASA_IM__0P/ASA_IM__0CNPAM20080427_092430_000000172068_00079_32197_3368.N1/rdf

5.3 Processing trigger script

In section 1.2, we have seen that each job must have a processing trigger which is specified in <streamingExecutable> element of the job template. In our example, this executable shall be a shell script:

# FIRST OF ALL, LOAD CIOP INCLUDES
source ${ciop_job_include}

# If you want to have a complete debug information during implementation
ciop-enable-debug

# All return codes are predefined
SUCCESS=0
ERR_BADARG=2
ERR_MISSING_PREPROC_BIN=3
ERR_MISSING_NEAR_PARAM=4
ERR_MISSING_RADIUS_PARAM=5
ERR_MISSING_MASTER_PARAM=6
ERR_MISSING_SAT_PARAM=7
ERR_MISSING_FD1_PARAM=8
ERR_MISSING_NUMPATCH_PARAM=9
ERR_INPUT_DATA_COPY=18
ERR_PREPROC_ERROR=19
ERR_NOOUTPUT=20
DEBUG_EXIT=66

# This functions handle the exit of the executable
# with the corresponding error codes and will return a short message
# with the termination reason. It is important to have a synthetic and brief
# message because it will be raised to many upper level of the computing framework
# up to the user interface
function cleanExit ()
{
   local retval=$?
   local msg="" 
   case "$retval" in
        $SUCCESS)
            msg="Processing successfully concluded";;
        $ERR_BADARG)
            msg="function checklibs called with non-directory parameter, returning $res";;
        $ERR_MISSING_PREPROC_BIN)
            msg="binary 'pre_proc' not found in path, returning $res";;
        $ERR_MISSING_NEAR_PARAM)
            msg="parameter 'near_range' missing or empty, returning $res";;
        $ERR_MISSING_RADIUS_PARAM)
            msg="parameter 'earth_radius' missing or empty, returning $res";;
        $ERR_MISSING_MASTER_PARAM)
            msg="parameter 'master' missing or empty, returning $res";;
        $ERR_MISSING_FD1_PARAM)
            msg="parameter 'fd1' missing or empty, returning $res";;
        $ERR_MISSING_SAT_PARAM)
            msg="parameter 'sat' missing or empty, returning $res";;
        $ERR_MISSING_NUMPATCH_PARAM)
            msg="parameter 'num_patch' missing or empty, returning $res";;
        $ERR_INPUT_DATA_COPY)
            msg="Unable to retrieve an input file";;
        $ERR_PREPROC_ERROR)
            msg="Error during processing, aborting task [$res]";;
        $ERR_NOOUTPUT)
            msg="No output results";;
        $DEBUG_EXIT)
            msg="Breaking at debug exit";;
   *)
      msg="Unknown error";;
   esac
   [ "$retval" != 0 ] && ciop-log "ERROR" "Error $retval - $msg, processing aborted" || ciop-log "INFO" "$msg" 
   exit "$retval" 
}

# trap an exit signal to exit properly
trap cleanExit EXIT

# Use ciop-log to log message at different level : INFO, WARN, DEBUG
ciop-log "DEBUG" '##########################################################'
ciop-log "DEBUG" '# Set of useful environment variables                    #'
ciop-log "DEBUG" '##########################################################'
ciop-log "DEBUG" "TMPDIR                  = $TMPDIR"                  # The temporary directory for the task.
ciop-log "DEBUG" "_JOB_ID                 = ${_JOB_ID}"               # The job id
ciop-log "DEBUG" "_JOB_LOCAL_DIR          = ${_JOB_LOCAL_DIR}"           # The job specific shared scratch space 
ciop-log "DEBUG" "_TASK_ID                = ${_TASK_ID}"              # The task id
ciop-log "DEBUG" "_TASK_LOCAL_DIR         = ${_TASK_LOCAL_DIR}"       # The task specific scratch space
ciop-log "DEBUG" "_TASK_NUM               = ${_TASK_NUM}"             # The number of tasks
ciop-log "DEBUG" "_TASK_INDEX             = ${_TASK_INDEX}"           # The id of the task within the job

# Get the processing trigger directory to link binaries and libraries
PREPROC_BASE_DIR=`dirname $0`
export PATH=$PREPROC_BASE_DIR/bin:$PATH
export LD_LIBRARY_PATH=$PREPROC_BASE_DIR/lib:$LD_LIBRARY_PATH

${_CIOP_APPLICATION_PATH}/GMTSAR/gmtsar_config

# Test that all my necessary binaries are accessible
# if not, exit with the corresponding error.
PREPROC_BIN=`which pre_proc_batch.csh`
[ -z "$PREPROC_BIN" ] && exit $ERR_MISSING_PREPROC_BIN

# Processor Environment
# definition and creation of input/output directory
OUTPUTDIR="$_TASK_LOCAL_DIR/output"            # results directory
INPUTDIR="$_TASK_LOCAL_DIR/input"            # data input directory
MASTERDIR="$_TASK_LOCAL_DIR/master" 
mkdir -p $OUTPUTDIR $INPUTDIR

# Processing Variables
# Retrieve the near variable
NUMPATCH=`ciop-getparam num_patches`
[ $? != 0 ] && exit $ERR_MISSING_NUMPATCH_PARAM
NEAR=`ciop-getparam near_range`
[ $? != 0 ] && exit $ERR_MISSING_NEAR_PARAM
RADIUS=`ciop-getparam earth_radius`
[ $? != 0 ] && exit $ERR_MISSING_RADIUS_PARAM
FD1=`ciop-getparam fd1`
[ $? != 0 ] && exit $ERR_MISSING_FD1_PARAM
SAT=`ciop-getparam SAT`
[ $? != 0 ] && exit $ERR_MISSING_SAT_PARAM
MASTER=`ciop-getparam master`
[ $? != 0 ] && exit $ERR_MISSING_MASTER_PARAM
STOPONERROR=`ciop-getparam stop_on_error`
[ $? != 0 ] && STOPONERROR=false

# Create the batch.config parameter file
cat >${_TASK_LOCAL_DIR}/batch.config << EOF

num_patches = $NUMPATCH
earth_radius = $RADIUS
near_range = $NEAR
fd1 = $FD1

EOF

# the parameter 'master' is a reference to a data file
# we need to copy it for the rest of our processing
# This parameter is at job level so if another prallel task on the same
# computing resource has already copied it, we save a useless copy
masterFile=`ciop-copy -c -o "$MASTERDIR" -r 10 "$MASTER"`
[[ -s $masterFile ]] || { 
    ciop-log "ERROR" "Unable to retrieve master input at $url" ; exit $ERR_INPUT_DATA_COPY ; 
}
ciop-log "INFO" "Retrieved master input at $masterFile" 

echo $masterFile >${_TASK_LOCAL_DIR}/data.in

# Begin processing loop
# Read line by line the input in url variable
while read url
do
        # First we copy the data in the INPUT dire
        ciop-log "INFO" "Copying data $url" "preproc" 
        # ciop-copy $url in $INPUTDIR and retry 10 times in case of failure
        # local path of the copied file is returned in the $tmpFile variable
        tmpFile=`ciop-copy -o "$INPUTDIR" -r 10 "$url"`
        [[ -s $tmpFile ]] || { 
            ciop-log "ERROR" "Unable to retrieve inputfile at $url" ; 
            [[ $STOPONERROR == true ]] && exit $ERR_INPUT_DATA_COPY ; 
        }
        ciop-log "INFO" "Retrieved inputfile $tmpFile" 

        echo $tmpFile >${_TASK_LOCAL_DIR}/data.in

done

# here we start the processing of the stack of data
ciop-log "INFO" "Processing stack of data" "preproc" 
ciop-log "DEBUG" "$PREPROC_BIN $SAT data.in batch.config" 

$PREPROC_BIN $SAT ${_TASK_LOCAL_DIR}/data.in ${_TASK_LOCAL_DIR}/batch.config >$OUTPUTDIR/preproc.log 2>&1
rcpp=$?

if [ "$rcpp" != 0 ]; then
    ciop-log "ERROR" "$PREPROC_BIN failed to process, return code $?" "preproc" 
    cat $OUTPUTDIR/preproc.log >&2
    exit $ERR_PREPROC_ERROR
fi

ciop-log "INFO" "Processing complete" "preproc" 

# The the results are "published" for next job
# Practically, the output shall be published to a job shared space
# and the directory referenced as an url for next job
ciop-publish $OUTPUTDIR/

exit 0

/\ !!! Keep in mind that the execution shall take place in a non-interactive environment so error catching and logging are very important. They enforce the robustness of your application and avoid loosing time later in debugging !!!

Here is a summary of the framework tools used in this script with their eventual online help:
  • source ${ciop_job_include} --> include library for many functions such as ciop-log, ciop-enable-debug and ciop-getparam
  • ciop-enable-debug --> this enable the DEBUG level for logging system, otherwise just INFO and WARN message are displayed
  • ciop-log --> log message both in interactive computing framework and in processing stdout/err files. ciop-log usage
  • ciop-getparam --> retrieve job parameter. ciop-getparam usage
  • ciop-catquery --> query the EO Catalogue of the Sandbox. ciop-catquery usage
  • ciop-copy --> copy remote file to local directory. ciop-copy usage
  • ciop-publish --> copy task result files in workflow shared space. ciop-publish usage

5.4 Simulating a single job of the workflow

ciop-simjob --> simulates the execution of one processing job of the work=flow. ciop-simjob usage

We will use it to test the first processing block of GMTSAR:

ciop-simjob -f PreProc

This will output to stdout the URL of the Hadoop Map/Reduce job. Open the link to check if the processing is correctly executed.
The command will show the progress messages:

Deleted hdfs://sb-10-10-14-24.lab14.sandbox.ciop.int:8020/tmp/sandbox/sample/input.0
rmr: cannot remove /tmp/sandbox/sample/PreProc/logs: No such file or directory.
mkdir: cannot create directory /tmp/sandbox/sample/PreProc: File exists
Deleted hdfs://sb-10-10-14-24.lab14.sandbox.ciop.int:8020/tmp/sandbox/sample/workflow-params.xml
Submitting job 25764 ...
12/11/21 12:26:56 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.
packageJobJar: [/var/lib/hadoop-0.20/cache/emathot/hadoop-unjar5187515757952179540/] [] /tmp/streamjob7738227981987732817.jar tmpDir=null
12/11/21 12:26:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/11/21 12:26:58 WARN snappy.LoadSnappy: Snappy native library not loaded
12/11/21 12:26:58 INFO mapred.FileInputFormat: Total input paths to process : 1
12/11/21 12:26:58 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop-0.20/cache/emathot/mapred/local]
12/11/21 12:26:58 INFO streaming.StreamJob: Running job: job_201211101342_0045
12/11/21 12:26:58 INFO streaming.StreamJob: To kill this job, run:
12/11/21 12:26:58 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=sb-10-10-14-24.lab14.sandbox.ciop.int:8021 -kill job_201211101342_0045
12/11/21 12:26:58 INFO streaming.StreamJob: Tracking URL: http://sb-10-10-14-24.lab14.sandbox.ciop.int:50030/jobdetails.jsp?jobid=job_201211101342_0045
12/11/21 12:26:59 INFO streaming.StreamJob:  map 0%  reduce 0%
12/11/21 12:27:06 INFO streaming.StreamJob:  map 17%  reduce 0%
12/11/21 12:27:07 INFO streaming.StreamJob:  map 33%  reduce 0%
12/11/21 12:27:13 INFO streaming.StreamJob:  map 67%  reduce 0%
12/11/21 12:27:18 INFO streaming.StreamJob:  map 83%  reduce 0%
12/11/21 12:27:19 INFO streaming.StreamJob:  map 100%  reduce 0%
12/11/21 12:27:24 INFO streaming.StreamJob:  map 100%  reduce 33%
12/11/21 12:27:27 INFO streaming.StreamJob:  map 100%  reduce 100%
^@12/11/21 12:28:02 INFO streaming.StreamJob:  map 100%  reduce 0%
12/11/21 12:28:05 INFO streaming.StreamJob:  map 100%  reduce 100%
12/11/21 12:28:05 INFO streaming.StreamJob: To kill this job, run:
12/11/21 12:28:05 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job  -Dmapred.job.tracker=sb-10-10-14-24.lab14.sandbox.ciop.int:8021 -kill job_201211101342_0045
12/11/21 12:28:05 INFO streaming.StreamJob: Tracking URL: http://sb-10-10-14-24.lab14.sandbox.ciop.int:50030/jobdetails.jsp?jobid=job_201211101342_0045
12/11/21 12:28:05 ERROR streaming.StreamJob: Job not successful. Error: NA
12/11/21 12:28:05 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
[INFO   ][log] All data, output and logs available at /share//tmp/sandbox/sample/PreProc

At this point you can use your browser to display the URL of the Traking URL

This is a single thread job, see the reduce in the table
You can click on the kill job in the reduce line. The page shows the task attempt, usually one.

In this case, the job ended with exit code 19

Click on the task link, the same info as before is shown but there are more details (e.g. log in the last column)

You can click on all

stout and stderr appears, you can debug the processing job with this information.

5.6 -- Simulating a complete workflow -- (Section under revision)

6. -- Application deployment -- (section under revision)

This section describes the procedure to deploy your application once ready and successfully integrated.

6.1 Deploy as a service

6.2 Test application in pre-operation

6.3 Plan the production

Updated by Herve Caumont over 11 years ago ยท 2 revisions