Esgf-client » History » Revision 5
Revision 4 (Herve Caumont, 2013-09-18 14:00) → Revision 5/7 (Herve Caumont, 2013-11-05 16:46)
h1. ESGFClient
{{>toc}}
h2. Overview
ESGFClient is a Command Line tool Interface written in C# able to download products from the "ESGF Data Search" getting as input the location of a RDF file or a list of single product's urls.
h2. Installation
The user can install the ESGFClient following these easy steps:
<pre><code class="XML">
yum install esgf-tools
</code></pre>
and test the installation with
<pre><code class="XML">
ESGFClient --help
</code></pre>
h2. Usage
These are the options given:
<pre><code class="XML">
Options:
-u, --uri=VALUE RDF's URI -f, --file=VALUE File RDF to parse and download
-o, --output=VALUE Output folder -F, --Force Force the whole rdf's resources
-r, --resource=VALUE Resource -O, --OpenDAP Download ONLY the OpenDAP resources
-u, --url=VALUE Url to download
-O, -o, --openid=VALUE OpenId used to access.
-p, --password=VALUE Password for OpenId used to access.
-b, --bbox=VALUE bounding box to restrict the file search and
the file size (only with OpenDAP service). Use
west, south, east, north .
-s, --dtstart=VALUE The the beginning of the time query to restrict
the subset of data to download. query. YYYY-MM-
DDTHH:mm:ssZ. DDTHH:mm:ssZ .
-e, --dtend=VALUE The the end of the time query to restrict query. YYYY-MM-DDTHH:mm:ssZ
.
-t, --basetime=VALUE the start of time coverage of the Dataset. YYYY-
subset of data to download. YYYY-MM-DDTHH:mm:ssZ. MM-DDTHH:mm:ssZ .
--zM, --zmax=VALUE Maximum level (z) -v, --variable=VALUE variable
--zm, --zmin=VALUE Minimum level (z). By default it's equal to 0
-h, --help Show show this message and exit
</code></pre>
h2. Download from rdf url
The Client --file option allows the user to download only automatically parse an rdf response from the OpenDAP ESGFGateway, in order to find all downloadable online resources. resources, HTTP or OpenDAP.
It tries In the case of OpenDAP resources, the Client try to retrieve from the RDF file all the parameters needed for a query: a time start and a time stop (for temporal queries), the level range (z dimension if it's provided by the variable) a bounding box (for spatial queries) and the variable to query.
All these files are protected by OpenID, so the user has to give his credentials by specifying the "--openid" option an --password. If you don't have an OpenID account visit the page https://pcmdi9.llnl.gov/esgf-web-fe/createAccount. Conversely the security certificates are automatically downloaded by the WGET script.
Example:
<pre><code class="XML">
ESGFClient -u "http://geowow.terradue.com/catalogue/esgf/thetao/rdf?time_frequency=mon&experiment=rcp85&ensemble=r1i1p1&institute=MRI&count=1" -O "https://pcmdi9.llnl.gov/esgf-idp/openid/user" -f "http://geowow.terradue.com/catalogue/esgf/zos/rdf?bbox=-111,42,-119,43&startTime=1962-10-01T12:00:00Z&stopTime=1963-01-01T12:00:00Z&startPage=0&count=3" -o "https://pcmdi9.llnl.gov/esgf-idp/openid/username" -p password -s 2006-01-17 -e 2008-12-16 -zmin 500 -zmax 700 -o "./tmp/" "password" -F
</code></pre>
In order to build these correct OpenDAP query, the input bounding box and the time search and level parameters are converted to a set of indexes needed to make a spatial and temporal query.
h2. Download user defined resources
The --url option is used to download a user defined resource, by specifying also the temporal coverage and spatial bounding box.
Example:
h3. HTTP resource:
<pre><code class="XML">
ESGFClient -u "http://bcccsm.cma.gov.cn/thredds/fileServer/cmip5_data/output/BCC/bcc-csm1-1/decadal2005/mon/ocean/zos/r1i1p1/zos_Omon_bcc-csm1-1_decadal2005_r1i1p1_200601-203512.nc" -o "https://pcmdi9.llnl.gov/esgf-idp/openid/username" -p "password"
</code></pre>
h3. OpenDAP resource:
<pre><code class="XML">
ESGFClient -u "http://bcccsm.cma.gov.cn/thredds/dodsC/cmip5.output1.BCC.bcc-csm1-1.decadal2005.mon.ocean.Omon.r1i1p1.zos.1.aggregation" --bbox "-111,42,-112,43" --dtstart "2006-02-16T12:00:00Z" --dtend "2006-05-16T12:00:00Z" --basetime "2006-01-16T12:00:00Z" -o "https://pcmdi9.llnl.gov/esgf-idp/openid/username" -p "password"
</code></pre>
The download is based on a revised wget script built upon a template called *wgetTemplate.sh*, filled with the URLs to be gotten.
Then the client run the WGET script, that saves the files into the file system.
All these files are protected by OpenID, so the user has to give his credentials by specifying the "--openid" The --Force option an --password. Conversely the security certificates are automatically downloaded by the WGET script.
After the .dods file is downloaded, an internal script converts it allows to a NetCDF file, according with the .das description file too.
h2. OpenDAP limits
Since this version of the ESGFClient is exclusively based on the OpenDAP servers, there is a size limit for every download. It is set to 500 MByte and specify if all the query reaches this limit, the server returns a 403 error HTTP and OpenDAP online resources from the RDF should be download fails. This limit can be respected by setting or simple the right temporal and level slices of your sub-datasets.
first.