Esgf-client » History » Revision 6
Revision 5 (Herve Caumont, 2013-11-05 16:46) → Revision 6/7 (Herve Caumont, 2013-11-05 17:28)
h1. ESGFClient
{{>toc}}
h2. Overview
ESGFClient is a Command Line tool written in C# able to download products from the "ESGF Data Search" getting as input the location of a RDF file or a list of single product's urls.
h2. Installation
The user can install the ESGFClient following these easy steps:
<pre><code class="XML">
yum install esgf-tools
</code></pre>
and test the installation with
<pre><code class="XML">
ESGFClient --help
</code></pre>
h2. Usage
These are the options given:
<pre><code class="XML">
Options:
-u, --uri=VALUE RDF's URI to parse and download
-o, --output=VALUE Output folder
-r, --resource=VALUE Resource to download
-O, --openid=VALUE OpenId used to access.
-p, --password=VALUE Password for OpenId used to access.
-s, --dtstart=VALUE The beginning of the time query to restrict
the subset of data to download. YYYY-MM-
DDTHH:mm:ssZ.
-e, --dtend=VALUE The end of the time query to restrict the
subset of data to download. YYYY-MM-DDTHH:mm:ssZ.
--zM, --zmax=VALUE Maximum level (z)
--zm, --zmin=VALUE Minimum level (z). By default it's equal to 0
-h, --help Show this message and exit
</code></pre>
h2. Download from rdf url
The Client allows the user to download only the OPeNDAP OpenDAP online resources.
It tries to retrieve from the RDF file all the parameters needed for a query: a time start and a time stop (for temporal queries), the level range (z dimension if it's provided by the variable) and the variable to query.
Example:
<pre><code class="XML">
ESGFClient -u "http://geowow.terradue.com/catalogue/esgf/thetao/rdf?time_frequency=mon&experiment=rcp85&ensemble=r1i1p1&institute=MRI&count=1" -O "https://pcmdi9.llnl.gov/esgf-idp/openid/user" -p password -s 2006-01-17 -e 2008-12-16 -zmin 500 -zmax 700 -o "./tmp/"
</code></pre>
In order to build these correct OPeNDAP OpenDAP query, the time search and level parameters are converted to a set of indexes needed to make a spatial and temporal query.
The download is based on a revised wget script built upon a template called *wgetTemplate.sh*, filled with the URLs to be gotten.
Then the client run the WGET script, that saves the files into the file system.
All these files are protected by OpenID, so the user has to give his credentials by specifying the "--openid" option an --password. Conversely the security certificates are automatically downloaded by the WGET script.
After the OPeNDAP files (.dods, .das, .lev, .lat etc..) are .dods file is downloaded, an internal script converts the .dods Binary Format file it to a NetCDF file, according to with the .das Dataset Attribute Structure description file. Once the NetDCF file is generated, the script also deletes the downloaded temporary files. too.
For more details on the OPeNDAP formats, refer to http://docs.opendap.org/index.php/UserGuideOPeNDAPMessages
h2. OPeNDAP OpenDAP limits
Since this version of the ESGFClient is exclusively based on the OPeNDAP OpenDAP servers, there is a size limit for every download. With the ESGF federation, this limit It is set to the default value (500 MBytes), 500 MByte and if the query reaches this limit, the server returns a 403 error and the download fails. You This limit can deal with this limit be respected by setting the right temporal and level slices of your sub-datasets.