Sample standalone tools

This page describes different external tools we want to wrap via the Standalone Tools Adapter

Sentinel 2 Sen2Core

This tool provides atmospheric correction of Sentinel 2 MSI L1C product.

It takes as input a S2-L1C product, and output a S2-L2A product. Both S2-L1C and S2-L2A are product types for which SNAP readers are implemented.

The software is delivered as a standalone package that must be installed externally. We might want to provide a SNAP module providing it but this is out of scope for our concerns here.

The tool can be called like this (following standard Unix conventions)

bash-4.1$ L2A_Process -h

usage: L2A_Process [-h] [--resolution {10,20,60}] [--sc_only] directory

Sentinel-2 Level-2A Prototype Processor (Sen2Cor), 0.1.0, created: 2014.03.29

positional arguments:

  directory               Directory where the Level-1C input files are located

optional arguments:

-h, --help                Show this help message and exit

--resolution {10,20,60}   Target resolution, must be 10, 20 or 60 [m]

--sc_only                 Perform only the scene classification at 60m resolution

<directory> can be any directory where a L1C User product is stored. The target directory can have any name, providing that a “L1C_” is part of the directory name. The processor will create an output directory of the same name, but replacing “L1C_” with “L2A_”.

The command line tool does not provide progress report for now.

Proper configuration of the tool on the system should set up a number of environment variable.

S2L2APPHOME : a path to the root of the install tree of the Sen2Core tool
S2L2APPCFG : $S2L2APPHOME/cfg containing the GIPP file (see below)
PATH : $S2L2APPHOME/bin:$PATH
GDAL_DATA : path to a data directory provided by GDAL

Additionnally, the tool can read advanced internal parameters from a GIPP file (an XML file) :

<?xml version="1.0" encoding="UTF-8"?>
<Level-2A_Ground_Image_Processing_Parameter>
    <Common_Section>
        <Trace_Level>INFO</Trace_Level> 
        <DN_Scale>4000.0</DN_Scale>
        <DEM_Directory>false</DEM_Directory> 
        <DEM_Reference>http://data_public:GDdci@data.cgiar-csi.org/srtm/tiles/GeoTIFF/</DEM_Reference> 
    </Common_Section>
    <Scene_Classification>
       <Snow_Map_Reference>GlobalSnowMap.tiff</Snow_Map_Reference>
        <Classificators>
            <NO_DATA>0</NO_DATA>
            <SATURATED_DEFECTIVE>1</SATURATED_DEFECTIVE>
            <DARK_FEATURES>2</DARK_FEATURES>
            <CLOUD_SHADOWS>3</CLOUD_SHADOWS>
            <VEGETATION>4</VEGETATION>
            <BARE_SOILS>5</BARE_SOILS>
            <WATER>6</WATER>
            <LOW_PROBA_CLOUDS>7</LOW_PROBA_CLOUDS>
            <MEDIUM_PROBA_CLOUDS>8</MEDIUM_PROBA_CLOUDS>
            <HIGH_PROBA_CLOUDS>9</HIGH_PROBA_CLOUDS>
            <THIN_CIRRUS>10</THIN_CIRRUS>
            <SNOW_ICE>11</SNOW_ICE>
        </Classificators>
        <Calibration>
            <Median_Filter>3</Median_Filter>
        </Calibration>
        <Thresholds>
            <T1_B02>0.18</T1_B02>
            <T2_B02>0.22</T2_B02>
            <T1_B04>0.08</T1_B04>
            <T2_B04>0.25</T2_B04>
            <T1_B8A>0.15</T1_B8A>
            <T2_B8A>0.35</T2_B8A>
            <T1_B10>0.012</T1_B10>
            <T2_B10>0.035</T2_B10>
            <T1_B12>0.25</T1_B12>
            <T2_B12>0.12</T2_B12>
            <T_B02_B12>0.018</T_B02_B12>
            <T_CLOUD_LP>0.0</T_CLOUD_LP>
            <T_CLOUD_MP>0.35</T_CLOUD_MP>
            <T_CLOUD_HP>0.65</T_CLOUD_HP>
            <T1_NDSI_CLD>-0.24</T1_NDSI_CLD>
            <T2_NDSI_CLD>-0.16</T2_NDSI_CLD>
            <T1_NDSI_SNW>0.20</T1_NDSI_SNW>
            <T2_NDSI_SNW>0.42</T2_NDSI_SNW>
            <T1_NDVI>0.36</T1_NDVI>
            <T2_NDVI>0.47</T2_NDVI>
            <T1_SNOW>0.12</T1_SNOW>
            <T2_SNOW>0.25</T2_SNOW>
            <T1_R_B02_B04>0.85</T1_R_B02_B04>
            <T2_R_B02_B04>0.95</T2_R_B02_B04>
            <T1_R_B8A_B03>1.50</T1_R_B8A_B03>
            <T2_R_B8A_B03>2.50</T2_R_B8A_B03>
            <T1_R_B8A_B11>0.90</T1_R_B8A_B11>
            <T2_R_B8A_B11>1.10</T2_R_B8A_B11>
            <T11_B02>-0.40</T11_B02>
            <T12_B02>0.46</T12_B02>
            <T11_R_B02_B11>0.8</T11_R_B02_B11>
            <T12_R_B02_B11>1.5</T12_R_B02_B11>
            <T21_R_B02_B11>2.00</T21_R_B02_B11>
            <T22_R_B02_B11>4.00</T22_R_B02_B11>
            <T21_B12>0.1</T21_B12>
            <T22_B12>-0.09</T22_B12>
            <RV_B2>6.96000</RV_B2>
            <RV_B3>5.26667</RV_B3>
            <RV_B4>5.37708</RV_B4>
            <RV_B8>7.52000</RV_B8>
            <RV_B11>5.45000</RV_B11>
            <RV_B12>2.55000</RV_B12>
            <T_SDW>0.75</T_SDW>
        </Thresholds>
    </Scene_Classification>
    <Atmospheric_Correction>
        <References>
            <Lib_Dir>lib</Lib_Dir>
            <Atm_Data_Filename>h99000_wv20_rura.atm</Atm_Data_Filename>
        </References>
        <Flags>
            <WV_Correction>1</WV_Correction>
            <VIS_Update_Mode>1</VIS_Update_Mode> 
            <WV_Watermask>1</WV_Watermask> 
            <Cirrus_Correction>1</Cirrus_Correction>
            <BRDF_Correction>0</BRDF_Correction>
            <BRDF_Lower_Bound>0.22</BRDF_Lower_Bound>
        </Flags>
        <Calibration>
            <DEM_Unit>0</DEM_Unit> 
            <Adj_Km>1.000</Adj_Km> 
            <Visibility>30.0</Visibility> 
            <Altitude>0.100</Altitude> 
            <Smooth_WV_Map>100.0</Smooth_WV_Map> 
            <WV_Threshold_Cirrus>0.25</WV_Threshold_Cirrus> 
        </Calibration>
        <Sensor>
            <Solar_Irradiance units = "[mW/cm2 micron]">
                <Band_List>
                    <wavelength fwhm="0.018800" e0="187.282">0.442250</wavelength>
                    <wavelength fwhm="0.064400" e0="195.995">0.492225</wavelength>
                    <wavelength fwhm="0.034800" e0="182.252">0.560310</wavelength>
                    <wavelength fwhm="0.028800" e0="151.628">0.663085</wavelength>
                    <wavelength fwhm="0.014400" e0="142.487">0.703959</wavelength>
                    <wavelength fwhm="0.014800" e0="128.363">0.742381</wavelength>
                    <wavelength fwhm="0.018800" e0="116.686">0.781725</wavelength>
                    <wavelength fwhm="0.014400" e0="104.008">0.833331</wavelength> 
                    <wavelength fwhm="0.022800" e0="095.447">0.865816</wavelength> 
                    <wavelength fwhm="0.019200" e0="081.983">0.942251</wavelength>
                    <wavelength fwhm="0.028800" e0="036.705">1.373680</wavelength>
                    <wavelength fwhm="0.088400" e0="024.734">1.609431</wavelength>
                    <wavelength fwhm="0.172800" e0="008.649">2.193888</wavelength>
                </Band_List>
            </Solar_Irradiance>
            <Calibration units = "(mW cm-2 sr-1 um-1), L = c0 + c1 * DN">
                <Band_List>
                    <wavelength c0="0.00000" c1="0.1">0.442250</wavelength>
                    <wavelength c0="0.00000" c1="0.1">0.492225</wavelength>
                    <wavelength c0="0.00000" c1="0.1">0.560310</wavelength>
                    <wavelength c0="0.00000" c1="0.1">0.663085</wavelength>
                    <wavelength c0="0.00000" c1="0.1">0.703959</wavelength>
                    <wavelength c0="0.00000" c1="0.1">0.742381</wavelength>
                    <wavelength c0="0.00000" c1="0.1">0.781725</wavelength>
                    <wavelength c0="0.00000" c1="0.1">0.833331</wavelength> 
                    <wavelength c0="0.00000" c1="0.1">0.865816</wavelength> 
                    <wavelength c0="0.00000" c1="0.1">0.942251</wavelength>
                    <wavelength c0="0.00000" c1="0.1">1.373680</wavelength>
                    <wavelength c0="0.00000" c1="0.1">1.609431</wavelength>
                    <wavelength c0="0.00000" c1="0.1">2.193888</wavelength>
                </Band_List>
            </Calibration>
        </Sensor>
    </Atmospheric_Correction>
</Level-2A_Ground_Image_Processing_Parameter>

We might want the Standalone Tool Adapter system to be able to configure this.

Orfeo Toolbox

OTB comes with a set of command line tools exposing a number of functionnalities of the C++ library as simple-to-use executables.

The command line interface is shared between all the tools.

A general introduction can be found here and the application reference documentation describing the all application parameters can be found here.

Training material for using the applications is available at here

Here is a sample call of an OTB executable for segmenting an image :

$ otbcli_Segmentation -in segmentation_small_xt_phr.tif
-filter cc -filter.cc.expr "distance < 30"
-mode raster -mode.raster.out first_cc.tif uint32

Additionnal notes

The parameter system of an OTB application has a certain complexity. Parameter names are keys with hierarchical meaning. For example, the parameter mode.raster.out above is used only if the parameter mode is set to the value raster. Typically the segmentation application has a quite high number of parameters, with dependencies between them. See here for the full list of parameters.

OTB applications have already been integrated with the Processing module of QGIS, which is an equivalent of the SNAP Standalone Tool Adapter for the QGIS platform. A more simple parameter model is used there (flat list of parameters), so exposing for example the full list of parameters of the Segmentation application as a single QGIS module was not user-friendly enough. A new more simple representation of OTB applications has been developed in the frame of this development.

The full source code of the OTB Processing module can be found on the GitHub repository. The subdir containing the OTB applications descriptors should be reusable in the context of OTB SNAP integration as an intermediate layer for developing OTB SNAP modules, where some OTB applications have been split into several Processing modules.

For example, the OTB segmentation application has been split into five different modules (one for each segmentation algorithm : connected component, meanshift, edison, wateshed, morphological profiles), each one exposing only the parameters specific to each algorithm.

Progress reporting

OTB command line tools report progress with a progress bar on the standard output.

An implementation of the parser can be found in the QGIS Processing module code here.

GDAL

GDAL provides a number of executable exposing or extending the library functionnalities. The full list of raster utilities is here, and the full list of vector utilities is here.

GDAL command line tools have no common interface between them, so we foresee that each tool will need a specific template and will need to be studied and handled independently.

Proper installation of GDAL consists in the library, all the executables (some developed in C/C++, some others in python available), and a data directory containing mainly coordinate system definitions. The data directory need to be properly configured via the GDAL_DATA environment variable (unless GDAL is installed system-wise on Unix system).

gdal_translate

converts raster data between different formats

This is a basic tool performing data conversion between formats. This tool can be used to greatly enhance the number of supported formats in SNAP, at both the reading and writing level.

A basic call converting any of the available format to GeoTIFF is

gdal_translate -of GTiff -co "TILED=YES" input.anything output.tif

gdal_rasterize

takes an input vector file and outputs a raster file.

A subset of the functionnalities of this tool is available in OTB with the Rasterization tool. Wrapping the OTB tool might be easier as the interfacing will be close to the other OTB tool.