{"type":"doc","content":[{"type":"heading","attrs":{"level":2},"content":[{"text":"Bulk Processing with GPT","type":"text"}]},{"type":"paragraph","content":[{"text":"This little tutorial gives an introduction on bulk processing with the command shell on Windows and Unix systems. The provided scripts try to stay very generic in order to serve multiple processing requirements. However, not every edge case can be covered. The intention is to cover at least the main use cases. The scripts can probably be improved at multiple points but they can give you a starting point to write your own scripts. If you know improvements to the scripts or have questions regarding the usage of the script you are kindly invited to the ","type":"text"},{"text":"SNAP Forum","type":"text","marks":[{"type":"underline"},{"type":"link","attrs":{"href":"http://forum.step.esa.int/"}}]},{"text":".","type":"text"},{"type":"hardBreak"},{"text":"A general introduction to GPT and graphs can be found at ","type":"text"},{"text":"Creating a GPF Graph","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/spaces/SNAP/pages/70503590"}}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"The four files mentioned below are attached for download.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"processDataset.bat","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/download/attachments/70503475/processDataset.bat?api=v2"}}]},{"text":" (Windows)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"processDataset.bash","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/download/attachments/70503475/processDataset.bash?api=v2"}}]},{"text":" (Unix)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resample_s2.xml","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/download/attachments/70503475/resample_s2.xml?api=v2"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"resample_20m.properties","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/download/attachments/70503475/resample_20m.properties?api=v2"}}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Table of Contents","type":"text"}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"4"},"style":{"value":"disc"}},"macroMetadata":{"macroId":{"value":"154c2d5c-d204-4760-a2b5-260d5bda96e6"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"f55c3ee9-bce2-4f99-8a99-c45d7529b1af"}},{"type":"heading","attrs":{"level":4},"content":[{"text":"The Windows Script","type":"text"}]},{"type":"mediaGroup","content":[{"type":"media","attrs":{"id":"d1cca35f-b50c-4cc7-851d-08a8d458005e","collection":"contentId-70503475","type":"file"}}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"The Unix Script","type":"text"}]},{"type":"mediaGroup","content":[{"type":"media","attrs":{"id":"8327b956-5089-4f60-a9d2-0057e08c0b3a","collection":"contentId-70503475","type":"file"}}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Description of the Scripts","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Unix: The first line tells the shell what interpreter to use to run the script. Here it is bash.","type":"text"},{"type":"hardBreak"},{"text":"Windows: The first thing done in the script is to enable delayed expansion. This allows the evaluation of variables within a loop and is needed when iterating over the source products later on.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Next the path to the gpt batch file is specified. This script is later called to process the data products. The path to the script has to be adapted by the user.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The five parameters which must be given to the script are stored in variables for easier readability.","type":"text"},{"type":"hardBreak"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The path to the XML graph file which defines the processing graph performed on the source data product.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The path to a parameter file. The parameters specified in this file are used to configure graph file. The parameters can be given in the plain ","type":"text"},{"text":"properties format","type":"text","marks":[{"type":"underline"},{"type":"link","attrs":{"href":"http://en.wikipedia.org/wiki/.properties"}}]},{"text":" format.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The path pointing to the directory which contains the source products.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The path pointing to the directory where the processed data shall be placed.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"A file prefix in order to alter the name of the source product and indicate the type of processing.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Unix: Some helper functions are defined. They are later used in the main processing section.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"An output directory is created to keep hold of the processed data.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Now the iteration over all source products starts. Here only products with the extension 'SAFE' and the prefix 'S2' are considered.","type":"text"},{"type":"hardBreak"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The absolute path of the current source file is retrieved.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The path to the target file is compiled using the target directory, the file prefix and the name of the source product without extension. The file extension for the BEAM-DIMAP format is always appended.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The command line is assembled using the path to the gpt.exe file, the XML graph file, the parameter file and the source file and target file. The ","type":"text"},{"text":"-e","type":"text","marks":[{"type":"code"}]},{"text":" option is added in order to get longer messages in case of an error.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"As last step the command line is executed.","type":"text"}]}]}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"The loop used in the windows batch script is specific for looping recursively over folders in a directory. If the loop should only consider specific files in a source directory the sytanx can be simpler.","type":"text"}]},{"type":"paragraph","content":[{"text":"for %%F in (%sourceDirectory%\\*.nc) do (","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"…","type":"text"}]},{"type":"paragraph","content":[{"text":")","type":"text"}]},{"type":"paragraph","content":[{"text":"This will loop over all *.nc files within the specified source directory.","type":"text"}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Known Limitations of the Scripts","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Naming of the target product is limited. Use cases might be that the name should remain, or a more complex pattern should be applied.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Which products of the source directory are used for processing is currently hard coded (only with '.SAFE' extension and 'S2' prefix). This should be configurable.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The format of the target product is not configurable.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Example Usage","type":"text"}]},{"type":"paragraph","content":[{"text":"A set of input Sentinel-2 products shall be processed with the Resample processor.","type":"text"},{"type":"hardBreak"},{"text":"Therefore a XML graph is defined. For the resampling parameters variables are used. The variables are set in a properties file. This makes it easier to change them. It is also possible to write them directly into the XML graph file, as done for the ","type":"text"},{"text":"resampleOnPyramidLevels.","type":"text","marks":[{"type":"code"}]}]},{"type":"heading","attrs":{"level":5},"content":[{"text":"XML Graph File for S2 Resampling (resample_s2.xml)","type":"text","marks":[{"type":"strong"}]}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"\n 1.0\n \n Resample\n \n ${sourceProduct}\n \n \n ${resolution}\n ${up}\n ${down}\n ${flag}\n false\n \n \n","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"When entering expressions directly in the XM file then they often need to be escaped. Some characters have special meanings in XML. If they are used for other purposes, for example in an expression they need to be escaped.","type":"text"}]},{"type":"paragraph","content":[{"text":"These characters are: ","type":"text"},{"text":"“ ' < > &","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"An expression like this: ","type":"text"},{"text":"B8 > 0 && B8 < 0.1","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"Needs to be written like: ","type":"text"},{"text":"B8 %gt; 0 && B8 < 0.1","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"More details are on ","type":"text"},{"text":"Stackoverflow","type":"text","marks":[{"type":"link","attrs":{"href":"https://stackoverflow.com/questions/1091945/what-characters-do-i-need-to-escape-in-xml-documents"}}]},{"text":".","type":"text"}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"In the parameter file the 4 used parameters are defined.","type":"text"}]},{"type":"heading","attrs":{"level":5},"content":[{"text":"Parameters File (resample_20m.properties)","type":"text","marks":[{"type":"strong"}]}]},{"type":"codeBlock","attrs":{"language":"plaintext"},"content":[{"text":"resolution=20\ndown=Min\nup=Bicubic\nflag=First\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Now as we have all information we can call the scripts from the command line.","type":"text"}]},{"type":"paragraph","content":[{"text":"On Windows it is assumed that the source products are located in the directory '","type":"text"},{"text":"C:\\Eodata\\toProcess","type":"text","marks":[{"type":"code"}]},{"text":"' and the XML graph file and the parameter are in the same directory as the batch file. The processed files will go to '","type":"text"},{"text":"C:\\Eodata\\toProcess\\output","type":"text","marks":[{"type":"code"}]},{"text":"' and have the prefix ","type":"text"},{"text":"resampled20m","type":"text","marks":[{"type":"em"}]}]},{"type":"paragraph","content":[{"text":">processDataset.bat resample_s2.xml resample_20m.properties C:\\Eodata\\toProcess C:\\Eodata\\toProcess\\output resampled20m","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"On Unix the directory of source products is assumed to be '","type":"text"},{"text":"/Eodata/toProcess","type":"text","marks":[{"type":"code"}]},{"text":"' while the XML graph file and the parameter are in the same directory as the script file. The processed files will go to '","type":"text"},{"text":"/Eodata/toProcess/output","type":"text","marks":[{"type":"code"}]},{"text":"' and have the prefix ","type":"text"},{"text":"resampled20m","type":"text","marks":[{"type":"em"}]}]},{"type":"paragraph","content":[{"text":">processDataset.bash resample_s2.xml resample_20m.properties \"/Eodata/toProcess\" \"/Eodata/toProcess/output\" resampled20m","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Helpful Links","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://www.tutorialspoint.com/batch_script/index.htm"}},{"text":" - Explain the usage of batch scripts very exhaustively.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/windows-commands"}},{"text":" - General introduction to batch files and the commands.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"http://www.dostips.com/DtTipsStringOperations.php"}},{"text":" - Gives examples for string operations in batch files.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://www.robvanderwoude.com/batchstart.php"}},{"text":" - Examples, How-To and explanations about batch files.","type":"text"}]}]}]},{"type":"paragraph"}],"version":1}