Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

"Too many open files" when reading big S2 datasets

Description

Opening a large S2 dataset I get an error when opj_decompress is executed :

opj_decompress process failed! :java.io.IOException: Cannot run program "/home/jmalik/.snap/auxdata/openjpeg/openjpeg-2.1.0-Linux-x64/bin/opj_decompress" (in directory "/home/jmalik/.snap/var/cache/s2tbx/l1c-reader/S2A_OPER_PRD_MSIL1C_PDMC_20151201T124142_R131_V20151129T011528_20151129T011528.SAFE"): error=24, Too many open files at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.esa.s2tbx.dataio.openjpeg.OpenJpegUtils.runProcess(OpenJpegUtils.java:90) at org.esa.s2tbx.dataio.s2.S2TileOpImage.decompressTile(S2TileOpImage.java:279) at org.esa.s2tbx.dataio.s2.S2TileOpImage.computeRect(S2TileOpImage.java:178) at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137) at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904) at javax.media.jai.OpImage.getTile(OpImage.java:1129) at com.sun.media.jai.opimage.TranslateIntOpImage.getTile(TranslateIntOpImage.java:132) at javax.media.jai.PlanarImage.cobbleUShort(PlanarImage.java:3052) at javax.media.jai.PlanarImage.getData(PlanarImage.java:2175) at javax.media.jai.PlanarImage.getExtendedData(PlanarImage.java:2440) at com.sun.media.jai.opimage.MosaicOpImage.computeTile(MosaicOpImage.java:432) at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904) at javax.media.jai.OpImage.getTile(OpImage.java:1129) at com.sun.media.jai.opimage.CropOpImage.getTile(CropOpImage.java:122)

resulting in lots of missing tiles, so a product that looks very bad.

Environment

None

GitHub Work

None

Activity

Show:

Cosmin Cara 7 December 2015 at 08:57

A quick fix was checked in (so that decompressed jp2 files are accordingly closed). A refactoring, as suggested by Julien, will follow.

Julien Malik 2 December 2015 at 10:13

Thanks for having a look !

How about tackling https://senbox.atlassian.net/browse/SIITBX-133#icft=SIITBX-133 first ? This way we factorize code.
Your JP2TileOpImage is much cleaner than our S2TileOpImage.

Cosmin Cara 2 December 2015 at 09:36

Indeed, I've seen that. I will try to come up with a correction.

Julien Malik 2 December 2015 at 09:22
Edited

In S2TileOpImage.readTileData, we open the decompressed pgx files but never close them (except in dispose).
This cannot end up well

Done

Details

Assignee

Reporter

Fix versions

Components

Priority

Created 2 December 2015 at 08:13
Updated 10 February 2023 at 14:26
Resolved 10 February 2023 at 14:22

Flag notifications