SRTM Download - Validity Check

Hi,

I just ran into an issue when downloading SRTM data. The download itself went well (no exception). But when trying to access a file it was not available. A lot of files were there. When restarting the import everything went fine.

I just skimmed through the SRTMProvider.java and found the following code: https://github.com/graphhopper/graphhopper/blob/master/core/src/main/java/com/graphhopper/reader/dem/SRTMProvider.java#L301

Shouldn’t we at least report if we fail three times? So my first guess is that somehow one file could not be downloaded and was therefore skipped.

This is my StackTrace:

Exception in thread "PBF Reader" java.lang.RuntimeException: Unable to read PBF file.
	at com.graphhopper.reader.pbf.PbfReader.run(PbfReader.java:53)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
	at com.graphhopper.reader.OSMInputFile.process(OSMInputFile.java:291)
	at com.graphhopper.reader.pbf.PbfDecoder.sendResultsToSink(PbfDecoder.java:107)
	at com.graphhopper.reader.pbf.PbfDecoder.processBlobs(PbfDecoder.java:172)
	at com.graphhopper.reader.pbf.PbfDecoder.run(PbfDecoder.java:185)
	at com.graphhopper.reader.pbf.PbfReader.run(PbfReader.java:49)
	... 1 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
	at com.graphhopper.reader.OSMInputFile.process(OSMInputFile.java:288)
	... 5 more
Feb 27, 2016 12:36:45 PM com.google.inject.internal.MessageProcessor visit
INFO: An exception was caught and reported. Message: java.io.FileNotFoundException: /myFolder/N36E044.hgt.zip (No such file or directory)
java.lang.IllegalStateException: Couldn't load graph
	at com.graphhopper.http.DefaultModule.configure(DefaultModule.java:84)
	at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
	at com.google.inject.AbstractModule.install(AbstractModule.java:122)
	at com.graphhopper.http.GHServer$1.configure(GHServer.java:117)
	at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
	at com.google.inject.spi.Elements.getElements(Elements.java:110)
	at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
	at com.google.inject.Guice.createInjector(Guice.java:96)
	at com.google.inject.Guice.createInjector(Guice.java:73)
	at com.google.inject.Guice.createInjector(Guice.java:62)
	at com.graphhopper.http.GHServer.start(GHServer.java:64)
	at com.graphhopper.http.GHServer.main(GHServer.java:50)
Caused by: java.lang.RuntimeException: Couldn't process file europe-latest.osm.pbf, error: java.io.FileNotFoundException: /myFolder/N36E044.hgt.zip (No such file or directory)
	at com.graphhopper.reader.OSMReader.writeOsm2Graph(OSMReader.java:327)
	at com.graphhopper.reader.OSMReader.readGraph(OSMReader.java:155)
	at com.graphhopper.GraphHopper.importData(GraphHopper.java:771)
	at com.graphhopper.GraphHopper.process(GraphHopper.java:737)
	at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:710)
	at com.graphhopper.http.DefaultModule.createGraphHopper(DefaultModule.java:56)
	at com.graphhopper.http.DefaultModule.configure(DefaultModule.java:69)
	... 14 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /myFolder/N36E044.hgt.zip (No such file or directory)
	at com.graphhopper.reader.dem.SRTMProvider.updateHeightsFromZipFile(SRTMProvider.java:281)
	at com.graphhopper.reader.dem.SRTMProvider.getEle(SRTMProvider.java:249)
	at com.graphhopper.reader.OSMReader.getElevation(OSMReader.java:624)
	at com.graphhopper.reader.OSMReader.addNode(OSMReader.java:609)
	at com.graphhopper.reader.OSMReader.processNode(OSMReader.java:584)
	at com.graphhopper.reader.OSMReader.writeOsm2Graph(OSMReader.java:297)
	... 20 more
Caused by: java.io.FileNotFoundException: /myFolder/N36E044.hgt.zip (No such file or directory)
	at java.io.FileInputStream.open(Native Method)
	at java.io.FileInputStream.<init>(FileInputStream.java:146)
	at com.graphhopper.reader.dem.SRTMProvider.getByteArrayFromZipFile(SRTMProvider.java:311)
	at com.graphhopper.reader.dem.SRTMProvider.updateHeightsFromZipFile(SRTMProvider.java:266)
	... 25 more

Not sure what and how this is happening. But yes, the error handling is ugly and we should report this problem or even exit?

Also as we now have a more stable hosting (thanks :slight_smile: !) we should avoid the FileNotFoundException catch and just return without the retry. I.e. retries then really only happen if the server does not respond and not if we hit ‘sea area’

Maybe we could set this up as robust as possible? E.G. if we cannot find the file. try to download it. if this is not possible, try with a fallback url, if this is not possible as well, report a warning or error and interpolate the height (use the height information that surrounds the area and create something that is close to the surrounding area)?

I think we should only report an error if there really is land with elevation data :slight_smile:

Reporting errors for all the non-existing water areas will be overwhelming and real error will be ‘hidden’ again

Yes sure :slightly_smiling:. Maybe the easy solution could be to use a fallback url if the data is not available on the first url?

I would throw an exception if no data is available (file not found) and handle it more properly upstream so that we create sea level HeightTiles. But if no server connection or similar, I would throw a runtimeexception which should stop the whole import.

See here in CGIARProvider: https://github.com/graphhopper/graphhopper/blob/master/core/src/main/java/com/graphhopper/reader/dem/CGIARProvider.java#L190 where we have this sea level handling.

Sounds reasonable. So we would throw an Exception after the third iteration at this point: https://github.com/graphhopper/graphhopper/blob/master/core/src/main/java/com/graphhopper/reader/dem/CGIARProvider.java#L186 ?

Yes, sounds good :slight_smile: !