SRTM Download - Validity Check

boldtrn · February 27, 2016, 12:55pm

Hi,

I just ran into an issue when downloading SRTM data. The download itself went well (no exception). But when trying to access a file it was not available. A lot of files were there. When restarting the import everything went fine.

I just skimmed through the SRTMProvider.java and found the following code: https://github.com/graphhopper/graphhopper/blob/master/core/src/main/java/com/graphhopper/reader/dem/SRTMProvider.java#L301

Shouldn’t we at least report if we fail three times? So my first guess is that somehow one file could not be downloaded and was therefore skipped.

This is my StackTrace:

Exception in thread "PBF Reader" java.lang.RuntimeException: Unable to read PBF file.
	at com.graphhopper.reader.pbf.PbfReader.run(PbfReader.java:53)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
	at com.graphhopper.reader.OSMInputFile.process(OSMInputFile.java:291)
	at com.graphhopper.reader.pbf.PbfDecoder.sendResultsToSink(PbfDecoder.java:107)
	at com.graphhopper.reader.pbf.PbfDecoder.processBlobs(PbfDecoder.java:172)
	at com.graphhopper.reader.pbf.PbfDecoder.run(PbfDecoder.java:185)
	at com.graphhopper.reader.pbf.PbfReader.run(PbfReader.java:49)
	... 1 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
	at com.graphhopper.reader.OSMInputFile.process(OSMInputFile.java:288)
	... 5 more
Feb 27, 2016 12:36:45 PM com.google.inject.internal.MessageProcessor visit
INFO: An exception was caught and reported. Message: java.io.FileNotFoundException: /myFolder/N36E044.hgt.zip (No such file or directory)
java.lang.IllegalStateException: Couldn't load graph
	at com.graphhopper.http.DefaultModule.configure(DefaultModule.java:84)
	at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
	at com.google.inject.AbstractModule.install(AbstractModule.java:122)
	at com.graphhopper.http.GHServer$1.configure(GHServer.java:117)
	at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
	at com.google.inject.spi.Elements.getElements(Elements.java:110)
	at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
	at com.google.inject.Guice.createInjector(Guice.java:96)
	at com.google.inject.Guice.createInjector(Guice.java:73)
	at com.google.inject.Guice.createInjector(Guice.java:62)
	at com.graphhopper.http.GHServer.start(GHServer.java:64)
	at com.graphhopper.http.GHServer.main(GHServer.java:50)
Caused by: java.lang.RuntimeException: Couldn't process file europe-latest.osm.pbf, error: java.io.FileNotFoundException: /myFolder/N36E044.hgt.zip (No such file or directory)
	at com.graphhopper.reader.OSMReader.writeOsm2Graph(OSMReader.java:327)
	at com.graphhopper.reader.OSMReader.readGraph(OSMReader.java:155)
	at com.graphhopper.GraphHopper.importData(GraphHopper.java:771)
	at com.graphhopper.GraphHopper.process(GraphHopper.java:737)
	at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:710)
	at com.graphhopper.http.DefaultModule.createGraphHopper(DefaultModule.java:56)
	at com.graphhopper.http.DefaultModule.configure(DefaultModule.java:69)
	... 14 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /myFolder/N36E044.hgt.zip (No such file or directory)
	at com.graphhopper.reader.dem.SRTMProvider.updateHeightsFromZipFile(SRTMProvider.java:281)
	at com.graphhopper.reader.dem.SRTMProvider.getEle(SRTMProvider.java:249)
	at com.graphhopper.reader.OSMReader.getElevation(OSMReader.java:624)
	at com.graphhopper.reader.OSMReader.addNode(OSMReader.java:609)
	at com.graphhopper.reader.OSMReader.processNode(OSMReader.java:584)
	at com.graphhopper.reader.OSMReader.writeOsm2Graph(OSMReader.java:297)
	... 20 more
Caused by: java.io.FileNotFoundException: /myFolder/N36E044.hgt.zip (No such file or directory)
	at java.io.FileInputStream.open(Native Method)
	at java.io.FileInputStream.<init>(FileInputStream.java:146)
	at com.graphhopper.reader.dem.SRTMProvider.getByteArrayFromZipFile(SRTMProvider.java:311)
	at com.graphhopper.reader.dem.SRTMProvider.updateHeightsFromZipFile(SRTMProvider.java:266)
	... 25 more

karussell · February 29, 2016, 11:44am

Not sure what and how this is happening. But yes, the error handling is ugly and we should report this problem or even exit?

Also as we now have a more stable hosting (thanks !) we should avoid the FileNotFoundException catch and just return without the retry. I.e. retries then really only happen if the server does not respond and not if we hit ‘sea area’

boldtrn · February 29, 2016, 1:23pm

Maybe we could set this up as robust as possible? E.G. if we cannot find the file. try to download it. if this is not possible, try with a fallback url, if this is not possible as well, report a warning or error and interpolate the height (use the height information that surrounds the area and create something that is close to the surrounding area)?

karussell · February 29, 2016, 1:43pm

I think we should only report an error if there really is land with elevation data

Reporting errors for all the non-existing water areas will be overwhelming and real error will be ‘hidden’ again

boldtrn · February 29, 2016, 2:10pm

Yes sure . Maybe the easy solution could be to use a fallback url if the data is not available on the first url?

karussell · February 29, 2016, 2:26pm

I would throw an exception if no data is available (file not found) and handle it more properly upstream so that we create sea level HeightTiles. But if no server connection or similar, I would throw a runtimeexception which should stop the whole import.

See here in CGIARProvider: graphhopper/core/src/main/java/com/graphhopper/reader/dem/CGIARProvider.java at master · graphhopper/graphhopper · GitHub where we have this sea level handling.

boldtrn · February 29, 2016, 4:51pm

Sounds reasonable. So we would throw an Exception after the third iteration at this point: https://github.com/graphhopper/graphhopper/blob/master/core/src/main/java/com/graphhopper/reader/dem/CGIARProvider.java#L186 ?

karussell · February 29, 2016, 5:06pm

Yes, sounds good !