Error when importing planet .osm.pbf

Shahoo · January 11, 2021, 7:44pm

Hi,

I’ve been trying to import planet .osm but get the following exception after about a couple of hours.

Exception in thread “PBF Reader” java.lang.RuntimeException: Unable to read PBF file.
at com.graphhopper.reader.osm.pbf.PbfReader.run(PbfReader.java:50)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
at com.graphhopper.reader.osm.OSMInputFile.process(OSMInputFile.java:253)
at com.graphhopper.reader.osm.pbf.PbfDecoder.sendResultsToSink(PbfDecoder.java:96)
at com.graphhopper.reader.osm.pbf.PbfDecoder.processBlobs(PbfDecoder.java:151)
at com.graphhopper.reader.osm.pbf.PbfDecoder.run(PbfDecoder.java:162)
at com.graphhopper.reader.osm.pbf.PbfReader.run(PbfReader.java:47)
… 1 more
Caused by: java.lang.InterruptedException
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(Unknown Source)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.base/java.util.concurrent.LinkedBlockingQueue.put(Unknown Source)
at com.graphhopper.reader.osm.OSMInputFile.process(OSMInputFile.java:251)
… 5 more
2021-01-11 19:04:06.245 [main] ERROR io.dropwizard.cli.ServerCommand - Unable to start server, shutting down
java.lang.RuntimeException: Couldn’t process file planet-201228.osm.pbf, error: Can’t decode srtm_39_01.tif
at com.graphhopper.reader.osm.OSMReader.writeOsmToGraph(OSMReader.java:297)
at com.graphhopper.reader.osm.OSMReader.readGraph(OSMReader.java:148)
at com.graphhopper.GraphHopper.importData(GraphHopper.java:709)
at com.graphhopper.GraphHopper.readData(GraphHopper.java:688)
at com.graphhopper.GraphHopper.process(GraphHopper.java:675)
at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:640)
at com.graphhopper.http.GraphHopperManaged.start(GraphHopperManaged.java:125)
at io.dropwizard.lifecycle.JettyManaged.doStart(JettyManaged.java:27)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:407)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:371)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:45)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
at io.dropwizard.cli.Cli.run(Cli.java:78)
at io.dropwizard.Application.run(Application.java:94)
at com.graphhopper.http.GraphHopperApplication.main(GraphHopperApplication.java:36)
Caused by: java.lang.RuntimeException: Can’t decode srtm_39_01.tif
at com.graphhopper.reader.dem.CGIARProvider.generateRasterFromFile(CGIARProvider.java:112)
at com.graphhopper.reader.dem.AbstractTiffElevationProvider.getEle(AbstractTiffElevationProvider.java:136)
at com.graphhopper.reader.osm.OSMReader.getElevation(OSMReader.java:528)
at com.graphhopper.reader.osm.OSMReader.addNode(OSMReader.java:504)
at com.graphhopper.reader.osm.OSMReader.processNode(OSMReader.java:482)
at com.graphhopper.reader.osm.OSMReader.writeOsmToGraph(OSMReader.java:264)
… 20 common frames omitted
Caused by: java.io.EOFException: null
at org.apache.xmlgraphics.image.codec.util.SeekableStream.readUnsignedShort(SeekableStream.java:483)
at org.apache.xmlgraphics.image.codec.tiff.TIFFDirectory.getNumDirectories(TIFFDirectory.java:590)
at org.apache.xmlgraphics.image.codec.tiff.TIFFImageDecoder.getNumPages(TIFFImageDecoder.java:81)
at org.apache.xmlgraphics.image.codec.tiff.TIFFImageDecoder.decodeAsRenderedImage(TIFFImageDecoder.java:85)
at org.apache.xmlgraphics.image.codec.util.ImageDecoderImpl.decodeAsRaster(ImageDecoderImpl.java:136)
at org.apache.xmlgraphics.image.codec.util.ImageDecoderImpl.decodeAsRaster(ImageDecoderImpl.java:122)
at com.graphhopper.reader.dem.CGIARProvider.generateRasterFromFile(CGIARProvider.java:110)
… 25 common frames omitted
java.lang.RuntimeException: Couldn’t process file planet-201228.osm.pbf, error: Can’t decode srtm_39_01.tif
at com.graphhopper.reader.osm.OSMReader.writeOsmToGraph(OSMReader.java:297)
at com.graphhopper.reader.osm.OSMReader.readGraph(OSMReader.java:148)
at com.graphhopper.GraphHopper.importData(GraphHopper.java:709)
at com.graphhopper.GraphHopper.readData(GraphHopper.java:688)
at com.graphhopper.GraphHopper.process(GraphHopper.java:675)
at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:640)
at com.graphhopper.http.GraphHopperManaged.start(GraphHopperManaged.java:125)
at io.dropwizard.lifecycle.JettyManaged.doStart(JettyManaged.java:27)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:407)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:371)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:45)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
at io.dropwizard.cli.Cli.run(Cli.java:78)
at io.dropwizard.Application.run(Application.java:94)
at com.graphhopper.http.GraphHopperApplication.main(GraphHopperApplication.java:36)
Caused by: java.lang.RuntimeException: Can’t decode srtm_39_01.tif
at com.graphhopper.reader.dem.CGIARProvider.generateRasterFromFile(CGIARProvider.java:112)
at com.graphhopper.reader.dem.AbstractTiffElevationProvider.getEle(AbstractTiffElevationProvider.java:136)
at com.graphhopper.reader.osm.OSMReader.getElevation(OSMReader.java:528)
at com.graphhopper.reader.osm.OSMReader.addNode(OSMReader.java:504)
at com.graphhopper.reader.osm.OSMReader.processNode(OSMReader.java:482)
at com.graphhopper.reader.osm.OSMReader.writeOsmToGraph(OSMReader.java:264)
… 20 more
Caused by: java.io.EOFException
at org.apache.xmlgraphics.image.codec.util.SeekableStream.readUnsignedShort(SeekableStream.java:483)
at org.apache.xmlgraphics.image.codec.tiff.TIFFDirectory.getNumDirectories(TIFFDirectory.java:590)
at org.apache.xmlgraphics.image.codec.tiff.TIFFImageDecoder.getNumPages(TIFFImageDecoder.java:81)
at org.apache.xmlgraphics.image.codec.tiff.TIFFImageDecoder.decodeAsRenderedImage(TIFFImageDecoder.java:85)
at org.apache.xmlgraphics.image.codec.util.ImageDecoderImpl.decodeAsRaster(ImageDecoderImpl.java:136)
at org.apache.xmlgraphics.image.codec.util.ImageDecoderImpl.decodeAsRaster(ImageDecoderImpl.java:122)
at com.graphhopper.reader.dem.CGIARProvider.generateRasterFromFile(CGIARProvider.java:110)
… 25 more

I’ve tried with both the stable version and the master version but no luck. I’ve got a server with 128 GB ram. I also tried to use the default config file but still, it didn’t help I’m afraid.
I can import berlin .osm with no issues. And the planet .osm is downloaded from the official open street map.
I wonder what else I can try?

Thank you.

easbar · January 11, 2021, 9:05pm

Your error seems to be related to elevation. What are your elevation settings? Especially graph.elevation.provider and graph.elevation.cache_dir? Is there a srtm_39_01.tif file in this directory? The elevation data is downloaded on the fly during the import, so you need to make sure your internet connection is working, but it also might be related to a problem with the elevation data source.

easbar · January 11, 2021, 9:06pm

As a quick workaround you can of course disable elevation support by commenting out graph.elevation.provider in your config file.

Shahoo · January 11, 2021, 11:46pm

Thank you for you response.

Here is my elevation setting:
I had the cache directory set to an empty directory so for the elevation to be cached there for future uses. Now commented out that setting and trying again:

  graph.elevation.provider: cgiar

 # default location for cache is /tmp/srtm
 # graph.elevation.cache_dir: C:\graphhopper\executable\elevation

I’m afraid I cannot skip elevation support, so wondering how do I verify cgiar is responding ok whilst the file is being imported.

easbar · January 12, 2021, 7:52am

Something seems wrong with the CGIAR provider, I can reproduce your error with this code:

    @Test
    public void test_srtm_39_01() {
        new CGIARProvider("tmp_cache").getEle(57.5, 10);
    }

So for now your best option is using another elevation provider if you cannot skip elevation support altogether.
@karussell any idea what could be wrong with CGIAR here? The download url seems to be: http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF//srtm_39_01.zip

karussell · January 12, 2021, 12:13pm

The provider does something strange for this file. It will redirect to a URL without the slash before /wp-content.

But using https seems to fix the problem. Not sure if it will make the download slower or break something else. Would be nice if someone can host these files like kurviger hosts the SRTM files for us

@Shahoo Can you try the provider URL in the config via:

graphhopper:
  ...
  graph.elevation.base_url: https://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/

If this works we’ll have to fix this in master branch and remove this useless code

        String baseURL = ghConfig.getString("graph.elevation.base_url", "");
        if (baseURL.isEmpty())
            ghConfig.getString("graph.elevation.baseurl", "");

karussell · January 12, 2021, 12:20pm

Please use

graph.elevation.base_url

the baseurl won’t work (see the mentioned bug)

Shahoo · January 12, 2021, 5:56pm

I thought before applying the suggested elevation changes, to import the whole planet without elevation data but after 4 hours it threw this exception:

[5:49 PM] Shahin Kiassat

java.lang.IllegalStateException: graph is empty after in-place removal but was 146509886
at com.graphhopper.storage.BaseGraph.inPlaceNodeRemove(BaseGraph.java:797)
at com.graphhopper.storage.GraphHopperStorage.optimize(GraphHopperStorage.java:238)
at com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks.optimize(PrepareRoutingSubnetworks.java:90)
at com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks.doWork(PrepareRoutingSubnetworks.java:84)
at com.graphhopper.GraphHopper.cleanUp(GraphHopper.java:1340)
at com.graphhopper.GraphHopper.process(GraphHopper.java:676)
at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:640)
at com.graphhopper.http.GraphHopperManaged.start(GraphHopperManaged.java:125)
at io.dropwizard.lifecycle.JettyManaged.doStart(JettyManaged.java:27)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:407)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:371)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:45)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
at io.dropwizard.cli.Cli.run(Cli.java:78)
at io.dropwizard.Application.run(Application.java:94)
at com.graphhopper.http.GraphHopperApplication.main(GraphHopperApplication.java:36)
java.lang.IllegalStateException: graph is empty after in-place removal but was 146509886
at com.graphhopper.storage.BaseGraph.inPlaceNodeRemove(BaseGraph.java:797)
at com.graphhopper.storage.GraphHopperStorage.optimize(GraphHopperStorage.java:238)
at com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks.optimize(PrepareRoutingSubnetworks.java:90)
at com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks.doWork(PrepareRoutingSubnetworks.java:84)
at com.graphhopper.GraphHopper.cleanUp(GraphHopper.java:1340)
at com.graphhopper.GraphHopper.process(GraphHopper.java:676)
at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:640)
at com.graphhopper.http.GraphHopperManaged.start(GraphHopperManaged.java:125)
at io.dropwizard.lifecycle.JettyManaged.doStart(JettyManaged.java:27)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:407)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:371)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:45)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
at io.dropwizard.cli.Cli.run(Cli.java:78)
at io.dropwizard.Application.run(Application.java:94)
    at com.graphhopper.http.GraphHopperApplication.main(GraphHopperApplication.java:36)

And here is the log before the error was thrown:

2021-01-12 16:29:00.405 [main] INFO com.graphhopper.reader.osm.OSMReader - finished way processing. nodes: 146509886, osmIdMap.size:1407394312, osmIdMap:16584MB, nodeFlagsMap.size:313465, relFlagsMap.size:14048885, zeroCounter:0 totalMB:30688, usedMB:27348
2021-01-12 16:29:00.405 [main] INFO com.graphhopper.reader.osm.OSMReader - time pass1:4295s, pass2:7907s, total:12203s
2021-01-12 16:29:00.408 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - Start removing subnetworks (prepare.min_network_size:200) totalMB:30688, usedMB:27348
2021-01-12 16:29:00.408 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - Subnetwork removal jobs: [car|edge-based]
2021-01-12 16:29:00.409 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - Graph nodes: 146 509 886
2021-01-12 16:29:00.410 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - Graph edges: 0
2021-01-12 16:29:00.410 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - — vehicle: ‘car’
2021-01-12 16:29:00.418 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - Found 0 subnetworks (0 single edges and 0 components with more than one edge, total nodes: 0), took: 0.008324608s
2021-01-12 16:29:00.419 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - Removed 0 subnetworks (biggest removed: 0 edges) → 0 subnetwork(s) left (smallest: 0, biggest: 0 edges), total removed edges: 0, took: 6.77376E-4s
2021-01-12 16:29:05.690 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - Removed 146509886 nodes from the graph as they aren’t used by any vehicle after removing subnetworks
2021-01-12 16:29:05.913 [main] WARN com.graphhopper.storage.BaseGraph - More than a half of the network should be removed!? Nodes:146509886, remove:146509886
2021-01-12 16:29:12.189 [main] ERROR io.dropwizard.cli.ServerCommand - Unable to start server, shutting down

Wondering what went wrong and how can I fix it?

system · April 12, 2021, 5:56pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.