Build europan route graph with 8gb RAM

Open Source Routing Engine
1

Hi guys and thanks in advance for your time.

I have a centOS 6.6 machine with 8gb RAM and I tried to launch graphhopper on european-map.osm.pbf with the following command:

nohup java -Xmx8g -Xms8g -jar graphhopper-web-0.4.1-with-dep.jar jetty.resourcebase=webapp config=config-example.properties osmreader.osm=europe-latest.osm.pbf 2>>start_graph.log 1>>err_graph.log &

I obtain an OutOfMemory error. Are there some ways to solve this without increase RAM capacity?? Maybe using swap space or with different command launch…

Thanks a lot

2

Without increasing RAM capacity you can try graph.dataaccess=MMAP_STORE setting for the import, but this could be a lot slower.

BTW: Why are you using an old GH version?

3

I have similar issue OutOfMemoryError, trying to build whole planet, I have 64gb RAM. I set graph.dataaccess=MMAP. Do I need more RAM?

`2017-08-01 17:31:08,104 [main] WARN com.graphhopper.reader.osm.OSMReader - Parsing error in way with OSMID=428718022 : Cannot parse duration tag value: 00:20h
2017-08-01 17:31:08,105 [main] WARN com.graphhopper.reader.osm.OSMReader - Parsing error in way with OSMID=428718025 : Cannot parse duration tag value: 00:55h
2017-08-01 17:31:08,106 [main] WARN com.graphhopper.reader.osm.OSMReader - Parsing error in way with OSMID=428718028 : Cannot parse duration tag value: 00:20h
2017-08-01 17:34:19,101 [main] WARN com.graphhopper.routing.util.AbstractFlagEncoder - Unrealistic long duration ignored in way with way ID=435002819 : Duration tag value=P15M (=655200 minutes)
2017-08-01 17:34:19,101 [main] WARN com.graphhopper.routing.util.AbstractFlagEncoder - Unrealistic long duration ignored in way with way ID=435002819 : Duration tag value=P15M (=655200 minutes)
2017-08-01 17:48:29,711 [main] WARN com.graphhopper.reader.osm.OSMReader - Parsing error in way with OSMID=460994819 : Cannot parse duration tag value: '00:05’
2017-08-01 17:52:51,105 [main] WARN com.graphhopper.routing.util.AbstractFlagEncoder - Unrealistic long duration ignored in way with way ID=469672195 : Duration tag value=1:00 (=60 minutes)
2017-08-01 18:03:57,328 [main] INFO com.graphhopper.reader.osm.OSMReader - 4 400 000 000, locs:991 822 956 (0) totalMB:15219, usedMB:12439
2017-08-01 18:13:01,585 [main] INFO com.graphhopper.reader.osm.OSMReader - 4 416 743 003, now parsing relations
2017-08-01 18:13:37,431 [main] INFO com.graphhopper.reader.osm.OSMReader - finished way processing. nodes: 169339701, osmIdMap.size:992782833, osmIdMap:11692MB, nodeFlagsMap.size:1441947, relFlagsMap.size:1923325, zeroCounter:946185 totalMB:15219, usedMB:13876
2017-08-01 18:13:43,433 [main] INFO com.graphhopper.reader.osm.OSMReader - time pass1:2213s, pass2:27523s, total:29736s
2017-08-01 18:13:43,480 [main] INFO com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks - start finding subnetworks (min:200, min one way:200) totalMB:15219, usedMB:12170
2017-08-01 18:37:39,636 [main] INFO com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks - car findComponents time:1436.1173, size:1349204
2017-08-01 18:38:47,141 [main] INFO com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks - 23452891 subnetworks found for car, totalMB:9394, usedMB:6422
2017-08-01 19:03:15,033 [main] INFO com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks - foot findComponents time:1467.8918, size:1220117
2017-08-01 19:04:17,047 [main] INFO com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks - 6066381 subnetworks found for foot, totalMB:11155, usedMB:5688
2017-08-01 19:04:31,456 [main] INFO com.graphhopper.routing.subnetwork.PrepareRoutingSubnetworks - optimize to remove subnetworks (23452891), unvisited-dead-end-nodes (8255307), maxEdges/node (34)
2017-08-01 19:55:37,051 [main] INFO com.graphhopper.http.GraphHopperModule$1 - edges: 222851262, nodes 164908482, there were 23452891 subnetworks. removed them => 4431219 less nodes
2017-08-01 20:03:58,398 [main] INFO com.graphhopper.storage.index.LocationIndexTree - location index created in 501.30847s, size:193 298 451, leafs:39 706 147, precision:300, depth:6, checksum:164908482, entries:[64, 64, 64, 64, 16, 4], entriesPerLeaf:4.8682246
2017-08-01 20:04:02,239 [main] INFO com.graphhopper.routing.ch.CHAlgoFactoryDecorator - 1/2 calling CH prepare.doWork for fastest|car … (totalMB:11114, usedMB:6139)
2017-08-01 20:04:02,243 [main] INFO com.graphhopper.routing.ch.CHAlgoFactoryDecorator - 2/2 calling CH prepare.doWork for fastest|foot … (totalMB:11114, usedMB:6139)
Exception in thread “main” java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.graphhopper.routing.ch.CHAlgoFactoryDecorator.prepare(CHAlgoFactoryDecorator.java:307)
at com.graphhopper.GraphHopper.prepareCH(GraphHopper.java:1168)
at com.graphhopper.GraphHopper.postProcessing(GraphHopper.java:847)
at com.graphhopper.GraphHopper.process(GraphHopper.java:650)

`

4

Depending on the number of encoders this should work even for 32GB and with graph.dataaccess=RAM_STORE

What are your -Xmx settings? What is your config.properties file? See the docs: https://github.com/graphhopper/graphhopper/blob/master/docs/core/deploy.md

5

Thank you, one step further, but now I’m getting
IllegalStateException: Setting weights smaller than 0.001 is not allowed in CHGraphImpl#setWeight
What’s can be a problem? I’m using default settings, only added foot. Tried few times, every time downloaded new osm file. I can make it work with small maps, but not with planet.

my config.properties file:

graph.flag_encoders=car,foot
prepare.ch.weightings=fastest
prepare.min_network_size=200
prepare.min_one_way_network_size=200
routing.non_ch.max_waypoint_distance = 1000000
graph.dataaccess=RAM_STORE

Command I ran:
export JAVA_OPTS="-server -Xconcurrentio -Xmx55000m -Xms55000m" && nohup ./graphhopper.sh web planet-latest.osm.pbf > log.txt &

OSM image from https://planet.osm.org/, Latest Weekly Planet PBF File

log:
java.lang.IllegalStateException: Setting weights smaller than 0.001 is not allowed in CHGraphImpl#setWeight. You passed: 0.0 for the edge from 39.803126785229,-105.17421488108221 to 39.803126785229,-105.17421488108221

6

Either use the very recent master (fixed yesterday ;)) or a stable like 0.9 :slight_smile: