Hi,
we’re trying to get Graphhopper running with road data (nodes: 1 687 333, edges: 1 037 892, according to graphhopper). We use the postgres adapter (graphhopper/reader-postgis/src at postgis-4.0 · mbasa/graphhopper · GitHub) and a simple (default) config:
graphhopper:
profiles:
- name: car
vehicle: car
weighting: fastestprofiles_ch:
- profile: carprofiles_lm:
prepare.min_network_size: 200
routing.non_ch.max_waypoint_distance: 1000000
graph.dataaccess: RAM_STORE
The problem seems to be memory/performance related.
- When
sortGraph=false
and increasing the following parameters:
private int preciseIndexResolution = 300*100; // 300
private int maxRegionSearch = 4*40; // 4
// for prepare
private int minNetworkSize = 200; // original: 200
We reach to a point where LocationIndexTree#prepareInMemConstructionIndex
is executed.
The problem however is that this is painfully slow: it takes about 1second for 10 random edge iterations. It seems inMem.addToAllTilesOnLine
in this loop (graphhopper/core/src/main/java/com/graphhopper/storage/index/LocationIndexTree.java at 27b9ad3edf8c2e8c07c946f6293b1c265b3aa6b4 · graphhopper/graphhopper · GitHub) is slow:
loop for 44points took=1614ms
loop for 9points took=487ms
loop for 28points took=1263ms
loop for 1points took=0ms
loop for 21points took=952ms
Since we have 1mil+ edges this is not a feasible approach. We also needed to increase these parameters because otherwise we get an error of the class SpatialKeyAlgo
: allBits is too big for this implementation: 52
.
- When
sortGraph=true
, we get anjava.lang.OutOfMemoryError
whenIndexStructureInfo.create
is executed. We assumed that sorting the graph might speedup, but are unable to get it working wit 8GB ram.
We are not sure how this sorting impacts Graphhopper, but our main problem is that we cannot get past the index creation for our dataset because it is very slow.
Below is a partial log output. Anyone knows how to optimise or what can be done to get it working?
/Users/…/Library/Java/JavaVirtualMachines/openjdk-15.0.1/Contents/Home/bin/java -ea -Xmx15096m -Xmx8096m -Xms8096m -Duser.language=en -Didea.test.cyclic.buffer.size=4194304 -Dfile.encoding=UTF-8 -classpath /Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar: …
2022-01-03 14:22:47 INFO GraphHopperPostgis:625 - version 4.0|2022-01-03T12:44:56Z (7,20,5,4,5,7)
2022-01-03 14:22:47 INFO GraphHopperPostgis:627 - graph car|RAM_STORE|2D|no_turn_cost|, details:edges:0(0MB), nodes:0(0MB), name:(0MB), geo:0(0MB), bounds:1.7976931348623157E308,-1.7976931348623157E308,1
…
2022-01-03 14:23:05 INFO OSMPostgisReader:146 - 8 900 000 (junctions), junctionMap:8 894 341 totalMB:8096, usedMB:1319
2022-01-03 14:23:05 INFO OSMPostgisReader:164 - Number of junction points : 1687332
processed junctions
2022-01-03 14:23:05 INFO PostgisReader:134 - Opening DB connection to postgis localhost:5432 to database locatieservices2 schema public
2022-01-03 14:23:05 INFO PostgisReader:103 - Getting the feature iterator for graphhopper
2022-01-03 14:23:23 INFO OSMPostgisReader:227 - 1 000 000 (edges) totalMB:8096, usedMB:2692
processed roads
2022-01-03 14:23:23 INFO PostgisReader:134 - Opening DB connection to postgis localhost:5432 to database locatieservices2 schema public
2022-01-03 14:23:23 INFO PostgisReader:103 - Getting the feature iterator for graphhopper
processed restrictions
2022-01-03 14:23:33 INFO OSMPostgisReader:351 - Finished reading. Zero Counter 0 totalMB:8096, usedMB:3311
finished reading
imported osm
2022-01-03 14:23:33 INFO PrepareRoutingSubnetworks:90 - Start marking subnetworks, prepare.min_network_size: 200, nodes: 1 687 333, edges: 1 037 892, jobs: [my_car_subnetwork|fastest|car], totalMB:8096, usedMB:3311
2022-01-03 14:23:34 INFO PrepareRoutingSubnetworks:108 - my_car - Found 649686 subnetworks (0 single edges and 649686 components with more than one edge, total nodes: 2075784), took: 0.56943756s
2022-01-03 14:23:34 INFO PrepareRoutingSubnetworks:150 - my_car - Marked 649685 subnetworks (biggest: 210 edges) → 1 components(s) remain (smallest: 240, biggest: 240 edges), total marked edges: 1037772, took: 0.27964166s
2022-01-03 14:23:34 INFO PrepareRoutingSubnetworks:95 - Finished finding and marking subnetworks for 1 jobs, took: 0.8502388s, totalMB:8096, usedMB:2895
2022-01-03 14:23:34 INFO GraphHopperPostgis:1176 - nodes: 1 687 333, edges: 1 037 892
cleanup
sortGraph=false
interpolateBridgesTunnelsAndFerries
initLocationIndex
LocationIndexTree
setResolution
setMaxRegionSearch
loadExisting
ensureWriteAccess
prepareIndex
2022-01-03 14:26:34 INFO LocationIndexTree:187 - creating IndexStructureInfo.create(bounds=3.0E-6,261489.846,5.03E-4,260803.533, minResolutionInMeter=30000)
2022-01-03 14:26:34 INFO LocationIndexTree:189 - creating InMemConstructionIndex=IndexStructureInfo{entries=10, shiftssize=10, shifts=[4, 4, 4, 4, 4, 4, 4, 4, 4, 2], pixelGridTraversal=com.graphhopper.storage.index.PixelGridTraversal@40238dd0, keyAlgo=com.graphhopper.geohash.SpatialKeyAlgo@7776ab, bounds=3.0E-6,261489.846,5.03E-4,260803.533, parts=524288})