Hey all,
I’m still trying to build a car+foot index. I’m building from the latest github master branch. Today it’s not running out of memory (gave it 80g), but it does seem pretty stuck. The last output I saw from the builder was ~7 hours ago.
I’m wondering if I should give up on trying to build a combined index and instead build two separate ones and run them on separate boxes? Any advice on how to get this to work?
I’m running it with
JAVA_OPTS="-Xmx80g -Xms60g" ./graphhopper.sh -a web -i …/planet-191202.osm.pbf
My most recent output (it’s now 16:21 UTC on this box)
2019-12-11 17:51:46.623 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - foot.access|version=-1709808141|bits=1|index=1|shift=3|store_both_directions=true findComponents time:2370.195, size:1616251
2019-12-11 17:53:15.349 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - 7696784 subnetworks found for foot, totalMB:59504, usedMB:49376
2019-12-11 17:53:37.718 [main] INFO c.g.r.s.PrepareRoutingSubnetworks - optimize to remove subnetworks (34568457), unvisited-dead-end-nodes (10692578), maxEdges/node (26)
2019-12-11 19:51:34.897 [main] INFO c.g.reader.osm.GraphHopperOSM - edges: 303 384 329, nodes 226 055 746, there were 34 568 457 subnetworks. removed them => 5 742 554 less nodes
2019-12-11 19:57:03.009 [main] INFO c.g.storage.index.LocationIndexTree - location index created in 328.10974s, size:292 222 247, leafs:47 305 753, precision:300, depth:8, checksum:226055746, entries:[16, 16, 16, 16, 16, 16, 16, 4], entriesPerLeaf:6.177309
2019-12-11 19:57:05.166 [main] INFO c.g.r.ch.CHAlgoFactoryDecorator - 1/2 calling CH prepare.doWork for fastest|car|edge_based=false|u_turn_costs=-1 ... (totalMB:59504, usedMB:41231)
2019-12-11 19:57:05.167 [main] INFO c.g.r.ch.CHAlgoFactoryDecorator - 2/2 calling CH prepare.doWork for fastest|foot|edge_based=false|u_turn_costs=-1 ... (totalMB:59504, usedMB:41231)
2019-12-11 20:06:02.785 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 226 055 746, shortcuts: 0, updates: 0, checked-nodes: 0, t(total): 537.61, t(period): 535.30, t(lazy): 0.00, t(neighbor): 0.00, t(contr): 0.00, t(other) : 2.32, dijkstra-ratio: 38.14%, meanDegree: 1.00, dijkstras: 860 061 244, mem: 3449MB, totalMB:61232, usedMB:54488
2019-12-11 20:43:19.585 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 180 844 597, shortcuts: 2 988 818, updates: 1, checked-nodes: 45 211 149, t(total): 2774.42, t(period): 2599.43, t(lazy): 0.00, t(neighbor): 111.34, t(contr): 32.57, t(other) : 31.08, dijkstra-ratio: 73.37%, meanDegree: 1.00, dijkstras: 1 577 395 207, mem: 3449MB, totalMB:67216, usedMB:54984
2019-12-11 20:59:59.497 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 135 633 448, shortcuts: 23 380 558, updates: 2, checked-nodes: 90 422 298, t(total): 3774.33, t(period): 3032.33, t(lazy): 0.00, t(neighbor): 340.31, t(contr): 332.54, t(other) : 69.14, dijkstra-ratio: 69.75%, meanDegree: 0.00, dijkstras: 2 232 434 830, mem: 3449MB, totalMB:67216, usedMB:47712
2019-12-11 21:40:06.825 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 90 422 299, shortcuts: 49 906 123, updates: 3, checked-nodes: 135 633 447, t(total): 6181.66, t(period): 4753.03, t(lazy): 0.00, t(neighbor): 642.82, t(contr): 678.48, t(other) : 107.33, dijkstra-ratio: 75.06%, meanDegree: 1.87, dijkstras: 2 805 825 230, mem: 3449MB, totalMB:67216, usedMB:53464
2019-12-11 22:38:59.374 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 45 211 150, shortcuts: 104 809 751, updates: 4, checked-nodes: 180 844 596, t(total): 9714.20, t(period): 6481.69, t(lazy): 0.00, t(neighbor): 1492.08, t(contr): 1574.93, t(other) : 165.50, dijkstra-ratio: 79.15%, meanDegree: 2.43, dijkstras: 3 405 937 463, mem: 3449MB, totalMB:67216, usedMB:49635
2019-12-12 00:16:44.211 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 7 096 284, shortcuts: 166 667 695, updates: 5, checked-nodes: 226 055 745, t(total): 15579.04, t(period): 7353.65, t(lazy): 1537.67, t(neighbor): 3428.62, t(contr): 3024.73, t(other) : 234.38, dijkstra-ratio: 83.34%, meanDegree: 3.35, dijkstras: 4 246 369 624, mem: 3449MB, totalMB:67216, usedMB:45508
2019-12-12 01:48:11.472 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 0, shortcuts: 188 368 465, updates: 5, checked-nodes: 236 195 035, t(total): 21066.30, t(period): 7353.65, t(lazy): 3344.76, t(neighbor): 6202.33, t(contr): 3905.00, t(other) : 260.57, dijkstra-ratio: 86.39%, meanDegree: 0.64, dijkstras: 4 818 183 537, mem: 3449MB, totalMB:67216, usedMB:46303
2019-12-12 01:48:11.472 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - new shortcuts: 188 368 465, initSize:226 055 746, fastest|car, periodic:20, lazy:10, neighbor:20, t(total): 21066.30, t(period): 7353.65, t(lazy): 3344.76, t(neighbor): 6202.33, t(contr): 3905.00, t(other) : 260.57, dijkstra-ratio: 86.39%, lazy-overhead: 4%, totalMB:67216, usedMB:46304
2019-12-12 01:48:11.472 [fastest_car_node] INFO c.g.r.c.PrepareContractionHierarchies - took: 21066s, graph now - num edges: 303 384 329, num nodes: 226 055 746, num shortcuts: 188 368 465
2019-12-12 01:59:42.580 [fastest_foot_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 226 055 746, shortcuts: 0, updates: 0, checked-nodes: 0, t(total): 691.11, t(period): 689.50, t(lazy): 0.00, t(neighbor): 0.00, t(contr): 0.00, t(other) : 1.61, dijkstra-ratio: 41.65%, meanDegree: 1.00, dijkstras: 1 148 938 020, mem: 3449MB, totalMB:67216, usedMB:59960
2019-12-12 02:47:56.334 [fastest_foot_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 180 844 597, shortcuts: 276, updates: 1, checked-nodes: 45 211 149, t(total): 3584.86, t(period): 3436.35, t(lazy): 0.00, t(neighbor): 102.56, t(contr): 17.26, t(other) : 28.69, dijkstra-ratio: 76.46%, meanDegree: 1.00, dijkstras: 2 136 817 451, mem: 3449MB, totalMB:69024, usedMB:56248
2019-12-12 03:45:20.802 [fastest_foot_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 135 633 448, shortcuts: 27 165 051, updates: 2, checked-nodes: 90 422 298, t(total): 7029.33, t(period): 6021.79, t(lazy): 0.00, t(neighbor): 464.41, t(contr): 471.66, t(other) : 71.47, dijkstra-ratio: 80.62%, meanDegree: 2.21, dijkstras: 3 064 168 512, mem: 3449MB, totalMB:69024, usedMB:53944
2019-12-12 03:54:30.792 [fastest_foot_node] WARN com.graphhopper.storage.CHGraphImpl - Setting weights smaller than 0.001 is not allowed in CHGraphImpl#setWeight. You passed: 0.0 for the edge -1 nodeA 30.110038931147134,104.62920303559012 nodeB 30.110038931147134,104.62920303559012
2019-12-12 04:46:49.229 [fastest_foot_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 90 422 299, shortcuts: 67 581 769, updates: 3, checked-nodes: 135 633 447, t(total): 10717.76, t(period): 8450.67, t(lazy): 0.00, t(neighbor): 1027.85, t(contr): 1118.98, t(other) : 120.26, dijkstra-ratio: 82.44%, meanDegree: 2.70, dijkstras: 3 895 063 592, mem: 3449MB, totalMB:69024, usedMB:51584
2019-12-12 06:14:01.596 [fastest_foot_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 45 211 150, shortcuts: 138 617 376, updates: 4, checked-nodes: 180 844 596, t(total): 15950.12, t(period): 10988.81, t(lazy): 0.00, t(neighbor): 2385.68, t(contr): 2388.73, t(other) : 186.90, dijkstra-ratio: 84.34%, meanDegree: 3.24, dijkstras: 4 896 192 984, mem: 3449MB, totalMB:69024, usedMB:54685
2019-12-12 09:05:37.598 [fastest_foot_node] INFO c.g.r.c.PrepareContractionHierarchies - nodes: 9 400 522, shortcuts: 215 987 009, updates: 5, checked-nodes: 226 055 745, t(total): 26246.13, t(period): 13004.81, t(lazy): 2608.63, t(neighbor): 5906.20, t(contr): 4464.18, t(other) : 262.30, dijkstra-ratio: 87.48%, meanDegree: 4.48, dijkstras: 6 608 939 892, mem: 3449MB, totalMB:69024, usedMB:51817
my config.yml
graphhopper:
# OpenStreetMap input file
# datareader.file: some.pbf
##### Vehicles #####
# More options: foot,bike,bike2,mtb,racingbike,motorcycle (comma separated)
# bike2 takes elevation data into account (like up-hill is slower than down-hill) and requires enabling graph.elevation.provider below.
graph.flag_encoders: car,foot
# Enable turn restrictions for car or motorcycle.
# graph.flag_encoders: car|turn_costs=true
# Add additional information to every edge. Used for path details.
# If road_environment is added and elevation is enabled then also a tunnel and bridge interpolation is done, see #798.
# More options are: surface,max_width,max_height,max_weight,max_axle_load,max_length,hazmat,hazmat_tunnel,hazmat_water,toll,track_type
graph.encoded_values: road_class,road_class_link,road_environment,max_speed,road_access
##### Elevation #####
# To populate your graph with elevation data use SRTM, default is noop (no elevation)
# graph.elevation.provider: srtm
# default location for cache is /tmp/srtm
# graph.elevation.cache_dir: ./srtmprovider/
# If you have a slow disk or plenty of RAM change the default MMAP to:
# graph.elevation.dataaccess: RAM_STORE
#### Speed, hybrid and flexible mode ####
# By default the speed mode with the 'fastest' weighting is used. Internally a graph preparation via
# contraction hierarchies (CH) is done to speed routing up. This requires more RAM/disc space for holding the
# graph but less for every request. You can also setup multiple weightings, by providing a comma separated list.
# To enable finite u-turn costs use something like fastest|u_turn_costs=30, where 30 are the u-turn costs in seconds
# (given as integer). Note that since the u-turn costs are given in seconds the weighting you use should also
# calculate the weight in seconds. The u-turn costs will only be applied for edge_based, see below.
prepare.ch.weightings: fastest
# To enable turn-costs in speed mode (contraction hierarchies) edge-based graph traversal and a more elaborate
# pre-processing is required. Using this option you can either turn off the edge-based pre-processing (choose 'off'),
# use edge-based pre-processing for all encoders/vehicles with turn_costs=true (choose 'edge_or_node') or use node-based
# pre-processing for all encoders/vehicles and additional edge-based pre-processing for all encoders/vehicles with
# turn_costs=true (choose 'edge_and_node').
prepare.ch.edge_based: off
# Disable the speed mode. Should be used only with routing.max_visited_nodes or when the hybrid mode is enabled instead
# prepare.ch.weightings: no
# To make CH preparation faster for multiple flagEncoders you can increase the default threads if you have enough RAM.
# Change this setting only if you know what you are doing and if the default worked for you.
# prepare.ch.threads: 1
# The hybrid mode can be enabled with
# prepare.lm.weightings: fastest
# To tune the performance vs. memory usage for the hybrid mode use
# prepare.lm.landmarks: 16
# Make landmark preparation parallel if you have enough RAM. Change this only if you know what you are doing and if the default worked for you.
# prepare.lm.threads: 1
# avoid being stuck in a (oneway) subnetwork, see https://discuss.graphhopper.com/t/93
prepare.min_network_size: 200
prepare.min_one_way_network_size: 200
##### Routing #####
# You can define the maximum visited nodes when routing. This may result in not found connections if there is no
# connection between two points within the given visited nodes. The default is Integer.MAX_VALUE. Useful for flexibility mode
# routing.max_visited_nodes: 1000000
# If enabled, allows a user to run flexibility requests even if speed mode is enabled. Every request then has to include a hint ch.disable=true.
# Attention, non-CH route calculations take way more time and resources, compared to CH routing.
# A possible attacker might exploit this to slow down your service. Only enable it if you need it and with routing.maxVisitedNodes
# routing.ch.disabling_allowed: true
# If enabled, allows a user to run flexible mode requests even if the hybrid mode is enabled. Every such request then has to include a hint routing.lm.disable=true.
# routing.lm.disabling_allowed: true
# Control how many active landmarks are picked per default, this can improve query performance
# routing.lm.active_landmarks: 4
# You can limit the max distance between two consecutive waypoints of flexible routing requests to be less or equal
# the given distance in meter. Default is set to 1000km.
routing.non_ch.max_waypoint_distance: 1000000
##### Storage #####
# configure the memory access, use RAM_STORE for well equipped servers (default and recommended)
graph.dataaccess: RAM_STORE
# will write way names in the preferred language (language code as defined in ISO 639-1 or ISO 639-2):
# datareader.preferred_language: en
# Sort the graph after import to make requests roughly ~10% faster. Note that this requires significantly more RAM on import.
# graph.do_sort: true
##### Spatial Rules #####
# Spatial Rules require some configuration and only work with the DataFlagEncoder.
# Spatial Rules require you to provide Polygons in which the rules are enforced
# The line below contains the default location for these rules
# spatial_rules.location: core/files/spatialrules/countries.geo.json
# You can define the maximum BBox for which spatial rules are loaded.
# You might want to do this if you are only importing a small area and don't need rules for other countries.
# Having less rules, might result in a smaller graph. The line below contains the world-wide bounding box, uncomment and adapt to your need.
# spatial_rules.max_bbox: -180,180,-90,90
# Uncomment the following to point /maps to the source directory in the filesystem instead of
# the Java resource path. Helpful for development of the web client.
# Assumes that the web module is the working directory.
#
# assets:
# overrides:
# /maps: web/src/main/resources/assets/
# Dropwizard server configuration
server:
applicationConnectors:
- type: http
port: 8989
# for security reasons bind to localhost
bindHost: localhost
requestLog:
appenders: []
adminConnectors:
- type: http
port: 8990
bindHost: localhost
# See https://www.dropwizard.io/1.3.8/docs/manual/configuration.html#logging
logging:
appenders:
- type: file
timeZone: UTC
currentLogFilename: logs/graphhopper.log
logFormat: "%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"
archive: true
archivedLogFilenamePattern: ./logs/graphhopper-%d.log.gz
archivedFileCount: 30
neverBlock: true
- type: console
timeZone: UTC
logFormat: "%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"