Foot+car build stuck?

Hey all,

I’m still trying to build a car+foot index. I’m building from the latest github master branch. Today it’s not running out of memory (gave it 80g), but it does seem pretty stuck. The last output I saw from the builder was ~7 hours ago.

I’m wondering if I should give up on trying to build a combined index and instead build two separate ones and run them on separate boxes? Any advice on how to get this to work?

I’m running it with
JAVA_OPTS="-Xmx80g -Xms60g" ./graphhopper.sh -a web -i …/planet-191202.osm.pbf

My most recent output (it’s now 16:21 UTC on this box)

2019-12-11 17:51:46.623 [main] INFO  c.g.r.s.PrepareRoutingSubnetworks - foot.access|version=-1709808141|bits=1|index=1|shift=3|store_both_directions=true findComponents time:2370.195, size:1616251
2019-12-11 17:53:15.349 [main] INFO  c.g.r.s.PrepareRoutingSubnetworks - 7696784 subnetworks found for foot, totalMB:59504, usedMB:49376
2019-12-11 17:53:37.718 [main] INFO  c.g.r.s.PrepareRoutingSubnetworks - optimize to remove subnetworks (34568457), unvisited-dead-end-nodes (10692578), maxEdges/node (26)


2019-12-11 19:51:34.897 [main] INFO  c.g.reader.osm.GraphHopperOSM - edges: 303 384 329, nodes 226 055 746, there were 34 568 457 subnetworks. removed them => 5 742 554 less nodes
2019-12-11 19:57:03.009 [main] INFO  c.g.storage.index.LocationIndexTree - location index created in 328.10974s, size:292 222 247, leafs:47 305 753, precision:300, depth:8, checksum:226055746, entries:[16, 16, 16, 16, 16, 16, 16, 4], entriesPerLeaf:6.177309
2019-12-11 19:57:05.166 [main] INFO  c.g.r.ch.CHAlgoFactoryDecorator - 1/2 calling CH prepare.doWork for fastest|car|edge_based=false|u_turn_costs=-1 ... (totalMB:59504, usedMB:41231)
2019-12-11 19:57:05.167 [main] INFO  c.g.r.ch.CHAlgoFactoryDecorator - 2/2 calling CH prepare.doWork for fastest|foot|edge_based=false|u_turn_costs=-1 ... (totalMB:59504, usedMB:41231)
2019-12-11 20:06:02.785 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 226 055 746, shortcuts:          0, updates:  0, checked-nodes:          0, t(total): 537.61,  t(period): 535.30, t(lazy):   0.00, t(neighbor):   0.00, t(contr):   0.00, t(other) :   2.32, dijkstra-ratio:  38.14%, meanDegree: 1.00, dijkstras: 860 061 244, mem:     3449MB, totalMB:61232, usedMB:54488
2019-12-11 20:43:19.585 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 180 844 597, shortcuts:  2 988 818, updates:  1, checked-nodes: 45 211 149, t(total): 2774.42,  t(period): 2599.43, t(lazy):   0.00, t(neighbor): 111.34, t(contr):  32.57, t(other) :  31.08, dijkstra-ratio:  73.37%, meanDegree: 1.00, dijkstras: 1 577 395 207, mem:     3449MB, totalMB:67216, usedMB:54984
2019-12-11 20:59:59.497 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 135 633 448, shortcuts: 23 380 558, updates:  2, checked-nodes: 90 422 298, t(total): 3774.33,  t(period): 3032.33, t(lazy):   0.00, t(neighbor): 340.31, t(contr): 332.54, t(other) :  69.14, dijkstra-ratio:  69.75%, meanDegree: 0.00, dijkstras: 2 232 434 830, mem:     3449MB, totalMB:67216, usedMB:47712
2019-12-11 21:40:06.825 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 90 422 299, shortcuts: 49 906 123, updates:  3, checked-nodes: 135 633 447, t(total): 6181.66,  t(period): 4753.03, t(lazy):   0.00, t(neighbor): 642.82, t(contr): 678.48, t(other) : 107.33, dijkstra-ratio:  75.06%, meanDegree: 1.87, dijkstras: 2 805 825 230, mem:     3449MB, totalMB:67216, usedMB:53464
2019-12-11 22:38:59.374 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 45 211 150, shortcuts: 104 809 751, updates:  4, checked-nodes: 180 844 596, t(total): 9714.20,  t(period): 6481.69, t(lazy):   0.00, t(neighbor): 1492.08, t(contr): 1574.93, t(other) : 165.50, dijkstra-ratio:  79.15%, meanDegree: 2.43, dijkstras: 3 405 937 463, mem:     3449MB, totalMB:67216, usedMB:49635
2019-12-12 00:16:44.211 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes:  7 096 284, shortcuts: 166 667 695, updates:  5, checked-nodes: 226 055 745, t(total): 15579.04,  t(period): 7353.65, t(lazy): 1537.67, t(neighbor): 3428.62, t(contr): 3024.73, t(other) : 234.38, dijkstra-ratio:  83.34%, meanDegree: 3.35, dijkstras: 4 246 369 624, mem:     3449MB, totalMB:67216, usedMB:45508
2019-12-12 01:48:11.472 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes:          0, shortcuts: 188 368 465, updates:  5, checked-nodes: 236 195 035, t(total): 21066.30,  t(period): 7353.65, t(lazy): 3344.76, t(neighbor): 6202.33, t(contr): 3905.00, t(other) : 260.57, dijkstra-ratio:  86.39%, meanDegree: 0.64, dijkstras: 4 818 183 537, mem:     3449MB, totalMB:67216, usedMB:46303
2019-12-12 01:48:11.472 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - new shortcuts: 188 368 465, initSize:226 055 746, fastest|car, periodic:20, lazy:10, neighbor:20, t(total): 21066.30,  t(period): 7353.65, t(lazy): 3344.76, t(neighbor): 6202.33, t(contr): 3905.00, t(other) : 260.57, dijkstra-ratio:  86.39%, lazy-overhead: 4%, totalMB:67216, usedMB:46304
2019-12-12 01:48:11.472 [fastest_car_node] INFO  c.g.r.c.PrepareContractionHierarchies - took: 21066s, graph now - num edges: 303 384 329, num nodes: 226 055 746, num shortcuts: 188 368 465
2019-12-12 01:59:42.580 [fastest_foot_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 226 055 746, shortcuts:          0, updates:  0, checked-nodes:          0, t(total): 691.11,  t(period): 689.50, t(lazy):   0.00, t(neighbor):   0.00, t(contr):   0.00, t(other) :   1.61, dijkstra-ratio:  41.65%, meanDegree: 1.00, dijkstras: 1 148 938 020, mem:     3449MB, totalMB:67216, usedMB:59960
2019-12-12 02:47:56.334 [fastest_foot_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 180 844 597, shortcuts:        276, updates:  1, checked-nodes: 45 211 149, t(total): 3584.86,  t(period): 3436.35, t(lazy):   0.00, t(neighbor): 102.56, t(contr):  17.26, t(other) :  28.69, dijkstra-ratio:  76.46%, meanDegree: 1.00, dijkstras: 2 136 817 451, mem:     3449MB, totalMB:69024, usedMB:56248
2019-12-12 03:45:20.802 [fastest_foot_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 135 633 448, shortcuts: 27 165 051, updates:  2, checked-nodes: 90 422 298, t(total): 7029.33,  t(period): 6021.79, t(lazy):   0.00, t(neighbor): 464.41, t(contr): 471.66, t(other) :  71.47, dijkstra-ratio:  80.62%, meanDegree: 2.21, dijkstras: 3 064 168 512, mem:     3449MB, totalMB:69024, usedMB:53944
2019-12-12 03:54:30.792 [fastest_foot_node] WARN  com.graphhopper.storage.CHGraphImpl - Setting weights smaller than 0.001 is not allowed in CHGraphImpl#setWeight. You passed: 0.0 for the edge -1 nodeA 30.110038931147134,104.62920303559012 nodeB 30.110038931147134,104.62920303559012
2019-12-12 04:46:49.229 [fastest_foot_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 90 422 299, shortcuts: 67 581 769, updates:  3, checked-nodes: 135 633 447, t(total): 10717.76,  t(period): 8450.67, t(lazy):   0.00, t(neighbor): 1027.85, t(contr): 1118.98, t(other) : 120.26, dijkstra-ratio:  82.44%, meanDegree: 2.70, dijkstras: 3 895 063 592, mem:     3449MB, totalMB:69024, usedMB:51584
2019-12-12 06:14:01.596 [fastest_foot_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes: 45 211 150, shortcuts: 138 617 376, updates:  4, checked-nodes: 180 844 596, t(total): 15950.12,  t(period): 10988.81, t(lazy):   0.00, t(neighbor): 2385.68, t(contr): 2388.73, t(other) : 186.90, dijkstra-ratio:  84.34%, meanDegree: 3.24, dijkstras: 4 896 192 984, mem:     3449MB, totalMB:69024, usedMB:54685
2019-12-12 09:05:37.598 [fastest_foot_node] INFO  c.g.r.c.PrepareContractionHierarchies - nodes:  9 400 522, shortcuts: 215 987 009, updates:  5, checked-nodes: 226 055 745, t(total): 26246.13,  t(period): 13004.81, t(lazy): 2608.63, t(neighbor): 5906.20, t(contr): 4464.18, t(other) : 262.30, dijkstra-ratio:  87.48%, meanDegree: 4.48, dijkstras: 6 608 939 892, mem:     3449MB, totalMB:69024, usedMB:51817

my config.yml

graphhopper:

  # OpenStreetMap input file
  # datareader.file: some.pbf

  ##### Vehicles #####


  # More options: foot,bike,bike2,mtb,racingbike,motorcycle (comma separated)
  # bike2 takes elevation data into account (like up-hill is slower than down-hill) and requires enabling graph.elevation.provider below.
  graph.flag_encoders: car,foot

  # Enable turn restrictions for car or motorcycle.
  # graph.flag_encoders: car|turn_costs=true

  # Add additional information to every edge. Used for path details.
  # If road_environment is added and elevation is enabled then also a tunnel and bridge interpolation is done, see #798.
  # More options are: surface,max_width,max_height,max_weight,max_axle_load,max_length,hazmat,hazmat_tunnel,hazmat_water,toll,track_type
  graph.encoded_values: road_class,road_class_link,road_environment,max_speed,road_access

  ##### Elevation #####


  # To populate your graph with elevation data use SRTM, default is noop (no elevation)
  # graph.elevation.provider: srtm


  # default location for cache is /tmp/srtm
  # graph.elevation.cache_dir: ./srtmprovider/


  # If you have a slow disk or plenty of RAM change the default MMAP to:
  # graph.elevation.dataaccess: RAM_STORE



  #### Speed, hybrid and flexible mode ####


  # By default the speed mode with the 'fastest' weighting is used. Internally a graph preparation via
  # contraction hierarchies (CH) is done to speed routing up. This requires more RAM/disc space for holding the
  # graph but less for every request. You can also setup multiple weightings, by providing a comma separated list.
  # To enable finite u-turn costs use something like fastest|u_turn_costs=30, where 30 are the u-turn costs in seconds
  # (given as integer). Note that since the u-turn costs are given in seconds the weighting you use should also
  # calculate the weight in seconds. The u-turn costs will only be applied for edge_based, see below.
  prepare.ch.weightings: fastest

  # To enable turn-costs in speed mode (contraction hierarchies) edge-based graph traversal and a more elaborate
  # pre-processing is required. Using this option you can either turn off the edge-based pre-processing (choose 'off'),
  # use edge-based pre-processing for all encoders/vehicles with turn_costs=true (choose 'edge_or_node') or use node-based
  # pre-processing for all encoders/vehicles and additional edge-based pre-processing for all encoders/vehicles with
  # turn_costs=true (choose 'edge_and_node').
  prepare.ch.edge_based: off


  # Disable the speed mode. Should be used only with routing.max_visited_nodes or when the hybrid mode is enabled instead
  # prepare.ch.weightings: no


  # To make CH preparation faster for multiple flagEncoders you can increase the default threads if you have enough RAM.
  # Change this setting only if you know what you are doing and if the default worked for you.
  # prepare.ch.threads: 1


  # The hybrid mode can be enabled with
  # prepare.lm.weightings: fastest

  # To tune the performance vs. memory usage for the hybrid mode use
  # prepare.lm.landmarks: 16

  # Make landmark preparation parallel if you have enough RAM. Change this only if you know what you are doing and if the default worked for you.
  # prepare.lm.threads: 1


  # avoid being stuck in a (oneway) subnetwork, see https://discuss.graphhopper.com/t/93
  prepare.min_network_size: 200
  prepare.min_one_way_network_size: 200



  ##### Routing #####


  # You can define the maximum visited nodes when routing. This may result in not found connections if there is no
  # connection between two points within the given visited nodes. The default is Integer.MAX_VALUE. Useful for flexibility mode
  # routing.max_visited_nodes: 1000000


  # If enabled, allows a user to run flexibility requests even if speed mode is enabled. Every request then has to include a hint ch.disable=true.
  # Attention, non-CH route calculations take way more time and resources, compared to CH routing.
  # A possible attacker might exploit this to slow down your service. Only enable it if you need it and with routing.maxVisitedNodes
  # routing.ch.disabling_allowed: true


  # If enabled, allows a user to run flexible mode requests even if the hybrid mode is enabled. Every such request then has to include a hint routing.lm.disable=true.
  # routing.lm.disabling_allowed: true

  # Control how many active landmarks are picked per default, this can improve query performance
  # routing.lm.active_landmarks: 4


  # You can limit the max distance between two consecutive waypoints of flexible routing requests to be less or equal
  # the given distance in meter. Default is set to 1000km.
  routing.non_ch.max_waypoint_distance: 1000000


  ##### Storage #####


  # configure the memory access, use RAM_STORE for well equipped servers (default and recommended)
  graph.dataaccess: RAM_STORE


  # will write way names in the preferred language (language code as defined in ISO 639-1 or ISO 639-2):
  # datareader.preferred_language: en


  # Sort the graph after import to make requests roughly ~10% faster. Note that this requires significantly more RAM on import.
  # graph.do_sort: true



  ##### Spatial Rules #####
  # Spatial Rules require some configuration and only work with the DataFlagEncoder.


  # Spatial Rules require you to provide Polygons in which the rules are enforced
  # The line below contains the default location for these rules
  # spatial_rules.location: core/files/spatialrules/countries.geo.json

  # You can define the maximum BBox for which spatial rules are loaded.
  # You might want to do this if you are only importing a small area and don't need rules for other countries.
  # Having less rules, might result in a smaller graph. The line below contains the world-wide bounding box, uncomment and adapt to your need.
  # spatial_rules.max_bbox: -180,180,-90,90


# Uncomment the following to point /maps to the source directory in the filesystem instead of
# the Java resource path. Helpful for development of the web client.
# Assumes that the web module is the working directory.
#
# assets:
#  overrides:
#    /maps: web/src/main/resources/assets/

# Dropwizard server configuration
server:
  applicationConnectors:
  - type: http
port: 8989
# for security reasons bind to localhost
bindHost: localhost
  requestLog:
  appenders: []
  adminConnectors:
  - type: http
port: 8990
bindHost: localhost
# See https://www.dropwizard.io/1.3.8/docs/manual/configuration.html#logging
logging:
  appenders:
  - type: file
timeZone: UTC
currentLogFilename: logs/graphhopper.log
logFormat: "%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"
archive: true
archivedLogFilenamePattern: ./logs/graphhopper-%d.log.gz
archivedFileCount: 30
neverBlock: true
  - type: console
timeZone: UTC
logFormat: "%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"

watched pot, meet not boiling.

This completed successfully like an hour later.

1 Like