Hey all,
Thanks again to the super responsive folks on this board.
I’m building planet.osm GH indexes with car|turn_costs, bike2 and foot. On an ec2 m5d.8xlarge (32 vCPU, 128 “elastic compute units”, 128gb RAM, 2 x 600 NVMe SSD), I give the GH build 80gb of ram and run everything off the SSDs, and a full world import takes about 24 hours. Is that a reasonable build time? Is there anything else I could be tuning to speed it up?
I tried to set “prepare.ch.threads=4” which appeared to deadlock the build into an out of memory state. That’s the only major optimization I’ve tried. Experimenting with those variables is a bit painful because it takes so long to figure out if they’re going to work.
I get that building three routing graphs for the entire planet is a ton of work. If 24 hours is what it takes, so be it. Just wondering if it can get any faster.
Here’s my config.yml
graphhopper:
# OpenStreetMap input file
# datareader.file: some.pbf
##### Vehicles #####
# More options: foot,bike,bike2,mtb,racingbike,motorcycle (comma separated)
# bike2 takes elevation data into account (like up-hill is slower than down-hill) and requires enabling graph.elevation.provider below.
graph.flag_encoders: car|turn_costs=true,foot,bike2
# Enable turn restrictions for car or motorcycle.
# graph.flag_encoders: car|turn_costs=true
# Add additional information to every edge. Used for path details.
# If road_environment is added and elevation is enabled then also a tunnel and bridge interpolation is done, see #798.
# More options are: surface,max_width,max_height,max_weight,max_axle_load,max_length,hazmat,hazmat_tunnel,hazmat_water,toll,track_type
graph.encoded_values: road_class,road_class_link,road_environment,max_speed,road_access
##### Elevation #####
# To populate your graph with elevation data use SRTM, default is noop (no elevation)
graph.elevation.provider: srtm
# default location for cache is /tmp/srtm
graph.elevation.cache_dir: ./srtmprovider/
# If you have a slow disk or plenty of RAM change the default MMAP to:
# graph.elevation.dataaccess: RAM_STORE
#### Speed, hybrid and flexible mode ####
# By default the speed mode with the 'fastest' weighting is used. Internally a graph preparation via
# contraction hierarchies (CH) is done to speed routing up. This requires more RAM/disc space for holding the
# graph but less for every request. You can also setup multiple weightings, by providing a comma separated list.
# To enable finite u-turn costs use something like fastest|u_turn_costs=30, where 30 are the u-turn costs in seconds
# (given as integer). Note that since the u-turn costs are given in seconds the weighting you use should also
# calculate the weight in seconds. The u-turn costs will only be applied for edge_based, see below.
prepare.ch.weightings: fastest
# To enable turn-costs in speed mode (contraction hierarchies) edge-based graph traversal and a more elaborate
# pre-processing is required. Using this option you can either turn off the edge-based pre-processing (choose 'off'),
# use edge-based pre-processing for all encoders/vehicles with turn_costs=true (choose 'edge_or_node') or use node-based
# pre-processing for all encoders/vehicles and additional edge-based pre-processing for all encoders/vehicles with
# prepare.ch.turn_costs: edge_and_node
prepare.ch.edge_based: edge_and_node
# Disable the speed mode. Should be used only with routing.max_visited_nodes or when the hybrid mode is enabled instead
# prepare.ch.weightings: no
# To make CH preparation faster for multiple flagEncoders you can increase the default threads if you have enough RAM.
# Change this setting only if you know what you are doing and if the default worked for you.
# prepare.ch.threads: 1
# The hybrid mode can be enabled with
# prepare.lm.weightings: fastest
# To tune the performance vs. memory usage for the hybrid mode use
# prepare.lm.landmarks: 16
# Make landmark preparation parallel if you have enough RAM. Change this only if you know what you are doing and if the default worked for you.
# prepare.lm.threads: 1
# avoid being stuck in a (oneway) subnetwork, see https://discuss.graphhopper.com/t/93
prepare.min_network_size: 200
prepare.min_one_way_network_size: 200
##### Routing #####
# You can define the maximum visited nodes when routing. This may result in not found connections if there is no
# connection between two points within the given visited nodes. The default is Integer.MAX_VALUE. Useful for flexibility mode
# routing.max_visited_nodes: 1000000
# If enabled, allows a user to run flexibility requests even if speed mode is enabled. Every request then has to include a hint ch.disable=true.
# Attention, non-CH route calculations take way more time and resources, compared to CH routing.
# A possible attacker might exploit this to slow down your service. Only enable it if you need it and with routing.maxVisitedNodes
# routing.ch.disabling_allowed: true
# If enabled, allows a user to run flexible mode requests even if the hybrid mode is enabled. Every such request then has to include a hint routing.lm.disable=true.
# routing.lm.disabling_allowed: true
# Control how many active landmarks are picked per default, this can improve query performance
# routing.lm.active_landmarks: 4
# You can limit the max distance between two consecutive waypoints of flexible routing requests to be less or equal
# the given distance in meter. Default is set to 1000km.
routing.non_ch.max_waypoint_distance: 1000000
##### Storage #####
# configure the memory access, use RAM_STORE for well equipped servers (default and recommended)
graph.dataaccess: RAM_STORE
# will write way names in the preferred language (language code as defined in ISO 639-1 or ISO 639-2):
# datareader.preferred_language: en
# Sort the graph after import to make requests roughly ~10% faster. Note that this requires significantly more RAM on import.
# graph.do_sort: true
##### Spatial Rules #####
# Spatial Rules require some configuration and only work with the DataFlagEncoder.
# Spatial Rules require you to provide Polygons in which the rules are enforced
# The line below contains the default location for these rules
# spatial_rules.location: core/files/spatialrules/countries.geo.json
# You can define the maximum BBox for which spatial rules are loaded.
# You might want to do this if you are only importing a small area and don't need rules for other countries.
# Having less rules, might result in a smaller graph. The line below contains the world-wide bounding box, uncomment and adapt to your need.
# spatial_rules.max_bbox: -180,180,-90,90
# Uncomment the following to point /maps to the source directory in the filesystem instead of
# the Java resource path. Helpful for development of the web client.
# Assumes that the web module is the working directory.
#
# assets:
# overrides:
# /maps: web/src/main/resources/assets/
# Dropwizard server configuration
server:
applicationConnectors:
- type: http
port: 8989
# for security reasons bind to localhost
bindHost: 0.0.0.0
requestLog:
appenders: []
adminConnectors:
- type: http
port: 8990
bindHost: 0.0.0.0
logging:
level: INFO
loggers:
"com.graphhopper.resources": WARN