Faster request /route-custom

Hi all,

I was wondering if it is possible to get a faster response for requesting a route via the /route-custom endpoint. I am currently using graphhopper version 3.0.
My config.yml:

graphhopper:
  datareader.file: planet-latest.osm.pbf # OSM input file
  graph.location: graph-gh
  profiles:
    - name: motorcycle
      vehicle: motorcycle
      weighting: custom
      custom_model_file: empty
    - name: bike
      vehicle: bike
      weighting: custom
      custom_model_file: empty
    - name: foot
      vehicle: foot
      weighting: custom
      custom_model_file: empty
  graph.flag_encoders: foot,bike,motorcycle|turn_costs=true
  graph.encoded_values: toll, surface
  graph.elevation.provider: srtm
# PRODUCTION
  graph.elevation.cache_dir: /data/srtm/
  graph.dataaccess: RAM_STORE

server:
  application_connectors:
    - type: http
      port: 8989
      # DEVELOPMENT
#      bind_host: localhost
  admin_connectors:
    - type: http
      port: 8990
      # DEVELOPMENT
#      bind_host: localhost

# PRODUCTION
logging:
  appenders:
    - type: file
      time_zone: UTC+1
      current_log_filename: /data/logs/graphhopper.log
      log_format: "%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"
      archive: true
      archived_log_filename_pattern: /data/logs/graphhopper-%d.log.gz
      archived_file_count: 7
      never_block: true
    - type: console
      time_zone: UTC+1
      log_format: "%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"

My Java OPTS:

JAVA_OPTS="-Xmx50g -Xms40g"

A route with 4 points through the Netherlands, Spain, Croatia and Germany takes around 1 minute to calculate.
The post data of the request:

{
    "priority" : null,
    "profile" : "motorcycle",
    "points": [
        [
            5.060888546692342,
            52.02194174355072
        ],
        [
            -5.0900140325881695,
            37.58033672669711
        ],
        [
            16.06932114914624,
            44.26399668840436
        ],
        [
            12.698546113905273,
            52.35620034883152
        ]
    ],
    "ch_disable": true,
    "points_encoded": false,
    "elevation": true
}

The result:

{
    "hints": {
        "visited_nodes.sum": 28756208,
        "visited_nodes.average": 9585403.0
    },
    "info": {
        "copyrights": [
            "GraphHopper",
            "OpenStreetMap contributors"
        ],
        "took": 46776
    },
    "paths": [
        {...........}

image

Can the calculation be made faster by adjusting certain graphhopper java process settings? What do you recommend to get a faster response time?

I look forward to the response.

Greetings,
kit cat

Yes, you can either enable LM or CH. Something like:

  profiles_ch:
    - profile: motorcycle
    - profile: bike
    - profile: foot

or the same using profiles_lm. This will speed up the routing by orders of magnitude, but you will need more memory. You should try these settings with a small OSM extract to determine your hardware requirements. Since you are using turn_costs=true for motorcycle you might prefer LM instead of CH (which is especially expensive when turn costs are enabled). Also better use a recent GH version to benefit from all performance improvements (most recent official version is 5.3, but if you feel experimental you could also try latest master which since Stop removing elements from the priority queue, speeds up flexible algorithms by easbar · Pull Request #2571 · graphhopper/graphhopper · GitHub also offers faster query times without CH/LM).

Another thing you could try to speed up the queries (by a bit) is running separate GH instances for motorcycle and bike/foot. Motorcycles aren’t allowed on many roads where bike and foot can be used (and vice versa) so the resulting graph will be smaller which results in faster queries. But nowhere near as much faster as when you enable CH/LM.

1 Like

Thank you so much! profiles_lm really did the trick. And indeed, upgrading to the newer version of GH is the next step on my todo list :slight_smile:

1 Like

@easbar hi hi, I have another question related to the conversation above.

I tried to start the process on an instance that uses the planet-latest

Unfortunately it throws an error:
Caused by: java.util.concurrent.ExecutionException: com.carrotsearch.hppc.BufferAllocationException: Not enough memory to allocate buffers for rehashing: 67,108,864 -> 134,217,728

My instance specifications are:
AWS r5.4xlarge 16 vCPU en 128Gb memory

My JAVA OPTS is:
JAVA_OPTS="-Xmx100g -Xms60g"

Below the log:
graphhopper-2.log (145.4 KB)

Do you recommend increasing the capacity of the instance or is there another trick to solve this problem?

This should be possible on a 128GB machine. You could try increasing Xmx further (maybe 110g?). However, I think a better solution is setting prepare.lm.threads: 2, which will only run two LM preparations at a time (or if that does not work either, try setting it to one of course). So it will take longer but also the peak memory requirement will be lower. Also for large imports like this you are better off using the import command, rather than the server command. Internally this will call GraphHopper#importAndClose rather than GraphHopper#importOrLoad which means resources that are no longer needed for the import will be freed as soon as possible. Once the import is done you can use the server command to actually start the server using the imported data.

Also take a look here: graphhopper/deploy.md at master · graphhopper/graphhopper · GitHub
and here: update deployment guide by karussell · Pull Request #2576 · graphhopper/graphhopper · GitHub

1 Like

Thank you! I had not informed myself well enough about the different ways of importing. I’ll try this in combination with the prepare.lm.threads :grin:

Hi, it’s me again :sweat_smile:,

So the prepare.lm.threads:2 looked like they worked, but I’m facing another issue now.

022-05-22 07:43:50.734 [foot] INFO  c.g.routing.lm.LMPreparationHandler - LM foot finished totalMB:189879, usedMB:124370
2022-05-22 07:43:50.734 [main] INFO  c.g.routing.lm.LMPreparationHandler - Finished LM preparation, totalMB:189879, usedMB:124643
2022-05-22 07:43:50.734 [main] INFO  c.g.reader.osm.GraphHopperOSM - flushing graph foot,bike,motorcycle|RAM_STORE|3D|turn_cost|5,17,4,3,5, details:edges:412 773 192(15747MB), nodes:318 760 641(6080MB), name:(753MB), geo:3 384 530 257(12911MB), bounds:-180.0,180.0,-85.05112867916662,82.5254022894517,-929.65,6837.0, totalMB:189879, usedMB:124643)
2022-05-22 07:46:46.779 [main] ERROR io.dropwizard.cli.ServerCommand - Unable to start server, shutting down
java.lang.RuntimeException: Couldn't store bytes to /data/map/planet-latest.osm-gh/string_index_vals
	at com.graphhopper.storage.RAMDataAccess.flush(RAMDataAccess.java:181)
	at com.graphhopper.search.StringIndex.flush(StringIndex.java:322)
	at com.graphhopper.storage.BaseGraph.flush(BaseGraph.java:456)
	at com.graphhopper.storage.GraphHopperStorage.flush(GraphHopperStorage.java:291)
	at com.graphhopper.GraphHopper.flush(GraphHopper.java:1142)
	at com.graphhopper.GraphHopper.process(GraphHopper.java:665)
	at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:627)
	at com.graphhopper.http.GraphHopperManaged.start(GraphHopperManaged.java:124)
	at io.dropwizard.lifecycle.JettyManaged.doStart(JettyManaged.java:27)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)
	at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:45)
	at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
	at io.dropwizard.cli.Cli.run(Cli.java:78)
	at io.dropwizard.Application.run(Application.java:94)
	at com.graphhopper.http.GraphHopperApplication.main(GraphHopperApplication.java:36)
Caused by: java.io.IOException: No space left on device
	at java.io.RandomAccessFile.writeBytes(Native Method)
	at java.io.RandomAccessFile.write(RandomAccessFile.java:512)
	at com.graphhopper.storage.RAMDataAccess.flush(RAMDataAccess.java:175)
	... 21 common frames omitted

log:
graphhopper-2022-05-22.log (78.1 KB)

Any tips on how I can solve this? I already tried changing the Xmx settings

Thanks,

Well, No space left on device sounds like your hard disk is full.

Hi, I’m almost there…

So I’ve successfully built GH on an AWS r5.4xlarge server 16 vCPU and 128Gb memory.
After building I’ve tried changing the server back to an r5.2xlarge 8 vcpu / 64Gi RAM to start running.
Unfortunately, this immediately gives a java heap space error when starting up.

2022-05-24 11:49:55.536 [main] INFO  o.e.jetty.setuid.SetUIDListener - Opened application@6ea94d6a{HTTP/1.1, (http/1.1)}{0.0.0.0:8989}
2022-05-24 11:49:55.537 [main] INFO  o.e.jetty.setuid.SetUIDListener - Opened admin@28486680{HTTP/1.1, (http/1.1)}{0.0.0.0:8990}
2022-05-24 11:49:55.538 [main] INFO  org.eclipse.jetty.server.Server - jetty-9.4.35.v20201120; built: 2020-11-20T21:17:03.964Z; git: bdc54f03a5e0a7e280fab27f55c3c75ee8da89fb; jvm 1.8.0_312-b07
2022-05-24 11:52:30.944 [main] INFO  c.g.routing.lm.LMPreparationHandler - Creating LM preparations, totalMB:58112, usedMB:37725
2022-05-24 11:52:31.027 [main] INFO  c.g.r.u.s.SpatialRuleLookupBuilder - Created the SpatialRuleLookup with the following rules: [SpatialRule [getId()=ru, getPriority()=100], SpatialRule [getId()=tr, getPriority()=100], SpatialRule [getId()=ir, getPriority()=100], SpatialRule [getId()=africa, getPriority()=100], SpatialRule [getId()=eu, getPriority()=100], SpatialRule [getId()=south-east asia, getPriority()=100]]
2022-05-24 11:54:15.942 [main] ERROR io.dropwizard.cli.ServerCommand - Unable to start server, shutting down
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
	at com.graphhopper.routing.lm.LMPreparationHandler.loadOrDoWork(LMPreparationHandler.java:220)
	at com.graphhopper.GraphHopper.loadOrPrepareLM(GraphHopper.java:1101)
	at com.graphhopper.reader.osm.GraphHopperOSM.loadOrPrepareLM(GraphHopperOSM.java:99)
	at com.graphhopper.GraphHopper.postProcessing(GraphHopper.java:940)
	at com.graphhopper.GraphHopper.load(GraphHopper.java:792)
	at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:625)
	at com.graphhopper.http.GraphHopperManaged.start(GraphHopperManaged.java:124)
	at io.dropwizard.lifecycle.JettyManaged.doStart(JettyManaged.java:27)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)
	at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:45)
	at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
	at io.dropwizard.cli.Cli.run(Cli.java:78)
	at io.dropwizard.Application.run(Application.java:94)
	at com.graphhopper.http.GraphHopperApplication.main(GraphHopperApplication.java:36)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.graphhopper.routing.lm.LMPreparationHandler.loadOrDoWork(LMPreparationHandler.java:216)
	... 20 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.graphhopper.storage.RAMDataAccess.loadExisting(RAMDataAccess.java:142)
	at com.graphhopper.routing.lm.LandmarkStorage.loadExisting(LandmarkStorage.java:676)
	at com.graphhopper.routing.lm.PrepareLandmarks.loadExisting(PrepareLandmarks.java:112)
	at com.graphhopper.routing.lm.LMPreparationHandler.lambda$loadOrDoWork$0(LMPreparationHandler.java:197)
	at com.graphhopper.routing.lm.LMPreparationHandler$$Lambda$174/1591954203.run(Unknown Source)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

with JAVA_OPTS

JAVA_OPTS="-Xmx60g -Xms40g

Are there any other possible settings to make it start on a r5.2xlarge 8 vcpu / 64Gi RAM?
Or is upgrading to a bigger server the only solution?

Many thanks,

How large is the graphhopper folder?

Also have a look into the updated deployment guide that @easbar linked.

image (23)

172GB

You only need the planet-latest.osm-gh folder, so 95G. The rest is mainly elevation data /srtm, 14G you no longer need (unless you run a new import and want to prevent downloading it again) and probably the OSM file itself in /map, which you also no longer need unless you run the import again.

Still 95G on a 64G machine won’t work. Unless you do not load all profiles at once, or you use MMAP. Unfortunately, this is still not really mentioned in deploy.md. It says we can use MMAP for the import but need to accept a much longer import time. But another use-case might be the present one: We can also run the import on a larger machine (without MMAP) and then later load the graph using MMAP on a smaller one. @karussell ?

We can also run the import on a larger machine (without MMAP) and then later load the graph using MMAP on a smaller one. @karussell ?

Yes :slight_smile:

When using MMAP I highly recommend to still load as much as possible into memory using the options introduced here: DataAccess config for type and load by karussell · Pull Request #2440 · graphhopper/graphhopper · GitHub

btw: I wonder why the topic is about /route-custom? I recommend to use the latest version @kitcat

1 Like

Sorry for my late response. I totally agree, will update GH as soon as it is possible.

Powered by Discourse