Process stuck at 100% - Java heap space OutOfMemory

Open Source Routing Engine
1

Hi,

I just experienced the following exception on my GH setup. The Server has some RAM left for requests and I handle CH and Non-CH requests. This seems to be related to https://github.com/graphhopper/graphhopper/issues/304

But I did not need to use pkill -9, a regular kill was enough.

Max visited nodes is currently set quite high:
routing.max_visited_nodes = 15000000

I haven’t tuned GC.

(GHErrorHandler.java:44) ERROR: Java heap space, via:http://localhost:8989/route, java.lang.OutOfMemoryError: Java heap space at gnu.trove.map.hash.TIntObjectHashMap.rehash(TIntObjectHashMap.java:157) at 
gnu.trove.impl.hash.THash.postInsertHook(THash.java:388) at 
gnu.trove.map.hash.TIntObjectHashMap.doPut(TIntObjectHashMap.java:248) at 
gnu.trove.map.hash.TIntObjectHashMap.put(TIntObjectHashMap.java:222) at 
com.graphhopper.routing.DijkstraBidirectionRef.fillEdges(DijkstraBidirectionRef.java:204) at 
com.graphhopper.routing.DijkstraBidirectionRef.fillEdgesFrom(DijkstraBidirectionRef.java:155) at 
com.graphhopper.routing.AbstractBidirAlgo.runAlgo(AbstractBidirAlgo.java:72) at 
com.graphhopper.routing.AbstractBidirAlgo.calcPath(AbstractBidirAlgo.java:63) at 
com.graphhopper.routing.AbstractRoutingAlgorithm.calcPaths(AbstractRoutingAlgorithm.java:120) at 
com.graphhopper.routing.template.ViaRoutingTemplate.calcPaths(ViaRoutingTemplate.java:111) at 
com.graphhopper.GraphHopper.calcPaths(GraphHopper.java:1201) at 
com.graphhopper.GraphHopper.route(GraphHopper.java:1065) at 
com.graphhopper.http.GraphHopperServlet.doGet(GraphHopperServlet.java:149) at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) at 
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) at 
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) at 
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at 
com.graphhopper.http.IPFilter.doFilter(IPFilter.java:63) at 
com.graphhopper.http.CORSFilter.doFilter(CORSFilter.java:38) at 
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at 
com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at 
com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at 
com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at 
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
2

15mio nodes is indeed not small so multiple non-CH queries could indeed currently lead to this behaviour. If the last queries were non-CH queries, can you find out the rough distance?

To investigate this further (when this happens again if it is a GC or another issue) can you attach

-XX:+HeapDumpOnOutOfMemoryError -Xloggc:logs/your.log -XX:+PrintGCDateStamps

Be aware that HeapDumpOnOutOfMemoryError produces very big files, so enough disc space is required (at least size of your RAM but if multiple OOMs are thrown you need more).

To improve GC behaviour you can try a different GC -XX:+UseG1GC see the deployment guide

3

Thanks, I will reduce the max_visited_nodes and added the GC options. I will report if this happens again.

Yes, but there was nothing really special, I’d guess. I just checked the last 10 non-ch requests. All were below 600km (but with waypoints in between) and should have been finished at the time the exception occurred.

4

Please verify (via executing them again) that all of them are working. Maybe some query was stuck in a subnetwork?

5

I did now for the last 15 non-ch requests, and all executed well, but the system is not under high load right now.