I am trying to make an import of the whole world since a couple of weeks and it keeps on failing and failing and failing.
My machine is I guess very strong. We have 295GB(!!) of RAM but still I always end up getting killed. Most likely due to memory failures. All I see right before it crashes, is that memory consumption is topped out.
Now I am wondering what to do. Everything works fine if I don’t do CH. I can then get away without MMAP and everything imports in only a couple of hours. But as routing requests are rather slow that way I would like to enable CH. I have 5 vehicles to calculate.
I start everything up using first sysctl -w vm.max_map_count=500000 and then also -Xmx275g -Xms200g
Also… I tried to startup with CH disabled and MMAP disabled. That works fast I have like an savepoint. I then quite it and start it again with CH enabled and also MMAP enabled. Still… it will go OOM after around 4 days.
We are also still using version 6 of Graphhopper. Were there many changes going to version 7 regarding memory consumption?
Any help would be greatly appreciated, as a new try, with new parameter will then take 4 days again until it fails.
This is how memory consumption looks like when doing the import. There are some memory-releases between the importing of the profiles. But I’d say it keeps growing. Also the time is increasing quite a bit.
I will now try to create only one CH at a time and hope that that way I have more “safe-points” so that I don’t always loose 4 days of calculation time.
And what kind ouf ‘out of memory’ errors do you see? In some cases you might run out of JVM heap memory and get a Java OutOfMemory exception, but in others the JVM might be killed by the OS (the OOM killer, you need to check your system logs for this). The JVM uses quite a bit of memory outside the heap, especially for the garbage collector. So in some cases you might even use a too high value for the Java heap space such that there isn’t enough memory left for the off-heap stuff.
I will now try to create only one CH at a time and hope that that way I have more “safe-points” so that I don’t always loose 4 days of calculation time.
You can use the prepare.ch.threads parameter to control how many CHs are calculated simultaneously and you can add one at a time. I.e. you can start with the non-CH import: Simply setup your profiles and run the import with an empty profiles_ch section, and then add one profile at a time to profiles_ch. When you run the import GraphHopper will check which CHs were prepared already and only calculate those that do not exist yet.
Were there many changes going to version 7 regarding memory consumption?
I don’t think so.
Also note that CH for profiles with turn costs enabled takes much longer and more memory. I’m not sure if this is an option for you, but turning off turn costs will speed up the import a lot.
