I am trying to make an import of the whole world since a couple of weeks and it keeps on failing and failing and failing.
My machine is I guess very strong. We have 295GB(!!) of RAM but still I always end up getting killed. Most likely due to memory failures. All I see right before it crashes, is that memory consumption is topped out.
Now I am wondering what to do. Everything works fine if I don’t do CH. I can then get away without MMAP and everything imports in only a couple of hours. But as routing requests are rather slow that way I would like to enable CH. I have 5 vehicles to calculate.
I start everything up using first sysctl -w vm.max_map_count=500000
and then also -Xmx275g -Xms200g
Also… I tried to startup with CH disabled and MMAP disabled. That works fast I have like an savepoint. I then quite it and start it again with CH enabled and also MMAP enabled. Still… it will go OOM after around 4 days.
We are also still using version 6 of Graphhopper. Were there many changes going to version 7 regarding memory consumption?
Any help would be greatly appreciated, as a new try, with new parameter will then take 4 days again until it fails.
This is how memory consumption looks like when doing the import. There are some memory-releases between the importing of the profiles. But I’d say it keeps growing. Also the time is increasing quite a bit.
I will now try to create only one CH at a time and hope that that way I have more “safe-points” so that I don’t always loose 4 days of calculation time.
But every other tip is sooooo much welcome.