I’m currently having an issue while generating a graph using the europe.pbf file with 9 profiles. The process crashes with the following error:
Error while creating graph java.util.concurrent.ExecutionException: com.carrotsearch.hppc.BufferAllocationException: Not enough memory to allocate buffers for rehashing: 33,554,432 -> 67,108,864
java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.carrotsearch.hppc.BufferAllocationException: Not enough memory to allocate buffers for rehashing: 33,554,432 -> 67,108,864
That made me start wondering: do I really need that many profiles in GraphHopper?
For example, my “Foot” and “Foot Tourist” profiles only differ slightly in their CustomModel definitions. So I’m considering switching to a setup where I define a shared base profile, e.g., "Base Foot", and apply different CustomModels per request instead of predefining them as separate profiles.
Would this approach have any performance drawbacks, especially when using different CustomModels dynamically per request? I never used CH, because it needed a lot of RAM, so my configuration is using LM
So I’m considering switching to a setup where I define a shared base profile, e.g., "Base Foot", and apply different CustomModels per request instead of predefining them as separate profiles.
So basically you use one LM preparation for another profile.
Alternatively you could also use the custom model in the request with the same result, I think.
because it needed a lot of RAM
This should not be much different, especially for outdoor profiles that are without turn restriction. Or do you mean the preparation needed a lot of RAM? Did you ensure that only one ch thread was running?
Yes crash is happening on preparation. We have 128 GB RAM server which has:
Instance of webserver with running Graphhopper in RAM
Some other projects that also take a little RAM (not much, but still)
Once a week we start new Instance of webserver which is creating new graph
Then process of creating new graph crashes.
About CH thread I’m not sure, but we don’t want to use CH right know because we need flexability given by LM.
About this “inbuilt” functionality, can you share any example how to do it in code? We don’t use graphhoper .jar, we have everything in out codebase.
In Java you’d create a LMProfile and set the preparationProfile to the profile where you want to use the preparation of. But I suggest using the config.yml especially for the preparation.
I read the documentation and saw that it’s possible to use either RAM or disk. However, I was under the impression that there’s a hybrid option — it uses RAM by default, and when memory runs low, it switches to disk instead of crashing. Does such an option still exist, or did I misunderstand something?
Also, docs says that
For example if you use different profiles for various types of cars, trucks and motorcycles there will be only a small increase of the required memory, because a lot of data will be shared between these different profiles.
This refers to the memory requirements once the graph is already built, right? Because during the preparation phase, I noticed a significant difference in memory usage.
Also, how can I check the default or explicitly set number of threads for LM in the code? If I understood your response and the documentation correctly, the more threads are used, the more memory is required — is that right?
The MMAP option basically does this with a slow down depending on the missing RAM. There is also a rather advanced config to fine tune which parts of the graph should be forced to stay in-memory (e.g. geometry or location index might not be so important to be always in-memory), but this is likely not that effective for the import or preparation (only for routing, I think).
This refers to the memory requirements once the graph is already built, right?
Well, yes. The more profiles you use the more likely it is that more encoded values are used which (slightly) increase the base graph. But compared to the memory requirements for the CH or LM preparation this is basically nothing
If I understood your response and the documentation correctly, the more threads are used, the more memory is required — is that right?
Yes, if you use more threads you need more memory. The CHPreparationHandler and LMPreparationHandler uses 1 as default. So if you didn’t change this it should already consume the smallest amount of memory possible atm.
Using LM you could also tweak the number of landmarks and similar stuff, which might slightly reduce the response speed but also decrease the memory needs. But by default we already have chosen a good balance between the two.
Okey, great. I think that MMAP option should be enough for us right know, because it crashes at 7th from 9 profiles. Could you tell me how to change it from code? In previous version I had to use reflection, but now I have no idea to setup MMAP instead of RAM
