Running out of 120GB of memory when loading planet-latest.osm.pbf

Open Source Routing Engine
#1

Hi,

I’m trying to load the whole planet in for car routing. I read in the docs (https://github.com/graphhopper/graphhopper/blob/master/docs/core/deploy.md) that loading the global route network would take around 22GB. I’m running on a machine with 120GB, and I run out of memory after around 3 hours of processing. I do the following:

GraphHopperOSM hopper = new GraphHopperOSM();
hopper.forServer();
hopper.setOSMFile("/data/planet/planet-latest.osm.pbf");
hopper.setGraphHopperLocation("/data/planet/processed");
hopper.setEncodingManager(new EncodingManager("car"));
hopper.setEnableInstructions(false);
hopper.setEnableCalcPoints(false);
hopper.setCHEnabled(true);
hopper.setPreciseIndexResolution(50);
hopper.importOrLoad();
return hopper;

Is there something special I need to do to get it to load in a reasonable amount of ram. Or, in my case, load at all. Is there some flag or setting I’m missing? Should I load the data in a different format? How would you normally load the whole planet file for car queries?

#2

Why not load the network only for the routing area?
Does it make sense to load the entire planet?

#3

We do need to do the whole planet, yes. Otherwise we’d have issues with cross-border things when we go to use the data. I’ve actually found a repository of continent files, and that will help. The borders between most continents are small, after all. However, the main problem remains the same: how can I load these files into memory correctly. The Europe file is still almost 19GB.

#4

Yes, this should be still possible ±5GB or so. We do world wide stuff with multiple vehicle profiles and CH+LM enabled.

Avoid changing the defaults for the index:

hopper.setPreciseIndexResolution(50);
#5

Do you think the difference lies in that resolution? Do you think I could load the whole planet in a sane amount of memory if I just set that to 300 (which is the default)?

What are the parameters you use to load the planet file?

#6

I suggest to remove the method instead of setting the value explicitly.

What are the parameters you use to load the planet file?

As it is described in the deploy.md doc

#7

I mean, do you think the 300 (default) vs 50 is what makes it take up all that extra RAM? Is that the make-or-break thing here?

#8

As it is described in the deploy.md doc
Do you mean the referenced config in https://raw.githubusercontent.com/graphhopper/graphhopper/master/config-example.yml?

#9

If you follow https://github.com/graphhopper/graphhopper/blob/master/docs/core/deploy.md and adapt the config-examples to your needs, this should be sufficient to make it working. If not, let us know. We also have elevation enabled: https://github.com/graphhopper/graphhopper/blob/master/docs/core/elevation.md

I mean, do you think the 300 (default) vs 50 is what makes it take up all that extra RAM? Is that the make-or-break thing here?

Highly likely

#10

What effect does changing the resolution have, btw? I have it at 50, because that’s that the original author of the code explicitly changed it to, and I’m a bit afraid of changing this, because I don’t want to change the semantics of the code too much.

#11

In the end, processing individual region files and reducing the precision for one of them (asia), we managed to get the job to run. For future reference, the europe file, which is almost 19GB, requires ~80GB of ram to load at a precision of 50 meters.
This is just unpacking the OSM file. We don’t yet know how it’ll behave when we actually use it, but that’s a problem for tomorrow =)

#12

I also ran into this problem. @karussell How do you run the world with 22 GB of ram and ch prepared? I have europe with 5 flagencoder and 3 weightings and starting graphhopper takes more than 64GB of ram and it runs with 56GB after initial load.

Did it improve so much after 0.9.0?

#13

The deploy.md speaks about one vehicle profile, one weighting and one thread for CH preparation.

The more encoders and weightings you need the more RAM the CH preparation requires and the longer it takes. To reduce this time you can do CH preparation in parallel but this requires even more RAM. E.g. we use servers with >200GB RAM to make the import fast.

Running the service is a different story and usually requires less RAM, but also dependents on the number of weightings and vehicle profiles. For running the service there is a rule of thumb: the disk size (of graph-cache-gh) plus 2GB. (The higher the request volume the more RAM you should add)

Powered by Discourse