I am trying understand how the stores are meant to be used.
I have generated a graph (car - shortest) on the planet osm file.
The graph was generated and contracted in memory. It’s then flushed on the disk (RAM_STORE).
The ghraph’s files were then moved in another server that provides route calculation (using graphhoper-0.5) over JMS. So the graph can be read only but route calculation must be thread safe.
If I load the whole graph in RAM everithing works like a charm.
Problem comes when I try to use the graph without loading it in RAM.
I thought that I just had to switch the “DAType” from “RAM_STORE” to “MMAP_RO” when I load my graph but this does not seem to work that way.
Caused by: java.io.FileNotFoundException: /data/used_graphs/osm/car-shortest-gh/original_edges_shortest_car (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:122)
at com.graphhopper.storage.MMapDataAccess.initRandomAccessFile(MMapDataAccess.java:70)
... 53 more
And indeed this files does not exist so do I have to generate the graph another way (so that this missing file will be there) in order to load and use it from the disk ?
Indeed there is a flaw that this file is ‘created’ but not used if no contraction happens, e.g. if one just loads the graph. And normally the file is just silently created but that is not possible with “RO” (read only)
Workaround would probably be to create this file or remove “_RO”. Furthermore a real fix should be made.
I’ve overlooked a bit, please use MMAP_SYNC or do you need “read only”? Then use MMAP_SYNC_RO, should work too, not 100% sure -> see 315)
minor BTW: RAM_STORE does not touch the disc on dataAccess.create()
Using MMAP_SYNC work.
However I had to give write permission on the graph files and directory and the file ‘original_edges_shortest_car’ was created.
What are the purpose of the “sync” and “read only” options of the DAType?
Sounds like a trivial question but my case is that the files are shared among several instances of the server (they are even on a drive shared by two debian server) so there are concurrent read access. I did not expect that any of the processes would have to write in theses files.
Yeah, but in your case MMAP_RO_SYNC would be best and will make the least trouble. Please create an issue to avoid touching the disc in this case where the graph is only loaded, this should be fixed.
BTW: why are you using shortest?
Sounds like a trivial question
No problem to answer as this is really only documented in the code
The _RO appendix is for devices with low RAM and e.g. shared disc. E.g. on Android there were problems accessing the data without the read only option
The _SYNC appendix is important if you have multiple threads accessing the same dataAccess object at the same time. E.g. memory restricted server environments or if you develop custom multi threaded algorithms.
so there are concurrent read access
With the memory mapped storage access from different processes e.g. different JVMs should not be a problem, even for writes and even without _SYNC (although I have not tested this). And as you just query there are no writes and you should be really safe, the only write towards the original_edge DataAccess object is not critical as only the file is created, no content is generated (only used for contraction on import).
Now for different threads (concurrent access to one DataAccess object of GraphHopper) you are only safe if you use the _SYNC appendix.
I hope this is not too confusing. Keep in mind that MMAP is not that recommend as it is in general several times slower, there are lots of opportunities to make it faster (e.g. via unsafe) but no plans for it yet as 90% of the users set RAM_STORE.
Many thanks for this clear answer, it helps a lot.
Only to leave our end users the possibility to perform shortest routing but it’s barely not used so I suggested to put this graph in MMAP to save some RAM.
However it indeed turned on that a world graph in MMAP is way too slow so we’ll stick around with RAM_STORE.
BTW: with 0.5 you can use multiple profiles (even if CH is enabled) in one GraphHopper instance saving lots of memory. So shortest car & fastest car etc should be possible at the same time.
