Hey Community,
first of all thanks for the great work!
I´m playing with Graphhopper for some time. I tried to make a app for Germany, Austria and Switzerland. For this purpose i created one single graph with 2 CH-profiles. When i try do load the graph in my app following exception is thrown:
“An error happend while creating graph:Couldn’t map buffer 117 of 250 at
position 122683492 for 262143900 bytes with offset 100, new
fileLength:262144100”
Graph size on disk: 1,47 GB
I think its basically an OOM Issue. Furthermore i think its because of the large spatial extent of the graph and the small amount of RAM which is usable in an android device.
Simple Question: It is possible to use Graphhopper (Offline & on android) for such an huge extend e.g. to cover complete europe?
Hello Sebastian,
I cannot give you a YES or NO answer about using whole Europe.
But maybe you can find that out on your own:
Do a series of tests where you start generating the needed data only for a smaller part of Europe, and see how your memory will last.
In case of success, take a bigger extract als basis, maybe by adding one more country.
How do you initialize GraphHopper in the app?
You can try GraphHopper.forMobile for using memory mapped storage.
Emux
that´s the way i did. Germany works well. Germany + Austria +Switzerland is to large…
@devemux86: I use it already. i even tried to adapt the preciseIndexResolution.
Is there a simple way to reduce the needed memory amount? e.g. decrease the geometry quality?
Can you post your device spec (32/64bit?) and the Android version, as well as the full stack trace?
I´m using a OnePlus One with Android 5.1.1 (Cyanogen Mod). Its 32 bit. and im sorry, there is no more stacktrace.
the exception comes from this class
There is always a full stack trace but depends a bit on your log config, often with a message like here IOException: mmap failed: EPERM (Operation not permitted) additionally to the buffer message.
Do you use the largeHeap option for the app? See also this issue
(OutOfMemoryError)
yeah you´re right. Here it is:
11-19 10:09:39.607: E/...(977): Graph loading failure 11-19 10:09:39.607: E/...(977): java.lang.RuntimeException: Couldn't map buffer 170 of 250 at position 178258020 for 262143900 bytes with offset 100, new fileLength:262144100 11-19 10:09:39.607: E/...(977): at com.graphhopper.storage.MMapDataAccess.mapIt(MMapDataAccess.java:161) 11-19 10:09:39.607: E/...(977): at com.graphhopper.storage.MMapDataAccess.loadExisting(MMapDataAccess.java:245) 11-19 10:09:39.607: E/...(977): at com.graphhopper.storage.CHGraphImpl.loadExisting(CHGraphImpl.java:674) 11-19 10:09:39.607: E/...(977): at com.graphhopper.storage.GraphHopperStorage.loadExisting(GraphHopperStorage.java:286) 11-19 10:09:39.607: E/...(977): at com.graphhopper.GraphHopper.load(GraphHopper.java:826) 11-19 10:09:39.607: E/...(977): at com.example.....TaskLoadGraph.doInBackground(TaskLoadGraph.java:57) 11-19 10:09:39.607: E/...(977): at com.example.....TaskLoadGraph.doInBackground(TaskLoadGraph.java:1) 11-19 10:09:39.607: E/...(977): at android.os.AsyncTask$2.call(AsyncTask.java:292) 11-19 10:09:39.607: E/...(977): at java.util.concurrent.FutureTask.run(FutureTask.java:237) 11-19 10:09:39.607: E/...(977): at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231) 11-19 10:09:39.607: E/...(977): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112) 11-19 10:09:39.607: E/...(977): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587) 11-19 10:09:39.607: E/...(977): at java.lang.Thread.run(Thread.java:818) 11-19 10:09:39.607: E/...(977): Caused by: java.io.IOException: mmap failed: ENOMEM (Out of memory) 11-19 10:09:39.607: E/...(977): at java.nio.MemoryBlock.mmap(MemoryBlock.java:125) 11-19 10:09:39.607: E/...(977): at java.nio.FileChannelImpl.map(FileChannelImpl.java:257) 11-19 10:09:39.607: E/...(977): at com.graphhopper.storage.MMapDataAccess.newByteBuffer(MMapDataAccess.java:179) 11-19 10:09:39.607: E/...(977): at com.graphhopper.storage.MMapDataAccess.mapIt(MMapDataAccess.java:153) 11-19 10:09:39.607: E/...(977): ... 12 more 11-19 10:09:39.607: E/...(977): Caused by: android.system.ErrnoException: mmap failed: ENOMEM (Out of memory) 11-19 10:09:39.607: E/...(977): at libcore.io.Posix.mmap(Native Method) 11-19 10:09:39.607: E/...(977): at libcore.io.ForwardingOs.mmap(ForwardingOs.java:111) 11-19 10:09:39.607: E/...(977): at java.nio.MemoryBlock.mmap(MemoryBlock.java:122) 11-19 10:09:39.607: E/...(977): ... 15 more
android:largeHeap=“true” is enabled for the Application.
I think it´s related to Issue #1
Are here any Improvements or hints?
Hmmh, this is indeed due to a OOM while graph loading. But as you have 3GB this is strange and normally the ‘mapping process’ does not necessary allocate RAM. Can you somehow find out how much RAM your app consumes without GH?
Which storage are you using for the mapping - an external SD?
And can you restart your device and try again? (If that fixes the problem then the problem is the clean up of the mmap stuff under Android. Or a potentially fragmented RAM is cleaned up too.)
See the references issue there (506)
I never tried it for more than Germany to be honest, but a per-process limit for virtual memory on your device might be the problem, see here.
Application with GH (Graph loaded, no Route calculated):
Heap: 79,565 MB Allocated: 63,565 MB Free: 16,000 MB % Used: 79,89%
Application WITHOUT GH:
Heap: 79,421 MB Allocated: 63,421 MB Free: 16,000 MB % Used: 79,85%
No significant difference. But the MMAP is nevertheless outside of the heap of my application or?
No my smartphone doesnt support SD-Cards.
Dont change anything…
The Issue occurs when i try to load the Graph on my device. So no calculation is even possible.
hmm possible. Maybe there is a way to increase it or make a Workaround?
Indeed 
can you measure the RAM usage of your device when your app starts with and without GH somehow?
I would first try to find out if that is the reason. Maybe just use one CH profile to come under 1.7GB?
If that is the cause one could reduce required space or move the location index lookup in another process or similar.
After a delay here we go:
Availabe Memory on my device ( 3 GB Ram Overall present), measured after the graph is fully loaded and the App is fully started:
with large Graph (Germany) loaded:
1786 MB
without Graph (Graphhopper):
1818 MB
With described Error (Exception while Graph loading:
1807 MB
So MMAP dont influence much the Ram usage. As expected.
And a additional fact: a graph with a size of 1.15 GB is loadable and routable! but a Graph with 1,47 not.
It would be interesting to know if native applications have this limit too and then:
- If yes, then the only real solution would be to handle swapping out/in inside a new file-based DataAccess implementation. Not trivial as you need it fast.
- If no, one could try to implement MMapDataAccess based on minor native code where the mapping takes place.
- Or it is another issue, but really hard to say.
- Workarounds to reduce memory usage of GH exist but are also not that easy to implement: e.g. delta encoding of the geometry and the way name compression are lower hanging fruits.
Another interesting test would be to see if it is possible to start two applications with a 1.15GB graph each
good question.
I tested this simultaneously with your demo app and my app (both with the 1.15 GB Graph) and it works. The graphs are loaded and i can create routs even when i switch between them. Seems to be the per-process limit 
I think this might be the best solution. Is this on the agenda right now? 
No, but you can try workarounds e.g. load one CH profile per request only or avoid loading the ‘names’ DataAccess and do the MMAP only if you create the request or something. Maybe @devemux86 has some ideas how to avoid or workaround these issues?
Unfortunately I have not found yet a solution.
See #499, even if we manage to load a large country graph, OOM can occur later during country sized routing.
BTW it’s important to distinguish the device’s physical RAM from VM heap (which the application obtains).
Emux
If we can implement a simple native memory mapping or memory allocation - do you think that could solve the issue? As often there is plenty of RAM on the mobile devices these days.
Yes that would be a nice ‘side’ solution.
Peter since you’re the expert in GH memory management, I suppose we have exhausted all possibilities with the Java memory mapping way?
Emux
I do not know 
One possible trick could be to point into a memory mapped region via the unsafe utility but this is not available on Android (?)
Another trial could be to increase the segmentSize to reduce the number of memory mapped ByteBuffers. Also maybe mapping it via MapMode.PRIVATE could help? The problem is that I do not understand why exactly this fails. Is that an artificial dalvik limitation or a OS limit?
We could also reduce the required memory for the graph and learn from other projects but that is also currently out of reach for me.
I’m afraid more reading is needed in order to properly answer. 
http://developer.android.com/training/articles/memory.html
But AFAIK heap size is device dependent and not constant.
Emux
