Can't import gtfs-reader into android project

I’m trying to use gtfs-reader in an android project. This is the content of my build.gradle:

apply plugin: 'com.android.application'



android {
compileSdkVersion 27
buildToolsVersion "27.0.1"
defaultConfig {
    applicationId "com.graphhopper.android"
    minSdkVersion 10
    targetSdkVersion 22
}
buildTypes {
    all {
        minifyEnabled true
        useProguard false
        proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
    }
}
lintOptions {
    /* CGIARProvider refers to java.awt
     * Helper7 refers to java.lang.management
     * HeightTile refers to javax.imageio and java.awt
     * OSMElement refers to javax.xml.stream
     */
    disable 'InvalidPackage'
}
productFlavors {
}
}

/** only necessary if you need to use latest SNAPSHOT
configurations.all {
// check for updates every build
resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
}
 **/

dependencies {
implementation(group: 'com.graphhopper', name: 'graphhopper-core', version: '0.10-SNAPSHOT') {
    exclude group: 'com.google.protobuf', module: 'protobuf-java'
    exclude group: 'org.openstreetmap.osmosis', module: 'osmosis-osm-binary'
    exclude group: 'org.apache.xmlgraphics', module: 'xmlgraphics-commons'
}

implementation 'org.mapsforge:vtm:0.9.2'
implementation 'org.mapsforge:vtm-android:0.9.2'
implementation 'org.mapsforge:vtm-android:0.9.2:natives-armeabi'
implementation 'org.mapsforge:vtm-android:0.9.2:natives-armeabi-v7a'
implementation 'org.mapsforge:vtm-android:0.9.2:natives-arm64-v8a'
implementation 'org.mapsforge:vtm-android:0.9.2:natives-x86'
implementation 'org.mapsforge:vtm-android:0.9.2:natives-x86_64'
implementation('org.mapsforge:vtm-jts:0.9.2') {
    exclude group: 'com.vividsolutions', module: 'jts'
}
implementation 'org.mapsforge:vtm-themes:0.9.2'
implementation 'com.caverock:androidsvg:1.2.2-beta-1'

implementation 'org.slf4j:slf4j-api:1.7.25'
implementation 'org.slf4j:slf4j-android:1.7.25'
implementation 'com.graphhopper:graphhopper-reader-gtfs:0.10.alpha3'
}

When I sync the gradle, I get these errors:

/home/eduardo/github/graphhopper/graphhopper/android/app/build.gradle

Error:Error:Failed to resolve: org.geotools:gt-api:14.0
Error:Error:Failed to resolve: org.geotools:gt-epsg-hsql:11-beta
Error:Error:Failed to resolve: com.conveyal:jackson2-geojson:0.8

What should I do to solve it?

The GTFS functionality is not yet tried and tested on Android.

I’ve also tried to use the gtfs reader in an android app. As initial project I’ve used the android project from the graphhopper repository and changed the following things (this guide isn’t completed yet). I’ve forked the repository and you can find all my changes here: https://github.com/RaimundWege/graphhopper/commit/d22715e880694a6407c6d657357a39ea046172ef

1.) In the build.gradle of the android project add two repositories to resolve the missing dependencies (the order of the repositories is important).

2.) In the build.gradle of the app module you need to:
set the ‘minSdkVersion’ and ‘targetSdkVersion’ to at least ‘26’,
set ‘minifyEnabled’ to ‘false’,
add the property ‘multiDexEnabled’ and set ‘true’ as value,
enable java version 1.8,
add the graphhopper-gtfs depenceny,
and the android support multidex dependency:

android {
    // ...
    defaultConfig {
        // ...
        minSdkVersion 26
        targetSdkVersion 26
    }
    buildTypes {
        all {
            // ...
            minifyEnabled false
            multiDexEnabled true
        }
    }
    // ...
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
}
// ...
dependencies {
    // ...
    implementation 'Sorry, new users can only put 2 links in a post.'
    implementation 'Sorry, new users can only put 2 links in a post.'
}

3.) In the android manifest you need to add the permission to read from an external storage.

4.) Also in the android manifest you need to add the android:hardwareAccelerated=“true” and the android:largeHeap=“true” application attribute.

5.) In the MainActivity replace the doInBackground method of the loadGraphStorageMethod to (‘my-gtfs.zip’ is the uploaded feed):

protected Path saveDoInBackground(Void... v) throws Exception {
    String path = new File(mapsFolder, currentArea).getAbsolutePath() + "-gh";
    final PtFlagEncoder ptFlagEncoder = new PtFlagEncoder();
    final FootFlagEncoder footFlagEncoder = new FootFlagEncoder();
    final CarFlagEncoder carFlagEncoder = new CarFlagEncoder();
    EncodingManager encodingManager = new EncodingManager(Arrays.asList(ptFlagEncoder, footFlagEncoder, carFlagEncoder), 8);
    GHDirectory directory = GraphHopperGtfs.createGHDirectory(mapsFolder.getAbsolutePath() + "/graph-cache");
    GtfsStorage gtfsStorage = GraphHopperGtfs.createGtfsStorage();
    GraphHopperStorage graphHopperStorage = GraphHopperGtfs.createOrLoad(directory, encodingManager, ptFlagEncoder, gtfsStorage, Collections.singleton(path + "/my-gtfs.zip"), Collections.<String>emptyList());
    LocationIndex locationIndex = GraphHopperGtfs.createOrLoadIndex(directory, graphHopperStorage);
    GraphHopperGtfs tmpHopp = GraphHopperGtfs.createFactory(ptFlagEncoder, GraphHopperGtfs.createTranslationMap(), graphHopperStorage, locationIndex, gtfsStorage).createWithoutRealtimeFeed();
    tmpHopp.load(path);
    hopper = tmpHopp;
    return null;
}

6.) When I run the app now it fails after a while with the error ‘Could not instantiate class’ which is raised in the mapdb dependency…

Error Log:
10-10 15:37:20.329 9345-9345/com.graphhopper.android W/System.err: java.lang.RuntimeException: Could not instantiate class
10-10 15:37:20.330 9345-9345/com.graphhopper.android W/System.err: at org.mapdb.SerializerPojo.deserializeUnknownHeader(SerializerPojo.java:483)
at org.mapdb.SerializerBase.deserialize3(SerializerBase.java:1216)
at org.mapdb.SerializerBase.deserialize(SerializerBase.java:1132)
at org.mapdb.SerializerBase.deserialize(SerializerBase.java:867)
at org.mapdb.SerializerPojo.deserialize(SerializerPojo.java:701)
at org.mapdb.HTreeMap$2.deserialize(HTreeMap.java:135)
at org.mapdb.HTreeMap$2.deserialize(HTreeMap.java:121)
at org.mapdb.Store.deserialize(Store.java:297)
at org.mapdb.StoreDirect.get2(StoreDirect.java:486)
at org.mapdb.StoreDirect.get(StoreDirect.java:439)
at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:333)
at org.mapdb.Caches$HashTable.get(Caches.java:246)
at org.mapdb.HTreeMap.getInner(HTreeMap.java:458)
10-10 15:37:20.331 9345-9345/com.graphhopper.android W/System.err: at org.mapdb.HTreeMap.getPeek(HTreeMap.java:436)
at org.mapdb.HTreeMap.containsKey(HTreeMap.java:300)
at com.graphhopper.reader.gtfs.GtfsReader.addTrip(GtfsReader.java:484)
at com.graphhopper.reader.gtfs.GtfsReader.addTrips(GtfsReader.java:431)
at com.graphhopper.reader.gtfs.GtfsReader.lambda$buildPtNetwork$3$GtfsReader(GtfsReader.java:201)
at com.graphhopper.reader.gtfs.GtfsReader$$Lambda$0.accept(Unknown Source:4)
at java.lang.Iterable.forEach(Iterable.java:75)
at com.graphhopper.reader.gtfs.GtfsReader.buildPtNetwork(GtfsReader.java:175)
at com.graphhopper.reader.gtfs.GtfsReader.readGraph(GtfsReader.java:143)
at com.graphhopper.reader.gtfs.GraphHopperGtfs.createOrLoad(GraphHopperGtfs.java:369)
at com.graphhopper.android.MainActivity$6.saveDoInBackground(MainActivity.java:401)
at com.graphhopper.android.MainActivity$6.saveDoInBackground(MainActivity.java:392)
at com.graphhopper.android.GHAsyncTask.doInBackground(GHAsyncTask.java:12)
at android.os.AsyncTask$2.call(AsyncTask.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
10-10 15:37:20.332 9345-9345/com.graphhopper.android W/System.err: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)
Caused by: java.lang.NoSuchMethodException: []
at java.lang.Class.getConstructor0(Class.java:2320)
at java.lang.Class.getConstructor(Class.java:1725)
at org.mapdb.SerializerPojo.createInstanceSkippinkConstructor(SerializerPojo.java:579)
at org.mapdb.SerializerPojo.deserializeUnknownHeader(SerializerPojo.java:467)
… 32 more

This is an area that we have not tried before and it might be that there are incompatibilities with MapDB or similar.

You can try to do the heavy import on a desktop or server only and then use the created GraphHopper folder instead of gtfs+pbf files on Android.

I’ve already tried that but the result is the same. :confused:

It seems the issue happens when mapdb tries to initiate the GtfsStorage.Validity class. MapDB can’t find a constructor for it and tries to use the empty constructor but Validity has no empty constructor.

Now I’ve copied the java source from the reader directly into the app and added a default public constructor for the Validity and FeedIdWithTimezone classes of the GraphStorage class. With that it is possible to generate the cache on the device and request a route!

But the cache must be smaller than 512 MB. Also the cache can’t be reused after a restart or upload because of the ‘Could not instantiate class’ issue even when its generated on the device. This time it fails during the field initialization of an agency. The first values seems ok but the agency_url throws an exception.

The current progress can be found here: https://github.com/RaimundWege/graphhopper/tree/gtfs_reader_source

Great - it is nice to read about your progress!

Currently I fear we have no priority to push this ourself, but we’d appreciate if you can make your improvements like the two public constructors available as pull requests.

Sure, I will see if I can make a request next week. Finally I had to add even more empty constructors for all entities and errors of the conveyal.gtfs package. For some constructors I had to specify values for final fields … but at least it is possible to create and reload the graph from the cache. Also I’ve done a package implementation for flutter (android only) and I think about to publish it. But first I have to implement some more mappings to provide basic functionality :slight_smile:

1 Like

The first integration of the gtfs-reader in our app is done and here are the limitations:

  • only supported for Android 8.0+ (much use of Java 8 streams and other classes)
  • external storage required (I guess this can be fixed)
  • the graph-cache needs to be smaller than 512 MB
  • first initialization lasts up to ten minutes
  • when an *.pbf file is involved and the storage isn’t closed properly (in case of an app crash) the graph-cache needs to be initialized again because the database is corrupted (for now we open and close the storage for each route request)

In the next step I want to provide the dependency for lower API levels but I guess I’ve to change to many things. So it will not be possible to create a PR for the changes.

This sounds really interesting :slight_smile: Thanks for taking the time!

We would highly appreciate if you could contribute those improvements in some form. Maybe you can try to split those changes in smaller PRs like the missing constructors? Or we can discuss together a certain set of problems and how to solve them best?

when an *.pbf file is involved and the storage isn’t closed properly (in case of an app crash) the graph-cache needs to be initialized again

Were you succesful to import GTFS with PBF on Android? Or what do you mean here? Also why is open&close for each route request necessary?

BTW: what is you intent to make public transit working offline? Is it e.g. an app for countries (like Germany :wink: ) with poor mobile internet connection?

Currently I’m still in a research phase and looking for solutions with minimal changes. Where is the right place to discuss that? (git issue, git pr or this forum?)

Yes, I’ve created and loaded a GraphHopperGtfs instance with a gtfs.zip and an *.pbf file. First I’ve tested it only with a gtfs.zip and when I started to use an *.pbf file I discovered the database issue which happens when the app doesn’t close the GraphHopperStorage, GtfsStorage and the LocationIndex properly. Furthermore it was necessary to set MAX_WALK_DISTANCE_PER_LEG to 500 m otherwise the public transportation were ignored. This may be due to a bad gtfs file.

The alternative is to open it on app start and close it on app termination but in the case of an unexpected app crash the storage will not be closed properly and the database gets corrupted (this only happens when an *.pbf file is part of the configuration). A corrupted database includes a reinitialization and that takes a long time and drains the battery. With the open and close mechanism there is only a short time window where the database is vulnerable through an app crash.

Something similar. We’re trying to provide an app for the public transportation in cochabamba. In order to realize that we’re faced with many other issue too e.g. they don’t have stops and there is also no timetable and only a few roads have names etc. but thats the issue of the data team :wink:

1 Like

Discussion should be here. Concrete issues or PR or code at Github.

Yes, I’ve created and loaded a GraphHopperGtfs instance with a gtfs.zip and an *.pbf file.

This is awesome :slight_smile: … and completely out of the specs in a positive meaning :slight_smile: . I did not even know that the import is working for road routing on Android and the import for public transit routing is currently even heavier. The usual way to do low resource routing is to do the import on a bigger server and then just load the resulting folder on Android. Is this not a better option for you?

(btw: there was even an iOS port but since a few releases it needs a bit more attention)

the database gets corrupted (this only happens when an *.pbf file is part of the configuration)

Can you describe in which way it is corrupted? Do you get an error message? Does it happen only if the import is interrupted and the resulting folder is not 100% ready? Or at routing time?

Because a routing query should be read-only and should not result in storage changes or even corruption.