Graphopper-reader and Protobuf 3

Is there a way or workaround to use graphopper-reader with protobuf 3?

In my gradle I use
compile('com.graphhopper:graphhopper-reader-osm:0.9.0')
and it depends on protobuf-java:2.6.1
But in my project I need protobuf-java:3.3.1 and that causes a conflict and exception is thrown because 2.6.1 is overriden by 3.3.1.

So, is there any way to use it along with protobuf v3?

I do not think a tweak via just the dependencies would work. But probably it is better to update the dependencies in the sources? I can see in the github repo that it should work with protobuf 3 … see this PR

maybe you can find out if there is a release of osm-binary and test everything and provide a PR :slight_smile: ? This would be highly appreciated!

With what Java the 3.x versions are built?
(though probably not needed on Android)

It looks like the 3.0.0 upgrade happened after the upgrade to java 8 but java 8 should not be a problem for reader-osm dependencies?

Well, let me provide more details.

My project is compiled using java 1.8_131; everything is compiled fine and no errors occur. Then my project runs without any issues even with
compile 'com.graphhopper:graphhopper-reader-osm:0.9.0' compile 'com.google.protobuf:protobuf-java:3.3.1' dependencies.

But in case a fresh import of osm.pbf file has to be performed during startup, an exception is thrown:

Exception in thread “PBF Reader” java.lang.VerifyError: class org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader overrides final method getParserForType.()Lcom/google/protobuf/Parser;

The reason is:
|+--- com.graphhopper:graphhopper-reader-osm:0.9.0 | +--- com.graphhopper:graphhopper-core:0.9.0 (*) | +--- org.openstreetmap.osmosis:osmosis-osm-binary:0.45 | | \--- com.google.protobuf:protobuf-java:2.6.1 -> 3.3.1 +--- com.google.protobuf:protobuf-java:3.3.1

The latest version of osmosis-osm-binary in maven repository is 0.45.
I’ve already seen osmosis-osm-binary repo, but I can not find any way to import it using maven, it seems not to be published.

If I just change the protobuf dependency from 3.3.1 to 2.6.1 then import of osm.pbf file runs fine and without any exceptions.

As a workaround I think it is possible to precompile :osmosis-osm-binary from the repository and just add it as jar to my dependencies.

But for the time being I’m trying to find a better solution.

Yes, that will be only possible if the upgrade to protobuf 3 will be released as a new version in maven central. See the PR where I asked about this.

Regarding reader-osm since it’s used on desktop, should be fine.

Osmosis 0.45 is before all these updates (May 2016).
In Mapsforge map / poi creation, we also use the same Osmosis release.

So, I tried to compile the latest osmosis-osm-binary from osmosis repo to add a new osmosis-osm-binary-0.45-52.jar dependency to my project, and protobuf:3.3.1 compatibility problem still persists:

Exception in thread “pool-4-thread-1” java.lang.NoSuchMethodError: org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock.getRequiredFeaturesList()Lcom/google/protobuf/ProtocolStringList;
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.processOsmHeader(PbfBlobDecoder.java:83)
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.runAndTrapExceptions(PbfBlobDecoder.java:340)
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.run(PbfBlobDecoder.java:355)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

This happens because of osmosis-osm-binary uses protobuf-java:3.0.0 and is not compatible with protobuf-java:3.3.1

I see that it’s not a graphhopper issue; I just wanted to let you know about this kind of problem.

Well, after some more experiments I’m totally lost.

For testing purpose I copied osmosis-osm-binary project into my project as a module. Recompiled their .proto files with protoc 3.3.0, added dependency for protobuf 3.3.0 to my project and osmosis-osm-binary module.

These are full dependencies for graphhopper:
+--- com.graphhopper:graphhopper-core:0.9.0 | +--- com.carrotsearch:hppc:0.7.2 | +--- org.apache.xmlgraphics:xmlgraphics-commons:2.1 | | +--- commons-io:commons-io:1.3.1 -> 2.4 | | \--- commons-logging:commons-logging:1.0.4 -> 1.2 | +--- com.fasterxml.jackson.core:jackson-annotations:2.8.4 | \--- com.vividsolutions:jts:1.13 +--- com.graphhopper:graphhopper-reader-osm:0.9.0 | +--- com.graphhopper:graphhopper-core:0.9.0 (*) | \--- log4j:log4j:1.2.17 +--- project :osmosis-osm-binary | \--- com.google.protobuf:protobuf-java:3.3.0
project :osmosis-osm-binary - that is my subproject with cloned code from original osmosis-osm-binary

And I still see exception

Exception in thread “pool-4-thread-1” java.lang.NoSuchMethodError: org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock.getRequiredFeaturesList()Lcom/google/protobuf/ProtocolStringList;
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.processOsmHeader(PbfBlobDecoder.java:83)

in this line of code: for (String feature : header.getRequiredFeaturesList()) {

but the most strange thing is, that during debug if I set breakpoint there I can evaluate expression
header.getRequiredFeaturesList() and get response with List of Strings and method really exists. But as soon as I ‘Step Into’ - the NoSuchMethodError is thrown.

I checked all dependencies - they all look fine and org.openstreetmap.osmosis group is fully excluded from com.graphhopper:graphhopper-reader-osm.

Here is a screenshot where I can evaluate during debug the same loop where exception happens. But as soon as I hit continue - exception is thrown.

Did you recompile the GH modules with the new dependency?

I’m not sure I understand what you mean.
GH is added as a dependency in gradle file.
After I added subproject osmosis-osm-binary, I invalidated caches, ran clean task and only then compiled my project.
So I don’t think there is any possibility GH could use an old osmosis-osm-binary, though it looks like it does.

There are probably source incompatibilities between new osmosis-osm-binary and GH. Did you recompile GH with the new one and see also if all tests were passing?

No, I didn’t. I’ll try to do it.

Thanks for the fix at https://github.com/graphhopper/graphhopper/pull/1175 @akamuza!