GraphHopper.com | Forum | GitHub | Maps | Blog

Graphopper-reader and Protobuf 3


#1

Is there a way or workaround to use graphopper-reader with protobuf 3?

In my gradle I use
compile('com.graphhopper:graphhopper-reader-osm:0.9.0')
and it depends on protobuf-java:2.6.1
But in my project I need protobuf-java:3.3.1 and that causes a conflict and exception is thrown because 2.6.1 is overriden by 3.3.1.

So, is there any way to use it along with protobuf v3?


#2

I do not think a tweak via just the dependencies would work. But probably it is better to update the dependencies in the sources? I can see in the github repo that it should work with protobuf 3 … see this PR

maybe you can find out if there is a release of osm-binary and test everything and provide a PR :slight_smile: ? This would be highly appreciated!


#3

With what Java the 3.x versions are built?
(though probably not needed on Android)


#4

It looks like the 3.0.0 upgrade happened after the upgrade to java 8 but java 8 should not be a problem for reader-osm dependencies?


#5

Well, let me provide more details.

My project is compiled using java 1.8_131; everything is compiled fine and no errors occur. Then my project runs without any issues even with
compile 'com.graphhopper:graphhopper-reader-osm:0.9.0' compile 'com.google.protobuf:protobuf-java:3.3.1' dependencies.

But in case a fresh import of osm.pbf file has to be performed during startup, an exception is thrown:

Exception in thread “PBF Reader” java.lang.VerifyError: class org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader overrides final method getParserForType.()Lcom/google/protobuf/Parser;

The reason is:
|+--- com.graphhopper:graphhopper-reader-osm:0.9.0 | +--- com.graphhopper:graphhopper-core:0.9.0 (*) | +--- org.openstreetmap.osmosis:osmosis-osm-binary:0.45 | | \--- com.google.protobuf:protobuf-java:2.6.1 -> 3.3.1 +--- com.google.protobuf:protobuf-java:3.3.1

The latest version of osmosis-osm-binary in maven repository is 0.45.
I’ve already seen osmosis-osm-binary repo, but I can not find any way to import it using maven, it seems not to be published.

If I just change the protobuf dependency from 3.3.1 to 2.6.1 then import of osm.pbf file runs fine and without any exceptions.

As a workaround I think it is possible to precompile :osmosis-osm-binary from the repository and just add it as jar to my dependencies.

But for the time being I’m trying to find a better solution.


#6

Yes, that will be only possible if the upgrade to protobuf 3 will be released as a new version in maven central. See the PR where I asked about this.


#7

Regarding reader-osm since it’s used on desktop, should be fine.

Osmosis 0.45 is before all these updates (May 2016).
In Mapsforge map / poi creation, we also use the same Osmosis release.


#8

So, I tried to compile the latest osmosis-osm-binary from osmosis repo to add a new osmosis-osm-binary-0.45-52.jar dependency to my project, and protobuf:3.3.1 compatibility problem still persists:

Exception in thread “pool-4-thread-1” java.lang.NoSuchMethodError: org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock.getRequiredFeaturesList()Lcom/google/protobuf/ProtocolStringList;
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.processOsmHeader(PbfBlobDecoder.java:83)
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.runAndTrapExceptions(PbfBlobDecoder.java:340)
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.run(PbfBlobDecoder.java:355)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

This happens because of osmosis-osm-binary uses protobuf-java:3.0.0 and is not compatible with protobuf-java:3.3.1

I see that it’s not a graphhopper issue; I just wanted to let you know about this kind of problem.


#9

Well, after some more experiments I’m totally lost.

For testing purpose I copied osmosis-osm-binary project into my project as a module. Recompiled their .proto files with protoc 3.3.0, added dependency for protobuf 3.3.0 to my project and osmosis-osm-binary module.

These are full dependencies for graphhopper:
+--- com.graphhopper:graphhopper-core:0.9.0 | +--- com.carrotsearch:hppc:0.7.2 | +--- org.apache.xmlgraphics:xmlgraphics-commons:2.1 | | +--- commons-io:commons-io:1.3.1 -> 2.4 | | \--- commons-logging:commons-logging:1.0.4 -> 1.2 | +--- com.fasterxml.jackson.core:jackson-annotations:2.8.4 | \--- com.vividsolutions:jts:1.13 +--- com.graphhopper:graphhopper-reader-osm:0.9.0 | +--- com.graphhopper:graphhopper-core:0.9.0 (*) | \--- log4j:log4j:1.2.17 +--- project :osmosis-osm-binary | \--- com.google.protobuf:protobuf-java:3.3.0
project :osmosis-osm-binary - that is my subproject with cloned code from original osmosis-osm-binary

And I still see exception

Exception in thread “pool-4-thread-1” java.lang.NoSuchMethodError: org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock.getRequiredFeaturesList()Lcom/google/protobuf/ProtocolStringList;
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.processOsmHeader(PbfBlobDecoder.java:83)

in this line of code: for (String feature : header.getRequiredFeaturesList()) {

but the most strange thing is, that during debug if I set breakpoint there I can evaluate expression
header.getRequiredFeaturesList() and get response with List of Strings and method really exists. But as soon as I ‘Step Into’ - the NoSuchMethodError is thrown.

I checked all dependencies - they all look fine and org.openstreetmap.osmosis group is fully excluded from com.graphhopper:graphhopper-reader-osm.

Here is a screenshot where I can evaluate during debug the same loop where exception happens. But as soon as I hit continue - exception is thrown.


#10

Did you recompile the GH modules with the new dependency?


#11

I’m not sure I understand what you mean.
GH is added as a dependency in gradle file.
After I added subproject osmosis-osm-binary, I invalidated caches, ran clean task and only then compiled my project.
So I don’t think there is any possibility GH could use an old osmosis-osm-binary, though it looks like it does.


#12

There are probably source incompatibilities between new osmosis-osm-binary and GH. Did you recompile GH with the new one and see also if all tests were passing?


#13

No, I didn’t. I’ll try to do it.


#14

Thanks for the fix at https://github.com/graphhopper/graphhopper/pull/1175 @akamuza!