Protobuf class collisions running map-matching 0.10.0 in a Spark 2.x EMR envronment

Developers
1

Hi,

I am trying to run map matching in a parallel processing environment using Spark on AWS EMR.
I am encountering the following issue during osm map initialization and wondering if anyone has run into this issue before.
After a few attempts, I’ve tried to downgrade to an earlier version of protobuf but am still running into missing method declarations. Running the library locally works for me.

com.google.protobuf protobuf-java 2.6.1 org.openstreetmap.osmosis osmosis-osm-binary 0.45 com.graphhopper graphhopper-reader-osm ${graphhopper.version}

18/07/17 17:28:30 INFO OSMGraphhopper: using CH|car|RAM_STORE|2D|NoExt|, memory:totalMB:1857, usedMB:412
Exception in thread “pool-24-thread-1” java.lang.NoSuchMethodError: com.google.protobuf.LazyStringList.getUnmodifiableView()Lcom/google/protobuf/LazyStringList;
at org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock.(Osmformat.java:300)
at org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock.(Osmformat.java:185)
at org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock$1.parsePartialFrom(Osmformat.java:321)
at org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock$1.parsePartialFrom(Osmformat.java:316)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at org.openstreetmap.osmosis.osmbinary.Osmformat$HeaderBlock.parseFrom(Osmformat.java:749)
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.processOsmHeader(PbfBlobDecoder.java:78)
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.runAndTrapExceptions(PbfBlobDecoder.java:340)
at com.graphhopper.reader.osm.pbf.PbfBlobDecoder.run(PbfBlobDecoder.java:355)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
18/07/17 17:29:28 INFO ExecutorAllocationManager: Request to remove executorIds: 2
18/07/17 17:29:28 INFO YarnAllocator: Driver requested a total number of 0 executor(s).
18/07/17 17:29:28 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 0 executors.
18/07/17 17:29:28 INFO ExecutorAllocationManager: Request to remove executorId

Thanks,
Trang

2

It looks like this is a dup of Calling hopper.importOrLoad from Spark
but unfortunately no fix was posted.

3

Closing the issue. This was a conflict between protobuf and AWS EMR. One fix is to use maven’s shade plugin to relocate the protobuf classes to a different package. See: http://www.waitingforcode.com/apache-spark/shading-solution-dependency-hell-spark/read

1 Like