In my application I would like to process the postal code for an edge using custom logic, in a similar way as can be done with the streetname of (custom) flags.
Ideally edges would be attributed with a string postcode, and would be retrieved with String postcode = edge.getPostCode(), just like you do with String streetName = edge.getName()
The postal_codes as can be found in OSM waypoint tags are ok for this. They can be easily parsed by extending OSMReader (e.g. extending EncodingManager with a custom public void applyWayTags(ReaderWay way, EdgeIteratorState edge) but I couldn’t store them in a Storage (RAM) and present them to the application without touching the code base all over.
What would be the best / most elegant way to tackle this?
1 Like
Hi,
probably the easiest way to do this would be to use a similar approach as shown here. In this example we show how to store a osmid per edge. It should be quite easy to change this example to store postal codes.
You can also have a look at the NameIndex, as this uses a DataAccess to store Strings.
Best,
Robin
Thanks for the useful hints robin, job done! What I did:
-
I departed from https://github.com/karussell/graphhopper-osm-id-mapping like you said.
-
I made a custom public static class PostCodeIndex implements Storable which I copied from NameIndex like you said, and changed few things to make it unique. I also added a lookup map to store the pointers for each postcode, so that one entry occurs only once in the map.
-
I removed the overlay for protected void storeOsmWayID(int edgeId, long osmWayId) as this only contains the osmWayId and I need the postal code too. Instead I overlayed public void applyWayTags(ReaderWay way, EdgeIteratorState edge) in the FlagEncoder since it gives me wayId and all the flags in one go.
-
The PostCodeIndex needs pointers for the edge. To do so, I extended the edgeMapping a bit so that it contains two longs: one for the osmWayId and one for the postalcode pointer into the index. Since the encoder needs to access the global edgeMapping and postCodeMapping, I made the Encoder class local to the GraphHopper class so that it can do:
long postCodePointer = postCodeMapping.put(postalCode);
long edgePointer = 16L * edgeId;
edgeMapping.ensureCapacity(edgePointer + 16L);
edgeMapping.setInt(edgePointer, bitUtil.getIntLow(osmWayId));
edgeMapping.setInt(edgePointer + 4, bitUtil.getIntHigh(osmWayId));
edgeMapping.setInt(edgePointer + 8, bitUtil.getIntLow(postCodePointer));
edgeMapping.setInt(edgePointer + 12, bitUtil.getIntHigh(postCodePointer));
The test class looks like this:
public class StoreOSMWayIDTest {
private static Logger logger = LoggerFactory.getLogger(StoreOSMWayIDTest.class);
private static TestGraphHopper hopper;
private static AlgorithmOptions algoOptions;
private static Graph graph;
public static void main(String [] args) throws InterruptedException, ExecutionException, FileNotFoundException {
// create hopper instance with CH enabled
hopper = new TestGraphHopper();
CarFlagEncoder encoder = hopper.new TestEncoder();
hopper.setDataReaderFile("./map-data/berlin-latest.osm.pbf");
hopper.setGraphHopperLocation("./map-data/berlin-latest");
hopper.setEncodingManager(new EncodingManager(encoder));
hopper.importOrLoad();
// force CH
algoOptions = AlgorithmOptions.start()
.maxVisitedNodes(1000)
.hints(new PMap().put(Parameters.CH.DISABLE, false))
.build();
logger.info("edge 30 -> " + hopper.getOSMWay(30) + ", " + hopper.getGraphHopperStorage().getEdgeIteratorState(30, Integer.MIN_VALUE).fetchWayGeometry(2));
GHResponse rsp = new GHResponse();
List<Path> paths = hopper.calcPaths(new GHRequest(52.498668, 13.431473, 52.48947, 13.404007).
setWeighting("fastest").setVehicle("car"), rsp);
Path path0 = paths.get(0);
for (EdgeIteratorState edge : path0.calcEdges()) {
int edgeId = edge.getEdge();
String vInfo = "";
if (edge instanceof VirtualEdgeIteratorState) {
// first, via and last edges can be virtual
VirtualEdgeIteratorState vEdge = (VirtualEdgeIteratorState) edge;
edgeId = vEdge.getOriginalTraversalKey() / 2;
vInfo = "v";
}
logger.info("(" + vInfo + edgeId + ") " + hopper.getOSMWay(edgeId) + "(" + edge.getName()+ " / "+hopper.getPostalCode(edgeId)+ "), " + edge.fetchWayGeometry(2));
}
hopper.close();
}
public static class PostCodeIndex implements Storable<PostCodeIndex> {
private static final long START_POINTER = 1;
private final DataAccess postcodes;
private long bytePointer = START_POINTER;
// hash the postcodes to avoid multiple entries with the same postcode
private HashMap<String,Long> map = new HashMap<String,Long>();
public PostCodeIndex(Directory dir) {
postcodes = dir.find("postcodes");
}
@Override
public PostCodeIndex create(long initBytes) {
postcodes.create(initBytes);
return this;
}
@Override
public boolean loadExisting() {
if (postcodes.loadExisting()) {
bytePointer = BitUtil.LITTLE.combineIntsToLong(postcodes.getHeader(0), postcodes.getHeader(4));
return true;
}
return false;
}
/**
* @return the byte pointer to the name
*/
public long put(String postCode) {
if (postCode == null || postCode.isEmpty()) {
return 0;
}
Long pointer = map.get(postCode);
if (pointer != null) {
return pointer;
}
byte[] bytes = getBytes(postCode);
pointer = bytePointer;
postcodes.ensureCapacity(bytePointer + 1 + bytes.length);
byte[] sizeBytes = new byte[]{
(byte) bytes.length
};
postcodes.setBytes(bytePointer, sizeBytes, sizeBytes.length);
bytePointer++;
postcodes.setBytes(bytePointer, bytes, bytes.length);
bytePointer += bytes.length;
map.put(postCode, pointer);
return pointer;
}
private byte[] getBytes(String name) {
byte[] bytes = null;
for (int i = 0; i < 2; i++) {
bytes = name.getBytes(Helper.UTF_CS);
// we have to store the size of the array into *one* byte
if (bytes.length > 255) {
String newName = name.substring(0, 256 / 4);
logger.info("Postal Code is too long: " + name + " truncated to " + newName);
name = newName;
continue;
}
break;
}
if (bytes.length > 255) {
// really make sure no such problem exists
throw new IllegalStateException("Postal Code is too long: " + name);
}
return bytes;
}
public String get(long pointer) {
if (pointer < 0)
throw new IllegalStateException("Pointer to access PostalCodeIndex cannot be negative:" + pointer);
// default
if (pointer == 0)
return "";
byte[] sizeBytes = new byte[1];
postcodes.getBytes(pointer, sizeBytes, 1);
int size = sizeBytes[0] & 0xFF;
byte[] bytes = new byte[size];
postcodes.getBytes(pointer + sizeBytes.length, bytes, size);
return new String(bytes, Helper.UTF_CS);
}
@Override
public void flush() {
postcodes.setHeader(0, BitUtil.LITTLE.getIntLow(bytePointer));
postcodes.setHeader(4, BitUtil.LITTLE.getIntHigh(bytePointer));
postcodes.flush();
map.clear();
}
@Override
public void close() {
postcodes.close();
}
@Override
public boolean isClosed() {
return postcodes.isClosed();
}
public void setSegmentSize(int segments) {
postcodes.setSegmentSize(segments);
}
@Override
public long getCapacity() {
return postcodes.getCapacity();
}
public void copyTo(PostCodeIndex postCodeIndex) {
postcodes.copyTo(postCodeIndex.postcodes);
}
}
static class TestGraphHopper extends GraphHopperOSM {
int wayPoints = 0;
// mapping of internal edge ID to OSM way ID
private DataAccess edgeMapping;
private PostCodeIndex postCodeMapping;
private BitUtil bitUtil;
private List<Path> paths;
class TestEncoder extends CarFlagEncoder {
public void applyWayTags(ReaderWay way, EdgeIteratorState edge) {
super.applyWayTags(way,edge);
String postalCode = way.getTag("postal_code");
long osmWayId = way.getId();
int edgeId = edge.getEdge();
long postCodePointer = postCodeMapping.put(postalCode);
long edgePointer = 16L * edgeId;
edgeMapping.ensureCapacity(edgePointer + 16L);
edgeMapping.setInt(edgePointer, bitUtil.getIntLow(osmWayId));
edgeMapping.setInt(edgePointer + 4, bitUtil.getIntHigh(osmWayId));
edgeMapping.setInt(edgePointer + 8, bitUtil.getIntLow(postCodePointer));
edgeMapping.setInt(edgePointer + 12, bitUtil.getIntHigh(postCodePointer));
wayPoints++;
}
}
TestGraphHopper(){
super();
getCHFactoryDecorator().setDisablingAllowed(true);
}
@Override
public boolean load(String graphHopperFolder) {
boolean loaded = super.load(graphHopperFolder);
Directory dir = getGraphHopperStorage().getDirectory();
bitUtil = BitUtil.get(dir.getByteOrder());
edgeMapping = dir.find("edge_mapping");
postCodeMapping = new PostCodeIndex(dir);
if (loaded) {
edgeMapping.loadExisting();
postCodeMapping.loadExisting();
}
return loaded;
}
@Override
protected DataReader createReader(final GraphHopperStorage ghStorage) {
OSMReader reader = null;
reader = new OSMReader(ghStorage) {
{
edgeMapping.create(1000);
postCodeMapping.create(2000);
}
@Override
protected void finishedReading() {
super.finishedReading();
edgeMapping.flush();
postCodeMapping.flush();
}
};
return initDataReader(reader);
}
public long getOSMWay(int internalEdgeId) {
long pointer = 16L * internalEdgeId;
return bitUtil.combineIntsToLong(edgeMapping.getInt(pointer), edgeMapping.getInt(pointer + 4L));
}
public String getPostalCode(int internalEdgeId) {
long pointer = 16L * internalEdgeId;
long postCodePointer = bitUtil.combineIntsToLong(edgeMapping.getInt(pointer + 8L), edgeMapping.getInt(pointer + 12L));
return postCodeMapping.get(postCodePointer);
}
public void close(){
super.close();
edgeMapping.close();
postCodeMapping.close();
}
}
}
Cheers,
Philip
3 Likes
Hi Philip,
thanks for the detailed reply. I think this will help others that face a similar issue, thanks for sharing.
Cheers,
Robin