How to extend edges with OSM postal_code tags

Open Source Routing Engine
#1

In my application I would like to process the postal code for an edge using custom logic, in a similar way as can be done with the streetname of (custom) flags.

Ideally edges would be attributed with a string postcode, and would be retrieved with String postcode = edge.getPostCode(), just like you do with String streetName = edge.getName()

The postal_codes as can be found in OSM waypoint tags are ok for this. They can be easily parsed by extending OSMReader (e.g. extending EncodingManager with a custom public void applyWayTags(ReaderWay way, EdgeIteratorState edge) but I couldn’t store them in a Storage (RAM) and present them to the application without touching the code base all over.

What would be the best / most elegant way to tackle this?

1 Like
#2

Hi,

probably the easiest way to do this would be to use a similar approach as shown here. In this example we show how to store a osmid per edge. It should be quite easy to change this example to store postal codes.

You can also have a look at the NameIndex, as this uses a DataAccess to store Strings.

Best,
Robin

#3

Thanks for the useful hints robin, job done! What I did:

  1. I departed from https://github.com/karussell/graphhopper-osm-id-mapping like you said.

  2. I made a custom public static class PostCodeIndex implements Storable which I copied from NameIndex like you said, and changed few things to make it unique. I also added a lookup map to store the pointers for each postcode, so that one entry occurs only once in the map.

  3. I removed the overlay for protected void storeOsmWayID(int edgeId, long osmWayId) as this only contains the osmWayId and I need the postal code too. Instead I overlayed public void applyWayTags(ReaderWay way, EdgeIteratorState edge) in the FlagEncoder since it gives me wayId and all the flags in one go.

  4. The PostCodeIndex needs pointers for the edge. To do so, I extended the edgeMapping a bit so that it contains two longs: one for the osmWayId and one for the postalcode pointer into the index. Since the encoder needs to access the global edgeMapping and postCodeMapping, I made the Encoder class local to the GraphHopper class so that it can do:

     		long postCodePointer = postCodeMapping.put(postalCode);
 		long edgePointer = 16L * edgeId;
 		edgeMapping.ensureCapacity(edgePointer + 16L);
 		edgeMapping.setInt(edgePointer, bitUtil.getIntLow(osmWayId));
 		edgeMapping.setInt(edgePointer + 4, bitUtil.getIntHigh(osmWayId));
 		edgeMapping.setInt(edgePointer + 8, bitUtil.getIntLow(postCodePointer));
 		edgeMapping.setInt(edgePointer + 12, bitUtil.getIntHigh(postCodePointer));

The test class looks like this:
public class StoreOSMWayIDTest {

private static Logger logger = LoggerFactory.getLogger(StoreOSMWayIDTest.class);


private static TestGraphHopper hopper;
private static AlgorithmOptions algoOptions;
private static Graph graph;



public static void main(String [] args) throws InterruptedException, ExecutionException, FileNotFoundException {

	// create hopper instance with CH enabled
	hopper = new TestGraphHopper();
	CarFlagEncoder encoder = hopper.new TestEncoder();
	hopper.setDataReaderFile("./map-data/berlin-latest.osm.pbf");
	hopper.setGraphHopperLocation("./map-data/berlin-latest");
	hopper.setEncodingManager(new EncodingManager(encoder));
	hopper.importOrLoad();

	// force CH
	algoOptions = AlgorithmOptions.start()
			.maxVisitedNodes(1000)
			.hints(new PMap().put(Parameters.CH.DISABLE, false))
			.build();

	logger.info("edge 30 -> " + hopper.getOSMWay(30) + ", " + hopper.getGraphHopperStorage().getEdgeIteratorState(30, Integer.MIN_VALUE).fetchWayGeometry(2));

	GHResponse rsp = new GHResponse();
	List<Path> paths = hopper.calcPaths(new GHRequest(52.498668, 13.431473, 52.48947, 13.404007).
			setWeighting("fastest").setVehicle("car"), rsp);
	Path path0 = paths.get(0);
	for (EdgeIteratorState edge : path0.calcEdges()) {
		int edgeId = edge.getEdge();
		String vInfo = "";
		if (edge instanceof VirtualEdgeIteratorState) {
			// first, via and last edges can be virtual
			VirtualEdgeIteratorState vEdge = (VirtualEdgeIteratorState) edge;
			edgeId = vEdge.getOriginalTraversalKey() / 2;
			vInfo = "v";
		}

		logger.info("(" + vInfo + edgeId + ") " + hopper.getOSMWay(edgeId) + "(" + edge.getName()+ " / "+hopper.getPostalCode(edgeId)+ "), " + edge.fetchWayGeometry(2));
	}
	hopper.close();

}

public static class PostCodeIndex implements Storable<PostCodeIndex> {
	private static final long START_POINTER = 1;
	private final DataAccess postcodes;
	private long bytePointer = START_POINTER;
	// hash the postcodes to avoid multiple entries with the same postcode
	private HashMap<String,Long> map = new HashMap<String,Long>();

	public PostCodeIndex(Directory dir) {
		postcodes = dir.find("postcodes");
	}

	@Override
	public PostCodeIndex create(long initBytes) {
		postcodes.create(initBytes);
		return this;
	}

	@Override
	public boolean loadExisting() {
		if (postcodes.loadExisting()) {
			bytePointer = BitUtil.LITTLE.combineIntsToLong(postcodes.getHeader(0), postcodes.getHeader(4));
			return true;
		}

		return false;
	}

	/**
	 * @return the byte pointer to the name
	 */
	public long put(String postCode) {
		if (postCode == null || postCode.isEmpty()) {
			return 0;
		}
		Long pointer = map.get(postCode);
		if (pointer != null) {
			return pointer;
		}
		byte[] bytes = getBytes(postCode);
		pointer = bytePointer;
		postcodes.ensureCapacity(bytePointer + 1 + bytes.length);
		byte[] sizeBytes = new byte[]{
				(byte) bytes.length
		};
		postcodes.setBytes(bytePointer, sizeBytes, sizeBytes.length);
		bytePointer++;
		postcodes.setBytes(bytePointer, bytes, bytes.length);
		bytePointer += bytes.length;
		map.put(postCode, pointer);
		return pointer;
	}

	private byte[] getBytes(String name) {
		byte[] bytes = null;
		for (int i = 0; i < 2; i++) {
			bytes = name.getBytes(Helper.UTF_CS);
			// we have to store the size of the array into *one* byte
			if (bytes.length > 255) {
				String newName = name.substring(0, 256 / 4);
				logger.info("Postal Code is too long: " + name + " truncated to " + newName);
				name = newName;
				continue;
			}
			break;
		}
		if (bytes.length > 255) {
			// really make sure no such problem exists
			throw new IllegalStateException("Postal Code is too long: " + name);
		}
		return bytes;
	}

	public String get(long pointer) {
		if (pointer < 0)
			throw new IllegalStateException("Pointer to access PostalCodeIndex cannot be negative:" + pointer);

		// default
		if (pointer == 0)
			return "";

		byte[] sizeBytes = new byte[1];
		postcodes.getBytes(pointer, sizeBytes, 1);
		int size = sizeBytes[0] & 0xFF;
		byte[] bytes = new byte[size];
		postcodes.getBytes(pointer + sizeBytes.length, bytes, size);
		return new String(bytes, Helper.UTF_CS);
	}

	@Override
	public void flush() {
		postcodes.setHeader(0, BitUtil.LITTLE.getIntLow(bytePointer));
		postcodes.setHeader(4, BitUtil.LITTLE.getIntHigh(bytePointer));
		postcodes.flush();
		map.clear();
	}

	@Override
	public void close() {
		postcodes.close();
	}

	@Override
	public boolean isClosed() {
		return postcodes.isClosed();
	}

	public void setSegmentSize(int segments) {
		postcodes.setSegmentSize(segments);
	}

	@Override
	public long getCapacity() {
		return postcodes.getCapacity();
	}

	public void copyTo(PostCodeIndex postCodeIndex) {
		postcodes.copyTo(postCodeIndex.postcodes);
	}
}


static class TestGraphHopper extends GraphHopperOSM {

	int wayPoints = 0; 

	// mapping of internal edge ID to OSM way ID
	private DataAccess edgeMapping;
	private PostCodeIndex postCodeMapping;
	private BitUtil bitUtil;
	private List<Path> paths;




	class TestEncoder extends CarFlagEncoder {

		public void applyWayTags(ReaderWay way, EdgeIteratorState edge) {
			super.applyWayTags(way,edge);
			String postalCode =  way.getTag("postal_code");
			long osmWayId = way.getId();
			int edgeId = edge.getEdge();

			long postCodePointer = postCodeMapping.put(postalCode);
			long edgePointer = 16L * edgeId;
			edgeMapping.ensureCapacity(edgePointer + 16L);
			edgeMapping.setInt(edgePointer, bitUtil.getIntLow(osmWayId));
			edgeMapping.setInt(edgePointer + 4, bitUtil.getIntHigh(osmWayId));
			edgeMapping.setInt(edgePointer + 8, bitUtil.getIntLow(postCodePointer));
			edgeMapping.setInt(edgePointer + 12, bitUtil.getIntHigh(postCodePointer));

			wayPoints++;
		}
	}

	TestGraphHopper(){
		super();
		getCHFactoryDecorator().setDisablingAllowed(true);
	}


	@Override
	public boolean load(String graphHopperFolder) {
		boolean loaded = super.load(graphHopperFolder);

		Directory dir = getGraphHopperStorage().getDirectory();
		bitUtil = BitUtil.get(dir.getByteOrder());
		edgeMapping = dir.find("edge_mapping");
		postCodeMapping = new PostCodeIndex(dir);


		if (loaded) {
			edgeMapping.loadExisting();
			postCodeMapping.loadExisting();
		}

		return loaded;
	}

	@Override
	protected DataReader createReader(final GraphHopperStorage ghStorage) {
		OSMReader reader = null;

		reader = new OSMReader(ghStorage) {

			{
				edgeMapping.create(1000);
				postCodeMapping.create(2000);
			}

			@Override
			protected void finishedReading() {
				super.finishedReading();
				edgeMapping.flush();
				postCodeMapping.flush();
			}
		};

		return initDataReader(reader);
	}


	public long getOSMWay(int internalEdgeId) {
		long pointer = 16L * internalEdgeId;
		return bitUtil.combineIntsToLong(edgeMapping.getInt(pointer), edgeMapping.getInt(pointer + 4L));
	}


	public String getPostalCode(int internalEdgeId) {
		long pointer = 16L * internalEdgeId;
		long postCodePointer = bitUtil.combineIntsToLong(edgeMapping.getInt(pointer + 8L), edgeMapping.getInt(pointer + 12L));
		return postCodeMapping.get(postCodePointer);
	}

	public void close(){
		super.close();
		edgeMapping.close();
		postCodeMapping.close();	    	
	}
}

}

Cheers,

Philip

3 Likes
Feed custom properties while parsing osm pbf
Feed custom properties while parsing osm pbf
#4

Hi Philip,

thanks for the detailed reply. I think this will help others that face a similar issue, thanks for sharing.

Cheers,
Robin