Supported file,format and few coordinate problem

Hi graphhopper developpers

I am using this API for map-matching purpose.

I have a taxi dataset which contains a trace of mobility of drivers, an example of a trajectory is found in the file attachednew_oilrag.txt (30.7 KB)

I had parsed it to present the essential parts for my project out_new_oilrag.txt (32.0 KB)
my question is what is the suitable tool to use so it convert my file to supported GPX format for map matching

is their any additional parsing steps to apply before conversion to GPX?

I am asking this question, since I have always face the problem of few corrdinate

Thanks in advance

Best regards

They are in reverse order. So for GraphHopper you have the requirement that the second point has to happen after the first point and so on.

Another problem can be if there is a too huge gap like e.g. here, of 8 minutes:

point16,37.61456,-122.39435,2008-05-18 06:30:19
point17,37.61473,-122.39232,2008-05-18 06:22:04

This case should be doable as the distance is not that far, but in general this can make problems and you could try to preprocess the data and split them or wait for the following PR being merged: https://github.com/graphhopper/map-matching/pull/87

1 Like

ok Mr Karussell, a very clear reply except PR I don’t know what is means exactly
Thank you very much

1 Like

PR = pull request (in the context of git version control)

2 Likes

Another problem can be if there is a too huge gap like e.g. here, of 8 minutes

in lieu of the above PR, the map matching will still run over large separations, though you may need to set the maximum number of points visited to be quite large. Whether it is meaningful to do so is up to you.

1 Like

thank you for the explanation,
I would like to know what is the maximum gap allowed(permitted).
I have tried to split my dataset into files and if I set a small difeernce in time(50second) I get a big number of small files with few points --> risque of “Too few coordiante problem” and even when I set 20 minutes as gap between two successive points I get files but with also few coordinates problem with this file out_0new_ifeshce.txt.txt (2.5 KB). I really don’t know what to do ? The dataset I am using is a realistic dataset of Sanfransisco taxi cab http://crawdad.org/epfl/mobility/20090224/
Thank you in advance

There is no maximum distance (per se) and there is no need to split your file. All you need to do is set maxVisitedNodes (e.g. https://github.com/graphhopper/map-matching/blob/master/matching-core/src/test/java/com/graphhopper/matching/MapMatchingTest.java#L8) large enough. The above PR actually requires this too.

1 Like

Thank you for the reply, I appreciate

Did you have an explanation why I still get the problem of “too few coordinate” with the file attached in my previous post?

Best regards

If you’re using master, then see https://github.com/graphhopper/map-matching/blob/master/matching-core/src/main/java/com/graphhopper/matching/MapMatching.java#L170-L173 - i.e. the only way this should happen is if you provide a GPX list with less than two entries. If you’re not using master, it may be associated with the filtering - points are only used which differ by measurementErrorSigma. It looks like the points in your file are all very ‘close’, and hence would probably be filtered down to a single point. You can either fix this (as suggested above), or manually detect these errors - it may be fair to assume that the point is stationary (within measurementErrorSigma).

1 Like

Thank you for the reply

I see that is defined as measurementErrorSigma = 50.0;

What is the measurement unit used so I will find 1) the stay points(distance between successive GPS points are less then measurementErrorSigma and time is less then a threshold) 2) split the trajectory according to stay Points found

Thank you once again

Best regards

here is my distance function I will use

private static double distance(double lat1, double lon1, double lat2, double lon2) {
		double theta = lon1 - lon2;
		double dist = Math.sin(deg2rad(lat1)) * Math.sin(deg2rad(lat2)) + Math.cos(deg2rad(lat1)) * Math.cos(deg2rad(lat2)) * Math.cos(deg2rad(theta));
		dist = Math.acos(dist);
		dist = rad2deg(dist);
		dist = dist * 60 * 1.1515;
		dist = dist * 1.609344*1000;
		System.out.println("dist "+dist);
		return (Math.round(dist));
	}

	/*:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::*/
	/*::	This function converts decimal degrees to radians						 :*/
	/*:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::*/
	private static double deg2rad(double deg) {
		return (deg * Math.PI / 180.0);
	}

	/*:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::*/
	/*::	This function converts radians to decimal degrees						 :*/
	/*:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::*/
	private static double rad2deg(double rad) {
		return (rad * 180 / Math.PI);
	}

I had also check my GPX file and comprae it with one of the test GPX files in Graphhooper , I had found that the stucture of those file is the following

rtept lat="51.374982" lon="12.344148"><desc>turn left onto B 6</desc><extensions><gh:distance>57.5</gh:distance><gh:time>4599</gh:time><gh:direction>E</gh:direction><gh:azimuth>75</gh:azimuth></extensions></rtept>

Whereas mine is just way point

<wpt lat="37.75194" lon="-122.39564">
  <time>2008-05-20T05:07:11Z</time>
  <name>point132</name>
</wpt>

Best regards

You could do that @noura, or you could:

  • not worry about this at all (as above, there’s no need to split your files)
  • split your files (even though not needed) and manually handle too-few-coordinate cases. This will be equivalent to what you’re proposing (in which you’ll then have to manually handle single stationary points anyway).
  • split your file (even though not needed) and manually handle ‘stationary points’ (even though not needed), but use Graphhopper distance calculations instead of your one e.g. as in the filtering used

To your question: unit = meters.

1 Like

Thank you once again for the reply and the explanation

the challenge I am facing is that I have to map match many hundred of files in the dataset and then extract the set of edges ID traversed, thus manual filtering will be a hard task.

I have used the distance function in graphhopper defined as follow, and the result is quit different from the one I used before distance=53.0 VS 52.684626566411(with calcDist)

public static double calcDist(double fromLat, double fromLon, double toLat, double toLon) {
        double dLat = toRadians(toLat - fromLat);
        double dLon = toRadians(toLon - fromLon);
        // use mean latitude as reference point for delta_lon
        double tmp = cos(toRadians((fromLat + toLat) / 2)) * dLon;
        double normedDist = dLat * dLat + tmp * tmp;
        double R = 6372.797560856*1000;
        return R * sqrt(normedDist);
}

I add the value of radius R

I see the function “filterGPXEntries” is available only in master and not in 0.8 I am using. unfortunately I faced problems before with master, therefore, I jumped to 0.8.

Thank you very much for your time and help, I really appreciate, I have to map-match the dataset for my PhD project.

Thank you once again

I’ve already made my recommendation a few times, and no sense repeating it.

Regarding distances - the GH one is most likely to be correct (depending on which you need). That said, a difference of 0.3m is much likely to be less than the accuracy of your coordinates, and if that’s the case, it’s not material.

Regarding 0.8 - filterGPXEntries is really just here. Since it’s for your PhD, it’s probably worthwhile looking around the code to understand what it’s doing.

1 Like

hello again Graphhopper developpers
I split my file with definition of stay points, and I have also change the dataset with another realistc dataset
but, unfortunately, I get problems:ofjava.lang.IllegalStateException: Time difference between subsequent location measurements must be >= 0.
and when I ommit some points I get java.lang.IllegalStateException: No edge matches found for path. Too short? Sequence size 31

Here is my file, IAOK2.txt (134.0 KB)
Thank in advance

~ $ cat IAOK2.txt | grep -C1 '2016-11-02T10:01:10'
<trkpt lat="37.77605" lon="-122.39371"><time>2016-11-02T09:59:40+00:00</time></trkpt>
<trkpt lat="37.77609" lon="-122.39349"><time>2016-11-02T10:01:10+00:00</time></trkpt>
<trkpt lat="37.74259" lon="-122.46454"><time>2016-11-02T00:00:05+00:00</time></trkpt>

Look at the timestamps - the first error seems to be correct. I can’t verify the second error without knowing the data set you ran it with - though it appears something may be fundamentally wrong with your data set.

1 Like

Yes Mr @kodonnell, the timestamps aren’t well established, I didn’t remark(notice) this. This is due to the fact that the data are from multiple users in the same data file.iaok3.txt (5.7 KB)
this is attached file for example report the second error
I would like also to know how did you discover the previous error, is their any log file of the error or it is the jdk log file for example?
thank you

the data are from multiple users in the same data file

map-matching won’t be able to handle this (reasonably) - how is it to know that two points are from different devices and not to be connected? The input for map-matching should be a single GPX track (unless you know that it’s OK to abuse it etc.). In reality, you’re going to want to split it into separate tracks.

I would like also to know how did you discover the previous error, is their any log file of the error or it is the jdk log file for example?

The error java.lang.IllegalStateException: Time difference between subsequent location measurements must be >= 0 explains pretty clearly what the problem is, and I found the example using python.

Yes exactly,I will separate trajectories of different users.
than you for the explanation.
Excuse me but, did you know the reason of my second error, I attached the file with the problem in my last post

Thank you for your time

did you know the reason of my second error

No - at this stage I’ll leave the debugging to you. It could be a bug (as it’s not meant to happen), though it’d pay to test it hasn’t been fixed on master first.

1 Like