Warnings #setWeight while importing osm.pbf

engopy · December 30, 2020, 2:41pm

Hello,

In general what I do is getting my own osm.pbf edited map, merge that with official from geofabrik and upload back to graphhopper. I use osmosis merge method. Unfortunatelly After reading result file, my boundary box visible in graphhopper is wrong set and is somewhere on Athlantic Ocean (around 0,0 point).

What I did is check how graphhoppers logs looks like when reading geofabrik and my merge files. What I found was around 50 WARN logs:
[shortest_foot_node] WARN com.graphhopper.storage.CHGraphImpl - Setting weights smaller than 0.001 is not allowed in CHGraphImpl#setWeight. You passed: 0.0 for the edge -1 nodeA 0.25275871722605936,0.0972544796151179 nodeB 0.25275871722605936,0.0972544796151179

and also the same for [shortest_car_node]:
[shortest_car_node] WARN com.graphhopper.storage.CHGraphImpl - Setting weights smaller than 0.001 is not allowed in CHGraphImpl#setWeight. You passed: 0.0 for the edge -1 nodeA 0.2527585309615403,0.09725429335059882 nodeB 0.2527585309615403,0.09725429335059882

I’ve checked into source code but It’s not very easy for me and couldn’t exactly find out what is a problem. Only one I expect that it prints out coordinates of some point A and point B. I tried to find those longitude or latitude values in postgres db in table nodes, but I didn’t find any nodes with that values.
I also exported .osm.pbf file into .osm and used grep method on this file with one of coordinate value, but no results was given.

Can you give me any clue if setWeight method is connected with any speed tag/allowed weight on specific road etc.?
Any suggestions will be appreciated.

easbar · December 30, 2020, 4:44pm

The boundary issue should be independent from the CH preparation. The first thing I would do is remove profiles_ch from config.yml to disable CH and check if the boundary box works. But honestly I do not expect the setWeight warnings to be related to the boundary box issue. Which GH version are you using? Can you try running the import with latest master? This commit: https://github.com/graphhopper/graphhopper/commit/7881b5cccc09f1e825e0c67a8bfb6325869dfa96 introduce some more strict checks during import, maybe this also reveals an issue with your map file. Also you could try opening your pbf file with JOSM or osmium to see if there are any issues.

engopy · December 31, 2020, 7:56am

Thank you for you response.
I have found in pom.xml version 0.13-SNAPSHOT and I assume that is a application version (on github there is 3.0 nowadays).
I will try to run the same files on the newest version, but first I would try to solve the current problem, because whole process for me worked before. One thing which is different now is a addint new pathways for pedestrians in a city. Do you know if there maight be any connections with adding new paths or restrictions? During adding those items in map editor no errors appeared.

I don’t know exactly how that import problem is connected with bbox, but when I load official OSM map, no WARN appear in logs, and graphhopper center view and draw boundary box in correct positiona and correct shape.

easbar · December 31, 2020, 8:19am

If you use 0.13 you need to set the prepare.ch.weightings=no to disable CH (the profiles_ch section was only introduced in 1.0+).

I will try to run the same files on the newest version, but first I would try to solve the current problem, because whole process for me worked before

Ok, I was just trying to say: Running the import with the most recent version might reveal the problem so it will be easier to fix in the version you are using.

No I have no idea how this could be connected to the broken bounding box.

Which map editor are you using? Did you try JOSM?

Yes I still think this must be somehow related to something unusual in your (edited) OSM file.

engopy · December 31, 2020, 11:20am

After some tests I have found problem.
The problem lied step before -> reading database and write it into .pbf file using osmosis with granularity=20000 parameter:
(osmosis --read-apidb host=$DBHOST database=$DBNAME user=$DBUSER password=$DBPASSWORD validateSchemaVersion=no --log-progress --write-pbf file=$FINAL_PBF_NAME omitmetadata=true granularity=20000)

As osmosis documentation says:

The granularity or precision used to store coordinates. The default of 100 nanodegrees is the highest precision used by OSM, corresponding to about 1.1cm at the equator. In the current osmosis implementation, the granularity must be a multiple of 100. If map data is going to be exported to software that does not need the full precision, increasing the granularity to 10000 nanodegrees can save about 10% of the file size, while still having 1.1m precision.

I don’t need such a accurracy that’s why I used 20000 (2,2m precision).

Maybe you will better predict what happened.
Perhaps during truncating accuraccy of points, some crucial of them overlaps? Maybe as warning says -> weight of the edge between 2 nodes are less than 0.01. (I assume that distance can be took as a weight in i.e. dijkstra algorithm).

However I will omit granularity parameter.

Thank you very much for your help!

easbar · December 31, 2020, 4:35pm

I guess the granularity parameter controls the number of decimal points for coordinates in the OSM file. Can you use osmosis --read-pbf file="your_file.pbf" --write-xml file="your_file.osm" to make the file readable and have a look what the coordinates look like?

engopy · January 2, 2021, 2:30pm

I’ve exported db to osm.pbf file with and without granularity parameter. As it’s written in osmosis documentation and You expected granularity rounds coordinates values with demand accuraccy.
I’ve used BeyondCompare tool to compare both OSM files, and results for the same points:
(First value without granularity, second with granularity=20000 - around 2,2 meters of accuraccy)

<node id="21911894" version="-1" timestamp="1970-01-01T00:00:00Z" changeset="-1" lat="43.7377782" lon="7.4259476"/>
<node id="21911894" version="-1" timestamp="1970-01-01T00:00:00Z" changeset="-1" lat="43.73776" lon="7.42594"/>

<node id="21911901" version="-1" timestamp="1970-01-01T00:00:00Z" changeset="-1" lat="43.7377621" lon="7.4267099"/>
<node id="21911901" version="-1" timestamp="1970-01-01T00:00:00Z" changeset="-1" lat="43.73776" lon="7.4267"/>

easbar · January 2, 2021, 7:12pm

Ok thanks. But unfortunately that does not give me any clue why the bounding box is broken. Can you share the OSM file that produces the broken bounding box? I could have a look myself.

engopy · January 3, 2021, 12:07am

I tested it on Monaco map.

I don’t know if external file hosting are well seen but osm files are around 6mb so it’s impossible to upload them here. If that’s not acceptable I will transfer them different way and edit that post.
One file is exported with granularity=20000, other is exported without that parameter:

Granularity osm file
No granularity osm file

easbar · January 4, 2021, 11:22am

The file with granularity was deleted already. Can you try again?

engopy · January 5, 2021, 7:02am

Sure. Here it should stay for 30 days.
OSM with granularity
OSM without granularity

easbar · January 6, 2021, 8:46am

Unfortunately I could not reproduce your issue. I tried both version 0.13 and current master and each time the bounding box was shown correctly. Which version are you using and how exactly do you start GraphHopper / what does your config file look like?

engopy · January 12, 2021, 10:52pm

So it seems that problem lies in my graphhopper.
Actually our graphhopper is partly customized for our buissness demands, but I didn’t expect those modifications could cause that problem. Messages of WARN logs shown in while graphhopper starts didn’t track me that way. Modifications was made by previous developer, and I don’t entirely understand all of them.

Anyway I paste below our config file (I cut comment lines):

 graphhopper:

  ##### Vehicles #####
  graph.flag_encoders: car
  graph.encoded_values: road_class,road_class_link,road_environment,max_speed,road_access
  graph.bytes_for_flags: 4

  ##### Elevation #####

  #### Speed, hybrid and flexible mode ####
  prepare.ch.weightings: fastest
  prepare.ch.edge_based: off
  prepare.min_network_size: 200
  prepare.min_one_way_network_size: 200

  ##### Routing #####
  routing.non_ch.max_waypoint_distance: 1000000
  ##### Storage #####
  spatial_rules.max_bbox: -180, 180, -90, 90

server:
  applicationConnectors:
  - type: http
    port: 8989
    # for security reasons bind to localhost
    bindHost: localhost
  requestLog:
      appenders: []
  adminConnectors:
  - type: http
    port: 8990
    bindHost: localhost

logging:
  appenders:
  - type: file
    timeZone: UTC
    currentLogFilename: logs/graphhopper.log
    logFormat: "%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"
    archive: true
    archivedLogFilenamePattern: ./logs/graphhopper-%d.log.gz
    archivedFileCount: 30
    neverBlock: true
  - type: console
    timeZone: UTC
    logFormat: "%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"

Main modification in businnes logic is to use tags maxspeed:forward and maxspeed:backward to manipulate max speeds for concrete ways.
I can see that most of changes were done in CarFlagEncoder.class in the constructor which set default speeds for roads in //autobahn section and also in handleWayTags() method where maxspeed:forward and maxspeed:backward tags reading is added.

Anyway we have decided to upgrade our graphhopper version, and try to use custom vehicles whith speed rules on specific roads instead of editing OSM map file with adding earlier mentioned tags and use them in graphhopper.

Our main goal is to use different speed limits in the city depends on period of day (bigger traffic/lower). It’s my next task to make a research in that topic.

@easbar I think I will not go deeper in that problem, while it turned out that’s problem which we have created on our own. I would like to say you big thank you for support and your time

easbar · January 26, 2021, 8:56am

Good idea