GraphHopper.com | Forum | GitHub | Maps | Blog

Setup Continuous Performance Measurement


#1

Continued Discussion of #1163
Original issue:

My last post:

We can discuss here :slight_smile:


#2

Hi Martin,

thanks for posting here :).

I think your proposed solution sounds very interesting and in general I would like to learn more (or maybe test it, if it’s already runnable?). I think @karussell is right, we should set this up decoupled from GraphHopper and make it even possible for other projects to use it as well?

Cheers,
Robin


#3

Hi Robin,

okay :slight_smile: I didn’t set anything up yet. I just read your issue and your discussion with @karussell and Jenkins fits very well in there. You can think of jenkins as a selfhosted alternative to travis :slight_smile: It works with github webhooks, it has build queues, you can set up multiple slaves with multiple OS for testing, but you have to set it up yourself.

If you want to try jenkins, just install the jenkins docker image here and create a new freestyle job :slight_smile:

There are two different jobtypes at the moment:

  1. Freestyle jobs are jobs, which you create mostly with the gui. There is no easy readable code behind it or something like that. Most of the time you have one job for all of your branches or you have to create a new job for a new branch.
  2. Pipeline jobs are jobs, which are defined in code and a new job will be created at the moment of branch creation (or later if you set manually scan timer) and you can define each job seperatly. The advantage of these jobs is, that they follow the source code repository management. Your changed job will follow automatically with merges in your branch etc. So if you have a branch, where you switch for example from maven to gradle you can change the job in that branch and all other branches will be runned with maven, but yours will run gradle. If you merge that branch to master, master will run on gradle, but all other branches will run on maven until you sync them with the master.
    You can find out more about it here: https://jenkins.io/doc/book/pipeline/syntax/

The hardest or more intense part would be to set up the log file analyses. As I can see here you have something like that already, so you could reuse. I could imagine a comparison to master branch performance and a report back into the pull request of jenkins with comparisons.
To isolate noise and it depends on the jenkins workload, you can rerun a test X times, create a median or something like that to just eliminate noise. Just some sorts for later here :slight_smile:

All in all jenkins just is a tool to run the command you gave it like a shell script, but the infrastructure behind it supports all kind of things and also there is a big plugin support. If you need some special things, there is a big possibility that there is already a jenkins plugin for it.

If I can help you with something, just ask :slight_smile:

Cheers,
Martin


#4

Hi Martin,

thanks for the explanation, it’s nice to see that jenkins evolved, I haven’t heard about the freestyle jobs so far.

I used jenkins in a couple of projects several years ago.

It sounded like you already had a setup running and already a tweaked config for this.

Cheers,
Robin


#5

Hi Robin,

no sorry. If I have time, I can start to work on some things in my freetime, but I just hasn’t created anything yet. After reading your discussion, it was just a quick thought and I just wanted to charge my thoughts.

How are the measurements are done right now? :slight_smile:

I also don’t have the server capacity right now at home or as a vps to do something like that, but for testing I can create a docker on my machine.

Cheers,
Martin


#6

Hi Martin,

you can have a look at the Measurement class and also at graphhopper.sh. Currently you can run the measurements by executing graphhopper.sh measurement osm.pbf. The result of the measurement is a log file containing numbers for the different cases that have been run.

I think we can host this if it’s in a runnable state.

Help for this issue would be definitely appreciated, if you happen to have some time and interest in doing it :).

Cheers,
Robin