okay I didn’t set anything up yet. I just read your issue and your discussion with @karussell and Jenkins fits very well in there. You can think of jenkins as a selfhosted alternative to travis It works with github webhooks, it has build queues, you can set up multiple slaves with multiple OS for testing, but you have to set it up yourself.
If you want to try jenkins, just install the jenkins docker image here and create a new freestyle job
There are two different jobtypes at the moment:
- Freestyle jobs are jobs, which you create mostly with the gui. There is no easy readable code behind it or something like that. Most of the time you have one job for all of your branches or you have to create a new job for a new branch.
- Pipeline jobs are jobs, which are defined in code and a new job will be created at the moment of branch creation (or later if you set manually scan timer) and you can define each job seperatly. The advantage of these jobs is, that they follow the source code repository management. Your changed job will follow automatically with merges in your branch etc. So if you have a branch, where you switch for example from maven to gradle you can change the job in that branch and all other branches will be runned with maven, but yours will run gradle. If you merge that branch to master, master will run on gradle, but all other branches will run on maven until you sync them with the master.
You can find out more about it here: https://jenkins.io/doc/book/pipeline/syntax/
The hardest or more intense part would be to set up the log file analyses. As I can see here you have something like that already, so you could reuse. I could imagine a comparison to master branch performance and a report back into the pull request of jenkins with comparisons.
To isolate noise and it depends on the jenkins workload, you can rerun a test X times, create a median or something like that to just eliminate noise. Just some sorts for later here
All in all jenkins just is a tool to run the command you gave it like a shell script, but the infrastructure behind it supports all kind of things and also there is a big plugin support. If you need some special things, there is a big possibility that there is already a jenkins plugin for it.
If I can help you with something, just ask