I am also interested in this question. Am curious to know what you have tried / found out. I am also a user like you, not a developer.
I have looked at the map of a city and removed “tracks” and “footpaths” etc and reduced the size considerably. I only left primary, secondary, tertiary and a few other highways in. I found that it is indeed faster, by about 10%, and affects fastest more than shortest. It also depends on whether you use CH or not.