Just found a thread that confirms this behavior pretty clearly:
As Peter mentioned there, GraphHopper doesn’t support using a single Point geometry inside the areas section of a custom model.
You need to create a small Polygon (or MultiPolygon) around the location you want to affect deally large enough to intersect the road segment you’re trying to penalize or avoid.
That lines up with my tests too tiny polygons work fine as long as they touch the target edge.