Even if you have enough RAM at container level, you might need to use the -XmxYYg (with YY experimental deduced value) to allow enough RAM for the java process. The amount of RAM needed depends on the /data/merge.osm.pbf as well the config.xml .
Maybe you should try first without docker to get an estimation of the RAM needed.
Other than that you are not stating on which device you are running the docker compose. So if you would run the container israelhikingmap/graphhopper on a raspberry pi or an MacOS M1 for instance it would run emulated and be very sloooooow