Yes, you can combine the geocoding from the Directions API (unrelated to custom encoded values etc.) and the routing from your local server.
Are you using GH 6.x (with the olds map client) or the current master branch (with the new maps client)? For 6.x you can specify different hosts for geocoding and routing as I explained here: Regarding Hosting the Graph Hopper on Private Server - #4 by easbar, but for latest master maybe we need to make the configuration more flexible to do this first.