So i went ahead and did a test to make sure it was working. I spun up a passphrase free key and tested via
% docker run -dit -v /home/baz/.ssh_keyfree/:/rally/.ssh elastic/rally bash
% docker exec -it ddeb184e7c8c bash
rally@ddeb184e7c8c:~$ git clone git@github.com:elastic/rally-tracks.git
Cloning into 'rally-tracks'...
remote: Enumerating objects: 4225, done.
remote: Total 4225 (delta 0), reused 0 (delta 0), pack-reused 4225
Receiving objects: 100% (4225/4225), 1.20 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2898/2898), done.
And then tried again without importing the key directory and from in the container got
rally@442eb19e3a8c:~$ git clone git@github.com:elastic/rally-tracks.git
Cloning into 'rally-tracks'...
The authenticity of host 'github.com (140.82.114.3)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? ^C
Im running all this on linux. A colleague mentioned there is a long standing issue w/ macos detailed here that could be causing your issue, if you are on macos.
But rest assured that this does work in a linux+docker environment with the rally container, as per the example above.