The IdentitiesOnly yes is required to prevent the SSH default behavior of sending the identity file matching the default filename for each protocol. If you have a file named ~/.ssh/id_rsa that will get tried BEFORE your ~/.ssh/id_rsa.github without this option.

Try overriding this behaviour with this at the bottom of your .ssh/config file: Host * IdentitiesOnly yes

Basically, specifying IdentityFiles just adds keys to a current list the SSH agent already presented to the client.