Q: Why does the layer norm come before the MHA / before the FFN in Pre-LN?

Troubleshooting in March 2023 on an M2 Mac:

1. I could only get tensorflow 2 running on my M2 Mac, not tensorflow 1 like the setup.py file requires. To do so, I changed the dep from tensorflow to tensorflow-macos and removed the version constraint.

2. I also removed the version constraint for torch.

3. I then also had to change how tensorflow is imported e.g. in run.py:

import tensorflow.compat.v1 as tf tf.disable_v2_behavior()

By making these changes, I was able to successfully run the installtest.