The fact that you are opening a cabinet and picking up an object and rotating it isn’t important to the game world… What’s important is that through affecting the game world, you’re creating (p. 131) your understanding of it.
Giving the player agency through world interaction isn't about allowing them to have an impact on the world but rather allowing the player to develop their own understanding of the world.