The most visible difference between adventure games and walking sims is the removal of puzzles
While I agree there are fewer physical challenges in a walking sim than an adventure game, I think the factor of predetermined choices/option is a big difference. These choices refer to more than typical multiple-choice answers, but also choices to change spaces and to do certain actions. In a walking sim, the player is free to do whatever. There's no need to record one's input in order to build the algorithm and advance in the game. A player can explore as many or as few areas of the game and can still make progress and finish the game.