have to remember what the remote and branch are that I'm using
This is basically because under the hood, subtree uses branches to track entirely unrelated histories with an offset (prefix). This is the same as creating a new branch whose first commit (trunk?) is a clone of an entirely different remote, and then merging that branch into the main branch using a git subtree merge strategy with an offset=prefix.
This is at least git-like in the background but for some reason, the branches aren't uniquely ID'd like git can do with remote branches vs regular branches.