To effectively fuse the multi-view information, we propose a geometrically-guided projective attentionmechanism. Instead of applying full attention to densely aggregate features across spaces and views,it projects the estimated 3D joint into 2D anchor points for different views, and then selectivelyfuses the multi-view local features near to these anchors to precisely refine the 3D joint location. wepropose to encode the camera rays into the multi-view feature representations via a novel RayConvoperation to integrate multi-view positional information into the projective attention. In this way, thestrong multi-view geometrical priors can be exploited by projective attention to obtain more accurate3D pose estimation.