I think he was afraid they would dart down a side-street and out of his life forever.
It is surprising how Tom, who usually dominates every situation, appears genuinely afraid of losing control. His repeated glances back at Gatsby’s car reveal an unexpected vulnerability, as if Daisy might suddenly escape his life altogether.