gaffer
According to the OED, there are multiple potential meanings for this term when this poem was written. It could have been a general term of address, especially for an older man or it could have been a way to refer to a master, governor, or foreman. This stanza itself does not make the distinction clear, but considering the focus in the third stanza on "men of fourscore" (line 23) it could be the first definition; however, a foreman or master would produce the image of control Winter has over the speaker.
https://www.oed.com/dictionary/gaffer_n?tab=meaning_and_use#3371962