His "If not later, when?" offends Elio because it hints at his looking for a better moment to speak (to speak or to die) and their continuous delaying of approaching the matter at hand by speaking, maybe because speaking will change something between them, because speaking means something.

In this case, Oliver himself is picking at his own scab, as he always says "Later!" Him saying this to Elio is ironic because the two of them are both delaying admitting their true feeling and Oliver is telling Elio to hurry up and confront him, even when all signs say he doesn't want to.

Apricots are compared to "blushing with shame, shame comes with age" and then directly referenced to Oliver's butt. Clearly representing Oliver with age and reference to his butt, so it characterizes Oliver as two things; shameful, and "premature" and "ripen early", which is ironic considering his age -- what does this say about youth, identity, and contradictions?