I remembered seeing something related to annotating a video in Emacs, but can not quite recall where exactly. Probably one in the series by Uncle Dave. I'll have to re-check. UPDATE: probably just org-emms, you mentioned below.

Another one, Foxit, seems quite feature rich.