Have you tried Zathura? It’s a ui-less PDF reader with plugins for a lot of different formats, including the ones you mentioned. All navigation is done via keyboard and once you learn the shortcuts it’s a breeze.

Koreader maybe? Calibre is intended for library / e-reader device management, the viewer it has is mostly just for a quick preview look and not really intended for actually reading.