We present the Fourier lightfield multiview stereoscope (FiLM-Scope). This imaging device combines concepts from Fourier lightfield microscopy and multiview stereo imaging to capture high-resolution 3D videos over large fields of view. The FiLM-Scope optical hardware consists of a multicamera array, with 48 individual microcameras, placed behind a high-throughput primary lens. This allows the FiLM-Scope to simultaneously capture 48 unique 12.8 megapixel images of a 28×37 mm field-of-view, from unique angular perspectives over a 21 deg×29 deg range, with down to 22 μm lateral resolution. We also describe a self-supervised algorithm to reconstruct 3D height maps from these images. Our approach demonstrates height accuracy down to 11 μm. To showcase the utility of our system, we perform tool tracking over the surface of an ex vivo rat skull and visualize the 3D deformation in stretching human skin, with videos captured at up to 100 frames per second. The FiLM-Scope has the potential to improve 3D visualization in a range of microsurgical settings.