The $h$-version of the finite-element method ($h$-FEM) applied to the high-frequency Helmholtz equation has been a classic topic in numerical analysis since the 1990s. It is now rigorously understood that (using piecewise polynomials of degree $p$ on a mesh of a maximal width $h$) the conditions "$(hk)^p ρ$ sufficiently small" and "$(hk)^{2p} ρ$ sufficiently small" guarantee, respectively, $k$-uniform quasioptimality (QO) and bounded relative error (BRE), where $ρ$ is the norm of the solution operator with $ρ\sim k$ for non-trapping problems. Empirically, these conditions are observed to be optimal in the context of $h$-FEM with a uniform mesh. This paper demonstrates that QO and BRE can be achieved using certain non-uniform meshes that violate the conditions above on $h$ and involve coarser meshes away from trapping and in the perfectly matched layer (PML). The main theorem details how varying the meshwidth in one region affects errors both in that region and elsewhere. One notable consequence is that, for any scattering problem (trapping or nontrapping), in the PML one only needs $hk$ to be sufficiently small; i.e. there is no pollution in the PML. The motivating idea for the analysis is that the Helmholtz data-to-solution map behaves differently depending on the locations of both the measurement and data, in particular, on the properties of billiards trajectories (i.e. rays) through these sets. Because of this, it is natural that the approximation requirements for finite-element spaces in a subset should depend on the properties of billiard rays through that set. Inserting this behaviour into the latest duality arguments for the FEM applied to the high-frequency Helmholtz equation allows us to retain detailed information about the influence of $\textit{both}$ the mesh structure $\textit{and}$ the behaviour of the true solution on local errors in FEM.