Most people do the same. While it is technically correct that congressman applies to both houses, it’s generally understood that if you’re referring to a member of the senate they’re referred to as a senator rather than as a congressman and that therefore if you say “congressman” you’re implicitly not referring to a member of the senate but a member of the house of representatives.