Seasonal influenza viruses create a persistent global disease burden by evolving to escape immunity induced by prior infections and vaccinations. New antigenic variants have a substantial selective advantage at the population level, but these variants are rarely selected within-host, even in previously immune individuals. We find that the temporal asynchrony between within-host virus exponential growth and antibody-mediated selection make within-host antigenic adaptation rare. Instead, selection for new antigenic variants acts principally at the point of initial virus inoculation, where small virus populations encounter well-matched mucosal antibodies in previously infected individuals. Selection later in infection is rare. Our results explain how virus antigenic evolution can be highly selective at the global level but nearly neutral within hosts. They also suggest new avenues for improving influenza control.