Brown trout populations in the Hardanger Fjord, Norway, have declined drastically due to increased exposure to salmon lice from salmonid aquaculture. We studied contemporary samples from seven populations and historical samples (1972 and 1983) from the two largest populations, one of which has declined drastically whereas the other remains stable. We analysed 11 microsatellite loci, including one tightly linked to the UBA gene of the major histocompatibility class I complex (MHC) and another locus linked to the TAP2A gene, also associated with MHC. The results revealed asymmetric gene flow from the two largest populations to the other, smaller populations. This has important conservation implications, and we predict that possible future population recoveries will be mediated primarily by the remaining large population. Tests for selection suggested diversifying selection at UBA, whereas evidence was inconclusive for TAP2A. There was no evidence for temporally fluctuating selection. We assessed the distribution of adaptive divergence among populations. The results showed the most pronounced footprints of selection between the two largest populations subject to the least immigration. We suggest that asymmetric gene flow has an important influence on adaptive divergence and constrains local adaptive responses in the smaller populations. Even though UBA alleles may not affect salmon louse resistance, the results bear evidence of adaptive divergence among populations at immune system genes. This suggests that similar genetic differences could exist at salmon louse resistance loci, thus rendering it a realistic scenario that differential population declines could reflect differences in adaptive variation.