Quantitative genetic models are used to investigate the evolution of generalists and specialists in a coarse-grained environment with two habitat types when there are costs attached to being a generalist. The outcomes for soft and hard selection models are qualitatively different. Under soft selection (e.g., for juvenile or male-reproductive traits) the population evolves towards the single peak in the adaptive landscape. At equilibrium, the population mean phenotype is a compromise between the reaction that would be optimal in both habitats and the reaction with the lowest cost. Furthermore, the equilibrium is closer to the optimal phenotype in the most frequent habitat, or the habitat in which selection on the focal trait is stronger. A specialist genotype always has a lower fitness than a generalist, even when the costs are high. In contrast, under hard selection (e.g., for adult or female-reproductive traits) the adaptive landscape can have one, two, or three peaks; a peak represents a population specialized to one habitat, equally adapted to both habitats, or an intermediate. One peak is always found when the reaction with the lowest cost is not much different from the optimal reaction, and this situation is similar to the soft selection case. However, multiple peaks are present when the costs become higher, and the course of evolution is then determined by initial conditions, and the region of attraction of each peak. This implies that the evolution of specialization and phenotypic plasticity may not only depend on selection regimes within habitats, but also on contingent, historical events (migration, mutation). Furthermore, the evolutionary dynamics in changing environments can be widely different for populations under hard and soft selection. Approaches to measure costs in natural and experimental populations are discussed.