We propose a spatial equilibrium model with heterogeneous households holding general non-homothetic preferences over tradable goods and housing. In equilibrium, desirable and productive locations command high housing prices. So long as housing is a necessity, these locations are disproportionately inhabited by high-income earners who are relatively less affected by high housing prices. We clarify how this source of sorting complements other potential sorting forces in spatial equilibrium models, namely, comparative advantage in production and heterogeneous preferences for locations. We show how to measure changes in welfare inequality across income groups in a theoretically-consistent way when housing is a necessity, extending the approach popular in models with homothetic preferences. We use our framework to track the evolution of welfare inequality between college and non-college graduates in the United States between 1980 and 2020. We find that, accounting for change in prices, it has risen by more than nominal wage inequality, even as college graduates increasingly sort into cities with expensive housing over this time period.