I already addressed this in my original post. It seems obvious that the displacement of workers would hurt the economy. However, it does not seem to actually happen -- the cities with the highest housing prices have the highest growth of any cities in the world.
"That which is not seen" and all that...