At least traditionally, the driver had direct physical control over the vehicle through hydraulics. Obviously this has become less true over the past couple of decades with the introduction of electronic systems, but the traditional imprint remains in the popular mind.
(Truly) self-driving cars are, inversely, completely outside the control of the "driver". Anyone who has considered it even briefly has realized that governments will inevitably mandate lockout systems preventing people from using their own car if they are a wanted felon/drunk/a dissident. Such restrictions are completely impractical on traditional vehicles.
They're already in use in the case of repeat drunk drivers and it's not a stretch to imagine similar technology in other ways with traditional vehicles. Self driving is completely orthogonal to this concern.
(Truly) self-driving cars are, inversely, completely outside the control of the "driver". Anyone who has considered it even briefly has realized that governments will inevitably mandate lockout systems preventing people from using their own car if they are a wanted felon/drunk/a dissident. Such restrictions are completely impractical on traditional vehicles.