It can still be the cheapest option even after pricing in the externality. A lot of people forget this, because they advocate such Pigovian taxes with the intent of stamping out the good, irrespective of the (edit: net) value it provides.
I very rarely see anyone come in with the attitude of: "Hey, burn all the gas you want once we have the appropriate carbon taxes!"
One problem with environmental degradation is that effects are often non-linear whereas taxes are typically linear. So "do all the X you want" is not the appropriate response if the tax is simply used as a lever to move behaviour in one direction.
On the other hand, if the government is selling emission rights then this can model non-linear effects. Then it makes sense to say, "go on, use your rights to the full extent."
If you won't quantify how bad the damage is and how much you'd need to erase it, and therefore what at what tax rate you'd be fine with someone consuming to their heart's content, then you think the cost is infinite and therefore have no business proposing policy to begin with.
"Someone consuming to their heart's content" is not bounded.
It seems to me you're arguing that just because a tax is a blunt tool it shouldn't be used at all. But tax rate doesn't have to be set in stone, it can be adjusted dynamically in a feedback loop.
No, I'm saying that advocates should be clear about what the precise harms are and that taxes should be used to ameliorate the harms of the taxed activity, and that anyone unable to put such numbers on it is not advocating a serious policy we should listen to.
I very rarely see anyone come in with the attitude of: "Hey, burn all the gas you want once we have the appropriate carbon taxes!"