I believe you misunderstood my comment. Numbers who's origins are tainted by an illegal number can be illegal. Numbers who's origins do not have taint from an illegal number are not. It is not the number that is illegal, it is it's color. How it came into being, it's history, a permanent invisible mark that it carries with it, and transfers to whatever it touches.
You can't just dodge color by mutating the number via mathematical operations, color doesn't work like that. You can only get rid of the taint by using a fresh number without taint.
If there's an illegal number X and I give you a number Y with the intention that you obtain X from it by some transform then Y is also illegal because I "tainted" it when I did the original transformation from X.
If I just need Y (or X) for some operation that's not related to the "illegality" then those numbers are not illegal.
The essay that droffel linked to explains this very well.
It doesn't disregard mathematics at all. Quite the contrary — it separates mathematics from the intent of the mathematician.
The number itself isn't illegal, because it can come from any number of contexts. What's illegal is wilfully working towards generating that number while knowing its purpose as a key. It's just the principle of Mens Rea at work.
Therefore all numbers are illegal.