It's actually more complicated than that. CPython will cache constants within a scope, so
a = 500
b = 500
a is b
will give `True`. But the REPL forces each line to compile separately and CPython won't cache across them. However, there's a global cache for integers less than 256 so in that case they're `is`-equivalent even in the REPL.
`is` means "do these things reside in the same location in memory?", and the runtime is merely required to give a plausible answer.
For inequivalent values or `True`, `False` and `None`, the result is defined. Identity is also preserved across assignment. For everything else, the answer is "whatever the runtime feels like". PyPy, for instance, will always say True for integers and floats.
It's not just hard to predict - it's inherently unpredictable. The runtime gets free reign as long as you can't prove that it's lying.