Presumably, any State Attorney General will have gone to law school, and will thus know that the Federal Government is immune to suits from the states.
They are not actually immune, states sue the federal government (or at least departments) all the time. Look at the ACA cases for an example. They can also go after the individual people involved as long as they are not serving in the government.
The states can go for a variety of reasons when the feel the federal government is overstepping their bounds or has committed a constitutional violation. They can open investigations into federal behavior.
Who said anything about suing the USG, unless you are already assuming someone contributing to OpenSSL was hired by the USG in some relevant capacity. I don't see what would shield contributors to OpenSSL from a criminal investigation.