In its early stages Rust used to employ a lot of sigils (people joked that its a language which makes full use of all the keys on the keyboard), but the language has moved away with the cryptic operators and IMHO looks cleaner.
AFAIK, the sigils were used to express differences in data types (wrt to memory allocation scheme), which is quite different from ASCII operators. This doesn't bother me nearly as much, but it may be my Perl background.
You are right, they were mainly around pointers. As long as they are a handful and their meaning doesn't change (as opposed to using in a DSL), its easy to get used to them. Still, I find the explicit keywords a familiar territory and better to read.