Yes, certainly they do. Amazon even did that famous internal study showing that, other factors being equal, a page that loaded 200ms more slowly caused a statistically significant drop in revenue. (Even if end users aren't aware that they care, they care.)