Users that care, or even notice, are so few that you are noise compared to the number of users that will leave if the site loads slower, so the CDN is likely to be a net win many times over.