> Outside of finance I am familiar with many tech companies which (openly or otherwise), kickstarted what are now widely known enterprises through cleverly crawling or scraping massive amounts of data.
Google comes to my mind. I'm only partly joking, as it seems to me that the line between search engine web crawling and other forms of web scraping is very thin.
It's more than thin: just scrape a few more sites, make a public search interface and say you are working on a search engine. In the meantime, you might analyze the data for other (not so) related purposes.
Google comes to my mind. I'm only partly joking, as it seems to me that the line between search engine web crawling and other forms of web scraping is very thin.