This seems like a field that could pretty easily be commodified. I can imagine a service like the Google prediction API could meet the needs for this kind of tech for many companies.
So while it's certainly an interesting field, I wonder how many hackers are really going to need these skills.
The biggest problem with machine learning occurs when people subscribe to the belief that it's a black-box solution. The truth is that you can't just drag-and-drop your data into a pre-existing solution. The types of algorithms you use depend on the types of problems you're trying to solve (e.g., classification, regression, clustering). The data you collect depends on the algorithms you use.
Sure, prediction APIs could arise that give detailed use cases for each algorithm, but then there's a problem with the fringe cases: you might not know that two pieces of data are so heavily correlated that they completely shatter a conditional independence assumption, for example.
As a hacker who originally subscribed to the belief that a thorough understanding of machine learning was overkill, it is without hesitation that I admit being 100% wrong. The truth of the matter is that when it's done properly, artificial intelligence and machine learning ought to be inextricably linked with your core business processes.
Agree: I had a dataset for work no one had yet been able to use in categorizing two effects (one category was 98% of all the data). The values looked too "Gaussian normal" with everything mixed up. It couldn't be separated out, but a combination of SVM and in dept knowledge of the source of the data and I was able to find a generalized model that could accurately categorize parts 80%+ of the time for the small set, without misclassifying the other 98%. All other methodologies had failed up to that point and a blind approach with linear regression or SVMs resulted in at best 70% accuracy on all categories... not very good or implementable in a production setting (that means in the bulk of cases the 98% I was only correct 70% of the time).
I can certainly see a role for somebody that understands the tradeoffs of each of these algorithms and that understands how to properly select and prepare dataasets. But I wonder how many people will really need to be able to actually implement these algorithms.
I think the regress you're talking about is super important---black box AI only goes so far---but I also think there's great benefit to just applying the first layer of broken, incorrectly paired ML to a new field.
My prediction is that even the most black box ML, creatively applied, is and will be an incredible skill. Increasing levels of sophistication will continually kill off the current practices of black box ML, but the willingness to apply statistical pattern recognition to new and interesting areas can't help but be incredible.
As someone studying this intensely, it's quite the opposite. Basic ML can (and has) been commodified with good toolkits and APIs. Additionally, much of practical ML is just applications of already invented algorithms to fields that just haven't even seen them yet.
But that said, the deeper message is in interpretation and discovery from data. Large data, small data, highly structured data, or just regularized DB pulls. The heart of it is statistical pattern recognition and it's really just begun to be broached (even academically) in the last 25 years.
I respectfully disagree. Tools like Weka, nltk, etc. are okay for exploratory data analysis, but it's risky to use them for problems that scale, problems that differ from the norm, or homegrown solutions for data that does not yet exist. Because a large portion of HN users are interested in bringing their ideas to life, I'd suspect that the latter particularly resonates with them.
The problem facing people who intend to work with data that does not yet exist becomes one of feature selection: what data matters and how do we use it? For NLP tasks, does stemming matter? What about part-of-speech tagging? Some classification problems are not linearly separable, which makes certain kernel methods impossible without using (and knowing to use) the kernel trick.
In the end, I think my reply here is tautological: ML is too complex to be transformed into a set of APIs a la Google Maps and Google Search.
The problem facing people who intend to work with data that does not yet exist becomes one of feature selection: what data matters and how do we use it? For NLP tasks, does stemming matter? What about part-of-speech tagging?
Indeed, I worked on machine learning in NLP (fluency ranking, parse disambiguation). As a general rule, roughly 90% of the improvement of models is in clever feature engineering and exploiting the underlying system to get more interesting information that improves classification, 10% you get from using more clever machine learning techniques than, say a standard maxent learner with a gaussian prior (for linearly separable data).
For instance, the last relatively large boosts of the accuracy of the parser developed by our research group came from feature engineering:
I think we just disagree on what "basic" ML means. I think a lot of real problems have solutions which involve very simple applications of poorly tuned ML algorithms.
Engineering even a basic ML solution is challenging---feature engineering especially.
Actually, Google Prediction API is very simple and it covers supervised learning (regression and classification) already. I can imagine very simple extensions (of the API itself, the algorithms would be completely different) to cover a lot of the unsupervised and semi-supervised ground as well.
The algorithms are not disclosed, but the docs hint that they are properly regularized so throwing more features at them is always good.
You still need to be able to reformulate the problems so that they fit a standard ML setting and then know how to tune things, but it looks like the API can get you pretty far.
So while it's certainly an interesting field, I wonder how many hackers are really going to need these skills.