Machine Learning, Algorithms and Unicorns
Most people that talk about algorithm and machine learning have no idea what that even means. This is my conclusion after numerous discussions with startup founders that effusively explain how they are going to build some Über-engine for social gaming personal deal recommendations or other such thing. It is an admirable goal, but I wager there is a better chance of seeing unicorns than seeing something that vaguely matches their vision.
I really enjoyed reading a post recently by Stijn Debrouwere called Machine Learning Fairy Dust. I think fairy dust is an apt term. Having seen what is required in building blackbox trading algorithms over the past few years, I can attest to the fact that the gap between talking about machine learning and actually building a machine learning product are vast.
We are in an age when anyone could build and deploy a web application. Even building mobile apps is getting significantly easier with more frameworks and toolkits available. This is not to diminish the effort, as it requires experience and skillfulness to build something that is rock solid, secure and highly useful. Rather, the hurdle to build websites and apps has decreased over the past decade in terms of money, time and skills required.
The real challenge is in driving the intelligence behind the websites and data models. This means rethinking user interactivity and design to remove the appearance of complexity, understanding implicit user intention, providing relevant contextual recommendations, aggregating massive data sets to identify patterns, etc. All of these are squarely in the realm of mathematics and computational science. Just look at the explosion in job listings for “data scientists” in the past year to understand growth in this area of technology.
Most of what I have seen however in this recent generation of startups however a hopelessly naïve approach towards machine learning. They start out building their Über-engine, only to realize that it is 1) superhard and 2) superexpensive. It is for these very reasons that such brute-force approaches as Mechanical Turk, crowd sourcing and hiring human readers in the third world countries are such popular “hacks” to dealing with the problem of building intelligence into modern websites. Therefore, if you are ready to jump into the startup fray, it is important to understand a few things before you start building anything:
- First, the next wave in web technology is not more niche consumer apps with cool UI’s, it is building the guts of websites incorporating machine learning and driven by big data. Right now we have the dumb web, but very soon we are going to enter the age of the intelligent web. The startups that build these sites are going to wipe all of the “leaders” in this current wave of tech startups off the map.
- Second, if you do decide to build your Über-engine, start with the hack to prove the idea before you go ahead and build anything. As I mentioned before, it takes a lot of time, money, brain power and testing to get these systems right. The worst thing you could do is to put a ton of energy into writing code for something that ultimately does not really work. This is the proverbial lean startup approach.
- Third, once the concept has proven itself, get programmers with a strong mathematical and analytical bent. Many programmers are certainly capable of writing decent code, but it is the rare group that can conceive of complex models and weave them into reality. Essentially, you are looking for artists, not coders. These folks will be essential in developing the next generation of intelligent systems.
10 Notes/ Hide
- machinewisdom-blog reblogged this from marksbirch
- clintonwu-blog reblogged this from marksbirch
- adsahay reblogged this from marksbirch
- liamgooding liked this
- marksbirch posted this