How Algorithms & Big Data could predict the Future
We already know from the booming Algo-trading industry that yes, algorithms can indeed predict future events. Whether these events are self-prophecies or actual ‘future telling’ mathematical magic, is a sort of unknown? But, maybe a little of both. Either way, it’s an interesting idea to imagine data and algorithms can be put together to create a picture of our future or specific events that haven’t occurred. Of course, Algo-Trading tends to predict mainly on financial data so it is essentially data predicting data like an actuary or financial analyst. But, why couldn’t it be more than that? Just like “Deep Thought” in Hitchhiker’s Guide to the Galaxy – Why couldn’t there be an all knowing super computer with access to all the data (via internet, intranets, open big data & networks) any additional input we provide and armed with a sophisticated enough algorithm to interpret this data, analyze it’s trends and come up with statistical evidence to support predictions of possible ‘future’ events?
It sounds far fetched and while I pitched the concept a couple of years ago to friends and received mixed feedback. (I called it “Thought Pulse” or “Brain Pulse” a search engine that predicts social events based on network & big data). I feel like the possibility is actually becoming more feasible, with the rise of big data & the amount of activity that occurs & is increasing on the web (think – the world’s stream of consciousness.) The search industry is already in flux between question/answer search engines and search engines that actually interpret input – from natural language processing to aggregated data & probability calculations. One example is the Wolfram Mathematica Search. You can download a trial or use the online version (which is much less powerful) Wolfram Alpha Online Search. At first glance this isn’t ground breaking technology, in fact, I had some trouble with the Wolfram product at first where it didn’t even seem to understand my input. It’s really more the concept that intrigues, taking input and providing responses that go above and beyond what any human could provide – a super computer with access to all the data, calculations, mathematical abilities, satellite networks & people information to provide answers to questions we didn’t even know we could ask.
What would be really interesting about a search engine that provides predictive analysis is it’s ability to move beyond numbers & statistics, to apply the natural language processing techniques of search algorithms to the constantly changing landscape of social networks, news, forums, satellite reads in space, weather patterns, the sentient nature of human language; essentially reading the pulse of the world at any given millisecond and recording it. If the algorithm were created and the data (which is there) could be accessed, then the theory is less of a theory and really more a matter of time. It’s not exactly as magical as it sounds, it’s being done and on a niche-y scale there’s evidence that it’s possible – watch Nicholas Christakis Ted Talk about social data predicting epidemics
I think it would have to start out small, using niche, accessible information and focusing less on the ‘hard-to-gather’ data and more on a smart algorithm. One of the more ‘marketable’ or ‘fun’ ways I thought of creating something like this on a small scale would be predicting social behavior of friends or colleagues. So, you have this engine and you connect all your social networks, blog, etc. so now the engine knows who your fbook friends are, your twitter friends, it has already created a word cloud of your hashtags and interactions, (think Klout but smarter). It even knows readily available ‘big data’ like statistics, demographics, geo-stats, census info, etc. You then have the option to input other digital information (for example a chain of emails between you and a significant other or colleague or boss.) The engine takes the data and applies math, statistics, pattern recognition, natural language processing to answer ‘social’ questions in ways you never thought possible.
A girl sitting at home, day dreaming of her wonderful boyfriend & all the babies they have in their future, wonders what are the odds –
Input >> Will he break up with me?
[Engine takes your interaction w/ said person, their interactions with others on social networks, keywords and natural language processing of emails and any other interactions you’ve had with this person, age, ethnicity, genders, location, parents status (together, divorced), etc. The engine utilizes statistical data as well as pattern recognition from previous queries (it would get smarter with time.) ]
Answer >>“Evidence concludes, He will break up with you in 6 months to 1 year, with 87% confidence. Factors that contributed to conclusion..”
Lots of other more real-world like applications – the easiest is not surprisingly the one that is most similar to algo-trading: business. In algo-trading analysts work in buildings and offices as close as possible to network towers that transmit the information they are analyzing – so wall street, downtown nyc – the first trader that captures the data, passes it through their algorithm & makes the buy/sell/whatever is the most profitable trader. Same with business – like predicting industry trends based on news or internet activity or know in real-time the latest competition or specific events that effect your industry – in order to make profitable business decisions as soon as possible.
Crazy right? Crazy like a fox! Let’s build it, who’s with me.
Other Interesting References