“The future belongs to the companies and people that turn data into products”

Some excellent links and information from O’Reilly Radar…  things are falling into place, even if it is early days and things are not straight forward (yet).

The SMAQ stack for big data:

It is now economic to conduct extensive investigation into data, or create data products that rely on complex computations. The resulting explosion in capability has forever altered the landscape of analytics and data warehousing systems, lowering the bar to entry and fostering a new generation of products, services and organizational attitudes

What is data science?: (Get the full version in PDF format here.)

We’ve all heard a lot about “big data,” but “big” is really a red herring. Oil companies, telecommunications companies, and other data-centric industries have had huge datasets for a long time. And as storage capacity continues to expand, today’s “big” is certainly tomorrow’s “medium” and next week’s “small.” The most meaningful definition I’ve heard: “big data” is when the size of the data itself becomes part of the problem. We’re discussing data problems ranging from gigabytes to petabytes of data. At some point, traditional techniques for working with data run out of steam.

Big business for big data:

It’s not just the marketers that are throwing petabytes of information at problems. Scientists, intelligence analysts, governments, meteorologists, air traffic controllers, architects, civil engineers-nearly every industry or profession is touched by the era of big data.

Also:

and:

“Stream processing” of data to gain faster insight

I’m not an an IBM trip again, but… this sounds like an industry to be in.

• New York Times: I.B.M. Bids $1.7 Billion for Data Company:

Until recently, business intelligence had been seen as mainly a software technology. Information was stored in databases, and the software was sent in to mine the databases for answers…

Today, that approach is often too slow for fast-paced decision-making in both business and science. The traditional model is a rear view mirror. Increasingly, firms want real-time answers and smart predictions about things like buying trends, weather patterns and medical therapies…

The real-time model of business intelligence, though, requires not just software, but a tight integration with hardware… for what is called “stream processing” — parsing data in streams rather than after it is stored in data bases.

Most of the business intelligence companies are also working on so-called in-memory processing systems. Again, the goal is to decipher data on the fly, not after it has been stored in a database. “Massive amounts of data that can be analyzed instantly has tremendous value,” John Schwarz, the former chief executive of Business Objects, said in an interview months ago.

• IBM Snags Netezza in $1.7B Business Intelligence Play:

IBM intends to fold Netezza into its Information and Management software suite, padding out its Business Analytics and Organization Consulting group. Over the past four years, IBM has shelled out more than $12 billion to acquire 23 firms offering analytics products and technology. The company now dispatches more than 6,000 dedicated consultants to assist clients with the deployment of its analytics and BI software to help make sense of the proliferation of business data.

Machete trailer

• Looks even better than Planet Terror… roll on 26 November!

Firms and the fuzz use big data and Facebook to catch crooks and customers

Companies can spot … influencers, and work out all sorts of other things about their customers, by crunching vast quantities of calling data with sophisticated “network analysis” software… The market for such software is booming. By one estimate there are more than 100 programs for network analysis, also known as link analysis or predictive analysis. The raw data used may extend far beyond phone records to encompass information available from private and governmental entities, and internet sources such as Facebook.

• The Economist: Mining social networks: Untangling the social web

The Economist on the future of the internet

The Economist’s technology-related articles are always written and mindful of the audience, and this latest is well pitched indeed.  For the intelligent, interested but not all-knowing reader, this is a balanced view of the risks faced by the open internet that we have today.  The fact that it is a viewpoint from this side of the pond is refreshing.  Good on net neutrality – especially the idea that this would not be such a big issue if only the US had more competition in the telecoms market.  Well recommended.

• The Economist: The future of the internet: A virtual counter-revolution

• The Economist (leader comment): The web’s new walls