Asking the question posed in the title of this post – Just how amazing is data generation and analysis of one exabyte a day? – seems somewhat premature when you think that it is not even possible today efficiently.
The question came up in my mind when I read the following article and associated press release:
• Future telescope array drives development of exabyte processing (Ars Technica)
• IBM’s press release (direct link to PDF)
Here are some quotes:
…the proposed Square Kilometer Array (SKA) of radio telescopes promises to push well beyond the current computing ability of the entire planet
The initial investment… will be used to investigate the types of new processors, power supplies, storage systems, and networking technology necessary to handle the amount of data needed.
…the challenge is not merely to collect and store the data, but to process at least some of it in real time.
While handling data on this scale is possible with current technology simply by brute force, the energy requirement is prohibitive. Thus, the DOME collaboration will investigate new processors, optical networking techniques, and fast storage methods that are energy efficient.
From the press release:
Scientists estimate that the processing power required to operate the telescope will be equal to several millions of today’s fastest computers.
The next generation of large scientific instruments, of which the SKA is a key example, requires a high-performance computing architecture and data transfer links with a capacity that far exceeds current state-of-the-art technology.
This is Big Data Analytics to the extreme.
Only by basing the overall design on architectures that are beyond the current state- of-the-art will it be possible to handle the vast amounts of data produced by the millions of antenna systems of the SKA.
This is a massive undertaking that will benefit all of computing. However there is one sentence in the article that changed my impression of the project:
The projected date for the start of full operations is 2024.
This prompted me to ask another question: Just how amazing will data generation and analysis of one exabyte a day be in 12 years’ time?
The only credible answer I can give at the moment: much less remarkable than it is today.