The amazing amount of new data provided by next generation facilities will enable astronomers to tackle truly transformational science. Think along the lines of formation and evolution of the first stars and galaxies after the Big Bang, the role of cosmic magnetism, the nature of gravity, and possibly even life beyond Earth.
This data-intensive era of astronomy, however, presents a number of challenges. The enormous data volumes, rates of acquisition, and inherent data complexity are making new demands on data management, processing, and user access. More fundamentally perhaps, connecting researchers to their data and preserving the adaptive analysis paradigm at the core of traditional astronomical research is one of most challenging aspects of Big Data Astronomy.
In an era when even single datasets can span petabytes, the traditional scientific analysis paradigm is evolving rapidly. For many scientists, the standard analysis scenario involves a moderately sized dataset, obtained from a given facility, and analyzed personally on their individual workstation or other local facilities. This “single desktop” analysis paradigm allows astronomers to improve their results by additional processing or reprocessing, to correlate and compare with other existing data, and to explore interesting features all in an adaptive process that combines both pre-defined and interactive elements.
At exascales, the introduction of the high-performance computing infrastructure required to deal with the scale of the data can break this paradigm and make it difficult for researchers to engage with their data in familiar ways, thereby slowing or complicating the discovery process. One of the main goals is to provide researchers with a flexible and familiar analysis environment that can handle the extremes of exascale astronomy.
Michael Wise, project leader radio astronomy