Case Study: NCDC

NCDC needs downpours of broadband for weather analysis

Without high-speed broadband connectivity through NCREN, researchers at NOAA's National Climatic Data Center in Asheville would have a difficult time doing what they do best – monitoring the country’s weather.

BUSINESS DESCRIPTION

NOAA's National Climatic Data Center maintains the world's largest climate data archive and provides climatological services and data to every sector of the United States and to users worldwide. The center's mission is to preserve this data and make it available to the public, business, industry, government, and researchers. NCDC was incorporated with all civil weather entities as part of NOAA in 1970. Twelve years later, it was renamed the National Climatic Data Center and has remained housed at the Veach-Baley Federal Building since 1995.

"Broadband connectivity is the life-blood of our center, and we wouldn't be here without it. Right now, we house about 14 petabytes of data. That's the equivalent of two Kindle books for every person on the planet. And, the expectation five years from now is to have about 100 petabytes of data. That is an enormous amount of data."
--Thomas Karl
NCDC Director

CHALLENGE

NCDC has seen exorbitant spikes in data use and storage in recent years. On a typical day, about a terabyte of data comes in to be stored and archived – most in real-time. During major weather events, such as Hurricane Sandy in 2012, data access requests will jump three-to-six times that amount. NCDC recently initiated its Climate Data Record Program to continuously provide objective climate information derived from weather satellite data that NOAA has collected for more than 30 years. This data comprises the longest record of global satellite mapping measurements in the world, and is complemented by data from other sources including NASA and U.S. Department of Defense satellites as well as foreign satellites. For the first time, NOAA is applying modern data analysis methods to this historical global satellite data. This process will unravel climate trends and return new economic and scientific value from the records. In parallel, NCDC will maintain and extend these records. But without reliable, high-speed broadband connectivity, none of this would be possible to tackle in a timely fashion.

"Broadband is absolutely critical to what we do now. If we didn't have broadband, we wouldn't be able to move all this data in a timely manner and get it to researchers who need it … it is absolutely critical to be able to do that now, and broadband is a must to be able to do what we do."
--Alan Hall
NCDC IT Project Manager

SOLUTION

In fall 2011, NCDC received two 10G broadband connections as part of the build-out through the first phase of the of Golden LEAF Rural Broadband Initiative administered through MCNC. The connection enabled was a significant upgrade from two 1G connections that NCDC previously used. One main use of the 10G connections is for the Suomi National Polar-orbiting Partnership (NPP), which represents a critical first step in building next-generation Earth-observing satellite systems. The NPP is the result of a partnership between NASA, NOAA, and the U.S. Department of Defense. Since Oct. 2011, 1.3 petabytes of data has streamed into NCDC for consumption and storage from the NPP. In that process, a copy also is made of all data and is sent as a backup to Colorado. This means since this project went active, more than 2 petabytes of information has traveled over broadband-based pipes to researchers all over the world, again putting huge value on broadband connectivity.

BENEFITS

"As more data comes in and out every day, we need high-speed connections to realize all these data sets. We have better download rates now and as others get more speeds soon, we will be ready as we continue to future-proof our infrastructure."
--Preston Carter
NCDC IT Specialist for Network Operations & Storage

Today, data comes to NCDC from not only land-based stations but also from ships, buoys, weather balloons, radars, satellites, and even sophisticated weather and climate models. In the past 10 years, NCDC’s digital archive experienced a six-fold increase from 1 petabyte to 6 petabytes. With increasing sophistication of data collection equipment, data is expected to exceed 14 petabytes by 2020. The United States has made tremendous investments in Earth-observing satellites over the past five decades. Despite remarkable success, great potential remains in the nation’s archived measurements for climate change applications. NOAA’s new Climate Data Record Project promises to unleash the potential of this data to address critical climate questions. But again, doing this type of work today would not be possible without high-speed, low latency broadband.

At a glance

What they needed?

• Reliable high-speed, low-latency connectivity to store and share unprecedented amounts of data
• Bigger bandwidth/capacity to properly implement and add access to new CDR Program and Suomi NPP
• Larger broadband-based pipes to better utilize new and sophisticated climate data collection equipment

What they did?

• Leveraged NCREN connectivity to keep up with the exorbitant pace of reciprocal data sets worldwide
• Upgraded from two 1G broadband connections to two 10G connections

What they accomplished?

• More people can now access useful climate trends and data to return new economic and scientific value
• Expanded capabilities to be able to transfer important data to researchers in a more timely manner
• Designed the beginnings of a future-proof infrastructure to meet the needs of big data in the years come