United States' NOAA: Opening Up Global Weather Data in Collaboration with Businesses

Opening up weather data through the United States National Oceanic and Atmospheric Administration (NOAA) has significantly lowered the economic and human costs of weather-related damage through more accurate forecasts; the development of a multi-billion-dollar weather derivatives financial industry; and the growth of a million-dollar industry of tools and applications derived from NOAA’s real-time data. In many ways, the industry built around NOAA’s weather data is seen as the paradigmatic example of how the release of open data can yield major economic impacts. To further scale the impact of its data, NOAA has launched the Big Data Project (BDP), which provides an opportunity to combine NOAA’s tremendous volume of high-quality environmental data and advanced data products, private industry’s vast infrastructure and technical capacity, and the U.S. economy’s innovation and energy.

• The impacts of a given open data set can span an incredible array of sectors and users – with NOAA’s real-time data able to both help an individual decide whether or not to bring an umbrella on her commute or enable a farmer to better prepare for this season’s crop yield, to name just two.
• NOAA collects over 3.5 billion weather observations per day. Around 96 percent of the U.S. public obtains 301 billion forecasts each year, providing $31.5 billion in benefits, far exceeding the $5.1 billion spent annually by both private and public weather bureaus on generating forecasts.
• These estimated benefits are available because of NOAA’s use of a variety of quantitative estimates of the value of its products and services using the Value of Information (VOI) methodologies. This focus on assessing the impacts of information on decisions and on the effect of those decisions on real-world outcomes sets NOAA apart from many other open data initiatives in terms of quantified impacts.
• Collaborations between public data providers and private industry actors can create new value and opportunities from open data.
• Crowdsourcing ideas for how to best make use of open data could help to ensure that data owners build fruitful partnerships and/or release data in ways that can create maximum value without requiring maximum resource investment.

The National Oceanic and Atmospheric Administration (NOAA) traces its roots to 1807 as the Coast and Geodetic Survey.2 Over time, numerous other scientific and environmentally focused federal agencies formed. In 1970, President Nixon proposed the creation of NOAA to “unify the nation’s piece-meal environmental activities and provide a rational and systematic approach to understanding, protecting, developing and enhancing the total environment.”3 Today, NOAA is housed under the U.S. Department of Commerce, and its stated mission is “to understand and predict changes in Earth’s environment and conserve/manage coastal and marine resources to meet the nation’s economic, social and environmental needs.”4

With an annual budget over $5 billion, NOAA deploys weather-monitoring satellites and high-tech instrumentation and sensors to collect large amounts of climate and environmental data.5 Today, its operating units include the National Weather Service (which conducts forecasts, and broadcasts severe weather watches and warnings), the National Environmental Satellite and Data Service (which operates satellites and the atmospheric data center), the National Ocean Service (which performs hydrographic, geodesy and maritime services), the National Marine Fisheries Service (which manages commercial fisheries and protects marine species) and the Office of Oceanic and Atmospheric Research (which conducts applied research).6

Data is at the heart of NOAA’s mission and central to virtually all its activities. As of 2014, “about 100 petabytes of environmental data currently are stored in NOAA data centers,” with another 30 petabytes added every year.7 For perspective, 1 petabyte = 1 million gigabytes; or about the equivalent of 20 million four-drawer filing cabinets filled with texts, or 13.3 years of HD-TV video. The “entire written works of mankind from the beginning of recorded history” is about 50 petabytes.8 Over 3.5 billion observations are collected each day9 from over 90 operational and research observing systems as well as 100 real- and near-real-time information systems, including: satellite data, regional monitoring (e.g., coastal, arctic), model data and archives, ocean profiles, climatology, etc.

The resulting information underlies huge swathes of economic, social and political life in America (and, indeed, around the world). Among other activities, they inform weather forecasts, climate predictions, ship and aircraft navigation, and conservation work involving marine populations. Thus NOAA’s data helps guide decisions that are both momentous (e.g., how much electricity to produce on a given weekend) and more trivial (e.g., should I make outdoor plans this weekend?).

NOAA also plays a significant role in limiting losses from natural disasters by working with the Federal Emergency Management Agency (FEMA). In this partnership, NOAA doesn’t simply provide data. It is also involved in the analysis of information, converting raw data into an “index of severity” that helps guide FEMA’s planning for and responses to disasters.

NOAA’s data is widely consumed by actors in both the private and public sectors. In funding and supporting NOAA, and especially in opening up its information systems to private and public actors, the United States government is performing a vital social, economic, political and cultural service that could not easily be replaced by private sector actors. The data that NOAA publishes is best handled by a public sector entity for a variety of reasons: a) it is non-exclusive (anyone can observe and record the environment); b) it is non-rival (one actor using that data does not subsequently make it any less useful to others); c) it requires a high cost of the infrastructure (expensive scientific equipment); and d) it is reproducible at near zero-marginal cost (for most purely information goods, once produced, the marginal cost of redistribution is close to zero; therefore they cannot be created and produced by firms that use revenues obtained from sales to cover costs. For firms to make revenue, there must be a value-added service, as seen in The Weather Channel reformatting NOAA data into concise, localized formats for TV, websites and mobile phones.).10 As NOAA CIO Zach Goldstein describes, “It’s our job to get that data out there. The data doesn’t belong to us, it belongs to the American people.”11

Since its inception, NOAA has boasted a strong open data culture and is considered a leader in open data, if not the leading open data example among government agencies. When the Obama administration launched data.gov as part of its flagship Open Government Initiative in January 2009, NOAA was cited as the paradigmatic example as to how government agencies can both publish data and make that data accessible for the private sector to use and build a multi-billion-dollar industry.12

It follows that NOAA’s data products and services have always been user-driven, determined by demand among consumers and citizens who have a need for weather-related information. Demand has been particularly strong from certain sectors that are highly affected by weather events (e.g., energy, agriculture, water resources and emergency management). These sectors have in many ways helped shape the way NOAA provides data, and have been at the forefront of pushing the agency to open up its information pool and services. As NOAA Data Management Architect Jeff de La Beaujardiere explains, if NOAA measures something to predict the weather, users will get their weather forecast for tomorrow, but those observations might have other uses, like long-term forecasting for planting crops. “The broader societal objective is to make sure that data gets out there so that as many people and companies and stakeholders can get to them and make decisions based on the data, and make new information products based on the data they find.”13

In its early years, NOAA collected its data in a proprietary format, making it difficult for private users or companies to download and manipulate it independently. This primarily was due to technical limitations of the time. NOAA otherwise, culturally and policy-wise, has always prioritized making its data as accessible to users as possible.14 When NOAA first began developing a centralized open data portal in the early 2000s, its goal was to make the portal as technologically “cutting edge as possible.”15 In particular, the organization hoped to transition from being simply a clearing house for raw data to “actually a menu of html and formats to be pulled directly into applications,” in response to user demand.16

As advances in data technology proliferated, and the sophistication of users continued to increase, NOAA recognized the need for more standardized goals and policies regarding its relationship with users. In 2003, The National Research Council (NRC), the working arm of the U.S. National Academies, therefore conducted a study examining the respective roles of government, academic and private sectors in the weather industry. The study found that “advances in science and technology have blurred the distinctions between the sectors,” and provided recommendations on how “partnership can effectively move forward in an era of rapid advances.”17 The NRC identified the need for a policy that would “define processes for decision-making rather than defining roles.”18

In response, in 2004 NOAA announced a new partnership policy, which included the following provisions:19

• NOAA will adhere to applicable law regarding government information, based on the premise that government information is a valuable national resource and the benefits to society are maximized when such information is available to all.
• NOAA will carry out activities that contribute to its mission and provide open and unrestricted access to publicly funded information at the lowest possible cost.
• NOAA will provide information in forms accessible to the public as well as the underlying data in forms convenient to additional processing.
• NOAA will give due consideration to the ability of private sector entities and the academic and research community to provide diverse services and will consider the effects of its decision on the activities of these entities, to serve the public interest and advance the environmental information enterprise as a whole.
• NOAA will not haphazardly institute significant changes in existing information dissemination activities, or introduce new services without first carefully considering the full range of views and capabilities of all parties.
• NOAA will use appropriate mechanisms to encourage input from and collaboration with others on decisions affecting the environmental information enterprise.
• NOAA will promote the open and unrestricted exchange of environmental information worldwide.
• NOAA’s participation in the environmental information enterprise will be based on the principles of mission connection, consultation, open information dissemination, equitable dealings and recognition of the role of others.

These provisions were designed to help “nurture the growth of a complex and diverse environmental information enterprise and to serve the public interest by giving our nation the best environmental information services in the world,” as described by Conrad C. Lautenbacher, undersecretary of commerce for Oceans and Atmosphere and NOAA administrator at the time.20 Accordingly, in 2004, NOAA changed its data format to the widely used XML, significantly lowering barriers to data use and expanding the base of potential information consumers tremendously. These actions set the stage for NOAA to be a leader in open data policies, and more specifically, for the NOAA Data Catalog that is available today (data.noaa.gov/).21

Figure 1: NOAA’s Geoportal

Today, NOAA data is archived at the National Centers for Environmental Information (NCEI), which “is committed to full and open data access in support of its community of Data Producers and Data Consumers.” NCEI archives data in collaboration with a number of international and national organizations “dedicated to the exchange and open access of ocean- and climate-related data.” NOAA continues to work hard to make its forecasting information more relevant and precise, corresponding to advances in technology and in user feedback on data service or product needs. Once a service has been developed, NOAA promotes it through formal and informal avenues, such as its Web channels and list-servs.22

NOAA’s early attempts to use standard and modern Web services to distribute the data, as opposed to “dumping it all on an FTP site,” helped reinforce the perception of NOAA as an open data innovator, both externally and within internal culture.23

Figure 2: Oceanography Data Sets on NOAA’s Data Portal

Data sets and data products on the NOAA portal are organized:

• by categories
• Satellite Data
• Regional Products (e.g., Harmful Algal Blooms Observing Systems, Arctic Regional Climatology, Gulf of Mexico Data Atlas)
• Observational & Near-Real-Time Data
• Instrument Types (e.g., Acoustic Doppler Current Profiler Data, Buoy Data)
• Model Data (e.g., NOAA Operational Model Archive and Distribution System)
• Ocean Profile Data
• Ocean Climatology
• Project Data (e.g., Coral Reef Information System, Joint Archive for Sea Level)
• International Ocean Atlas Series (e.g., Climatic Atlas of the North Pacific Seas 2009)
• by data types (e.g., Temperature, Oxygen, Ocean Currents, Waves)
• by specific data services (e.g., Live Access Server, HTTP)24

However, of the 30 petabytes of NOAA data collected and archived every year, only about 10 percent is openly available for public access on its websites.25 Again, this is due to the technical challenges inherent in providing access to the 20 terabytes of data produced daily,26 rather than cultural or policy barriers. Still, NOAA’s home page, noaa.gov and the more weather-specific data site, weather.gov, are the most visited websites for the U.S. federal government overall.27 The average citizen looking for basic information about weather, such as beach forecasts or severe weather alerts, knows to go to those two sites, or to climate.gov for more detailed climate research data. Indeed, climate data has become “very important” to NOAA’s communications with both the public and the private sector. Officials are learning that the demand for this data is very high and the expectations surrounding the access, availability and products from NOAA data are increasing.28

Industry users (e.g., utilities, energy, agriculture, space) usually access NOAA data through a more advanced open data application called the Geoportal, which attempts to centralize access to distributed geospatial data, tools, applications and services, allowing users to search and access data via a master catalog.29 However, NOAA recognizes that much of its data remains siloed, which can make searching difficult for all users, and therefore launched a website redesign in late 2015, described in more detail below.

Users can directly use these sites and data sets on their own, or they can partner with NOAA to develop applications and other products to access and analyze the data to fit their specific needs. Some of these partnerships are listed below. These partnerships have changed NOAA’s own perceptions of its data, nudging it to improve its services and the quality of the information and tools on its site. Overall, say NOAA insiders, partnerships have greatly expanded the scope and the potential of the organization’s data. As Tim Owen, chief of the NOAA Climate Information Services Division, puts it: The move toward openness is “as much of a cultural change as it is a policy change. ... Open data brought collaborative channels to work that haven’t worked before, or weren’t even possible to work 10-15 years ago.”30

Private partnerships have also helped shift the internal culture to be more willing to allow industry and economic forces to find new ways to publish and use the data, with NOAA filling in the gaps as necessary. However, NOAA firmly believes that its data is a public good, open and already paid for by the taxpayers, and so, as NOAA CIO Zach Goldstein explains, “NOAA will always focus on advancing science and fulfilling its duty to serving the public interest.”31

NOAA Partnerships32,33,34,35

An example of NOAA’s partnerships can be seen in the Climate Data Initiative, launched in March 2014 by the Obama administration with the goal of leveraging the government’s data resources to stimulate innovation and entrepreneurship in response to climate change. As a part of the president’s Climate Action Plan, the Climate Data Initiative launched a number of significant commitments from federal agencies (including NOAA), private-sector collaborators and research institutions to combat climate change through data-driven innovation.36

There are many ways to measure the impact of NOAA’s data and its data portal. One fairly conventional way is simply to look at data consumption – to count visitors to the site and data downloads. NOAA officials remark that these usage numbers reveal an increase over time, as user interest in and sophistication with data increases, as does the amount of data and products NOAA offers.37 Tim Owen, chief of the NOAA Climate Information Services Division, explains that open data has “raised our awareness of the level of sophistication of the use-community and in turn forces the government to step up the response.”38

Economic Benefits

However, data consumption does not on its own provide an accurate indicator of value created. What matters is not how much data users download or analyze, but what they do with that data. More sectors of the U.S. economy are recognizing the impacts of weather, water and climate on their operations and are becoming more sophisticated at using weather-related information to make better decisions. NOAA’s National Weather Service (NWS) collects environmental information and provides services for other governmental agencies, emergency managers, the private sector, the public and the global community. NOAA’s National Ocean Service (NOS) provides ocean and coastal science, tools and services to address threats to coastal areas such as climate change, population growth, port congestion and contaminants in the environment, all working toward healthy coasts, coastal populations and coastal economies.39

To better understand the impact of its data, NOAA has made a number of quantitative estimates of the value of its products and services using the Value of Information (VOI) methodologies. These methodologies are based on the premise that the information informs decisions, and that these decisions have real-world economic outcomes.

Several methodologies can be employed to estimate VOI, including:40

• The first is to model and compare the outcomes of decisions taken with information vs. those taken without information.
• The second is to ask stakeholders (e.g., those who have taken decisions using information) to assess the value of the information they used – i.e., a self-assessment methodology.
• The third is by comparing data from actual events – i.e., observed effects of, for example, weather phenomena with and without forecasts or warnings.41,42

Based on these various methodologies (and sometimes a combination of them), several estimates have been made of the VOI of NOAA’s data products:43,44

• NOAA real-time data supplies a burgeoning private weather service industry with well over $700 million in value added annually.
• The United States’ $8-10 billion and growing annual Weather Derivatives financial industry relies on NOAA’s seasonal weather data and records.
• NOAA’s forecasts and warnings and associated emergency responses result in $3 billion in a typical hurricane season.
• United States electricity generators save $166 million annually using 24-hour temperature forecasts.
• Economic benefits of new investments in United States coastal ocean observing systems from improved coastal marine information is estimated at over $700 million annually.
• Benefits to world ship routing from NOAA polar satellite data is estimated at $95 million annually.
• Every dollar invested in mitigation of storm-surge effects on coastal communities saves the U.S. taxpayer four dollars in losses from natural hazards.
• Installation of NWS Doppler radars reduced tornado fatalities and injuries by 40 percent from the levels in late 80s and early 90s.

Other studies have attempted to quantify the value of accurate weather data over the years. Notable findings include:

• The agriculture industry has always been among the most important consumers of NOAA’s products. By providing more accurate climate forecasts and more timely warnings of adverse weather, NOAA has helped the industry improve decision-making and crop yields. According to one estimate, data provided by the NWS Climate Prediction Center – a component of NOAA45 – has benefited U.S. agriculture by over $460 million by helping guide planting decisions in El Niño, normal and La Niña years.46
• The U.S. electricity industry is estimated to save $166 million every year due to better forecasts that allow it to estimate and plan for demand. Each percentage point of improvement in the quality of forecasts saves an additional $1.4 million; each 1°C improvement results in approximately $59 million of additional savings annually.47
• According to a nationwide survey, 96 percent of the U.S. public obtains, either actively or passively, 301 billion forecasts each year. Based on an average annual household value of $286 placed on weather information,48 the American public collectively receives $31.5 billion in benefits from forecasts each year. These benefits far exceed the $5.1 billion spent annually by both private and public weather bureaus on generating forecasts.49
• Data from NOAA (and other government agencies) has also created value by spurring the development of a booming weather-related industry that creates jobs and new economic opportunities. According to Aneesh Chopra, the nation’s former chief technology officer, “Weather is a $5 billion-a-year industry.”50 Some prominent examples of companies built around weather data include The Weather Channel, which uses NOAA data to reach some 97.3 million American households; and the Climate Corporation, which used weather data to provide “weather insurance” to businesses and was sold in 2013 to Monsanto for $930 million.51
• Using the self-assessment methodology mentioned above to estimate VOI, a survey was conducted of households in states prone to hurricane damage to learn how much taxpayers would be willing to pay for enhanced hurricane forecasts. Researchers found that, on average, households in at-risk states were willing to pay an additional $14.34 per year.52

Additionally, some of NOAA’s impacts are less obviously economic in nature. But while fighting erosion and lessening toxins in coastal waters clearly have wider benefits, these results can help to bolster, for example, tourism industries and lessen disaster-related economic losses.

• Penny Pritzker, the U.S. Secretary of Commerce, recently pointed out that more accurate weather forecasts have saved lives by improving warning-times for events like tornadoes, resulting in individuals having more time to get to safety, and also for businesses to prepare their properties and activities to mitigate damage.53
• NOAA’s data has proved instrumental in a number of environmental and conservation efforts in recent years whether through one of operating units or other government agencies using NOAA data. For example, the data has been used to track and respond to coastal erosion;54 to predict and forecast areas at risk from wildfires, particularly in southern California;55 to protect ecosystems in the Gulf of Mexico that are adversely affected by toxins released by algae blooms in coastal waters;56 and in a host of additional settings and applications designed to protect water bodies, forests, animal life and other natural phenomena, key factors in many regional economies, including seafood, recreation, tourism, property and development.

To help scale, NOAA is also working to measure how open its data is by taking an inventory of all its data and analyzing how accessible that data is through its websites. Asking for example, how many data sets have an online link to the data? Where do those links lead to exactly: a standardized Web service that lets users download data or a project home page that requires additional navigation? NOAA hopes this assessment will provide a better understanding of exactly which data sets are open and to what extent that data meets accepted open standards and usability, in order to better scale and meet user needs – perhaps by writing software that automatically accesses the catalog holdings, services offered and formats used for each user need.58

Data Security

To address data security challenges, from hackers to the broader system risks inherent in such large volumes of data processing, NOAA has a dedicated Cyber Security Division, which serves as the NOAA IT security officer. The division oversees the accreditation of NOAA’s IT systems, including the “development of computer security plans, risk assessments, development and testing of contingency/disaster recovery plans, and system certification.” NOAA also has an Information Technology (IT) Security Program with a dedicated Computer Incident Response Team, which provides “reasonable and acceptable assurance that IT systems are performing as specified; that information is provided adequate protection; that data and software integrity is maintained; and, that unplanned disruptions of processing will not seriously impact mission accomplishment.” While data security will likely always be a threat, as it is with any data service, continued investment in the Cyber Security Division can help NOAA mitigate any risks.

Understanding User Needs

In order to keep growing and serving its users, NOAA must also develop more accurate data use measurement tools and better understand its user capabilities and needs. To those ends, NOAA has already expanded upon rudimentary measures of usage (e.g., volume and downloads) to more detailed customer analytics over the past decade, and has provided more data formats, visualizations and applications in response to user requests. To remain useful and meet its full potential for users, NOAA will have to continue investing in customer analytics and apply the insights learned from analyzing user behavior toward developing more products and services, as user needs inevitably evolve and grow.59

Better understanding of user needs is also one of the goals of the Big Data Project (detailed below.) By partnering with key private sector actors involved in big data, open data and cloud computing, NOAA can directly collaborate with experts representing different user bases and their specific needs regarding open data access, types, standards and products. Building an active “ecosystem” around the data, with an active feedback loop among partners, is “very high on [NOAA]’s list of priorities,”60 and will help NOAA better address this challenge as its open data initiatives continue to scale in size and scope.

Data Navigation and User Experience

NOAA is also seeking to improve user experience by redesigning its entire Web presence, a process launched in 2014. Given NOAA’s history as an aggregation of different agencies, its Web presence grew in a similar way, with many sub-sites with siloed information that may be difficult to find and/or repeat elsewhere, making navigation difficult. To make the Web experience more cohesive, NOAA has been working to promote itself both internally and externally as more of a unified entity, shifting from “this is my [agency’s] content, to this is our [NOAA’s] content.”61 Redesigning its websites and portals under this mindset will hopefully help unify NOAA’s messaging in addition to improving user experience and understanding of NOAA’s offerings.62

The other cultural driver to the website redesign is viewing NOAA as a service agency, with the “understanding that everything that we do is of service to someone. From kindergarten students to aviators, every spectrum of the public has something that NOAA touches their lives,” and therefore the website, and NOAA data, should be as accessible and open to users as possible.63

Data Awareness

Finally, raising and promoting data awareness remains a challenge. Although NOAA’s customer service teams help customers navigate data sets and tools they may not be aware of through Web channels, list-servs, targeted outreach initiatives and surveys,64 there may be new potential clients or users who could benefit from NOAA’s data, and who are currently unaware of the portal, do not have a formal partnership with NOAA, and/or are not as sophisticated as a typical NOAA data-user and so would need more assistance.

For the end user, who’s not necessarily the “expert data manipulator,” according to NOAA’s de La Beaujardiere: “The really key thing is to be able to see yourself in the data or have the data apply to you or where you are. If people are given a global figure, e.g., temperature around the globe is going to go up by a tenth of a degree, then that doesn’t seem to have much of an impact. If you can see what’s going to happen, has been happening, might happen at your location, that is probably the most important contextual variable.”65

Therefore, to reach these users, NOAA plans to continue growing its outreach and customer services teams to identify demands and needs that users may not even be aware of themselves.66 Additionally, NOAA plans to continue supporting social media and marketing efforts, particularly around the more day-to-day stories of not just how NOAA data is impacting people’s lives, but how the individual people of NOAA prioritize serving citizens.67

Key among NOAA’s strategies to meet these various challenges is a plan to pursue more partnerships with private sector companies. In a 2014 speech, the U.S. Secretary of Commerce Penny Pritzker extolled the virtues of NOAA’s work and the weather industry it had spurred. But she pointed out that, despite all this work, “the public has access to just 10 percent of NOAA’s more than 20 terabytes of data produced daily.” (Twenty terabytes of data is equivalent to 20,480 gigabytes – in other words, NOAA produces the storage capacity of 1,280 standard iPhones each day.) She went on to suggest that in order to tap into the potential of the remaining 90 percent, “we must partner with the private sector to make our data even more useful to businesses, communities, individuals and decision-makers of all types.”68 Her remarks point to a broader interest in so-called “data collaboratives,” innovative solutions for cross-sectoral data collaboration to benefit the public good.

At the heart of NOAA’s plans to develop more partnerships is its “Big Data Project.”69 In February 2014, NOAA issued a request for information (RFI) to businesses and researchers to solicit ideas from the private sector on ways to better improve access to, and the usability of, its data and data products,70 opening access to its 30 petabytes of annual environmental data to spur innovation and economic growth.71 The RFI focused on developing physical infrastructure for storing and sharing NOAA’s data within a publicly accessible cloud that would allow industry experts and scientists to extrapolate vast amounts of data. After receiving 70 responses from individuals, academia and industry organizations, NOAA developed an initial concept for its big data partnership business model called The Cooperative Research & Development Agreement (CRADA).72

Currently, there are two major pipelines of NOAA data. The first is station-based data via weather services channels, which has an established pipeline that can publish all data, in near real time.73 The other data, consisting of remote sensing, radar and other advanced forecast and climate research data, however, has a much higher volume and can only be handled by a supercomputer. Providing unlimited access therefore poses a problem, as NOAA currently does not have the resources to allow unlimited access without risking slowing down the entire data system. As NOAA CIO Zach Goldstein explains, “The last thing I want to do is run the risk of preventing citizens from getting information on hurricanes, for example. People’s lives depend on getting information from NOAA.”74

As part of CRADA, private companies IBM, Amazon, Google, Microsoft and nonprofit Open Cloud Consortium (OCC) will help NOAA publish this supercomputer data with almost no latency, while innovating, researching and providing feedback on the data.75 The CRADA model allows NOAA to explore what works from a cloud perspective without risking slowing down its current data publishing processes. Instead, the Big Data Project will take an iterative approach, starting with a few large data sets and allowing CRADA members themselves to determine how to most efficiently and effective publish the data into the public domain.76 NOAA chose this incremental approach to help maximize value by better understanding which data sets are most valuable, and to encourage a feedback loop among Big Data Project partners.77

CRADA is also innovative in that it asks the private companies involved to pay the marginal cost for the data, with Goldstein noting that “it may be open but it’s not free.”78 Dr. Maria Patterson of the Open Cloud Consortium (OCC) echoes this sentiment, noting that data being “freely available doesn’t mean there is not a cost associated with it,” and in order to have a lot of data open, someone has to be paying for storage, access and sharing. To balance this cost, therefore, NOAA will not charge users for access to the data itself, but if the cost of providing new access requires additional resources, CRADA members will absorb that marginal cost. If successful, the Big Data Project will represent a new and cost-effective approach to publishing and utilizing NOAA’s vast data resources.

It is important to highlight that in addition to offering an innovative public-private business model, the research component of the Big Data Project is equally valuable, illustrated by the fact that the partners involved are not “contractors” but participants in a “cooperative research and development agreement.”79 As Dr. Patterson explains, NOAA was very deliberate in developing the CRADA approach, and that “the entire project itself is its own research experiment – asking how can NOAA work with partners in a mutually beneficial arrangement to release data into an ecosystem. If the business model fails, the research project can still succeed in identifying what works and what does not work.”80 NOAA CIO Zach Goldstein expands upon this sentiment, “No one else in government has looked at building open data with big data in this way. What we’re doing that’s unique is putting it all together and updating it with today’s technology. It’s possible it won’t work, and NOAA may eventually have to resort to a more traditional pay-per-service model, but the only way we’ll fail is if we fail to learn – if we fail to try.”81

As anticipated, the partnership with the private sector combines three powerful resources: NOAA’s tremendous volume of high-quality environmental data and advanced data products, private industry’s vast infrastructure and technical capacity, and the American economy’s innovation and energy. The overall goal is to create a sustainable, market-driven ecosystem that lowers the cost barrier to data publication, spurs economic development and creates new jobs.

From its early, closed inception to today’s Data Portal to tomorrow’s Big Data Project, the evolution of NOAA’s data efforts represents a powerful illustration of how opening data can have significant economic, social, environmental and other impacts. The changes wrought by NOAA’s 30 petabytes of annual data have rippled through American (and, indeed, global) life, often under the surface, but no less powerful for that. With NOAA’s plans for expansion and new partnerships, there is every reason to believe that further transformations await.