Converging Power and Natural Gas Markets Raise More Questions for Data Management

Converging Power and Natural Gas Markets Raise More Questions for Data Management

December 31, 2014

The once fragmented US and European power markets have changed significantly over the past few years as a result of market expansion and convergence between practices. Many independent system operators (ISOs) have been expanding in order to enhance reliability and ease cross-border trade.

Similarly, in the case of the global LNG market, future expansion will result in increased cross-border trade among North America, Europe, Asia, and Australia. As such, LNG freight and shipping routes will evolve and become more flexible.

With these markets constantly shifting and expanding, data reports are becoming increasingly complex, which has the potential to mystify even the most seasoned analysts. As a result, there has been, and is expected to be, a greater reliance on effective enterprise data management (EDM) solutions as these events continue to unfold.

North American and European Power Markets Moving towards Convergence

North American and European electricity markets are undergoing a transition of expansion through organized marketplaces and the realignment of practices along interfaces among neighboring markets.

This phenomenon has been occurring since the deregulation of US electricity markets in the mid-1990s and, more progressively, in Europe in the early 2000s. The trend became more prominent, however, in 2013 and 2014 with the wave of mergers among North American ISOs and regional transmission organizations (RTOs) as well as with the unification of the European electricity market into a single entity.

These consolidations are seen as a strategic combination of assets and skills to improve economic direction of energy flows between market areas and to bring flatter regional price differentials and improved price transparency.

MISO Is Constantly Extending its Footprint

Initiatives to eliminate connections in interconnected grids through modifications of the existing tools and processes have been taken on by several system operators, especially by those located in the Midwestern and Southwestern interconnections.

On December 18, 2013, the Midcontinent Independent System Operator Inc. (MISO), the RTO for a large portion of the Midwestern United States, completed the integration of a four-state region of the electric grid across the South into its existing footprint in the Midwest. The change in control extends MISO’s operational market footprints from the Gulf of Mexico all the way to Manitoba, Canada.

Figure 1 illustrates the effects of convergence on location-based marginal price (LMP) data.

Figure 1: MISO LMP contour map following integration of the South (Source: MISO)

Figure 1: MISO LMP contour map following integration of the South (Source: MISO)


Over the past five years, the amount of market price data for only the day-ahead LMP market has been increasing considerably, ranging from 5,700 LMP data points in 2009 to 6,699 LMP data points in 2014, with a 21% increase in the last year alone. This is illustrated in Figure 2.

MISO Day-Ahead LMPs

Figure 2: MISO day-ahead market LMPs (December 2009 – December 2014) (Source: MISO)


PJM-NYISO Market-to-Market Coordination

In an effort to improve coordination of power transactions, enhance efficiency, and provide cost-savings to consumers, the New York Independent System Operator (NYISO), Pennsylvania-New Jersey-Maryland Interconnection (PJM), and ISO-New England (NE) have been attempting to eliminate trade barriers among the North American power markets through the Broader Regional Markets initiatives since 2010.[1]

On November 4, 2014, PJM and NYISO announced that they are streamlining the flow of energy across their mutual borders. The two grid operators implemented a coordinated transaction scheduling (CTS) system to help market participants access the latest source of power and more efficiently use the transmission lines connecting the two regions—lowering the combined energy production costs of the two systems. This increased flow of energy will ultimately translate into greater price convergence between the two regions.[2]

Expansion of SPP Market into Eastern and Western Grid Interconnections

Another major expansion occurred on November 10, 2014 with the Federal Energy Regulatory Commission’s (FERC) approval to expand the size of the Southwest Power Pool’s (SPP) footprint. SPP will more than double the size of its footprint to include three new members that will span the Eastern and Western connections of the US electric grid to include the Western Area Power Administration’s Upper Great Plains, Basin Electric Power Cooperative, and Heartland Consumers Power.[3]

The expansion will cover 9,500 miles of transmission lines, servicing three million customers across seven states, including Iowa, Minnesota, Montana, Nebraska, North Dakota, and South Dakota. SPP plans to take on reliability coordination of its three new members starting in June 2015, with full membership beginning in October 2015.

Renewable Generation and Energy Markets

A growing share of renewable resources in the generation portfolio has been pushing balancing authorities to evaluate and revise existing energy balancing practices. The US Pacific Northwest, with some of the most aggressive renewable generation portfolios, takes the lead.

First, the Bonneville Power Administration (BPA) implemented the Intra-Hour Scheduling Pilot Program in 2011, allowing for scheduling energy transmission in 15-minute intervals. Afterward, California Independent System Operator (CaISO) joined BPA’s initiative.

On November 1, 2014, CaISO and PacifiCorp began operating a real-time energy imbalance market (EIM), implemented to better integrate renewable generation into the grid across California, Oregon, Washington, Utah, Idaho, and Wyoming. NV Energy will also begin participating in the EIM in October 2015.

Figure 3 illustrates the extent of CaISO’s footprint across the Southwest.

CAISO footprint and PacifiCorp and NV electric service territories

Figure 3: CaISO footprint expanding throughout PacifiCorp’s electric service territory (Source: CaISO)


Energy transfers between balancing authorities have been predominantly scheduled according to hourly intervals. The dispatch of generation and load data at 15-minute intervals will further add to the data management challenge by creating more stress for the data exchange velocity and storage.

Power Markets Become Integrated in Europe

On the other side of the Atlantic, electricity market integration has been progressing since Denmark, Finland, Norway, and Sweden ratified a market splitting model following the liberalization of the electricity market in the 1990s. France, Belgium, and the Netherlands integrated their markets (Central Western Europe – CWE) in 2006. Germany, Luxemburg, and Great Britain joined the CWE in 2010 and 2011 respectively. Spain and Portugal coupled their markets in 2007, Czech Republic and Slovakia in 2011, and Italy and Slovenia in 2011. The latest step of European market integration took place on November 19, 2014, with the creation of acoupled day-ahead electricity marketof OTE in Czech Republic, HUPX in Hungary, OPCOM in Romania, and OKTE in Slovakia.

Just nine months prior, the price coupling of northwestern Europe occurred on February 4, 2014, which provided day-ahead trading in Belgium, Denmark, Finland, France, Germany, Austria, Latvia, Lithuania, Luxembourg, the Netherlands, Norway, Estonia, Sweden, and the United Kingdom.[4]

In Figure 4, the dark blue area represents a fully coupled day-ahead market as of February 4, 2014. Spanish and Portuguese markets coupled on May 13, 2014, which is represented by the orange area. Italy is expected to operate a fully coupled day-ahead market by 2015.

Fully coupled day-ahead electricity market in Europe

Figure 4: Fully coupled day-ahead electricity market in Europe (Source: ACER)


The expansion between these countries is considered a step towards enhancing reliability by easing cross-border trade and increasing competition between suppliers. The further expansion of interconnected borders will demand a greater need for coordination between market participants.

As European power markets converge, data sources associated with these markets will also go through changes. Market participants have to watch very closely how data reports, especially from the key data providers such as EPEX SPOT and APX, are being transformed. With consolidation occurring so quickly and frequently, this task can be very challenging for those with limited resources in dedicated data monitoring function.

But it isn’t only the power markets that are shifting; natural gas markets are also experiencing significant changes.

Global LNG Trade: More Opportunities and More Intricacies

Because of the sudden surge in shale gas production in recent years, the natural gas markets have undergone a fundamental shift. As of 2012, North America leads the world in shale gas production, growing by an average rate of 1.6% per year.[5] Up until 2009, the majority of total US shale gas production came from the Barnett shale. However, in the last few years, production from other plays, particularly the Haynesville and Marcellus, is increasing rapidly. Figure 5 represents the increase in US natural gas production between 2010 and 2013.

US natural gas production has been undergoing an increasing trend

Figure 5: US natural gas production has been undergoing an increasing trend (Data Source: EIA)


According to the EIA’s Annual Energy Outlook 2013, shale gas production is predicted to almost double between 2013 and 2040.[6]

The price of natural gas is an important factor in the rapid increase in shale gas production in the United States. Since 2006, the development of shale gas resources has spurred growth in natural gas production, with producers seeing opportunities as a result of growing demand from abroad.The EIA predicts that the United States will export approximately 3.6 trillion cubic feet of natural gas annually by 2040.[7]

In the case of the global LNG market, North American suppliers have been challenged to keep pace with demand. Europe and Asia are major areas of interest for North American LNG. North American LNG companies have been making efforts to enhance their positions in the European market by creating long-term LNG contracts that are beneficial for European consumers and that can be delivered to European spot markets.

For example, in June 2014, Gas Natural from Spain signed a 20-year contract with Cheniere that links natural gas prices to the US spot market for Henry Hub.[8] BG Group signed a similar contract with Cheniere in 2011. Growing exports of LNG from North America and a more globalized gas market will likely lead the US Henry Hub price benchmark to become the driver of natural gas prices in Europe. The graph in Figure 6 represents similar prices at global trading benchmarks for Henry Hub and National Balancing Point (NBP).

ZEMA graph comparing prices at global trading benchmarks for Henry Hub, Brent, and NBP

Figure 6: ZEMA graph comparing prices at global trading benchmarks for Henry Hub, Brent, and NBP (Data Source: NYMEX)


On November 21, 2011, Norwegian Statoil signed a 13 billion GBP supply agreement with UK utility Centrica that links natural gas prices to NBP benchmarks.[9] Centrica also signed an agreement with Qatar for 2.4 million tons (3.26 bcm) per year of LNG supplies.[10] Henry Hub may soon play the same role in Japan, thanks to approval of several LNG export facilities in the United States. In September 2014, the US Department of Energy approved the export of LNG to non-Free Trade Agreement countries, such as Japan and other Asian countries, representing a significant milestone for the industry. FERC approved the Sempra Energy’s LNG export terminal in Louisiana and Carib Energy’s smaller terminal in Florida.[11] They are the second and third facilities approved to export LNG to the Asia-Pacific region.

Canada is the fourth-largest exporter of natural gas in the world. Although Canada has intentions to export LNG, plans for seven LNG terminals have either been cancelled or suspended due to political and economic factors.[12] Australia is poised to become the largest supplier of LNG in the world and to become a leading exporter of LNG to the Asia-Pacific region once LNG export terminals come on line. Australian LNG exports are forecast to climb to 83 million tons a year by 2020, compared with 79 million tons for Qatar.[13] East Africa may emerge as a strong competitor to Qatar and Australia in the battle to capture key export markets in Asia.

As future LNG markets expand throughout the world, LNG routes will evolve and become more flexible. This is apparent in Figure 7, which represents current cross-border LNG trade among North America, Europe, Asia, and Australia.

Future global LNG trade routes

Figure 7: Future global LNG trade routes (Source: Citi Investment Research and Analysis)


As LNG supply, terminals, plants, shipping, liquefaction capacities, trade contracts, and net exports increase, so will the volumes of data associated with them.

This data boom leads to a special use case of the data management mess, where power, natural gas, and other energy and commodity market participants face a deluge of market data from multiple sources. Such a wealth of data can be exceedingly complicated to correlate and interpret using manual data collection and management processes. As the data requirements of these markets are complex and constantly expanding, finding ways to efficiently track, manage, organize, and store data is among the most significant challenges.

When it comes to addressing this challenge, many energy companies are confronted with the decision of implementing an Enterprise Data Management (EDM) solution to support their operational needs.

The Enterprise Data Management Solution

In order to meet the challenges of shifting energy markets and the volumes of accompanying data, many companies with strong IT departments are attracted to the idea of building one-off software solutions created solely for their business. However, creating software from scratch is, in most cases, an unnecessary risk. Research suggests that 31.1% of software development projects will be cancelled at some point during development, and more than half will cost at least 189% of organizations’ original estimates. On average, only 16% of build projects will be successful and will often have less functionality than originally intended.[14] As well, the average time-to-market of an EDM software development project is three years. Overshooting timelines is extremely common, as those projects that make it to some form of completion will often require re-design and revaluation. Restarts are another major issue that disrupts these projects, often resulting in their cancellation before they even near completion.

Buying a commercial, off-the-shelf EDM reduces the time-to-market risks inherent in building an EDM solution internally. Software companies exist solely for the purpose of developing robust and efficient software solutions. Even an organization with a very strong IT department will not have nearly the level of expertise and dedication as a business that is devoted entirely to developing software products. For this reason, when a business opts to purchase a solution rather than attempting to create it themselves, they aren’t just buying the product—they are also buying the thousands of hours of research and development that went into its creation and continual maintenance.

The majority of issues faced by businesses are not uncommon. As such, a software company with proven success developing solutions for a particular industry will have gained valuable insights from helping other businesses that have faced similar challenges. Nearly all software vendors incorporate suggestions from end-users to create continuous updates to functionality; the solution will continue to evolve, adding value without requiring constant investment of capital and resources.


Request ZE’s white paper “Weighing the Options: Build an Internal EDM System or Get ZEMA” to learn more.



[1]Edelson, David, “Broader Regional Markets,” New York Independent System Operator, May 24, 2014, accessed December 5, 2014,

[2]“PJM, NYISO Implement Coordinated Transaction Scheduling,” New York Independent System Operator, November 5, 2014, accessed December 5, 2014, NYISO Implement Coordinated Transaction Scheduling – 11 5 14 – FINAL.pdf.

[3]“Order Conditionally Accepting in Part, Rejecting in Part, Accepting and Suspending Filings in Part, Establishing Hearing and Settlement Judge Procedures and Consolidation Proceedings and Directing Compliance Filing,” United States of America Federal Energy Regulatory Commission, November 10, 2014, accessed December 5, 2014,

[4]“Market Coupling: A Major Step Towards Market Integration,” EPEX SPOT, accessed December 7, 2014,

[5]“Annual Energy Outlook 2014,” US Energy Information Administration, May 7, 2014, accessed December 8, 2014,

[6]“Annual Energy Outlook 2013,” US Energy Information Administration, April, 2013, accessed December 8, 2014,

[7] Ibid.

[8]Toyer, Julien, “Cheniere, Gas Natural LNG Deal worth $13 Billion: Source,” Reuters, June 2, 2014, accessed December 9, 2014,

[9]“Centrica Establishes Strategic Partnership with Statoil and Announces £1bn Norwegian Asset Acquisition,” Centrica, November 21, 2013, accessed December 9, 2014,

[10]“Centrica Signs £2bn Gas Deal with Qatar to Secure LNG for the UK,” Centrica, February 23, 2011, accessed December 9, 2014,

[11]“North American LNG Import /Export Terminals Approved,” United States of America Federal Energy Regulatory Commission, December 3, 2014, accessed December 10, 2014,

[12]Yee, Adrian K., “North American LNG Projects: Political and Economic Factors,” ZE DataWatch, November 28, 2014, accessed December 10, 2014,

[13]Sharples, Ben, “Australian LNG Contracts Under Pressure From U.S., Russia Supply,” Bloomberg, September 9, 2014, accessed December 10, 2014,

[14] “Chaos,” The Standish Group, 1995: 2, accessed November 26, 2014,



 Effective Date: December 30, 2014

Post a Comment


More Articles on In Depth

Big Schedule for Big Data

This article is the introductory part of the Big Schedule Optimization series addressing concerns of Big Data collection processes. In most cases, analytical tools assume that the data is already collected, unified and stored in a database. Business Intelligence (BI) systems ... Read more »

The Year-end Review and Future Projections: Awaiting More Synergy between the Energy Sector and IT

Lacking originality, we are closing 2015 with the yearly review and endeavored predictions of the near future; this time however, we will look at the energy sector in conjunction with data business and software solutions. The energy sector has been ... Read more »

Data Security + Data Surveillance = More Data Business

In today’s fast paced business environment, risk is everywhere and data security is rapidly changing. However, few things remain unchanged: hackers will create more sophisticated scripts to get around your security fortresses trying to steal your identity, money, and corporate ... Read more »

Weighing the Options: Build an Internal EDM System or Get ZEMA

The Role of Enterprise Data Management in Energy and Commodities Industries   Many large integrated oil and natural gas firms, commodity firms, and power utilities rely on an Enterprise Data Management (EDM) system to acquire the data required for their trade and ... Read more »

Natural Gas Coasts Along, while Other Energy Markets Test Rollercoasters

I do not recall more confusion and misalignment in the energy sector as is happening now. Any correlation and interdependency that existed previously among oil, coal, natural gas, and electricity markets seem to have evaporated; a multiplicity of independent events ... Read more »

Tesla and the Shift in Power: Part 2 – How Powerpack and Powerwall batteries will amplify the need for data management

Alongside hyperloops, rockets, and luxury electric vehicles, the Tesla Powerwall battery isn’t exactly the most sensational technology for which Elon Musk, CEO of Tesla Motors and SpaceX, has made headline news recently. Nonetheless, Powerwall has the potential to become Musk’s ... Read more »

Tesla and the Shift in Power: Part 1 – What Powerwall batteries could mean for energy consumers

On April 30, 2015, Elon Musk, CEO of Tesla Motors and SpaceX, took to the stage amidst cheering and rock music to announce, in his words, a “fundamental transformation of how the world works, how energy is delivered across the ... Read more »

 Effective Date: June 01, 2015

Price Reporting Assessments, Agencies, and Increased Asian Oil Consumption

Sales and purchases of varied commodities affect each country in the world, but few commodities have a global impact greater than that of oil. Price reporting agencies (PRAs) monitor the global oil industry carefully, producing price assessments which are meant ... Read more »

 Effective Date: April 28, 2015

Why Is Crisis so Paramount?

As the first quarter of 2015 draws to a close, we are beginning to get used to the concept of cheap oil. The question on everyone’s minds, of course, is how long it will last. March has seen the lowest ... Read more »

The Seven Ages of Oil Part 2: Boom and Bust, War and Peace, Growth and Decline

Part Two of Two: The Seven Ages Continued In Part One of our feature story from last month, we covered the first four of the “seven ages of oil.” 1859-1870: Illumination Births an Industry 1870-1911: Rockefeller Creates the Multinational Oil Standard 1911-1921: A Pax ... Read more »


Page 1 of 612345...Last »