Converging Power and Natural Gas Markets Raise More Questions for Data Management

Converging Power and Natural Gas Markets Raise More Questions for Data Management

December 31, 2014

The once fragmented US and European power markets have changed significantly over the past few years as a result of market expansion and convergence between practices. Many independent system operators (ISOs) have been expanding in order to enhance reliability and ease cross-border trade.

Similarly, in the case of the global LNG market, future expansion will result in increased cross-border trade among North America, Europe, Asia, and Australia. As such, LNG freight and shipping routes will evolve and become more flexible.

With these markets constantly shifting and expanding, data reports are becoming increasingly complex, which has the potential to mystify even the most seasoned analysts. As a result, there has been, and is expected to be, a greater reliance on effective enterprise data management (EDM) solutions as these events continue to unfold.

North American and European Power Markets Moving towards Convergence

North American and European electricity markets are undergoing a transition of expansion through organized marketplaces and the realignment of practices along interfaces among neighboring markets.

This phenomenon has been occurring since the deregulation of US electricity markets in the mid-1990s and, more progressively, in Europe in the early 2000s. The trend became more prominent, however, in 2013 and 2014 with the wave of mergers among North American ISOs and regional transmission organizations (RTOs) as well as with the unification of the European electricity market into a single entity.

These consolidations are seen as a strategic combination of assets and skills to improve economic direction of energy flows between market areas and to bring flatter regional price differentials and improved price transparency.

MISO Is Constantly Extending its Footprint

Initiatives to eliminate connections in interconnected grids through modifications of the existing tools and processes have been taken on by several system operators, especially by those located in the Midwestern and Southwestern interconnections.

On December 18, 2013, the Midcontinent Independent System Operator Inc. (MISO), the RTO for a large portion of the Midwestern United States, completed the integration of a four-state region of the electric grid across the South into its existing footprint in the Midwest. The change in control extends MISO’s operational market footprints from the Gulf of Mexico all the way to Manitoba, Canada.

Figure 1 illustrates the effects of convergence on location-based marginal price (LMP) data.

Figure 1: MISO LMP contour map following integration of the South (Source: MISO)

Figure 1: MISO LMP contour map following integration of the South (Source: MISO)


Over the past five years, the amount of market price data for only the day-ahead LMP market has been increasing considerably, ranging from 5,700 LMP data points in 2009 to 6,699 LMP data points in 2014, with a 21% increase in the last year alone. This is illustrated in Figure 2.

MISO Day-Ahead LMPs

Figure 2: MISO day-ahead market LMPs (December 2009 – December 2014) (Source: MISO)


PJM-NYISO Market-to-Market Coordination

In an effort to improve coordination of power transactions, enhance efficiency, and provide cost-savings to consumers, the New York Independent System Operator (NYISO), Pennsylvania-New Jersey-Maryland Interconnection (PJM), and ISO-New England (NE) have been attempting to eliminate trade barriers among the North American power markets through the Broader Regional Markets initiatives since 2010.[1]

On November 4, 2014, PJM and NYISO announced that they are streamlining the flow of energy across their mutual borders. The two grid operators implemented a coordinated transaction scheduling (CTS) system to help market participants access the latest source of power and more efficiently use the transmission lines connecting the two regions—lowering the combined energy production costs of the two systems. This increased flow of energy will ultimately translate into greater price convergence between the two regions.[2]

Expansion of SPP Market into Eastern and Western Grid Interconnections

Another major expansion occurred on November 10, 2014 with the Federal Energy Regulatory Commission’s (FERC) approval to expand the size of the Southwest Power Pool’s (SPP) footprint. SPP will more than double the size of its footprint to include three new members that will span the Eastern and Western connections of the US electric grid to include the Western Area Power Administration’s Upper Great Plains, Basin Electric Power Cooperative, and Heartland Consumers Power.[3]

The expansion will cover 9,500 miles of transmission lines, servicing three million customers across seven states, including Iowa, Minnesota, Montana, Nebraska, North Dakota, and South Dakota. SPP plans to take on reliability coordination of its three new members starting in June 2015, with full membership beginning in October 2015.

Renewable Generation and Energy Markets

A growing share of renewable resources in the generation portfolio has been pushing balancing authorities to evaluate and revise existing energy balancing practices. The US Pacific Northwest, with some of the most aggressive renewable generation portfolios, takes the lead.

First, the Bonneville Power Administration (BPA) implemented the Intra-Hour Scheduling Pilot Program in 2011, allowing for scheduling energy transmission in 15-minute intervals. Afterward, California Independent System Operator (CaISO) joined BPA’s initiative.

On November 1, 2014, CaISO and PacifiCorp began operating a real-time energy imbalance market (EIM), implemented to better integrate renewable generation into the grid across California, Oregon, Washington, Utah, Idaho, and Wyoming. NV Energy will also begin participating in the EIM in October 2015.

Figure 3 illustrates the extent of CaISO’s footprint across the Southwest.

CAISO footprint and PacifiCorp and NV electric service territories

Figure 3: CaISO footprint expanding throughout PacifiCorp’s electric service territory (Source: CaISO)


Energy transfers between balancing authorities have been predominantly scheduled according to hourly intervals. The dispatch of generation and load data at 15-minute intervals will further add to the data management challenge by creating more stress for the data exchange velocity and storage.

Power Markets Become Integrated in Europe

On the other side of the Atlantic, electricity market integration has been progressing since Denmark, Finland, Norway, and Sweden ratified a market splitting model following the liberalization of the electricity market in the 1990s. France, Belgium, and the Netherlands integrated their markets (Central Western Europe – CWE) in 2006. Germany, Luxemburg, and Great Britain joined the CWE in 2010 and 2011 respectively. Spain and Portugal coupled their markets in 2007, Czech Republic and Slovakia in 2011, and Italy and Slovenia in 2011. The latest step of European market integration took place on November 19, 2014, with the creation of acoupled day-ahead electricity marketof OTE in Czech Republic, HUPX in Hungary, OPCOM in Romania, and OKTE in Slovakia.

Just nine months prior, the price coupling of northwestern Europe occurred on February 4, 2014, which provided day-ahead trading in Belgium, Denmark, Finland, France, Germany, Austria, Latvia, Lithuania, Luxembourg, the Netherlands, Norway, Estonia, Sweden, and the United Kingdom.[4]

In Figure 4, the dark blue area represents a fully coupled day-ahead market as of February 4, 2014. Spanish and Portuguese markets coupled on May 13, 2014, which is represented by the orange area. Italy is expected to operate a fully coupled day-ahead market by 2015.

Fully coupled day-ahead electricity market in Europe

Figure 4: Fully coupled day-ahead electricity market in Europe (Source: ACER)


The expansion between these countries is considered a step towards enhancing reliability by easing cross-border trade and increasing competition between suppliers. The further expansion of interconnected borders will demand a greater need for coordination between market participants.

As European power markets converge, data sources associated with these markets will also go through changes. Market participants have to watch very closely how data reports, especially from the key data providers such as EPEX SPOT and APX, are being transformed. With consolidation occurring so quickly and frequently, this task can be very challenging for those with limited resources in dedicated data monitoring function.

But it isn’t only the power markets that are shifting; natural gas markets are also experiencing significant changes.

Global LNG Trade: More Opportunities and More Intricacies

Because of the sudden surge in shale gas production in recent years, the natural gas markets have undergone a fundamental shift. As of 2012, North America leads the world in shale gas production, growing by an average rate of 1.6% per year.[5] Up until 2009, the majority of total US shale gas production came from the Barnett shale. However, in the last few years, production from other plays, particularly the Haynesville and Marcellus, is increasing rapidly. Figure 5 represents the increase in US natural gas production between 2010 and 2013.

US natural gas production has been undergoing an increasing trend

Figure 5: US natural gas production has been undergoing an increasing trend (Data Source: EIA)


According to the EIA’s Annual Energy Outlook 2013, shale gas production is predicted to almost double between 2013 and 2040.[6]

The price of natural gas is an important factor in the rapid increase in shale gas production in the United States. Since 2006, the development of shale gas resources has spurred growth in natural gas production, with producers seeing opportunities as a result of growing demand from abroad.The EIA predicts that the United States will export approximately 3.6 trillion cubic feet of natural gas annually by 2040.[7]

In the case of the global LNG market, North American suppliers have been challenged to keep pace with demand. Europe and Asia are major areas of interest for North American LNG. North American LNG companies have been making efforts to enhance their positions in the European market by creating long-term LNG contracts that are beneficial for European consumers and that can be delivered to European spot markets.

For example, in June 2014, Gas Natural from Spain signed a 20-year contract with Cheniere that links natural gas prices to the US spot market for Henry Hub.[8] BG Group signed a similar contract with Cheniere in 2011. Growing exports of LNG from North America and a more globalized gas market will likely lead the US Henry Hub price benchmark to become the driver of natural gas prices in Europe. The graph in Figure 6 represents similar prices at global trading benchmarks for Henry Hub and National Balancing Point (NBP).

ZEMA graph comparing prices at global trading benchmarks for Henry Hub, Brent, and NBP

Figure 6: ZEMA graph comparing prices at global trading benchmarks for Henry Hub, Brent, and NBP (Data Source: NYMEX)


On November 21, 2011, Norwegian Statoil signed a 13 billion GBP supply agreement with UK utility Centrica that links natural gas prices to NBP benchmarks.[9] Centrica also signed an agreement with Qatar for 2.4 million tons (3.26 bcm) per year of LNG supplies.[10] Henry Hub may soon play the same role in Japan, thanks to approval of several LNG export facilities in the United States. In September 2014, the US Department of Energy approved the export of LNG to non-Free Trade Agreement countries, such as Japan and other Asian countries, representing a significant milestone for the industry. FERC approved the Sempra Energy’s LNG export terminal in Louisiana and Carib Energy’s smaller terminal in Florida.[11] They are the second and third facilities approved to export LNG to the Asia-Pacific region.

Canada is the fourth-largest exporter of natural gas in the world. Although Canada has intentions to export LNG, plans for seven LNG terminals have either been cancelled or suspended due to political and economic factors.[12] Australia is poised to become the largest supplier of LNG in the world and to become a leading exporter of LNG to the Asia-Pacific region once LNG export terminals come on line. Australian LNG exports are forecast to climb to 83 million tons a year by 2020, compared with 79 million tons for Qatar.[13] East Africa may emerge as a strong competitor to Qatar and Australia in the battle to capture key export markets in Asia.

As future LNG markets expand throughout the world, LNG routes will evolve and become more flexible. This is apparent in Figure 7, which represents current cross-border LNG trade among North America, Europe, Asia, and Australia.

Future global LNG trade routes

Figure 7: Future global LNG trade routes (Source: Citi Investment Research and Analysis)


As LNG supply, terminals, plants, shipping, liquefaction capacities, trade contracts, and net exports increase, so will the volumes of data associated with them.

This data boom leads to a special use case of the data management mess, where power, natural gas, and other energy and commodity market participants face a deluge of market data from multiple sources. Such a wealth of data can be exceedingly complicated to correlate and interpret using manual data collection and management processes. As the data requirements of these markets are complex and constantly expanding, finding ways to efficiently track, manage, organize, and store data is among the most significant challenges.

When it comes to addressing this challenge, many energy companies are confronted with the decision of implementing an Enterprise Data Management (EDM) solution to support their operational needs.

The Enterprise Data Management Solution

In order to meet the challenges of shifting energy markets and the volumes of accompanying data, many companies with strong IT departments are attracted to the idea of building one-off software solutions created solely for their business. However, creating software from scratch is, in most cases, an unnecessary risk. Research suggests that 31.1% of software development projects will be cancelled at some point during development, and more than half will cost at least 189% of organizations’ original estimates. On average, only 16% of build projects will be successful and will often have less functionality than originally intended.[14] As well, the average time-to-market of an EDM software development project is three years. Overshooting timelines is extremely common, as those projects that make it to some form of completion will often require re-design and revaluation. Restarts are another major issue that disrupts these projects, often resulting in their cancellation before they even near completion.

Buying a commercial, off-the-shelf EDM reduces the time-to-market risks inherent in building an EDM solution internally. Software companies exist solely for the purpose of developing robust and efficient software solutions. Even an organization with a very strong IT department will not have nearly the level of expertise and dedication as a business that is devoted entirely to developing software products. For this reason, when a business opts to purchase a solution rather than attempting to create it themselves, they aren’t just buying the product—they are also buying the thousands of hours of research and development that went into its creation and continual maintenance.

The majority of issues faced by businesses are not uncommon. As such, a software company with proven success developing solutions for a particular industry will have gained valuable insights from helping other businesses that have faced similar challenges. Nearly all software vendors incorporate suggestions from end-users to create continuous updates to functionality; the solution will continue to evolve, adding value without requiring constant investment of capital and resources.


Request ZE’s white paper “Weighing the Options: Build an Internal EDM System or Get ZEMA” to learn more.



[1]Edelson, David, “Broader Regional Markets,” New York Independent System Operator, May 24, 2014, accessed December 5, 2014,

[2]“PJM, NYISO Implement Coordinated Transaction Scheduling,” New York Independent System Operator, November 5, 2014, accessed December 5, 2014, NYISO Implement Coordinated Transaction Scheduling – 11 5 14 – FINAL.pdf.

[3]“Order Conditionally Accepting in Part, Rejecting in Part, Accepting and Suspending Filings in Part, Establishing Hearing and Settlement Judge Procedures and Consolidation Proceedings and Directing Compliance Filing,” United States of America Federal Energy Regulatory Commission, November 10, 2014, accessed December 5, 2014,

[4]“Market Coupling: A Major Step Towards Market Integration,” EPEX SPOT, accessed December 7, 2014,

[5]“Annual Energy Outlook 2014,” US Energy Information Administration, May 7, 2014, accessed December 8, 2014,

[6]“Annual Energy Outlook 2013,” US Energy Information Administration, April, 2013, accessed December 8, 2014,

[7] Ibid.

[8]Toyer, Julien, “Cheniere, Gas Natural LNG Deal worth $13 Billion: Source,” Reuters, June 2, 2014, accessed December 9, 2014,

[9]“Centrica Establishes Strategic Partnership with Statoil and Announces £1bn Norwegian Asset Acquisition,” Centrica, November 21, 2013, accessed December 9, 2014,

[10]“Centrica Signs £2bn Gas Deal with Qatar to Secure LNG for the UK,” Centrica, February 23, 2011, accessed December 9, 2014,

[11]“North American LNG Import /Export Terminals Approved,” United States of America Federal Energy Regulatory Commission, December 3, 2014, accessed December 10, 2014,

[12]Yee, Adrian K., “North American LNG Projects: Political and Economic Factors,” ZE DataWatch, November 28, 2014, accessed December 10, 2014,

[13]Sharples, Ben, “Australian LNG Contracts Under Pressure From U.S., Russia Supply,” Bloomberg, September 9, 2014, accessed December 10, 2014,

[14] “Chaos,” The Standish Group, 1995: 2, accessed November 26, 2014,



 Effective Date: December 30, 2014

Post a Comment


More Articles on In Depth

Natural Gas: High Stakes, High Uncertainty and Global Repercussions

Gas markets around the world have undergone some major shifts in the past decade. Production and storage levels have risen to historic levels in North America, where speculation over liquefied natural gas has reached fever pitch. Add environmental and political ... Read more »

The State of Fracking Globally: Who’s Playing Catch-Up on the US

Shale gas has been pitted as one of the main drivers for economic recovery in the US. Earlier this year, during his state of the Union address, President Barack Obama said the American oil and gas boom of recent times had ... Read more »

How Renewables Shape the Future by KYOS Energy Consulting

by Cyriel de Jong, Hans van Dijken and Emiliyan Enev, KYOS Energy Consulting Introduction In 2012, the German capacity of solar energy passed the 32 GW mark. The impact on the market prices is noticeable: summer prices go down, peak prices go ... Read more »

 Effective Date: January 29, 2013

2013 Energy Forecast by EOX Live

Despite the relatively sideways trading range and declining volatility that the last few months of 2012 has brought the oil markets, underlying developments have remained fairly dynamic. The November election in the U.S. and change of power in China will ... Read more »

Falling from a Height – the Impact of the Fiscal Cliff on the Energy Sector

The energy sector has recently become a bastion of hope for the US. Despite the year-on-year growth in the shale gas and oil industries, continued bickering in Washington is causing serious concern over what consequences no compromise could have on ... Read more »

The Revival of Western Oil and Gas Production – It Only Changes Everything

Crude oil and natural gas production in North America is booming, contributing to a massive reshaping of the global energy landscape. At the launch of the 2012 Edition of the World Energy Outlook (WEO) in London on November 12, the International ... Read more »

South East Asian Electricity Markets: a Long and Winding Road to Competition

Competitive markets promote efficiencies in operation for many sectors and the power industry is no different. Early adopters of deregulated power industries in Europe in the early 90s paved the way for similar implementations around the world. A competitive market ... Read more »

Shale Gas: More Complications for Price Forecast Developers. Part III.

The Demand Side of the Equation An unprecedented natural gas production revolution in the U.S. offers many benefits and at the same time poses many questions. We are presenting a series of articles that discuss how shale gas expansion adds to ... Read more »

U.S. Coal-Fired Generation Community Wins Important Battle Against EPA

By Brock Ramey, North American Power Research Manager for IIR On August 14, 2012, coal industry leaders were sitting in a crowded room listening to the keynote address at the opening of the Coal-Gen conference in Louisville, Kentucky, and the buzz ... Read more »

Shale Gas: More Complications for Price Forecast Developers. Part II: LNG

In this article, we continue to discuss how shale gas expansion in the U.S. affects natural gas price projections and what factors influence its supply and demand (part one in the May 2012 DataWatch). Liquefied natural gas (LNG) is frequently ... Read more »


Page 4 of 6« First...23456