Converging Power and Natural Gas Markets Raise More Questions for Data Management

Converging Power and Natural Gas Markets Raise More Questions for Data Management

December 31, 2014

The once fragmented US and European power markets have changed significantly over the past few years as a result of market expansion and convergence between practices. Many independent system operators (ISOs) have been expanding in order to enhance reliability and ease cross-border trade.

Similarly, in the case of the global LNG market, future expansion will result in increased cross-border trade among North America, Europe, Asia, and Australia. As such, LNG freight and shipping routes will evolve and become more flexible.

With these markets constantly shifting and expanding, data reports are becoming increasingly complex, which has the potential to mystify even the most seasoned analysts. As a result, there has been, and is expected to be, a greater reliance on effective enterprise data management (EDM) solutions as these events continue to unfold.

North American and European Power Markets Moving towards Convergence

North American and European electricity markets are undergoing a transition of expansion through organized marketplaces and the realignment of practices along interfaces among neighboring markets.

This phenomenon has been occurring since the deregulation of US electricity markets in the mid-1990s and, more progressively, in Europe in the early 2000s. The trend became more prominent, however, in 2013 and 2014 with the wave of mergers among North American ISOs and regional transmission organizations (RTOs) as well as with the unification of the European electricity market into a single entity.

These consolidations are seen as a strategic combination of assets and skills to improve economic direction of energy flows between market areas and to bring flatter regional price differentials and improved price transparency.

MISO Is Constantly Extending its Footprint

Initiatives to eliminate connections in interconnected grids through modifications of the existing tools and processes have been taken on by several system operators, especially by those located in the Midwestern and Southwestern interconnections.

On December 18, 2013, the Midcontinent Independent System Operator Inc. (MISO), the RTO for a large portion of the Midwestern United States, completed the integration of a four-state region of the electric grid across the South into its existing footprint in the Midwest. The change in control extends MISO’s operational market footprints from the Gulf of Mexico all the way to Manitoba, Canada.

Figure 1 illustrates the effects of convergence on location-based marginal price (LMP) data.

Figure 1: MISO LMP contour map following integration of the South (Source: MISO)

Figure 1: MISO LMP contour map following integration of the South (Source: MISO)


Over the past five years, the amount of market price data for only the day-ahead LMP market has been increasing considerably, ranging from 5,700 LMP data points in 2009 to 6,699 LMP data points in 2014, with a 21% increase in the last year alone. This is illustrated in Figure 2.

MISO Day-Ahead LMPs

Figure 2: MISO day-ahead market LMPs (December 2009 – December 2014) (Source: MISO)


PJM-NYISO Market-to-Market Coordination

In an effort to improve coordination of power transactions, enhance efficiency, and provide cost-savings to consumers, the New York Independent System Operator (NYISO), Pennsylvania-New Jersey-Maryland Interconnection (PJM), and ISO-New England (NE) have been attempting to eliminate trade barriers among the North American power markets through the Broader Regional Markets initiatives since 2010.[1]

On November 4, 2014, PJM and NYISO announced that they are streamlining the flow of energy across their mutual borders. The two grid operators implemented a coordinated transaction scheduling (CTS) system to help market participants access the latest source of power and more efficiently use the transmission lines connecting the two regions—lowering the combined energy production costs of the two systems. This increased flow of energy will ultimately translate into greater price convergence between the two regions.[2]

Expansion of SPP Market into Eastern and Western Grid Interconnections

Another major expansion occurred on November 10, 2014 with the Federal Energy Regulatory Commission’s (FERC) approval to expand the size of the Southwest Power Pool’s (SPP) footprint. SPP will more than double the size of its footprint to include three new members that will span the Eastern and Western connections of the US electric grid to include the Western Area Power Administration’s Upper Great Plains, Basin Electric Power Cooperative, and Heartland Consumers Power.[3]

The expansion will cover 9,500 miles of transmission lines, servicing three million customers across seven states, including Iowa, Minnesota, Montana, Nebraska, North Dakota, and South Dakota. SPP plans to take on reliability coordination of its three new members starting in June 2015, with full membership beginning in October 2015.

Renewable Generation and Energy Markets

A growing share of renewable resources in the generation portfolio has been pushing balancing authorities to evaluate and revise existing energy balancing practices. The US Pacific Northwest, with some of the most aggressive renewable generation portfolios, takes the lead.

First, the Bonneville Power Administration (BPA) implemented the Intra-Hour Scheduling Pilot Program in 2011, allowing for scheduling energy transmission in 15-minute intervals. Afterward, California Independent System Operator (CaISO) joined BPA’s initiative.

On November 1, 2014, CaISO and PacifiCorp began operating a real-time energy imbalance market (EIM), implemented to better integrate renewable generation into the grid across California, Oregon, Washington, Utah, Idaho, and Wyoming. NV Energy will also begin participating in the EIM in October 2015.

Figure 3 illustrates the extent of CaISO’s footprint across the Southwest.

CAISO footprint and PacifiCorp and NV electric service territories

Figure 3: CaISO footprint expanding throughout PacifiCorp’s electric service territory (Source: CaISO)


Energy transfers between balancing authorities have been predominantly scheduled according to hourly intervals. The dispatch of generation and load data at 15-minute intervals will further add to the data management challenge by creating more stress for the data exchange velocity and storage.

Power Markets Become Integrated in Europe

On the other side of the Atlantic, electricity market integration has been progressing since Denmark, Finland, Norway, and Sweden ratified a market splitting model following the liberalization of the electricity market in the 1990s. France, Belgium, and the Netherlands integrated their markets (Central Western Europe – CWE) in 2006. Germany, Luxemburg, and Great Britain joined the CWE in 2010 and 2011 respectively. Spain and Portugal coupled their markets in 2007, Czech Republic and Slovakia in 2011, and Italy and Slovenia in 2011. The latest step of European market integration took place on November 19, 2014, with the creation of acoupled day-ahead electricity marketof OTE in Czech Republic, HUPX in Hungary, OPCOM in Romania, and OKTE in Slovakia.

Just nine months prior, the price coupling of northwestern Europe occurred on February 4, 2014, which provided day-ahead trading in Belgium, Denmark, Finland, France, Germany, Austria, Latvia, Lithuania, Luxembourg, the Netherlands, Norway, Estonia, Sweden, and the United Kingdom.[4]

In Figure 4, the dark blue area represents a fully coupled day-ahead market as of February 4, 2014. Spanish and Portuguese markets coupled on May 13, 2014, which is represented by the orange area. Italy is expected to operate a fully coupled day-ahead market by 2015.

Fully coupled day-ahead electricity market in Europe

Figure 4: Fully coupled day-ahead electricity market in Europe (Source: ACER)


The expansion between these countries is considered a step towards enhancing reliability by easing cross-border trade and increasing competition between suppliers. The further expansion of interconnected borders will demand a greater need for coordination between market participants.

As European power markets converge, data sources associated with these markets will also go through changes. Market participants have to watch very closely how data reports, especially from the key data providers such as EPEX SPOT and APX, are being transformed. With consolidation occurring so quickly and frequently, this task can be very challenging for those with limited resources in dedicated data monitoring function.

But it isn’t only the power markets that are shifting; natural gas markets are also experiencing significant changes.

Global LNG Trade: More Opportunities and More Intricacies

Because of the sudden surge in shale gas production in recent years, the natural gas markets have undergone a fundamental shift. As of 2012, North America leads the world in shale gas production, growing by an average rate of 1.6% per year.[5] Up until 2009, the majority of total US shale gas production came from the Barnett shale. However, in the last few years, production from other plays, particularly the Haynesville and Marcellus, is increasing rapidly. Figure 5 represents the increase in US natural gas production between 2010 and 2013.

US natural gas production has been undergoing an increasing trend

Figure 5: US natural gas production has been undergoing an increasing trend (Data Source: EIA)


According to the EIA’s Annual Energy Outlook 2013, shale gas production is predicted to almost double between 2013 and 2040.[6]

The price of natural gas is an important factor in the rapid increase in shale gas production in the United States. Since 2006, the development of shale gas resources has spurred growth in natural gas production, with producers seeing opportunities as a result of growing demand from abroad.The EIA predicts that the United States will export approximately 3.6 trillion cubic feet of natural gas annually by 2040.[7]

In the case of the global LNG market, North American suppliers have been challenged to keep pace with demand. Europe and Asia are major areas of interest for North American LNG. North American LNG companies have been making efforts to enhance their positions in the European market by creating long-term LNG contracts that are beneficial for European consumers and that can be delivered to European spot markets.

For example, in June 2014, Gas Natural from Spain signed a 20-year contract with Cheniere that links natural gas prices to the US spot market for Henry Hub.[8] BG Group signed a similar contract with Cheniere in 2011. Growing exports of LNG from North America and a more globalized gas market will likely lead the US Henry Hub price benchmark to become the driver of natural gas prices in Europe. The graph in Figure 6 represents similar prices at global trading benchmarks for Henry Hub and National Balancing Point (NBP).

ZEMA graph comparing prices at global trading benchmarks for Henry Hub, Brent, and NBP

Figure 6: ZEMA graph comparing prices at global trading benchmarks for Henry Hub, Brent, and NBP (Data Source: NYMEX)


On November 21, 2011, Norwegian Statoil signed a 13 billion GBP supply agreement with UK utility Centrica that links natural gas prices to NBP benchmarks.[9] Centrica also signed an agreement with Qatar for 2.4 million tons (3.26 bcm) per year of LNG supplies.[10] Henry Hub may soon play the same role in Japan, thanks to approval of several LNG export facilities in the United States. In September 2014, the US Department of Energy approved the export of LNG to non-Free Trade Agreement countries, such as Japan and other Asian countries, representing a significant milestone for the industry. FERC approved the Sempra Energy’s LNG export terminal in Louisiana and Carib Energy’s smaller terminal in Florida.[11] They are the second and third facilities approved to export LNG to the Asia-Pacific region.

Canada is the fourth-largest exporter of natural gas in the world. Although Canada has intentions to export LNG, plans for seven LNG terminals have either been cancelled or suspended due to political and economic factors.[12] Australia is poised to become the largest supplier of LNG in the world and to become a leading exporter of LNG to the Asia-Pacific region once LNG export terminals come on line. Australian LNG exports are forecast to climb to 83 million tons a year by 2020, compared with 79 million tons for Qatar.[13] East Africa may emerge as a strong competitor to Qatar and Australia in the battle to capture key export markets in Asia.

As future LNG markets expand throughout the world, LNG routes will evolve and become more flexible. This is apparent in Figure 7, which represents current cross-border LNG trade among North America, Europe, Asia, and Australia.

Future global LNG trade routes

Figure 7: Future global LNG trade routes (Source: Citi Investment Research and Analysis)


As LNG supply, terminals, plants, shipping, liquefaction capacities, trade contracts, and net exports increase, so will the volumes of data associated with them.

This data boom leads to a special use case of the data management mess, where power, natural gas, and other energy and commodity market participants face a deluge of market data from multiple sources. Such a wealth of data can be exceedingly complicated to correlate and interpret using manual data collection and management processes. As the data requirements of these markets are complex and constantly expanding, finding ways to efficiently track, manage, organize, and store data is among the most significant challenges.

When it comes to addressing this challenge, many energy companies are confronted with the decision of implementing an Enterprise Data Management (EDM) solution to support their operational needs.

The Enterprise Data Management Solution

In order to meet the challenges of shifting energy markets and the volumes of accompanying data, many companies with strong IT departments are attracted to the idea of building one-off software solutions created solely for their business. However, creating software from scratch is, in most cases, an unnecessary risk. Research suggests that 31.1% of software development projects will be cancelled at some point during development, and more than half will cost at least 189% of organizations’ original estimates. On average, only 16% of build projects will be successful and will often have less functionality than originally intended.[14] As well, the average time-to-market of an EDM software development project is three years. Overshooting timelines is extremely common, as those projects that make it to some form of completion will often require re-design and revaluation. Restarts are another major issue that disrupts these projects, often resulting in their cancellation before they even near completion.

Buying a commercial, off-the-shelf EDM reduces the time-to-market risks inherent in building an EDM solution internally. Software companies exist solely for the purpose of developing robust and efficient software solutions. Even an organization with a very strong IT department will not have nearly the level of expertise and dedication as a business that is devoted entirely to developing software products. For this reason, when a business opts to purchase a solution rather than attempting to create it themselves, they aren’t just buying the product—they are also buying the thousands of hours of research and development that went into its creation and continual maintenance.

The majority of issues faced by businesses are not uncommon. As such, a software company with proven success developing solutions for a particular industry will have gained valuable insights from helping other businesses that have faced similar challenges. Nearly all software vendors incorporate suggestions from end-users to create continuous updates to functionality; the solution will continue to evolve, adding value without requiring constant investment of capital and resources.


Request ZE’s white paper “Weighing the Options: Build an Internal EDM System or Get ZEMA” to learn more.



[1]Edelson, David, “Broader Regional Markets,” New York Independent System Operator, May 24, 2014, accessed December 5, 2014,

[2]“PJM, NYISO Implement Coordinated Transaction Scheduling,” New York Independent System Operator, November 5, 2014, accessed December 5, 2014, NYISO Implement Coordinated Transaction Scheduling – 11 5 14 – FINAL.pdf.

[3]“Order Conditionally Accepting in Part, Rejecting in Part, Accepting and Suspending Filings in Part, Establishing Hearing and Settlement Judge Procedures and Consolidation Proceedings and Directing Compliance Filing,” United States of America Federal Energy Regulatory Commission, November 10, 2014, accessed December 5, 2014,

[4]“Market Coupling: A Major Step Towards Market Integration,” EPEX SPOT, accessed December 7, 2014,

[5]“Annual Energy Outlook 2014,” US Energy Information Administration, May 7, 2014, accessed December 8, 2014,

[6]“Annual Energy Outlook 2013,” US Energy Information Administration, April, 2013, accessed December 8, 2014,

[7] Ibid.

[8]Toyer, Julien, “Cheniere, Gas Natural LNG Deal worth $13 Billion: Source,” Reuters, June 2, 2014, accessed December 9, 2014,

[9]“Centrica Establishes Strategic Partnership with Statoil and Announces £1bn Norwegian Asset Acquisition,” Centrica, November 21, 2013, accessed December 9, 2014,

[10]“Centrica Signs £2bn Gas Deal with Qatar to Secure LNG for the UK,” Centrica, February 23, 2011, accessed December 9, 2014,

[11]“North American LNG Import /Export Terminals Approved,” United States of America Federal Energy Regulatory Commission, December 3, 2014, accessed December 10, 2014,

[12]Yee, Adrian K., “North American LNG Projects: Political and Economic Factors,” ZE DataWatch, November 28, 2014, accessed December 10, 2014,

[13]Sharples, Ben, “Australian LNG Contracts Under Pressure From U.S., Russia Supply,” Bloomberg, September 9, 2014, accessed December 10, 2014,

[14] “Chaos,” The Standish Group, 1995: 2, accessed November 26, 2014,



 Effective Date: December 30, 2014

Post a Comment


More Articles on In Depth

Reading the Commodity Market Barometer: Energy and Non-Energy Commodity Price Movements

As the world’s population has burgeoned over the past decade, particularly in regions like the Asia-Pacific, technological and infrastructural developments and improvements have also occurred. Increased economic and industrial activities worldwide have contributed to a growing global demand for energy,1 ... Read more »

Middle Eastern and Asian Commodity Exchanges Turn Up the Volume

Emerging markets in the Middle East and Asia are growing faster than markets in the western world, partly due to their large infrastructure projects and subsequent need for imported commodities. Commodity market participants in the Middle East and China, like their ... Read more »

 Effective Date: January 31, 2014

Price Reporting Agencies, Assessments, and Increased Asian Oil Consumption

Sales and purchases of varied commodities affect each country in the world, but few commodities have a global impact greater than that of oil. Price reporting agencies (PRAs) monitor the global oil industry carefully, producing price assessments which are meant ... Read more »

 Effective Date: December 23, 2013

Data Deluge and the Dread of Latency

In recent years, data has taken the business world by storm. The amount of information captured by enterprises along with a growth in technology has fuelled a global data explosion, and its effects on industry have been substantial. In a ... Read more »

In Search of Equilibrium: Growth of Capacity and Transmission in the NYISO

The New York Independent System Operator (NYISO) was founded in December of 1999, taking over from the New York Power Pool and following the nationwide unbundling of electricity supply and delivery in the nineties.[i] It is the responsibility of ... Read more »

A Brief History of Oil Prices and Middle East Tensions: Fear of a Black Gold Shortage

In the final weeks of summer 2013, tension over Syria’s alleged use of chemical weapons was followed by the threat of U.S. military action against the country, which sent oil prices† on a roller coaster ride. The price sensitivity of ... Read more »

Will the Rise of Distributed Energy Spell the End for Traditional Utilities?

It would be an understatement to say that utilities have been bracing for hard times in the last few years. Figure 1. Passing phase or permanent ... Read more »

Powering a Continent: The Multi-Billion Dollar Scramble for Africa

During an official visit by President Barack Obama to Africa last month, a Washington Post blogger traveling with the delegation reported that the Tanzanian hotel he was staying in had experienced ‘a few kinks’. During a nationwide blackout, the lights ... Read more »

Europe 2020: Bane or Boon to European Energy and Power Markets?

When the Intergovernmental Panel on Climate Change (IPCC) published its Fourth Assessment Report in 2007 and affirmed that global warming is anthropogenic, its climate model projections indicated that, without immediate intervention to reduce our greenhouse gas emissions, global surface temperature ... Read more »

Coordination of Gas and Electric Industries: Increasing Transparency?

Over the last decade, the electric power industry has been on an unprecedented rollercoaster ride throughout a whirlwind of market ups and downs, regulatory twists and turns, and environmental swings. It has gone through a series of major changes. Even ... Read more »


Page 3 of 612345...Last »