Collective Intelligence & the Energy Trading Floor of the New Decade

Are you tired of hearing how big data, artificial intelligence, and machine learning will eventually solve all complex decisions in the energy markets?

Blame it on baseball.

Seriously.

Sabermetrics began the quest to use data to predict the future. Once the Boston Red Sox embraced sabermetrics and finally conquered the “Curse of the Bambino”, Silicon Valley realized the impossible was possible.  

The age of algorithms had arrived.

Today, successful, forward-thinking organizations understand data analytics is now in just about everybody’s job description, and analyzing raw information to make conclusions colors every facet of daily business life. However, simply embracing big data analysis is only a first step, and even game-changers like machine learning will never fully ensure our ability to solve every intricate business decision, including those associated with problems as complex as energy portfolio optimization.

Once considered a bulwark against the forces of modernization, industry, and technological change is pushing energy trading organizations to consider a number of strategic changes to their front, middle, and back office functions that optimally engage people and systems to navigate our VUCA world.

The Way It Was

For roughly the past 25 years, energy trading operations could best be described as centralized, top-down control organizations typically divided into three distinct groups:

• The Front Office oversees transactional functions that include identifying market opportunities, buying low, selling high, and complying with the company’s risk policy.

• The Middle Office monitors the front office from the perspective of risk policy, as well as credit and regulatory compliance.

• The Back Office pays the bills and collects the money. Largely forensic in nature, it also corrects front office data entry errors and completes month-ending accounting requirements.

Bright red lines have segmented the roles and responsibilities among these groups, which has been one reason why the physical proximity of the front, mid, and back office teams remained relatively far apart. This time-tested organizational design of energy trading operations was ideal for market(s) of low complexity.

Enter today.

The New Energy Trading Floor:  Collectively Intelligent

“On today’s energy trading floor, everyone is now an analyst to some degree. Growth in the application of collective intelligence will help manage an increasingly complex operational and competitive landscape with constantly shrinking margins for error.

Front, middle, and back office personnel must increasingly collaborate, combining their unique perspectives and skill sets to help solve complex portfolio optimization challenges posed by a fast-changing industry.

• Regional organized markets have taken over most of the country. In areas where RTOs do not yet exist, rapidly expanding solar generation will almost certainly require some type of regional coordination to handle impending real-time price volatility.

• The explosion of intermittent generating resources has exponentially increased the systemic complexity of today’s energy markets. Systemic complexity is increasing faster than data volumes.

• Energy trading has largely become a commoditized service thanks in part to software automation.

• Data Warehouses have given market participants a central repository from which to access large amounts of information accumulated from a wide range of sources.

• Data scientists and analysts have risen to prominence as they employ progressively more powerful software that assimilates and interprets structured and unstructured data to help make ever more refined, granular adjustments to enterprise operations.

In fact, on today’s energy trading floor, everyone is now an analyst to some degree, a reality that has helped create a fertile environment for the growth of collective intelligence to help manage an increasingly complex operational and competitive landscape with constantly shrinking margins for error.

Collective intelligence, as originally defined by MIT Sloan School of Management Professor Thomas Malone, occurs when groups of individuals act collectively in ways that seem intelligent. With the advent and growth of artificial intelligence, the meaning of collective intelligence has evolved to describe interconnected groups of people and computers working in tandem to produce optimal decisions and results.

The most effective and efficient trading floors are embracing this perspective and can take a number of steps to ensure that all front, mid, and back office personnel are working in precise coordination to extract maximum value from their efforts and the data analytics driving them. Advancing the collective intelligence paradigm within an energy trading organization can include the following:

Redesign the Trade Floor

• Benefits
  • Closer physical proximity enhances communication & data analysis.

• Example #1
  • Placing an expert on settlement charges next to a market person provides quicker two-way communication and understanding of “why” a charge was assigned to the portfolio.
  • Post-analysis work will be cleaner and quicker.
  • Back-up staffing is enhanced.
  • Succession planning is easier.

• Example #2
  • Placing mid-office staff closer to daily operations streamlines communication on important questions in a time-sensitive, financially binding market.
  • This move can help change the perception of the middle office as the police to being a source of learning and guidance.
  • Market monitor questions can be addressed and answered more easily.

Redefine, Reassign Roles & Responsibilities

• Benefits
  • Redefining roles and responsibilities may reduce time spent on mundane tasks that can easily be automated, unleashing staff members to focus on higher value-added analytical work.

• Example #1
  • Redefine the role of front office staff to include RTO settlement analysis.
  • Providing immediate feedback on load and generation charges improves optimal day-ahead planning.

• Example #2
  • Re-define the role of analytics formerly exclusive to the middle office.
  • Load forecasting represents one of the biggest financial risks in an RTO market.
  • LMP forecasting represents another big risk when forecasting the gas needs for a combined cycle.
  • Having collective intelligence will lead to reduced forecast errors.

Utilize a Single Software Platform to Stage All Transactional & Analytical Duties

• Benefits
  • Easier, more accurate portfolio risk assessment.
  • Easier data reporting requirements.
  • Reductions in maintenance, testing, and troubleshooting.  
  • Business continuity during events such as hurricanes, floods, or pandemics.

• Example #1
  • Single-source cloud platforms offer easier work-from-home capabilities and do not require internal IT support for home offices or backup sites.

• Example #2
  • Studying load forecasting trends or meter data from the same platform assures all analysts fish from the same well.
  • Reporting becomes more manageable and promotes standardization across the organization.

Prevent Internal Silos from Clinging to the Status Quo

• Benefits
  • Silos oftentimes results in analytics and decision outcomes that are sub-optimal. Razing silos and shifting the overall team’s collective focus typically increases efficiency and encourages higher value-added work.

• Example #1
  • Human nature dictates a reluctance among some to give up task ownership for fear of becoming irrelevant.  
  • Effective leaders explain the individual and team benefits of automating lesser-value tasks to foster more collaboration and better tackle the big problems.
  • “Letting go” is required from leaders, as well as those managed by them.

Change is Easy. Transformation is Hard. Ask Bill James.

The father of sabermetrics, Bill James, helped take the Boston Red Sox from being the best-known “curse” in major professional sports to winning four World Series titles in a decade and a half. He made analytics cool, and his success in Boston validated his trailblazing approach that produced winners based on data and statistical analysis rather than on “gut feel.” James was a boat rocker whose approach and methodology threatened firmly entrenched beliefs about how things were done in a very mature business highly grounded in tradition.

A little like the utility business.

Today, data analytics is no longer new and is woven into nearly every facet of every business on earth. However, the variety, volume, and velocity of truly “big data” and being able to effectively analyze it is relatively new for utilities that have been around a long time but are now operating in an ultra-dynamic industry with multiple, changing value streams and some of the most complex optimization challenges in the history of commodity markets.

The energy trading floor of the new decade (and beyond) will be based on collective intelligence and the entirely new level of collaboration and interdependency it introduces for front, middle, and back office teams. Constructively, successfully migrating to a collective intelligence organizational model can be achieved in many ways, some of which have been presented in this article. However, what works best will depend on the unique characteristics of your enterprise and the people who comprise it.