I was recently paid a house call by a couple of door-to-door salesmen from an electric supplier outfit to inform me of my choice in selecting a supplier of my liking through ComEd, the electric distributor servicing my residential area, and the reasons why I should be choosing this particular supplier for reasons including lower cost, ‘cleaner’ alternative etc. More “Power to People” I guess.
The rise of alternative electric suppliers over the past several years has been due to gradual deregulation of the energy market, right from generation to supply and distribution. Deregulation is one of the many forces that are affecting the long-term viability of traditional utility industry, specifically the venerable power grid that is the bread and butter of the industry. These forces are coming together to form a perfect storm that is increasingly forcing traditional utility companies down a death spiral, indications of which have recently been seen with the German utility industry. Like many other industries, the utility industry has high fixed costs that it must spread across as many customers as possible. Any negative perturbation in the customer base – say due to deregulation – forces per unit charges for the remaining smaller customer base to go up, which in turn exacerbates the perturbation, setting the stage for the death spiral.
What may be an especially dangerous force in this dynamic system is the rapidly maturing alternative power generation technology, particularly solar and wind power generation technologies that enable ‘distributed power generation’ and enable generating consumers to get a free ride on existing grid infrastructure through billing mechanisms such as ‘net metering’. These technologies and the role they are playing in the utility industry remind us of the concept of ‘disruptive innovation’, a term coined by Harvard Business School professor, Clay Christiansen. Disruptive Innovations are innovations that provide value in entirely new ways and thus change the status quo in fundamental ways. Typically, such innovations start at the low-end of the market, at low performance-price ratios, but experience an inflection point due to a technological breakthrough or convergence of exogenous factors, which increases their viability to address the needs of the mass market (see graphic below). The high-tech industry, for example, is littered with examples of technologies that have been rendered obsolete and were ultimately supplanted by disruptive ones. A classic example is the traditional computer disc drive, which Christiansen has discussed at length in his treatment of disruptive innovations.
The utility industry is very different from the fast-moving high-tech industry, and the disruptive death spiral that industry developments are foreboding will likely not happen tomorrow. That should not be a reason, though, for industry executives to be complacent, especially since recent advances are rapidly bringing down the cost and improving the performance of solar and wind power generation, as well being complemented by advances in battery technology that will allow consumers to efficiently store energy for later consumption. Yet more power to the people.
The Economist recently featured a book review on The Second Machine Age by Erik Brynjolfsson and Andrew McAfee , two academics with MIT’s Center for Digital Business. The book is a wonderful treatment summarizing the impact of various technological revolutions of the past including steam engine and electrification, and more recently the so called “Second Machine Age” that began with the introduction of electronic computing in the 1960s. Reading the first few introductory chapters reminded me just how important technology has been to economic growth and betterment of humanity as a whole. If one were to plot the world GDP across time and overlay the major technological revolutions of the past (see graphic below), one cannot help but see the strong correlation between the two.
What is interesting about the data that Brynjolfsson and McAfee present, however, is that we are just getting started with the second machine age. Advances in electric power generation and transmission are continuing to improve growth and productivity to this day, and there is no reason to believe that we have seen everything there is to see with advances enabled by electronic computing. As the graphic from the book comparing the timelines of electrification era and second machine age era shows (see below), we are currently at the same level of productivity gains from electronic computing as we were in the 1930s with electrification.
The other important thing that Brynjolfsson and McAfee point out is how there is a lag effect between the time new technology is introduced and the time actual productivity benefits are realized. When initially deployed, technology may help automate and improve operations to a certain extent, however true benefits accrue when managerial innovation fully exploits the benefits of the new technology, as happened with electrification when simply replacing steam engines with electric motors did little to improve productivity in the beginning, but later delivered great benefits as overall work flows and operations were improved to leverage the new technology.
Enthusiasm around big data’s potential in creating business value has been driven to a great extent by technology innovations that have been shaping up in the database management and analytic platform market over the past decade or so. These innovations have been driven by three primary technology related factors:
- Limitations of Existing Data Management Platforms: As Internet entered the business mainstream, new models and paradigms for large scale data management became necessary. Vertical scaling through addition of raw computing power and disk became increasingly unfavorable as a feasible alternative. Traditional database and data warehousing platforms ran into cost effectiveness limitations in serving increasingly varying data processing workloads.
- Falling Cost of Technology: The price of all three workhorses of modern day data management i.e. disk, CPU, and network, continued to fall, enabling new price-performance characteristics. Use of solid state devices and flash memory is becoming commonplace, more CPU cores mean more processing threads, and Infiniband and other network interconnects allow for speedy movement of massive data sets.
- Computing Platform Innovations: A range of technologies from cloud computing to open source software has enabled the rise of a modern digital platform that is giving rise to new models for sourcing and delivery of data management and analytic capabilities. Google’s BigQuery service represents just the beginning of this evolution.
These factors have driven the emergence of a plethora of vendors and solution providers in the database management and analytic platform market over the past decade and a half. Venture capital investors have poured more than a billion dollars in funding data management start up companies. The market today is a hodge-podge of various players: from traditional data warehousing solution providers adding capabilities and/or acquiring data management startups to round out their product capabilities, to new challengers that are rethinking and re-architecting the data management and processing platform from the ground up. The state of the market today reminds many of the “database wars” of the 1990s when numerous relational database vendors were fighting to capture the database market, ultimately leading to the emergence of a few victors (Oracle, IBM and Microsoft) by the early 2000s. Similar such developments are bound to shape the database and analytic platform in the coming years. However, until that happens, the market is bound to be an interesting smorgasbord of products and solutions.
There are four major logical environments (see graphic below) related to data processing and information delivery that these products and solutions ultimately fall in and are competing to support. These four environments are interconnected by a fifth category of products and solutions: Data Connectors that consists of products and solutions providing integration paths between environments and across legacy and new tools.
In subsequent posts, I will delve more deeply in each of these categories. For each of these categories, we will look at key solution capabilities, and major vendors and their products and solutions that provide those capabilities. The categories above are a logical set of categories, i.e. we will find many vendors that offer an integrated single platform that on its own supports multiple environments. However, we will find it easier to understand this complex and evolving landscape if we attempt to deconstruct the picture logically one piece at a time.
Click here to go the blog homepage.