The Five D’s of Fintech: Disintermediation

Finance, as the adage goes, is the art of passing money from hand to hand until it finally disappears. Like giant spiders in the middle of a system of webs, banks have played a key role in intermediating money flows across the vast capitalistic system, and doing so in a highly profitable manner. Banking from yore – taking deposits from customers and making loans to borrowers – has given way to a lending system that is dominated by non-banking financials institutions and capital markets. This “banking disintermediation” – banks no longer holding the loans they originated on their balance sheets but selling them off; borrowers going directly to the capital markets rather than to banks to obtain a credit; or savers investing directly in securities – refers to credit disintermediation and has been in the making for a number of decades: banks moved from the boring low-profitability business of credit provisioning to high margin fee-based businesses (investment banking, M&A, insurance etc.). “Disintermediation” – the third theme in The Five D’s of Fintech – has taken on a renewed significance in the wake of the rising tide of fintech: this time around, it refers to how banks may potentially be disintermediated from their customers in their fee-based businesses. Customer relationship, the bedrock of origination and sales that dominates such fee-based businesses, is now under attack, and banks have started to take note.

Fintech led disintermediation has been palpable in areas where venture investment has traditionally poured into: peer-to-peer lending, remittances, payments, equity crowdfunding and even robo-advisory. Fintech use cases on disintermediation of traditional payment players are old news. By comparison, the impact of fintech led disintermediation of the banks in capital markets appears to be relatively small. By some estimates, only $4 billion of the $96 billion (around 4%) of the fintech investment since the beginning of the millennium has been directed to capital markets, and of 8000+ fintechs globally, only 570 (around 7%) are operating in capital markets (Fintech in Capital Markets: A Land of Opportunity).

This may be beginning to change not least because of the juicy returns that can potentially be plucked from the investment banks in origination and sales (which is the key activity in fee based business such as investment banking, M&A/payments, asset management etc.) which at an estimated 22% ROE (Cutting through the noise around financial technology), is a much more profitable business than credit provisioning (which is the key activity in core banking (lending, deposits, etc.)) with an estimated ROE of only 6%. Entry barriers to capital markets and investment banking are significant: the highly regulated nature of the business, the oligopolistic nature of the industry, vicious competition from incumbent banks, and complex operating environment. However, regulatory pressures, compressed margins and technology enabled opportunities are forcing incumbent banks to acknowledge fintech led opportunities.

The investment banking world is being buffeted by the forces of digitization, regulation, and competition. Continued electronification of OTC markets, for example, has meant greater transparency around the price discovery process for investors. Dealers have pulled back from markets where regulatory requirements have rendered them unprofitable for those banks, thus providing an opening wedge for non-bank players such as hedge funds and institutional investors to step in. The financial crisis has tarnished banks’ reputation. It is against this backdrop that fintech outfits are evolving from just providing solutions to automate and streamline bank transactions to being serious threats across asset classes (equity/debt/FX), markets (primary/secondary), and the value chain (origination through distribution and dealing). In the world of supply chain financing, technology-led disintermediation has begun to make inroads. Regulatory pressures, decreasing margins, and KYC/AML challenges have made it difficult for commercial banks to scale their supply chain finance business: the market is underserved and rapidly growing. This is changing with the emergence of such platforms as PrimeRevenue, an American supply chain financing platform, that connects 70 lenders, including 50-odd banks, to 25,000 suppliers with $7bn-worth of invoices a month. While such platforms are not overthrowing banks’ supply chain financing business completely, they are slowly but surely intermediating themselves into direct relationships with end customers: witness how C2FO has offered Citi more lending opportunities in return for access to its supply chain financing customers.

The origination market for new securities issuance is an area where incumbents are still strong but evolving conditions may favor fintech led innovation. Some large asset managers want to ensure that corporations structure securities that those asset managers want to buy, not necessarily those that a bank may structure: this may just be the kind of solution a fintech may develop. Platforms that connect investors directly with issuers are few and far between in the equity and debt primary markets, and even those that are there have dealers in the mix, but slowly and surely there outfits such as Origin (a private placement platform aiming to simplify and automate the issuance of Medium Term Notes) and Overbond (a Canadian platform provider aiming to streamline bond issuance for both private- placements and public offerings) that are going to market with automation and auditing solutions today, but which may in the future choose to intermediate themselves more prominently with investors and issuers by offering data assets and related value added services.

There are, however, more worrisome developments for banks in the securities origination market. Investors are discovering new avenues for investing, for example equity crowdfunding, which may affect bank’s traditional business in private placements and investment banking payments. There are indications that institutional investors have already started to use crowdfunding websites to gain stakes in new businesses, such as in the real estate sector. One website, for example, enables crowdfunders to pool their capital and compete with institutional investors or co-invest with venture capital funds. There is a perceived threat that more buy-side firms will tap crowdfunding sites. Most worrisome for banks is perhaps the specter of alternate fund raising models whereby conventional IPO gives way to alternate intermediated mechanisms such as electronic auctions, crowdfunding and initial coin offerings (ICOs).

New capital raising models represent the latest in how disintermediation is playing out at the very heart of the capitalistic system. Equity crowdfunding has the potential to disintermediate early stage financing and democratizing access to capital and investment opportunities (in spite of the regulatory and reporting burdens imposed by the JOBS Act): witness the rise of such platforms as Kickstarter and Indiegogo. Equity crowdfunding has been thriving for a few years now, but it is Initial Coin Offering (ICOs) that has attracted investor attention of late.

In contrast to a traditional venture funding or IPOs where investors typically get a slice of the equity and control in the business, ICOs involve raising money by selling a cryptocurrency – called a “token” – at a discount to investors who can then trade the tokens on a crypto exchange. ICOs are different from traditional fundraising and even equity crowdfunding: they are unregulated, and startups do not have to fork out a slice of their equity. Indeed, ICOs have been on a tear: a quarter of the blockchain investment over the past couple of years (approx. $250 million) has come from ICOs, two recent ICOs (Qtum and OMG) passed the unicorn mark in a matter of mere months, and storied VCs are investing in ICO funded startups (Cryptocurrency Hedge Fund Polychain Raises $10 million). ICOs offer yet another fund raising avenue for startups that are skittish about handing over control to outsiders, those that do not want to go through the regulatory hassle of taking their company public, or those that have been neglected by the market. For the investors, ICOs provide the lure of massive returns (although they are extremely high risk and many are scams).

Capital markets’ “Uber moment” may not be around the corner, but the capitalistic middlemen are going to have to increasingly define what value they bring in light of disintermediated banking models being championed by fintechs. It is not just capital raising where fintech startups are mushrooming, but in other areas of the capital markets as well, for example clearing and settlement where blockchain-based startups are providing capabilities for fully automated settlement without the need for a central counterparty. Symbiont, a New York based blockchain startup has demonstrated how “smart securities” can streamline post trade processing and settlement. This has huge implications for middle/back office functions in banks, and the wider capital market ecosystem including custodians, clearing houses and depository institutions. This redundancy of centralized processing in the fintech utopia will be the theme of the next installment of the 5 D’s of Fintech series.


The Five D’s of Fintech: Disaggregation


“Death by a thousand cuts”, “Sandblasting”, “Mastodons attacked by ants” and similar such metaphors have been used to describe the scourge of fintech and insurtech insurgents and their impact on incumbent banks and insurance companies. “Disaggregation” or “Unbundling” of products services lies behind fintech’s poaching of customers from banks, not singlehandedly but collectively, one product or service at a time. Stories and studies abound on how technology is unbundling not just products and services, but impacting entire value chains. This disaggregation is now well established in traditional banking (See CBInsight’s blog post as an example), but only now emerging in insurance. Disaggregation is the topic of this post, the second installment in the fintech series (see The Five D’s of Fintech: Introduction), in which we will look at where and how such disaggregation is taking place in the insurance industry.

The insurance industry is experiencing a technological double whammy of sorts: not only does technology enable creation and usage of new behavioral context which fuels new competition, but also it stands to threaten the underlying business economics of the industry. Insurance experts talk about “behavior disaggregation” to describe how consumer behaviors can be tracked and analyzed to engage with consumers directly in real-time, price risk accurately and discretely, and provide frictionless services. For example, It is not hard to imagine a “connected home” where various safety measures one might take e.g., installing a burglar alarm, are instantly used to recalibrate risk and thus adjust the homeowners or renters insurance. Tech savvy startups are already leveraging such behavior disaggregation: pay as you go car insurance, from Metromile is an example where driver behavior can be tracked to provide fit for purpose policies. Oscar, a health insurer in New York, gives all its policyholders a fitness tracker; whenever they hit a set goal (walking 10,000 steps in a day, say) they get a refund of a dollar. Where incumbent insurers put their consumers in a policy straightjacket based on blunt risk indicators such as age and occupation, insurtech players leverage technology to study behaviors down to the ‘last mile’ to offer highly flexible solutions, customized not just by the asset type insured but also by duration, for example through “micro duration” policies. These companies have the focus and the agility to build this specialization: like Metromile has done for car insurance, Trov has done for general merchandize, Ladder for life, and Hippo for home. Increasingly traditional insurers may find that insurtech’s win-win value propositions are hard to beat with traditional product bundles – hence the case for unbundling. But unbundling existing products is not enough. Unfortunately for traditional insurers, insurtech’s micro targeting of risk reduces incumbents’ existing risk and profit pools: they are thus forced to not just compete with the upstarts, but also seek out new revenue sources.

Insurtech is forcing disaggregation at an industry-level scale. Incumbents all have traditionally managed the entire value chain spanning product/service distribution, underwriting, claims management and investment/risk management. Using independent agency networks, conventional underwriting models and re-insurance for risk management, carriers have thrived in the marketplace. This value chain is now coming apart as each link in the chain is impacted by technology. New digital sources of distribution are threatening to disintermediate carriers from end consumers – just witness the rise of CoverHound for auto, PolicyGenius for life and disability, Abaris for annuities, and various others. Traditional risk pools are shrinking, risk is migrating from consumers to products, and the nature of risk is evolving thanks to self-driving cars, IoT technologies and the sharing economy — all this has led to emergence of new competitors offering alternate underwriting models such as Lemonade, Guevara, and Friendsurance. Upstarts such as Claimable seek to intermediate to provide easy claims settlement experience to end consumers. New arrangements such as Managing General Agents, catastrophe bonds and collaterized reinsurance are disaggregating the carrier/re-insurer relationship: now carriers can go directly to capital markets, and re-insurers can strike up business arrangements with startups focused on customer acquisition.   The neatly linked insurance value chain is slowly moving to horizontal stacks based structure (see BCG’s Philip Evans’ idea of stacks here).

Insurance is different from traditional financial services in that players in insurance, unless they are pure brokers, have to take on element of risk and hold associated capital, all of which comes with ton loads of regulatory requirements. Due to these reasons, insurtech has been slow to penetrate the $6 trillion insurance industry goliath. The pace however may accelerate. According to McKinsey, automation could leave up to 25 percent of the insurance industry’s current full-time positions consolidated or replaced over the next decade (see Automating the Insurance Industry). If nothing else, carriers should do everything in their power to prevent disintermediation, which will be the topic of the next installment in the fintech series.

The Five D’s of Fintech: Introduction

FinTech“Fintech” (a portmanteau of financial technology that refers to the disruptive application of technology to processes, products and business models in the financial services industry) is coming of age: two of the most prominent Fintechers, OnDeck and Lending Club, have gone public, many more are processing transactions to the order of billions of dollars, and outfits providing market intelligence in fintech are cropping up – there is even a newly minted index to track activity in marketplace lending. Banks are increasingly taking note of the Fintech movement, partnering with startups, investing in them or even acquiring them outright. Venture funding in fintech grew by 300% in one year to $12 billion in 2014.  According to the Goldman Sachs’s “Future of Finance” report, the total value of the market that can potentially be disrupted by Fintechers is an estimated $4.3 trillion.

Fintech is a complex market, spanning a broad swath of finance across individual and institutional markets and including market infrastructure providers as well. It is a broadly defined category for upstarts who have a different philosophy around how finance should function and how it should serve individuals and institutions. While some Fintechers seek to reduce transaction fees and improve customer experience, others exist to provide more visibility into the inner working of finance. In spite of this diversity, there are some common threads and recurring themes around why Fintech firms exist and what their market philosophy is. The 5 D’s of Fintech – Democratization, Disaggregation, Disintermediation, Decentralization and De-biasing – represent common themes around the mission, business models, values, and goals of many of these firms. In this series of posts on Fintech, we will look at each of the 5 D’s of Fintech, starting with Democratization — the mission of many a Fintech firm.

The Five D’s of Fintech

Fintech Slides


Technology has long enabled democratized access to financial services, however Fintech is taking the movement to another level by targeting specific market niches with customized value propositions. A central appeal of many Fintechers is their promise to bring to the masses resources and capabilities which heretofore have been the preserve of the wealthy, elite, or the privileged. This has been made possible by both by market opportunity and internal capability: market opportunity of serving a market whitespace, and the ability to do so economically through the use of data and advanced technologies.

The financial inclusion that Fintechers are now enabling is driven by their ability to clear obstacles, remove barriers, and enable access where none existed before, whether it is serving the unserved or underserved SMBs that have typically been shunned by traditional banks (Funding Circle), providing credit to the underbanked segment lacking the traditional credit scores (Kreditech), enabling investment advice without the need to rely on expensive financial advisors (Nutmeg or Betterment), or facilitating access to the capital markets by offering low-cost brokerage services (Robinhood). Financial services are now “for the people” and “by the people” as well: Quantiacs, a fintech startup with the aim of revolutionizing the hedge fund industry, is essentially a market place for quantitative trading strategies that enables anyone to market their quantitative skills and trading strategies. Or OpenFolio, which is an online community that allows one to link portfolios and measure investment performance against their communities and relevant benchmarks. Wealth management perhaps is a market ripest for democratization as shown by rapid emergence of a raft of outfits such as HedgeCoVest and iBillionaire (platforms that allow investors to mirror the trades of hedge funds and billionaires, respectively), Loyal3 (which offers no fee access to IPOs), Algomi and True Potential (which undo trading obstacles for investors).

As Vikas Raj with Accion Venture Lab notes, the real potential of fintech is in democratizing access to finance for the billions of low-income unbanked population in the emerging markets. The high complexity and low scale nature of this market is exactly the kind Fintechers are good at capitalizing on, and this is evident from the long list of companies that is emerging in this market beyond Silicon Valley and New York. Where traditional finance and government agencies have failed, Fintech has the promise and the potential to excel.

Other industries can learn a lot by observing how Fintech is driving democratization in finance. Whether it is healthcare, education, media or government services, there is potential value in market segments that are currently un/under served which a Fintech like movement can unlock. Adopting the technologies underlying Fintech is part of the story; what is needed first is the recognition of the potential for change, the support from the markets, and an entrepreneurial spirit to lead the movement.



Paths to Power: What can Architects Learn from Sources of America’s Global Dominance

In discussing American exceptionalism and influence in the modern world, political scientists and academics talk about various sources of American power, for example US military might or its economic dominance.  Walter Russell Mead, an academic and a foreign policy analyst, defines four kinds of American power: the sharp power of the US military that provides the foundation, the soft power of American pop culture and the sticky power of an American-led economic world order that build atop the foundation, and finally at the very top the hegemonic power of the US, a prerogative and privilege to provide credence and confidence to all things worldly, whether they be economic coalitions or other global pursuits.  Organizational theorists and researchers have done extensive study of power and influence of individuals and groups within organizations, however the political scientists’ lens provides an interesting framework to understand how power should be built and exercised.  An example of such an application is looking at how the architecture function wields and exercises power with an IT organization.

Architects typically exercise their power and exert influence through formal governance processes and mechanisms – the infamous “architecture review board” is one of the various architecture forums that have given many a sleepless night to IT program managers.  Architecture governance can be characterized as the architects’ sharp power that is exercised by poking and prodding the IT teams to behave in a certain manner.  Just as a nation’s sharp military power serves as an important foundation for its power structure, so too does the architecture governance framework for the architecture group.  But, a foundation is not enough to create and sustain power.  US’s overwhelming military power is a powerful deterrent for many, but it is far from enough to ensure American dominance in world affairs.  And this has been demonstrated time and again by many IT organizations where in spite of having strong governance processes, the architecture teams fails to have appropriate influence in decision-making.  What is needed to complement the sharp power of governance are the architecture team’s sticky and soft powers.

America’s sticky power rests on two foundations: a dollar based global monetary system and free trade.  A dollar-based system not only allows America to reap direct economic benefits e.g., seigniorage from issuing bank notes to lenders all over the world, but it also allows America to exert the influence of its monetary and fiscal policies far and wide.  Globalization and fee trade, championed by American based multinationals, has created an economic framework in which benefits accrue to all countries that participate.  There are strong incentives for a country to participate in such an economic system, and strong disincentives to leave.  Architects have an opportunity to wield sticky power by creating a system in which their work creates benefits for their key stakeholders – architects should reorient their work to facilitate the outcomes and key decision-making IT management would like to have but cannot due to political, organizational or technical reasons.  Be it brokering talks, striking alliances, or negotiating trade-offs, architects in this role will need to function as first-class deal makers.  And in this role, architects will be seen belonging to an elite group that knows how to get things done. In an ideal state, IT managers will have no choice but to engage with the architects, because of the assurance that such an engagement will provide for the achievement of the outcomes they desire.  Achieving initial success will create a self-reinforcing system in which stakeholders will be increasingly willing to work with the architecture team and increasingly hesitant to work against them.  Sticky power, however, needs an inducing agent, something that will draw others in.  That is where soft power comes in.

American soft power emanates from America’s ability to successfully export its culture, values, ideals, and ideas to the rest of the world.  Architecture’s ability to successfully project its image across the entire organization is critical to garnering soft power.  The typical perception of architecture work being just technically focused is a result of architecture’s team failure to project an image commensurate with the stature of true architecture work.  Architects need to build a culture, set of ideas, principles and work habits that are unique and suited to the problem-solving, deal-making, and relationship and influence building that architecture work demands.  It starts with hiring the best and the brightest – not just the technically savvy, but true leaders, strategic thinkers and doers.  But creating such a culture is not enough – it needs to be packaged and “exported” to the rest of the organization, with the goal of creating an environment where the rest actually want to go join the architecture team because it has a certain cool factor.  On this scale, the typical architecture team fails miserably – one has yet to come across an IT organization where he architecture team is considered one of the cool teams to join.

Hegemonic power, the holy grail on the road to building power and influence, is power that comes from powers of all other kinds – it is the cherry on the top of the icing that allows America to set the agenda and rules for world affairs, or be the “gyroscope of the world”.  The architecture team is in a prime position to be the gyroscope of the IT organization.  By combining the power of governance with the ability to create real value for their stakeholders and attracting the best talent, the architecture team can influence decision-making at the highest levels – it can set the agenda to facilitate its own goals and outcomes and thus perpetuate its power and influence.   The nature of architecture work is such that sustaining power and influence are crucial to ensuring long term success of architects.  Maintaining power on an ongoing basis however takes effort, wise decision-making, and moving with the times – just witness the case of Britain, which only around a hundred years ago, was the world’s leading power by far and wide, but which was gradually supplanted by America as it held onto its stubborn positions and made fatal policy mistakes.

Testing in a Continuous Delivery Environment

In his book Out of the Crisis, W. Edwards Deming cautioned “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place”. Ever since, ‘Building Quality In’ has become one of the central tenets of quality focused lean initiatives, including lean software development. The act of testing in software development is an example of inspection: inspection to find bugs and faults in the developed software; static code analysis is another example of inspection. Quality is important in the context of software development because software bugs cost both users and software providers dearly: a study conducted on behalf of the US National Institute of Standards and Technology estimated the cost of software bugs to the US economy to be around $60 billion. Perhaps the extent of this scourge is not surprising since in many organizations software testing is not effective: testing/QA teams run “quality gates” as an afterthought and even then testing does not necessarily translate into quality. When Agile came around, practitioners came up with new approaches to testing, aptly described under the banner of “Agile Testing”, that provided some improvement by driving more collaboration across teams and bringing testing up in the development cycle. Now with the advent of DevOps, testing specifically has taken on a new level of significance since continuous delivery is not just about delivering software rapidly, but software that works as well. A few have even coined a term for this new discipline: continuous testing. All that is well, but what does testing mean in a continuous integration/delivery environment?

blue devops_4

In a continuous delivery (CD) environment, quality becomes the responsibility of all. This does not mean that the QA and testing teams do not have a role to play in a CD environment. On the contrary, the QA and testing function moves into a strategic role, providing oversight, direction and leadership for diving overall quality. For example, instead of spending countless hours running manual tests, QA teams will invest resources and time to develop and implement a comprehensive test automation strategy, or they will spend effort putting in place governance processes, metrics and incentives to drive quality at every step. An example of how quality becomes everybody’s responsibility is what the development staff would do in such an environment. Development teams in a CD environment are empowered to take on quite a bit of testing upon themselves. In addition to a ‘test first’ approach, developers may also be required run pre commit testing that runs a suite of unit, component and integration tests. Indeed many CI servers provide the capability for ‘private builds’, which allows an individual developer to see if their code changes can be integrated into the main trunk for a successful build. Pre commit testing should enable developers to conduct a quick ‘smoke test’ to ensure that their work will not break the code in the main trunk. Therefore, pre commit testing may contain a selection of integration and acceptance tests. Once the developer checks in the code to the CI server after pre commit testing, the CI server runs the commit stage tests, which includes performing any static code analysis as required, component and integration testing, followed by system testing. Commit stage testing results are immediately fed back to the development team to get any errors or bugs addressed. Successful commit stage testing increases confidence in the build’s ability to be a candidate for acceptance testing. Builds failing commit stage testing do not progress to the next stage: the acceptance testing stage.

Acceptance testing is the domain of business analysts and business representatives assigned to the project team. However, this should not mean that development staff do not have any involvement in acceptance testing. Successful testing in a CD environment gives developers more ownership in driving quality by allowing them to conduct automated acceptance tests in their development environments. Common obstacles to enabling this, such as insufficient licenses and/or manual deployment and setup processes, need to be removed. Acceptance testing is a critical step in the deployment pipeline: a release is deemed acceptable for deployment only if it passes the acceptance test stage. The entire team should focus on fixing acceptance testing issues for a given release. A CD environment requires acceptance testing to be automated as much as possible: a fully automated acceptance testing suite enables the tests to be run for a build as when needed – this speeds up the development process and also enables creation of a powerful suite of regression tests that can be run over and over again. Some tools even offer capabilities to encode acceptance test criteria and to programmatically drive creation of acceptance testing based on those criteria: thus testing, and hence ultimately delivered software, can never be out of sync with evolving acceptance criteria and requirements.

If the system under development is a high performance system, some capacity and performance testing may become part of acceptance testing as well. Usually however capacity testing and testing for other ‘non-functional requirements’ is separate stage in a CD deployment pipeline. Although a CD environment requires such tests to be as automated as possible e.g. through the use of Recorded Interaction Templates and other devices, the success criteria for such tests is somewhat subjective – so although a release may fail automated capacity testing technically, it may still be greenlighted to go ahead based on human judgment. Ultimately, as the release completes the non-functional testing stage gate, it may then be put through more of the traditional manual testing. This is where human testers can excel and apply their expertise in UI testing, exploratory testing, and in creating unique testing conditions that automated testing may have not tested the software for. Manual testing effort thus is one of the last stages in the testing pipeline in a CD environment.

If testing is to indeed become ‘continuous’ in nature, there are several critical factors that need to be in place. Perhaps the most critical one is test automation, which is often times slammed by some practitioners to be difficult to do or non-value added. Whatever the reservations with automation, testing in a CD environment cannot possibly be efficient and effective without automation of testing – especially since testing is done in big numbers and it is done quite often. Automation is just one piece of the various test design and execution strategies to make testing execute efficiently and thus be successful in a CD environment. For example, CD practitioners recommend a commit testing stage lasting no more than 10 minutes – a hurdle that can be met only by adopting such strategies. Automation also applies to provisioning of and deployment to environments. ‘Push button’ deployments and provisioning of test environments is critical if developers are to conduct smoke acceptance tests to quickly test their work. Similarly, test data needs to be managed effectively. Test design and test isolation need to be such that data requirements for testing purposes are fit for purpose and parsimonious : wholesale replication of production data is neither feasible nor recommended in a CD environment. Data management, like environment management, needs to be fully automated with configurable design and push button techniques.

Testing has the opportunity to move from being a reactive purely static analysis function to being a proactive quality focused initiative. Achieving this requires making a number of tough decisions related to processes, division of responsibilities and organization of the testing effort. Making these tough decisions in a traditional environment is many times a choice. Moving to a CD environment however mandates those decisions to be made, which should be reason enough for organizations to start examining today how they can evolve and improve their testing efforts toward that ultimate model.

Transparency in Pharma and Drug Industries

With the passing of the Affordable Care Act and the recent ratification provided by the US Supreme Court in King Vs Burwell for federal tax subsidies, there is now a strengthening movement toward providing more transparency and accountability in the health care industry in America. Pharma and drug companies, an important part of the health value chain, suffer from a number of transparency issues. While the ACA made some progress in enabling this transparency by mandating that pharma companies make public drug related payments made to doctors, much remains to be done on other fronts such as publication of clinical trials data and full disclosure of drug side effects. A few, such as Dr. Ben Goldacre, the founder of AllTrials, have launched public movements to campaign for more data transparency in the pharma and drug industry. More data transparency, however, can be a double-edged sword. There are benefits, however, there are practical considerations as well.


Providing transparency around clinical data can be valuable. When clinical data on a class of antidepressants called selective serotonin-reuptake inhibitors (SSRIs) was analyzed, an increased risk of suicide among adolescents from the use of SSRIs was discovered. Similarly, when the raw clinical data of Tamiflu was analyzed, Tamiflu’s efficacy in fighting viral infections and reducing hospital admission rates was brought into question. Like any large scale statistical analyses of data, clinical data analysis upon which drug companies, regulators and government agencies depend for risk evaluation and approvals, can have anything ranging from egregious mistakes to subtle biases. These can stem from a number of factors, including selection bias in the controlled trials, or mistakes in interpreting statistical significance. The latter, in which the statistical model either lacks statistical power (thus increasing the likelihood of false negatives) and/or the threshold for detecting significance is not enough number of standard deviations (thus increasing the likelihood of false positives), are fairly common in the scientific research community. Couple this with other exacerbating factors, such as research scientists lacking appropriate skills in advanced statistical analysis, a prevalent tendency toward publishing positive hypotheses as opposed to negative ones, and ineffective peer reviews of clinical research findings, and one has a perfect storm in which such mistakes can be fairly common. Greater transparency of clinical data allows any external third party to review and validate the research findings and thus bring to light any potential issues and insights that may have escaped the research team’s due diligence or the government agency’s regulatory scrutiny. Thousands of clinical trials have never been registered with oversight agencies and results from around half of all clinical trials remain unpublished. Making that data available to statisticians would almost certainly lead to new discoveries and clinically useful findings (quoted directly from an article in The Economist).

The noble intention behind the push for greater transparency however may not translate into desirable effects and worse may have unintended consequences. One of the biggest fears is inappropriate analyses and interpretations of the clinical datasets. In a litigious environment, with pharma and drug companies already battling an image of being inhumane greedy corporates least concerned with the ability of the people to afford exorbitant drug prices, this can spell disaster. And it may serve as a strong innovation disincentive for the pharma industry in the long term, with the opportunity cost of not experimenting with novel treatment techniques ultimately being borne by consumers in the form of shortened life spans and/or degraded quality of life. Even when there is less room for misinterpretation, practical challenges with replicating the results of experiments may prevent one from exactly reproducing the results of clinical trials. It is a well-established fact that experimenters employ tacit knowledge and improvisations that are not always captured in experimental setup and process steps. Furthermore, many research teams may use proprietary models to analyze and interpret raw clinical data to arrive at their conclusions – models that they may be averse to sharing with the public, but which nonetheless are critical to arriving at the proper conclusions. The cost of full data disclosure for drug companies is not even discussed many times, but there is a non-trivial cost component to retaining and disclosing such data to the public at large.

So, mandating full disclosure of raw clinical data is just one of the items in an entire menu that needs to be put in place if indeed the objective is to improve the safety, efficacy and efficiency of the pharma and drug industry. The field of biostatistics can go long ways in educating researchers on correctly employing and interpreting clinical datasets. Independent data monitoring committees to oversee the construction and execution of clinical trials to ensure appropriate application of analytic techniques could be in place to provide guidance as the experiments are being conducted. Big data and modern statistical analytic techniques could be developed further to provide researchers with means to more effectively analyze data. In the process of doing all this, if we can help prevent even minor mistakes or incorrect interpretations of drug data, we will have made medicine that much safer for mankind.

Building in a Continuous Integration Environment

Continuous Integration (CI), which spans practices, processes and tools to drive continuous development and integration of software code, is a key building block of an organization’s DevOps methodology. An important CI component is the software build process.  The software build process traditionally has been a second-class citizen of the software development world, relegated to the background  as organizations spend limited resources on customer facing and project management functions.  Software development and delivery is inherently fragile, but one of the most fragile parts is the software build process because development managers have traditionally lacked clear visibility and control of the build process.  Too often, software builds break easily, are difficult to change, and resource intensive to troubleshoot.  With increasing pace of business change and higher delivery pressures, however, every link in the software development and delivery chain will need to be streamlined, and one of the key areas organizations will need to focus on as part of their CI journey is the software build process.

blue devops_4Building is the process of compiling raw software code, assembling and linking various program components, loading of external/third party libraries, testing to ensure that build has executed successfully, and packaging the code into a deployable package.  While this may seem simple and straightforward, building in a big program is complex enough that an overall build architecture is usually required to be defined first along with a dedicated team and infrastructure for ongoing management.  Building usually happens at multiple levels for different consumers: development builds focused on single component development and testing, integration builds across multiple components, system builds for all the system components, and release builds focusing on building customer releases.  Build architecture deals with making specific choices around what to build, how often to build, and how to build it.  Inefficient builds that take too long to finish, challenges with isolating shipped product bugs and issues and replicating them in the development environment, or challenges integrating multiple code streams into release builds efficiently are all symptoms of an inefficient build architecture.  Architecting the build process starts with identifying what to build in a software configuration management system.  Code branching and merging, dependency management, management of environment configuration and property files, and versioning of third party libraries, are all critical components of software configuration management that are closely tied to identifying what software components are needed in a build.  “How often to build” involves defining the build schedules for development, integration, system and release builds, depending upon a number of factors such as number of development work streams, the frequency of releases required, and the capacity of the project team to name a few.  The build schedule clearly identifies how the builds are promoted through the development pipeline, through successive stages of development, testing, quality assurance and deployment.  Last but not the least, having appropriate build process tooling and infrastructure allows building to be declaratively controlled, automated, and efficiently executed.  Build script automation, build parallelization and tracking and reporting of build metrics are all usually managed by a modern build management platform.

With so many moving parts to the build process, it is easy to see where and how things can go haywire.  While there may be multiple factors contributing to inefficient and slow software development and delivery, ineffective building almost always is one of the contributing factors.  Traditional build management suffers from a number of usual-suspect issues, and most of them are process issues, not tool issues.  One of the most common approaches to building has been the big-bang style of executing integration builds, where independent development work streams bring together work stream codebases together as part of an infrequent big bang event.  Compounding the problem is the tendency of development teams to throw code over-the-wall to the build team which is then painstakingly tasked with assembling sources, data, and other inputs to begin the building process.  The big-bang integration brings a big-bang of integration issues and broken builds, a.k.a. “integration hell” in the industry parlance.   Cultural and management issues play into this as well: build teams are not empowered to exercise and implement build discipline with development teams, which in turn do not always action on feedback from build team on broken builds in a timely manner, leading to lengthened cycle time for builds.  Lack of exhaustive pre-commit testing in the development phase, either because development teams are not incentivized to unit test or because effective unit testing harnesses are not available in the first place, leads to “bad commits” and downstream integration issues, putting pressure on the build team.  Many build issues can be traced to just poor build process management.  For example, complex source code branching and componentization schemes complicate the build tasks and make the build process error-prone.  Management of dependencies and build configuration frequently lacks sophistication which leads to challenges in implementing incremental builds, leading to build issues. Inadequate build automation and infrastructure can lead to a host of issues as well.  For example, manual environment setup and data management complicate the build tasks, making building error-prone and lengthening cycle times.  Build infrastructure often times is not adequate to execute complex from-the-scratch builds, which can take hours to complete, thus lengthening build cycle times.

Build management as part of CI aims to get around these challenges to streamline and turbo charge the build process, and ultimately improve the development process overall.  And it begins by fundamentally changing the traditional development and delivery mindset.  Whereas the usual approach involves “software craftsmen” working independently to create perfect fully functional modules that are then integrated over time, building in a CI environment espouses a much more agile approach in which team members come together to develop base product functionality as quickly as possible, incrementally building to deliver the full product over time.  Cultural change to drive close integration between development, testing/QA and build teams is key: in a continuous building environment, development team works hand in hand with the build and testing teams, and a “buildmeister” has the authority to direct development teams to ensure successful build outcomes.  Successful continuous building starts with overhauling the development effort, enacting such practices as test-driven development, and other precepts of Extreme Programming such as feedback loops that encourage development team to take ownership of ensuring successful testing and building. Development teams check-in and build often, sometimes frequently in a day.  And they follow strict practices around check-in, bug fixing and addressing broken builds.  A continuous building environment is characterized by the presence of a “build pipeline” – a conceptual structure which holds a series of builds, spanning the life cycle of the build development process, beginning with a developer’s private build all the way to a fully tested release build, each ready to be pulled by any team as needed.  To enable this, a CI server is used to automate and manage the build management process.  The CI server is a daemon process that continually monitors the source code repository for any updates and automatically processes builds to keep the build pipeline going.  The CI server allows builds to be pulled out of the pipeline by individual teams and for builds to be advanced through the pipeline as build promotion steps are successfully completed.  With each promotion step, the build becomes more robust and complete, and thus moves closer to being shipped to the customer.

To achieve success with continuous building, practitioners recommend a set of best practices across build management, processes and tools.  A few key ones are related to software configuration and environment management, specifically, that development be managed from one “global” source code branch (or reduce branching) and that all build components, including property and configuration files, be managed in a versioning system. Then there are process-related best practices, which deal with development teams following pre-commit testing and actioning on feedback for broken builds on a priority basis.  Automation is a key aspect of continuous building best practices as well: a CI server that manages builds and the build pipeline is a key component of the build automation infrastructure and is central to implementing continuous building.

Build management is a key part of CI, and one where a number of issues with traditional software development and delivery methodologies lie.  Achieving full CI, however, requires change to other key parts as well, for example, testing and deployment management.  In the future DevOps related posts, we will look at these other aspects of CI.