Technological trends featuring digital transformation: Big data, cloud and DevOps

Any revolution in history has always been strictly connected, somehow driven, by one or more technological evolutions, which offered the set of technical instruments to radically transform behaviors or processes.

How are technological trends like cloud, DevOps, microservices and big data influencing the revolution of the digital transformation? By going inside their stack, we’ll see how successful examples of digital businesses, like NetFlix, eBay, General Electric, Domino’s and Uber, took technological choices to re-imagine their traditional businesses or to improve the efficiency of their native ones.

Big data

The availability of huge amounts of data has changed the way companies fine tune their own business towards customers' needs. Big data offers digitalized companies the opportunity to take meaningful and strategic adjustments to reduce costs, maximize results and enable real time evaluations of business choices impacts on market. No less importantly, with data-analysis many companies are adopting refined customer engagement strategies to provide unique experiences to their clientele.

How eBay treats big data though Apache Spark: Apache Spark is a Unified Engine for big data processing. Since its release, it has seen a fast growing adoption by the most important digital enterprises for its speed, easy of use and set of sophisticated analytics.

Seshu Adunuthula, eBay’s head of analytics infrastructure, during the 2016 Apache Big Data Keynote in Vancouver, reported one of the biggest challenges for eBay. “A big technical challenge for eBay and every data-intensive business is to deploy a system that can rapidly analyse and act on data as it arrives into the streaming data.

“It allows the company’s data analysts to search information tags that have been associated with metadata and makes it consumable to as many people as possible with the right level of security and governance.”

The biggest problem with bigData is to extract a real value from huge amounts of information and use it in real time production. Apache Spark is leveraged at eBay through Hadoop YARN, a manager to run tasks on cluster resources. The leverage effect Ebay is able to exploit through this system is approaching the range of 2000 nodes, 100TB of RAM, and 20,000 cores. This massive data processing capability gives the auction platform the opportunity, for instance, to address typical problems like platform personalization, similar items recommendation, deal discovery, together with predictive models for fraud detection, and risk prediction.

Cloud

Flexibility is universally recognised as one of the key features of every prospering digital business. It’s easy to understand how the abstraction from a physical infrastructure can help organisations to promptly adapt to any operational scenario faced. By introducing a cloud based stack, an organization can optimise resources in few minutes, saving processing capabilities when not necessary or enwidening them when vital to pursue business success.

The Netflix example: Netflix started its journey to the cloud in 2008 when a serious disaster pushed the company towards “highly reliable distributed systems”, in this case represented by the AWS cloud platform.

In a recent article Yuri Izrailevsky, Netflix’s vice president for cloud and platform engineering, outlined the areas where an 8x growing rate organization had been able to experience benefits with this strategic migration:

  • Scalability: Elasticity of the cloud offered storage and processing extension abilities in a few minutes, instead of months when using a physical system.
  • Performance: overall availability steadily increased, nearing 99.99% uptime.
  • Cost optimization: Scale economies offered an overall reduction of infrastructural costs, and of course optimization of resources.

Domino’s

Domino’s has been one of the pioneers of digital transformation, starting their evolution from a traditional business into a digital one in 2010. Being pioneers, they had been able to foresee potentials of the network applied to their own business. They developed an ordering platform used by hundreds of thousands of customers all over the world, exploiting the advantages provided by PaaS, IaaS, on AWS cloud. Since the beginning of 2017 they have been moving further with a switch to Microsoft’s Azure platforms, while at the same time carrying out a performance and tuning optimization process for their management software.

Wayne McMahon, Domino’s CIO, recently revealed in an interview to iTNews their expectations. “By going over to PaaS we believe that we’ll be able to advance our regional high availability, better contain our development costs – because we’ll be able to utilise a lot of the standard Azure services – gain massive scalability, and bring the experience closer to users,” he said.

Netflix and Domino’s experiences highlight common key essential elements like scalability, cost reduction and resource optimization. These features should lie at the bottom of any process of business digitalization.

Microservices

Micro-service architecture structures an application as a collection of loosely coupled services interacting on a sub-stratus of http based protocol. The adoption of decoupled and collaborating services on the basis of a standard transport protocol predisposes this architecture for usage by any kind of equipment, from desktop personal computers to wearable devices and sensors.

Uber’s experience with microservices: Uber is one of the most commonly used examples when talking about digital transformation, or more properly of digital economy. Uber's stack is based on a micro-service architecture with more than 1200 production services active and running. Their stack features heterogeneous technologies like Java, Python, Node, Javascript, all interacting on an https based service structure.

In a recent conference Matt Ranney, Staff Engineer at Uber, outlined benefits of such an approach. He in particular emphasised the opportunity of independent releases and key features as isolation and compartization when dealing with an application running on hypothetically hundreds of different devices all over the world.

The overhead introduced when understanding a complicated distributed system is balanced by the aptitude to a quick and agile short term development. In addition, isolation of concerns offers easier integration, reducing the outlook when facing a new feature or when in need of identifying new development areas. 

DevOps

DevOps and more in general structured processes of deployment and automation streamline the software development phase and allow it to be made of shorter cycles. This means faster production deployment, with reduced global efforts and lower post-release activities.

The General Electric DevOps experience: General Electric is an American multinational corporation operating, among many others, in the appliances field, building innovative and energy-efficient equipments. Their catalogue is composed of thousands appliances, which means thousands of management and interaction applications all with their own specific technological requirements, and environments. 

In 2013 their operational setup required 6 weeks, in a best case scenario, to go from code running on local machine to code running in production. Every task was manual and non-reproducible. 

For this reason in 2014 they decided to move to a more modern infrastructure system embracing the cloud. This choice reduced time to production metrics from 6 weeks to 3 weeks: a huge reduction in terms of percentage, not in terms of absolute time. 

Only with the introduction of Containers - in particular Dockers running on the mesosphere platform - has GE been able to drastically reduce their time to production from 6 weeks to 1 hour, with completely repeatable and reproducible context. 

Thomas Barber and Brett Luckabaugh revealed benefits of such a strategic choice in a very interesting keynote held at DockerConv15:

  • DevOps reduced time and cost, optimized and simplified many aspects of software development and deployment.
  • Entry barrier when carrying out Ops have been lowered, writing a dockerFile to carry out deployment is as easy as using shell.
  • Each application owns a dedicated environment, built and run in any dependency context.
  • Both legacy and cutting edge assets can now be easily supported: 30 years old monolithic legacy applications run as well as 5 minutes old SoA.

Conclusion - what’s your strategy?

Digital transformation is a global and ongoing process affecting potentially any industry, from worldwide corporations to non-profits and small businesses. As we can understand from the previous examples, no predefined mix of technologies will ever guarantee to success the digital transformation.

Generally speaking, the introduction of any new technology has and will always come with ups and downsides. It is therefore mandatory for every organization to contextualize their adoption, balancing their mix, evaluating their cost/effectiveness with the aim of selecting the ones able to lead to an overall optimization of business processes. A customized digital transformation strategy, in which to plan the digital business goals alongside technological choices, is nowadays a must have for any digital or aspiring digital organization.

Related Stories

Leave a comment

Alternatively

This will only be used to quickly provide signup information and will not allow us to post to your account or appear on your timeline.