The Last Mile of Data
In this post we describe a challenge increasingly seen in organisations that have procured and rolled out powerful and complex cloud-scale systems (Data Lakes, Data Warehouses, Service Buses) but have failed to efficiently solve what we refer to as the ‘last mile of data.’
This is a condition wherein an organisation has successfully navigated the perils of moving from on-premises into the cloud and has sought to establish a ‘data’ function within the business but fails to render the value of Data systems to operational users.
Rather than achieving the dream of enabling users to become ‘data driven’ from top to bottom, the systems simply languish behind poor adoption or are only used by specialists. However, unlike the Last Mile problems origins in telecommunications grid planning and supply chain management, this Last Mile problem won’t be solved with fibre optics and drones but rather with improved interfaces for our cloud data systems.
James Dixon coined the term ‘Data Lake’ over ten years ago but only since cloud-computing has reached a high degree of price performance and maturity has the Data Lake concept been viable as such a ubiquitous part of our enterprise analytics landscape. Now we find Data Lakes or at least Data Warehouses spread across enterprises all the way down to small to medium business (SMB) and Startups, as businesses and governments seek to leverage the power of their data. Despite the incredible sophistication, complexity, and raw power of these systems, we at bolster feel that businesses often fail to reap the true potential of these IT investments due to their poor usability through operational personnel.
As former military operators ourselves, having spent time in combat zones where the old adage of ‘Keep it Simple Stupid’ (KISS) rules, we believe that there is great value in lowering the barrier to entry for non-technical, non-analytical, operational people to both generate and receive value from their data.
At bolster, we strike this balance between usability and capability by prioritising usability over feature richness, optimising the pluggability and flexibility of our interfaces, and using Infrastructure as Code (IaC) to stamp out integrations rapidly. This means that SMAQ can operate as a thin optimised user-experience (UX) membrane between an organisation’s data systems and its operational personnel, rather than via the typical technical and staffing bottlenecks to data access and innovation.
Rather than a social worker looking down at a phone trying to sift through dirty data via inflexible and manual email, chat, or voice interfaces, we provide teams with the ability to manage their own interfaces and interactions with their data directly. Any system that can work with event-oriented data (regardless of type, language, or size) and can run in the cloud, can leverage the efficiencies inherent in the SMAQ methodology to derive greater business value from their IT investments.
Our thesis is that most data systems are under-leveraged due to access management and/or metrics-related friction, and to resolve this pain we should:
- Work backwards from the interfaces currently embraced by users rather than reinventing the wheel where possible.
- Default to interfaces that require users to spend as little attention as possible, with the highest degree of perceived effectiveness.
- Optimise on elasticity and modularity under the assumption that the evolving model of your organisation will change and improve over time.
Rather than email or chat silos serving as the de facto method for dissemination of information from executive leadership - where team leaders are incentivised to be ‘front of mind’ to gather critical information for their people - SMAQ ensures that organisational data users are empowered to own their individual data feeds based on their stakeholder’s goals. They simply need to subscribe to event streams to which they have access, and they may adjust their insight development pipeline according to changing mission prerogatives.
Follow