Data Management remains a major hurdle to organizations striving to create a data-driven culture
Data management (DM) consists of the practices, architectural techniques, and tools for achieving consistent access to and delivery of data across the spectrum of data subject areas and data structure types in the enterprise, to meet the data consumption requirements of all applications and business processes.
Times gone by, data management was not something that business leaders needed to think about: it was built into the applications they purchased as an integral duty of the software vendor to determine how data was managed and provide administration tools to govern it.
These days, organizations want to maximize the value of data across their enterprise, and that means harvesting and re-using data. To achieve that, and do it well, organizations need to create an ‘enterprise-wide’ data management approach that extends beyond any single application.
Fragmented data sources, a problem that just won’t go away
Providing access to information quickly remains a fundamental challenge to data and knowledge management professionals.
A pivotal industry study by Accenture found that:
- Middle managers spend more than a quarter of their time searching for information necessary to their jobs, and when they do find it, it’s often wrong, and;
- Only half of all managers believe their companies do a good job in governing information distribution or have established adequate processes to determine what data each part of an organization needs!
The same study found:
- 59% say, that as a consequence of poor information distribution, they miss information that might be valuable to their jobs almost every day because it exists somewhere else in the company and they can’t find it.
- 42% say they accidentally use the wrong information at least once a week, and 53 percent said that less than half of the information they receive is valuable.
- 45% say gathering information about what other parts of their company are doing is a big challenge.
- More than half have to go to numerous sources to compile information
- 40% say other parts of the company are not willing to share information, and 36 percent said there is so much information available that it takes a long time to actually find the right piece of data.
Source: Accenture web-based survey of 1,009 managers in companies in the United States and United Kingdom with reported annual revenues of more than US$500 million. June 2006
Data Management – A growth enabler or sunk cost?
A logical question for leaders, is this: ‘Is Data Management a cost of doing business, or is it an enabler for growth?’
The potential of digital technologies to engage customers in new ways, and to provide self-service portals with which to orchestrate business models, has transformed leadership perceptions of the role of data management from being an IT administrative hygiene issue, to a primary competitive advantage.
Online retailers like Amazon are demonstrating how effective data management, in their case customer data, can result in unprecedented opportunities for growth. Amazon has set as its objective the desire to deliver awesome customer experience through the deepest knowledge possible of who customers are and what they care about. They see data management as an integral part of that story.
Airlines like easyjet, have invested considerably, although astutely, on inhouse developed systems based on the Microsoft .NET / Microsoft Azure platform that data management platforms like encanvas secure&live also use. Easyjet has streamlined flyer booking and their ‘from home to the gate’ experience, to make flying more convenient for passengers, while also supersizing their customer data capture opportunities from the self-service customer portals they have created.
Retailers like Tesco have established very effective voucher programs that motivate customers to buy ‘just a little more’ every time they shop by harvesting rich data about customers from in-store and online shopping behaviors. The Tesco Clubcard system affords Tesco a competitive advantage in its market by maximizing its ability to learn about what customers want by offering small voucher-based rewards.
Car Rental company HERTZ are past masters at membership schemes that reward repeat customers with benefits that improve their customer experience. It operates not one but several membership levels to target different audiences in its customer base. These card-based membership rewards schemes help HERTZ to tailor its offers, capture knowledge of the types of customer they have in their community, and learn how to tailor rewards to incentivize repeat business.
None of these initiatives would be possible without outstanding focus on data management, sponsored by data-savvy management teams who understand the value of data to their business as the ultimate competitive weapon.
Systems of Record (SoRs) don’t run businesses anymore
At the turn of the millennium, business leaders had the expectation they could purchase a ‘System of Record’ Enterprise Resource Planning (ERP) software application – like Oracle, Microsoft Dynamics, or SAP R3 – and they would have everything they needed ‘in the box’ with which to run their business. Again, the presumption was Data Management would be taken care of by the software provider. That hasn’t proven to be the case.
The footprint of any ERP system these days is barely 60% (if you’re lucky) of the IT systems needed to operate a company’s complete business model. The role of IT systems has extended beyond the use of internal stakeholders.
As the case examples given above show, businesses that qualify as ‘digital leaders’ are creating digital cloud platforms and ecosystems that unilaterally service the needs of customers, suppliers, shareholders, contractors and members of staff. These solutions have to be tailored and adapted to the specific audiences, operational behaviors and outcomes of the organization; something that slow-to-change ERP systems were never designed to do.
From departmental to enterprise data management
Perceptions of the role of data management have changed as organizations have come to realize the importance of maximizing the value of data. Business leadership teams are acknowledging the importance of making data-driven decisions. To achieve that, requires not only a data-driven culture, it needs data to be organized across the enterprise.
The starting point for most organizations on their data management journey is to construct a data catalog. As the name suggests, your data catalog holds a log record of all important data assets that exist in your enterprise IT systems.
Data platforms
In business, a data platform is any form of implementation of a database, data-mart or data warehouse used to manage business-critical data structures. More organizations these days want to house data in an enterprise data platform that maximizes its value and the ease to which data can be governed.
Customer data platforms
For many organizations, the thrust towards above and beyond customer experiences powered by data analytics means that Customer Data Platforms are becoming the first step towards enterprise data management.
A Customer Data Platform is an enterprise computer processing platform used to harvest, aggregate, cleanse, manage, process, analyze and output customer associated data. Data is pulled from multiple sources, cleaned and combined to create a single customer profile. This structured data is then made available to other marketing systems. Unlike a Customer Database, a Customer Data Platform extends its functionality to all aspects of the customer lifecycle. Normally it will include campaign management and the provisioning of multi-channel communications. Advanced systems will manage customer offers and promotions.
The CDP Institute defines a Customer Data Platform as “packaged software that creates a persistent, unified customer database that is accessible to other systems.” Basically it’s a prebuilt system that centralizes customer data from all sources and then makes this data available to other systems for marketing campaigns, customer service and all customer experience initiatives. Gartner defines CDPs as – ‘integrated customer databases managed by marketers that unify a company’s customer data from marketing, sales and service channels to enable customer modelling and drive customer experience.’ At encanvas, we see this as a narrow definition, given that Customer Data Platforms normally serve up insights to strategic teams and all departmental functions to shape processes and priorities.
Who owns the problem of data management?
This raises the challenges of ‘who owns the problem.’ Often, IT leaders are too busy concentrating on core IT systems and ‘keeping the lights on’ to then deal with data management and governance issues too.
Data quality continues to derail IT projects
Although tooling to enable data management technology has matured beyond recognition over the past decade, the fundamental challenge facing all organizations remains to be data quality issues. As organizations engage in projects to build an enterprise-wide view of data, and commence to task of harvesting data from operational systems, it becomes apparent how poorly business software applications manage the integrity of data they hold. These integrity issues happen for three main reasons:
- Applications are designed in isolation without giving consideration to future needs for enterprise data management. All systems architects will have an opinion on how best to organize data for their specific application, resulting in poor continuity between data types and structures.
- When applications are authored, designers focus on the usefulness of data designs for the application in question. Often, data tables and structures evolve in their role and use over time creating a paucity of poorly populated data table structures.
- There is always a trade-off between usability and enforcement of data capture rules. Often, data designers with sacrifice good data capture protocols (needed to protect data integrity) in pursuit of a pleasing user experience.
Silos of data
Always the run-up to data quality on lists of data management challenges, are the data management challenges brought about by the existence of systems and organizational silos.
Over time, businesses will evolve their organizational designs and data structures, normally one department and one system at a time. They become a collection of departments rather than one homogeneous enterprise data engine. Each will run systems to satisfy department data processing needs. Managing data from across the enterprise immediately exposes the challenge that systems were never designed to work together or share data. Important data identifiers that help data scientists to forge links between disparate databases – the gateways and bridges to data – may not exist, making the task of forming a single view of data extremely difficult.
Three steps to data management effectiveness
Industry practitioners agree that the three key steps to achieving effective enterprise data management are themselves not complicated. They are to:
- Discover where the useful data is and why it’s important
- Harvest it and create a new useful data structure based on a clearly thought-out Master Data Management model
- Install tools and a data-driven culture to make it valuable to the enterprise.
Although these steps seem trivial and obvious, it is a path paved with difficulties and a track-record of poor success. The list of failed projects includes some of the world’s largest companies and brands you might think should know better, or at least could afford to fix the issues they encountered along the way.
Whatever the size and scale of the data management project, it requires very clear and measurable reasons for doing it, and unrelenting management commitment. The reality is, that it will inevitably take longer and cost more than project leaders would like.
Good business reasons to get data organized on an enterprise scale
There are lots of good reasons why organizations should put a focus on organizing their data. We describe the top ones here.
Make data easier to find
Departmental managers and knowledge workers continue to complain about difficulties in accessing the data they need to discharge their roles. Searching for data that may or may not exist can feel like a huge waste of time for busy workers. Knowing where to look for data is normally a good start to improving accessibility to data. Tools like knowledge portals and Wikis can make a big difference to the ability of workers to find answers and content resources. Having ONE PLACE where knowledge is held offers the best starting point for organizations that today use multiple instances of content management and knowledge sharing systems. Social tools have the potential to change organizations, but only if those tools are implemented in a way that changes how individual employees work day to day.
Maximize data value to make smarter decisions and profit from the value of data
Executive teams that are unable to harvest data insights to answer their strategic questions have no other choice but to work off of hunches and gut-feel. Taking reports and insights from discreet operational systems creates a kaleidoscope view of operational realities. Executives team reliant on silo-reported data-sets claim to find it hard to gain a consistent impression of organizational performance and struggle to answer strategic questions. The cost of acquiring answers to strategic questions is extremely high, to the point of making it uneconomic.
Maximize data re-use
Without some form of enterprise data warehouse, re-using data and harvesting its value becomes extremely difficult, if not impossible.
Develop a richer understanding of important business landscapes
Having the ability to holistically understand your customers, systems, assets, products, supply-chain, partner channels, etc. makes a big difference to the ability of the enterprise to respond to change, opportunity and risk as it emerges. To achieve this, key data-sets need to be carefully structured and governed.
Minimize data risks
Not knowing where data is, and who can view and edit it, is a big risk to businesses. One of the challenges of data breaches is not knowing what data has been compromised in the event of a breached. Data security professionals need to know where precious data is and how users and user groups are governed to restrict privileges according to need and risk.
The remarkable influence data privacy has had on the data management discipline
The advent of the European Union’s General Data Protection Regulation (GDPR) has had a profound effect on the Data Management discipline. The risk of data loss has become a business continuity threat (fines of up to 4% of global turnover may be levied by litigators) and this has resulted in a compelling RoI argument for enterprise data management and data quality enrichment projects, perhaps for the first time.
The influence of the GDPR on Data Management doesn’t stop there. To implement data privacy safeguards, organizations are obliged to know where privacy data exists. In many organizations today, that simply isn’t possible owing to a fragmented departmental and systems view of data assets.
While organizations find it harder to justify investments based on growth potential, it is easier for financial professionals to justify spend on Data Management qualified by a measurable cost and risk implication.
Missing data – the digital DNA of your enterprise
When implementing an enterprise data catalog, some of the most important data is that which describes the enterprise; it’s companies, locations, organizational hierarchies, people, roles, processes, data, systems, risk and suppliers. Rarely is this information manage in a digital form in one place, if indeed it exists at all. The obstacle this presents ion data science is the lack of data context that exists without it.
Master Data Management (MDM)
Master data management is a tern used to describe methods used to holistically define and manage the critical data of an organization. The general priority of Master Data Management projects is to define a single version of the ‘data truth’ across the enterprise. Relatively few organizations have implemented robust MDM models, chiefly because of the costs involved and the absence of a clear return-on-investment. A good proportion of those organizations with effective MDM implementations have evolved their data management from the ground up, with strong executive leadership to make it happen. Retrofitting MDM across an existing enterprise IT architecture operating hundreds of applications is a non trivial and costly task.
Self-service reporting
Self-service reporting is ‘the customer’ of Data Management. Without the ability of machines and humans to harvest it, data serves little purpose.
Self-service reporting is an analytics paradigm that places the emphasis for data analysis and report-building on individual citizen users instead of highly trained statisticians or data scientists.
Advances in data visualization and business intelligence are making it progressively simpler to make sense of large data sets and to pull-out the things leaders, managers, workers and machines want to know. Technology is democratizing data analytics to bring its influence on an increasingly data aware, and data hungry enterprise. Whereas data management and analytics has been a specialist technical discipline, it’s entirely likely that a decade from now, we will see it as something every worker does.
Data science – The new industry
Data science is one of the buzz-terms of the enterprise computing industry. This realization, that data is important as a competitive differentiator and that a technical competency was needed to harness it, began with the evolution of big data and cloud computing. It quickly became apparent that the combination of big data and artificial intelligence would mean a substantive increase in the pace of digital business and the need to analyze, interpret and act on unimaginably large volumes data both faster, and more often. The industry of data science was born.
The term Data Science describes the broad set of scientific methods, processes, algorithms and systems used to extract knowledge and insights from many structural and unstructured data. Data science is related to data mining and big data.
The perpetual problem of shadow data and the long-tail of self-serviced applications
The true value and therefore effectiveness of Data Management remains compromised in organizations as the result of self-authored desktop applications that perpetuate shadow data, that exists unbeknown to IT.
Spreadsheets remain as the most prolific producer of shadow data. Use of spreadsheets means data scientists lose vital data assets because the data is held in desktop hard-drives in unusable structures.
To achieve optimal results in data management, organizations must somehow find ways to eradicate self-authored applications beyond the remit and control of IT.
The Author
Mason Alexander is a senior consultant specializing in helping organizational leadership teams to grow by implementing enterprise software platforms that improve data visibility, process agility; and organizational learning – creating an enterprise that learns and adapts faster. He writes on subjects of change management, organizational design, rapid development applications software, and data science. He can be contacted via his LinkedIn profile.