John Culkin, director of information management, Crown Records Management looks at how businesses can benefit from finally taking control of the great unknown.
Big Data
Just hearing the words ‘dark data’ can be enough for managers in pharmaceutical sector to feel a little uneasy - but despite its rather menacing name there has never been a better time to tackle the phenomenon head on.
When you think of the recent explosion in data it’s hardly surprising that businesses worry about how dark data - defined as ‘information assets that organisations collect, process and store during regular business activities, but generally fail to use for other purposes’ - will affect them in future. So why are so few doing anything about it?
The threats provided by an explosion in data are obvious; businesses which don’t know what information they have or how to find it are opening themselves up to the possibility of a data breach and in danger of falling foul of data protection regulation as it continues to evolve.
In pharma the problem is particularly acute. Regulation demands strict quality in manufacturing and research for safety reasons. However, raw clinical data or clinical trials are often in incompatible format, which then loses the ability to infer interesting insights from it.
Like all large organisations dependent on many complex systems, pharmaceutical companies will strive to meet their compliance needs - however they are still likely to have data hidden in their systems that is either not necessary to keep or could contain potentially useful information.
By finding and exposing the data and finding all the places it hides, gains can actually be realised. But the bottom line is organisations need to know what dark data is and where it is in the first place. If you don’t understand it how can you possibly be in control of it, let alone realise value hidden in it?
People fill their attics and garages with ‘useful’ things that could be needed in the future. The corporate equivalent is when people say ‘keep the data just in case’. How much money around the world is wasted due to those few words?
Top tips to prevent dark data accumulating – and derive benefit from it
1. Perhaps the first place to start for those in pharma is to realise that data may go to waste or sit unused if it is not properly managed – or if systems are not properly designed in the first place. This can include data which may contain potential insight for the future.
2. All industries are likely to see an explosion in data as the Internet of Things takes hold – and for pharmaceuticals it seems inevitable. The vast amount of data that genomics and personalised medicine will generate in future massively increases the volume of data being stored, creating further challenges.
3. Reducing the amount of dark data kept is crucial; so don’t accept that because data storage is cheap everything can be kept.
4. Develop policies and continuous training for all staff about managing data (including data on local drives, laptops, shared drives, removable devices and mobile devices).
5. Understand which regulations require what data to be kept for how long. Don’t keep it longer than necessary unless clear benefit can be derived from it. A quarter of respondents in a Crown Records Management survey this year said they were ‘not confident’ they knew how long they were legally obliged to keep data. This is a major concern.
6. Don’t believe all data in enough volume is “Big Data” and therefore has value. Much of it will not be useful in big data projects, especially as it’s often unstructured and in various formats.
7. Use File Level Analysis software, to analyse the content of data rather than just its creation and expiry date. It can help a business understand information content, type, size and location.
8. Map data sources generated by internal systems or received from external sources – this is about documenting ‘where’ and ‘why’ the data lives and how it was derived.
9. Have an Information Governance programme in place supported by the most senior management levels and practiced every day.
10. Talk and communicate with people because not everything will be officially documented or known about. Work-arounds and short-term fixes often become routine in the business place, so make people aware of the impact of not being able to manage everyone’s data effectively - it affects everyone.