The case against ground-up data solution developments

70% of data projects fail. It’s an incredible and worrying statistic when you consider all that wasted time and resource.


With Einstein’s definition of insanity in mind (doing the same thing over and over again and expecting different results), we’ve analysed failed projects so you can avoid the same mistakes.


The vast majority of failed projects are ground-up developments, and this approach leaves your development processes and resulting application vulnerable. Here are 5 reasons why.


1. Initially, most organisations only build what’s absolutely necessary, leaving out what are considered to be useful rather than essential features 


These omissions often cause major problems down the line, particularly when it comes time to release the application and operate it ongoing.


Common features in this category include process audit logging, error handling, documentation and streamlined handling of data quality. Another such feature is incremental processing, which is often required down the line when the solution and processing time go beyond the available window.


2. Developments become increasingly complex and opaque 


When you develop from the ground up, it can take a long time before you deliver something tangible. This is because the approach is reliant on individuals, and over time the developments become difficult to maintain due to the different approaches taken by different developers. Also, generic technologies don’t encourage reuse, which means they don’t encourage transparency.


3. A change in source or requirements destination derails the project


Ground-up solutions are invariably targeted at a specific source or a specific requirements destination, and a change in either can lead to major upheaval and, potentially, wholesale redevelopment.


4. The process of releasing the application is problematic


Everything seems simple in terms of running the isolated development environment and using it for testing. However, once the application moves into production and a separate production environment, the problems multiply.


Once in production, there’s rarely time to build anything in the way of runtime administration consoles to allow operations staff to takeover the application.


5. There are ongoing data governance challenges


Due to the complexity that ensues from a ground-up development – and the opaqueness of the routines and objects developed – data governance (via, for example, lineage) verges on the impossible.


Automation addresses many of these issues, helping put you in the elite 30% of projects that succeed


By automation, I mean using a product that significantly enhances the generic technology on which it runs. Automation products do this by supporting the generation of code to implement the data management process. As a result, you benefit from:


  • Faster development timescales
  • Lower testing overheads
  • Extensive out-of-the-box functionality, including methodologies
  • Flexibility around different types of source data and outputs
  • Transparency, for easy governance and maintenance



Like what you see here and want some more information? Request a call back and one of our team will be in touch.