Tackling the success conundrum with data warehouse automation

It's a notable characteristic of large-scale decision support systems — data warehouses, data marts, business intelligence applications — that after initial costs of extending and operating, these environments scale as fast, or faster, than the growth of their user communities. It's often referred to as “the success conundrum”.

The explanation is straightforward, intuitive and undoubtedly correct. The more you give people, in terms of data and analytical functionality, the more they want. Demand, where data-driven decision-making is concerned, is not only perennial, it’s insatiable.

This is compounded by the fact that no matter how sophisticated, comprehensive or useful the current data warehouse is, users perceive it as not meeting their newest requirements. Decision makers want what they don’t yet have and that becomes their focus. In other words, the decision support infrastructure is outdated as soon as its introduced.

But while the problem of rising user expectations has always been with us, the gap between what users expect and what IT organisations can provide has widened over time as they have become constrained by lower budgets for entitlement, new application spending controlled by the business and reduced IT headcount.

As IT organisations struggle to maintain hard-won gains in business intelligence with fewer people and smaller budgets, the emergence of the Big Data revolution threatens to create an entirely new class of demand for data and applications within business and functional groups.

What can be done?

Smart IT organisations are seeking to spend their scarce human talent — their collective ingenuity — building value that internal customers within the organisation perceive to be valuable: new, enabling, differentiating. Today, that ingenuity is trapped because IT teams are spending most or all of their time performing routine internally- focused tasks, many of them invisible to internal business constituents.

Most of those tasks are either susceptible to automation (so that no human labour is spent performing them) or acceleration (so that less human labour is spent performing them).

So what can be done? IT organisations need to move to a model that focuses on acceleration and automation, replacing human labour with software to reduce cycle time and cost.

IT departments can't keep saying “Give me more money, more people and more time, and I’ll do what I have done in the past”. Instead, they need to change how they produce decision support infrastructure to work effectively with fewer people, less money and less time and produce more than they were expected to deliver in the past. In this context, shifting to acceleration and automation is easy to understand and has obvious benefits for IT leadership.

Automation is the answer

According to IT leaders who have made the change, the secret is to place the emphasis for the transformation on ingenuity. Data warehouse automation (DWA) frees trapped ingenuity, liberating an IT organisation to pursue strategies for differentiating the company through data-driven decision-making. It’s a strategy for shifting human and capital resources away from repetitive tasks to value creation, and restoring data warehousing teams to relevance and leadership in their organisations.

Data warehouse automation is an integrated platform of tools designed to automate routine IT tasks associated with designing, building, operating and modifying data warehouses and data marts, or accelerating tasks that cannot be completely automated.

Implementing data warehouse automation methods and tools allows IT teams to:
· Respond to business requests in days with accurate time, cost and resource estimates
· Deliver completed data warehouses, data marts and BI environments in far less time
· Rework existing data warehouses, data marts and BI environments in response to business changes in hours or days.

Choosing Your First DWA project

The key to a successful transition to data warehouse automation is choosing an initial project carefully and increasing the usage consistently over time. First projects for data warehouse automation and acceleration should be ones that internal customers believe are critical to their success and where the ability to deliver value rapidly is particularly important.

Initial projects can be a ground-up build of a new data warehouse, a migration of an existing data warehouse to a new technology infrastructure, construction of a data mart, consolidation of multiple data marts, or data warehouse augmentation.

Automation has the potential to free scarce resources for high-value tasks in data warehousing, business intelligence and big data teams. It can also enable IT teams to rebuild cordial, collegial relationships with internal customers in business units and functional groups at a time when their attention and budgets are shifting toward other sources of ingenuity for business intelligence and advanced analytical needs.

Successful data warehouse automation initiatives will result in more customer-perceived value delivered faster, with fewer resources and less rework. Doing a better job in less time, for a lower cost and at reduced risk? Now that's the kind of success conundrum everyone likes!