On Data Infrastructure Automation. Q&A with Neil Barton
Q1. What is a “Data Vault”?
Neil: Data vaults are designed to provide long-term storage of data from all sorts of different operational systems. Data vaults use a different modeling approach than traditional data warehouses for greater resiliency to future business and technology changes. Developed by Dan Linstedt, the current model, Data Vault 2.0, was designed to offer IT teams greater flexibility, scalability, consistency and adaptability than traditional modeling approaches.
While completed data vaults have many benefits, designing and developing them by hand takes significant time, effort and money. Data vault automation helps businesses produce data-driven results faster and with less risk. For instance, our customer, Aptus Health, looked to WhereScape Data Vault Express for Snowflake to fast-track their delivery of the organization’s first version of their data vault.
Q2. How did Aptus Health eliminate a decade’s worth of data silos and legacy data warehouse integration challenges?
Neil: Aptus Health had experienced rapid organic and acquisition-related orgaizational growth and as a result was faced with the challenge of how to seamlessly integrate a number of siloed data sources within the 12-year-old legacy data warehouse environment to produce the analytics they needed.
First, Aptus Health attempted to address data warehouse integration challenges with a tool for extraction, transformation and loading (ETL) processes, and discovered manual ETL was far too time-consuming, complex and prone to human error to be effective. This led them to begin searching for a better way to develop data infrastructure quickly. WhereScape Data Vault Express for Snowflake, data warehouse automation software built to best leverage Data Vault 2.0 and all of the benefits of cloud data warehousing on Snowflake, was that answer.
Through the combination of WhereScape and Snowflake, Aptus Health was able to design the first version of its data vault in just three days, and within three months, deliver a production version in the cloud. Aptus Health is able to now respond more quickly to business questions, better evaluate historical trends and get a more complete picture of our many data sets inside one, fully documented data warehouse.
Q3. How do Snowflake and WhereScape relate to each other?
Neil: Snowflake is the data warehouse platform built for the cloud, and WhereScape provides the data infrastructure automation to help Snowflake customers deliver projects on Snowflake much more quickly to the business. In many ways it’s a match made in heaven.
Let me explain why – by automating the repetitive and time-intensive aspects of the design and development of data infrastructure and big data projects, WhereScape automation helps teams get more done quickly with reliable results. We’ve done this on-premises for many years, and as our customers began to adopt cloud initiatives, it was natural for us to look at how we could help them easily evolve and achieve these same productivity benefits in the cloud. Working with Snowflake was a no brainer given their ability to provide businesses the flexibility of big data platforms and the elasticity of the cloud at a fraction of the cost of traditional solutions.
WhereScape automation for Snowflake speeds up a business’ ability to start using Snowflake for new and ongoing cloud data infrastructure projects and begin reaping all of the benefits of Snowflake sooner.
Q4. Why should enterprises migrate existing data warehouses, data vaults and data marts from legacy data platforms to Snowflake?
Neil: In my experience, enterprises who move to Snowflake can deliver insights and value to the business faster. With better performance and elastic scalability from the cloud, organizations can more easily adjust based on the business needs, providing a positive financial impact.
The big question for many isn’t so much “why Snowflake” as “how on earth do we get there?” In the past, migrating existing data warehouses to new environments has taken months and required a team of developers needed to discover and understand the existing data infrastructure, map it to the new data structure, write and test scripts, recreate metadata – all in addition to transferring and validating the data. It’s complicated, time-consuming, and incredibly vulnerable to human error.
It doesn’t have to be so challenging, though. Today, many organizations are able to reduce project costs and months of additional effort by automating the development and operation of new data infrastructure and the migration of existing data infrastructure as well. Part of our WhereScape automation for Snowflake offering – WhereScape Migration Express for Snowflake – automates the migration of existing data and data infrastructure to Snowflake, so that organizations can begin taking full advantage of Snowflake cloud data warehousing benefits sooner. That means all the benefits of traditional data warehousing – faster insights, quicker time to value – but in the modern, flexible cloud environment with all the additional agility it provides.
Q5. When is it more appropriate to choose Snowflake’s data warehouse-as-a-service on Amazon Web Services or Microsoft Azure?
Neil: The specific needs of customers drive them to select the best platform or platforms based on the unique environments of their organizations. It is clear that the future is multi-cloud. Today, 81 percent of companies have a multi-cloud strategy and actively run applications within four or more clouds, according to research by RightScale. Organizations are searching for flexibility across their cloud strategy, whether it’s a choice of Amazon Web Services or Microsoft Azure.
Neil Barton is the Chief Technology Officer for WhereScape, the leading provider of data infrastructure automation software, where he leads the long-term architecture and technology vision for the company’s software products. Barton has held a variety of roles over the past 20 years, including positions at Oracle Australia and Sequent Computer Systems, focused on Software Architecture, Data Warehousing and Business Intelligence. Barton is a co-inventor of three US patents related to Business Intelligence software solutions.