Data Warehouse Governance Best Practices

Build A Business Case

Getting buy-in and sponsorship from leaders who will be part of the process is key when building a data governance practice, but buy-in alone wont fully support the effort and ensure success. Build a strong business case by identifying the benefits and opportunities that data quality will bring to the organization and show the improvements that can be gained, like an increase in revenue, better customer experience, and efficiency. Help everyone involved see and understand both the energy required and the eventual benefits to be successful. Most leaders can be convinced that poor data quality and poor data management is a problem, but data governance plans can fall short if leadership isnt committed to driving change.

Best Practices For Data Warehouse Maintenance

• 22 October 2019

The foundation of any organization’s data analytics stack is its data warehouse. Chances are you put a lot of thought into the structure and contents of yours, but have you given any thought to data warehouse maintenance that is, making sure the data warehouse columns, tables, views, and schemas are accurate and up to date?

As a companys data warehouse ages, you’ll need to:

• track new metrics, and stop tracking some old ones
• grant and remove permissions
• optimize modeling

Let’s look at how a data engineer can address these routine maintenance issues.

When Did Data Governance Become A Thing

Timeline and key milestones in the space.

For the past twenty years, the challenge around data was to build an infrastructure to store and consume data efficiently and at scale. Producing data has become cheaper and easier over the years with the emergence of cloud data warehouses and transformation tools like dbt. Access to data has been democratized thanks to BI tools with BI tools like Looker, Tableau or Metabase. Now, building nice dashboards is the new normal in Ops and Marketing team. This gave rise to a new problem: decentralized, untrustworthy & irrelevant data and dashboards.

Even the most data-driven companies still struggle to get value from data – up to 73% of all enterprise data goes unused.

â 1990-2010: emergence of the 1st regulation on data privacy

In the 1970s the first data protection regulation in the world was vetted in Hessen, Germany. Since then data regulation has kept increasing. The 1990’s mark the first regulations regarding data privacy with the EU directive on data protection.

Yet, compliance with regulation really became a worldwide challenge in the second half of the 2010s with the emergence of GDPR, HIPAA, and other regional regulations on personal data privacy. These first regulations drove data governance for large enterprises. This created an urgency to build tools to handle these new requirements.

â 2020+: Towards an automated and actionable data governance

You May Like: Best Resume For Government Jobs

What Is Data Governance

Data governance is a core component of good data management and assists with defining the policies involved in maintaining and protecting data. It is critical in ensuring that the organisational data is compliant with regulatory procedures and doesnt get misused. With data leakages becoming a common occurrence in the current climate think the Facebook/Cambridge Analytica incident governance is an increasingly important priority in establishing security and privacy for valuable data. Organisations now have to go beyond just managing their data effectively but must also govern the access of users to certain data.

A data governance framework is the blueprint to constructing standards of data and is an integral part of overall data management strategy. It encompasses the policies, processes, technology and people needed to protect and make appropriate use of data assets. By working hand in hand with data management, organisations will be able to establish clear lines of accountability and technical boundaries to support transparency and integrity.

Ensuring your data is compliant with standards and organisational policies is a crucial step in your data journey and Antares data strategy services can help you build trust in your data by maximising its value. Contact us on +61 2 8275 8811 to find out how we can create an integrated data strategy that drives operational efficiency and decision making.

What Is Data Governance And Why Does It Matter

• Jack Vaughan,Senior News Writer

Data governance is the process of managing the availability, usability, integrity and security of the data in enterprise systems, based on internal data standards and policies that also control data usage. Effective data governance ensures that data is consistent and trustworthy and doesn’t get misused. It’s increasingly critical as organizations face new data privacy regulations and rely more and more on data analytics to help optimize operations and drive business decision-making.

A well-designed data governance program typically includes a governance team, a steering committee that acts as the governing body, and a group of data stewards. They work together to create the standards and policies for governing data, as well as implementation and enforcement procedures that are primarily carried out by the data stewards. Executives and other representatives from an organization’s business operations take part, in addition to the IT and data management teams.

Also Check: How To Buy Short Term Government Bonds

Data Governance Is Not Master Data Management

Master data management focuses on identifying an organizations key entities and then improving the quality of this data. It ensures you have the most complete and accurate information available about key entities like customers, suppliers, medical providers, etc. Because those entities are shared across the organization, master data management is about reconciling fragmented views of those entities into a single viewa discipline that gets beyond data governance.

However, there is no successful MDM without proper governance. For example, a data governance program will define the master data models , detail the retention policies for data, and define roles and responsibilities for data authoring, data curation, and access.

Measure Results And Communicate Successes

For each of the specific outcomes defined above, establish measures in advance that can be used as a yardstick for success. As with any project or initiative, some outcomes are easier to measure than others, but data governance leaders should not shy away from defining and communicating the key metrics that establish the success or failure of their initiatives.

Data governance requires a long-term organizational commitment. Defining, measuring, and communicating the resulting benefits helps to ensure that stakeholders throughout your organization understand and appreciate the value of data governance programs over the long haul.

Don’t Miss: Government Grants For Wheelchair Vans

Roles And Responsibilities On A Data Governance Team

Some organizations have a dedicated data governance team, while others have employees assume the additional responsibilities in addition to their normal duties. Theres a common misconception that data governance is only an IT job. The reality is, while IT teams are responsible for providing solutions and developing infrastructure services, other team members are just as vital for example, they provide guidance on data governance policies and rules.

The key data governance roles that need to be filled include:

• Data steward This is an operational duty that focuses on implementing and coordinating policies and procedures. Data stewards manage corporate data projects, make data-related decisions, issue recommendations and develop relevant policies.
• Data governance council This team is responsible for setting up the data governance program, measuring success and gathering metrics.
• Data stakeholders These are the people who own and use specific data assets. They usually include individuals and teams in the human resources, IT, risk management, compliance and legal departments. Their insights and needs should be considered in decisions about policies, procedures, business rules and technology approaches.

The Top 12 Best Data Warehousing Books You Should Consider Reading

Our editors have compiled this directory of the best data warehousing books based on Amazon user reviews, rating, and ability to add business value.

There are loads of free resources available online and those are great, but sometimes its best to do things the old fashioned way. There are few resources that can match the in-depth, comprehensive detail of one of the best data warehousing books.

The editors at Solutions Review have done much of the work for you, curating this comprehensive directory of the best data warehousing books on Amazon. Titles have been selected based on the total number and quality of reader user reviews and ability to add business value. Each of the books listed in the first section of this compilation have met a minimum criteria of 15 reviews and a 4-star-or-better ranking.

Below you will find a library of titles from recognized industry analysts, experienced practitioners, and subject matter experts spanning the depths of data warehousing for beginners all the way to data lake best practices for the largest data volumes. This compilation includes publications for practitioners of all skill levels. Weve also included a new section below that features recent and upcoming data warehouse book selections that are worth checking out.

Read Also: Cash Government Check Without Bank Account

Data Warehousing Fundamentals For It Professionals

OUR TAKE: This title was specifically written for professionals responsible for designing, implementing, or maintaining data warehousing systems. It is also relevant for those working in research and information management.

This practical Second Edition highlights the areas of data warehousing and business intelligence where high-impact technological progress has been made. Discussions on developments include data marts, real-time information delivery, data visualization, requirements gathering methods, multi-tier architecture, OLAP applications, Web clickstream analysis, data warehouse appliances, and data mining techniques. The book also contains review questions and exercises for each chapter, and is appropriate for self-study or classroom work.

Data Management: An It Practice

Lets start with the more basic piecedata management. After all, if you dont have solid data management in place, the rest of the data world is a beyond your reach. Data management is best seen as an IT practice with the goal of organizing and controling your data resources so that it is accessible, reliable, and timely whenever users call on it.

Viewed from this administrative perspective, the IT teams responsible for data management may rely on a comprehensive, customized collection of practices, theories, processes, and systems an entire suite of tools that collect, validate, store, organize, protect, process, and otherwise maintain data. After all, if data isnt treated appropriately, the data can become corrupt or unusable, becoming completely useless.

Importantly, data management encompasses the entire lifecycle of a data asset, from the very initial creation of the data to the final retirement of the data. Data management can include many related fields and categories, including any of the following as relevant to your company:

• Data governance and data stewardship
• Data architecture
• Data security management

You May Like: Government Loans For Working Parents

How To Deprecate Old Metrics

First, let users know when metrics are no longer useful via email or through your BI tool so theyre not caught off-guard. Then update the names for these objects to something like _deprecated or _do_not_use. If you’re still concerned about people using the data incorrectly you can make new tables, or views that no longer contain the deprecated columns.

Naming conventions for old metrics should be incorporated into your companys style guide. In this case, naming plays an integral role in keeping users from querying data warehouse objects incorrectly.

Key Challenge: Balancing Speed & Risk

Governance has traditionally focused on the management of finished data such as financial close metrics, regulatory submissions, and key performance indicators. This type of data requires formal definitions and high data quality.

But todays advanced data science and data analytics often use raw and semi-finished data. And this creates a tension between data providers and data consumers. Providers work hard to provision data responsibly, to everyone, without putting the business at risk. Consumers want data for their projects immediately.

The tiered system shown below offers a solution to this challenge. The funnel addresses different user needs with different types of data, applying increasing scrutiny and quality standards as the data works its way through the system.

This system helps the enterprise governance function focus on a breadth of understanding across the enterprise, including enabling restrictions to sensitive data, as well as a depth of understanding for a smaller number of critical data assets.

Also Check: How To Get Free Grant Money From The Government

What Is Data Quality

Data quality is the degree to which data is error-free and able to serve its intended purpose. Certain properties of data contribute to its quality. Data must be:

• Accurate
• Complete with data in every field unless explicitly deemed optional
• Unique so that there is only one record for a given entity and context
• Formatted the same across all data sources
• Trusted by those that rely on the data

When organizations achieve consistently high quality data, they are better positioned to make strategic business decisions that yield valuable business insights as well as drive revenue.

Invest In Internal Training

Attaining good data quality is a difficult task. It requires a deep understanding of data quality principles, processes, and technologies. This knowledge is best obtained through formal training. Following the training track for a data management certification such as Certified Data Management Professional , Certified Information Management Professional , or Certified Data Steward would provide a good road map.

Encourage data quality staff to earn the certification, to better inform them on:

• Basic concepts, principles, and practices of quality management
• How quality management principles are applied to data
• How to think through both the benefits of high-quality data and the costs of poor quality
• How to create, deliver, and sell a business case for data quality
• The key principles in building data quality organizations
• Basic concepts, principles, and practices of a data stewardship program
• The data quality challenges that are inherent in data integration

You May Like: Government Program To Buy House

How To Find The Best Data Integration Tool

Transitioning to a data lake can be complicated, but a data integration tool can help overcome most of the challenges youre likely to encounter. When choosing a solution, look for one that can support every step of enterprise data management from data ingestion to data sharing. A data management tool should:

• Connect to unlimited data sources and allow you to add new sources easily
• Process data in a high-performance and secure fashion
• Process batch and real-time data at any speed
• Include built-in machine learning and data quality tools
• Include built-in data governance, metadata management, and data lineage tracking
• Offer self-service tools accessible to everyone from business users to skilled data scientists
• Run on any cloud or on-site platform
• Include built-in data vault capabilities and services

Finding a tool that hits all of these checkpoints will not only assist in a successful data lake setup, it will help you easily and efficiently maintain your data lake in a way that works best for your business.

Data Governance Is Not Optional

Organizations today have incredible amounts of data about customers, clients, suppliers, patients, employees, and more. When this information is properly used to better understand the market and your target audience, an organization will be more successful. The same data governance will also ensure this data is trusted, well-documented, and easy to find and access within your organization, and that it is kept secure, compliant, and confidential.

Make certain that your organization is positioned to maximize data governance investments and minimize risk of data breaches. Take a look at our data governance solutions when youre ready to get started.

Also Check: Florida Government Grants Small Business

Data Governance Challenges Are Not The Same For Everyone

Diverse governance’s use-cases based on industry needs and organizations size

There are two main drivers for data governance programs:

• Level of regulation needed in the industry

Data regulation push the minimum bar of data governance processes higher. It requires business to add controls, security, reporting and documentation. Organizations set up a governance program to ensure transparency over sometimes unclear processes.

• Level of complexity of the data assets

Having a strong governance become increasingly important with the exponential growth of data resources, tools and people in a company.

The level of complexity increases with the scope of business operations , the velocity of data creation or the level of automation based on data.

Business Benefits To Earn

There are multiple reasons why good data warehouse governance is a must, and it goes beyond the need for better data collection and management. Yes, having an efficient system for collecting and processing data allows the enterprise to benefit from lower data management costs in the long run, but the business implications are no less significant.

For starters, data can be fully integrated and processed holistically. Data relating to financial activities of the company, for instance, can be made more valuable when compounded with external data about market growth, competitors actions, and industry average values. For example, sales data can be analyzed in a deeper way within the context of market performance and changes.

The result is a healthier data-driven decision-making process, and one that encourages collaboration between departments. When a thorough analysis is performed, multiple aspects can be taken into account from the start. When deciding to expand the manufacturing line, for instance, market insights can be just as valuable as data from the sales and marketing teams.

There is also the possibility of increasing revenue from good data warehouse governance, both from the reduction of CAPEX and OPEX, and from the increase in revenue through the discovery of new opportunities. These are objectives that can be achieved through better data management and more accurate decision-making processes.

Opinions expressed by DZone contributors are their own.

Don’t Miss: How Are Governments Using Blockchain Technology

Building A Data Warehouse: With Examples In Sql Server

OUR TAKE: The reviews on this text speak for themselves, with one reader saying this title is everything you need on data warehouseing, while another says excellent roadmap book for building a data warehouse.

Here is the ideal field guide for data warehousing implementation. This book first teaches you how to build a data warehouse, including defining the architecture, understanding the methodology, gathering the requirements, designing the data models, and creating the databases. Coverage then explains how to populate the data warehouse and explores how to present data to users using reports and multidimensional databases and how to use the data in the data warehouse for business intelligence and customer relationship management.