data engineering

Open Metadata: Discover, Manage and Collaborate on your Data

Hazem Abbas

Jul 9, 2023 — 4 min read

Photo by fabio / Unsplash

OpenMetaData is a comprehensive platform that offers a range of functionalities, including data discovery, data lineage, data quality, observability, governance, and team collaboration. It is an open-source project that has gained immense popularity among companies across various industry verticals, thanks to its vibrant community and adoption.

OpenMetaData is built on a centralized metadata store that utilizes Open Metadata Standards/APIs. The platform supports connectors to a wide range of data services, enabling end-to-end metadata management. This gives users the freedom to unlock the full potential of their data assets and derive insights that can support better decision making.

With OpenMetaData, users can easily discover their data assets, understand their lineage, and ensure data quality. The platform also offers observability features that allow users to monitor their data pipelines and ensure they are running smoothly. Governance is another key feature of OpenMetaData, enabling users to manage their data assets and ensure compliance with regulatory requirements.

Collaboration is also a central aspect of OpenMetaData, with features that allow teams to work together seamlessly, share insights, and collaborate on data-driven projects.

In summary, OpenMetaData is an all-in-one metadata management platform that enables users to discover, manage, and collaborate on their data assets. Its open-source nature and support for Open Metadata Standards/APIs, combined with its extensive functionality, make it a popular choice among companies across various industry verticals.

Components

OpenMetadata includes the following:

Metadata Schemas - Defines core abstractions and vocabulary for metadata with schemas for Types, Entities, and Relationships between entities. This is the foundation of the Open Metadata Standard. Also supports the extensibility of entities and types with custom properties.
Metadata Store - Stores metadata graph that connects data assets, user, and tool-generated metadata.
Metadata APIs - For producing and consuming metadata built on schemas for User Interfaces and Integration of tools, systems, and services.
Ingestion Framework - A pluggable framework for integrating tools and ingesting metadata to the metadata store, supporting about 55 connectors. The ingestion framework supports well know data warehouses like Google BigQuery, Snowflake, Amazon Redshift, and Apache Hive; databases like MySQL, Postgres, Oracle, and MSSQL; dashboard services like Tableau, Superset, and Metabase; messaging services like Kafka, Redpanda; and pipeline services like Airflow, Glue, Fivetran, Dagster, and many more.
OpenMetadata User Interface - A single place for users to discover and collaborate on all data.

Features

Here are some of the supported features in a nutshell:

Data Collaboration - Get event notifications with Activity feeds. Send alerts & notifications using webhooks. Add Announcements to notify the team of upcoming changes. Add Tasks to request descriptions or glossary term approval workflows. Add user mentions and collaborate using conversation threads.
Data Quality and Profiler - Standardized tests and data quality metadata. Groups related tests as Test Suites. Supports custom SQL data quality tests. Has an interactive dashboard to drill down to the details.
Data Lineage - Supports rich column-level lineage. Effectively filters queries to extract lineage. Edit lineage manually as required and connect the entities with a no-code editor.
Comprehensive Roles and Policies - Handles complex access control use cases and hierarchical teams.
Webhooks - Supports webhook integrations. Integrate with Slack, Microsoft Teams and Google Chat.
Connectors - Supports 55 connectors to various databases, dashboards, pipelines, and messaging services.
Glossary - Add a Controlled Vocabulary to describe important concepts and terminologies within your organization. Add Glossaries, Terms, Tags, Descriptions, and Reviewers.
Data Security - Supports Google, Okta, custom OIDC, Auth0, Azure, Amazon Cognito, and OneLogin as identity providers for SSO. Also, supports SAML-based authentication for AWS SSO and Google.
Secrets Manager Interface - Communicates with any key management store.
And lots more...