connectors

No menu items for this category
OpenMetadata Documentation
Dagster

Dagster

PROD
Available In
Feature List
Pipelines
Pipeline Status
Tags
Usage
Owners
Lineage

In this section, we provide guides and references to use the Dagster connector.

Configure and schedule Dagster metadata and profiler workflows from the OpenMetadata UI:

To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing Airflow host, you can follow this guide.

If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to run the Ingestion Framework in any orchestrator externally.

OpenMetadata is integrated with dagster up to version 1.0.13 and will continue to work for future dagster versions.

The ingestion framework uses dagster graphql python client to connect to the dagster instance and perform the API calls

Click Settings in the side navigation bar and then Services.

The first step is to ingest the metadata from your sources. To do that, you first need to create a Service connection first.

This Service will be the bridge between OpenMetadata and your source system.

Once a Service is created, it can be used to configure your ingestion workflows.

Visit Services Page

Select your Service Type and Add a New Service

Click on Add New Service to start the Service creation.

Create a new Service

Add a new Service from the Services page

Select Dagster as the Service type and click Next.

Select Service

Select your Service from the list

Provide a name and description for your Service.

OpenMetadata uniquely identifies Services by their Service Name. Provide a name that distinguishes your deployment from other Services, including the other Dagster Services that you might be ingesting metadata from.

Note that when the name is set, it cannot be changed.

Add New Service

Provide a Name and description for your Service

In this step, we will configure the connection settings required for Dagster.

Please follow the instructions below to properly configure the Service to read from your sources. You will also find helper documentation on the right-hand side panel in the UI.

Configure Service connection

Configure the Service connection by filling the form