Skip to main content

Deploying Dagster as a service

This guide will walk you through deploying Dagster as a service on a single machine. It includes instructions for setting up the Dagster webserver and daemon. This approach is suitable for small-scale deployments or for testing purposes. For production environments, consider using containerized deployments or cloud-based solutions.

Prerequisites

Before following the steps in this guide, you will need to create a Dagster project.

Running the Dagster webserver​

The Dagster webserver is the core component of any Dagster deployment. It serves the Dagster UI and responds to GraphQL queries.

Step 1: Install the Dagster webserver​

First, install the Dagster webserver:

pip install dagster-webserver

Step 2: Start the Dagster webserver​

Before starting the webserver, set the DAGSTER_HOME environment variable, which tells Dagster where to store its persistent data and logs.

export DAGSTER_HOME="/home/yourusername/dagster_home"

Then, to run the webserver, use the following command:

dagster-webserver -h 0.0.0.0 -p 3000

This configuration will:

  • Set the DAGSTER_HOME environment variable, which tells Dagster where to store its persistent data and logs
  • Write execution logs to $DAGSTER_HOME/logs
  • Listen on 0.0.0.0:3000

Running the Dagster daemon​

The Dagster daemon is necessary if you're using schedules, sensors, backfills, or want to set limits on the number of runs that can be executed simultaneously.

Step 1: Install the Dagster daemon​

Install the Dagster daemon:

pip install dagster

Step 2: Start the Dagster daemon​

Make sure you've set the DAGSTER_HOME environment variable, see Running the Dagster Webserver for instructions. Then, run the Dagster daemon with this command:

dagster-daemon run

The dagster-daemon process will periodically check your instance for:

  • New runs to be launched from your run queue
  • Runs triggered by your running schedules or sensors
tip

Ensure that the dagster-daemon process has access to:

  • Your dagster.yaml file
  • Your workspace.yaml file
  • The components defined on your instance
  • The repositories defined in your workspace

Monitoring the Dagster daemon​

You can check the status of your dagster-daemon process in the Dagster UI:

  1. Navigate to the Instance tab in the left-hand navigation bar.
  2. View the daemon status.
important

A deployment can have multiple instances of dagster-webserver, but should include only a single dagster-daemon process.