Skip to main content

SSH / SFTP (dagster-ssh)

This library provides an integration with SSH and SFTP.

class dagster_ssh.SSHResource [source]
beta

This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.

A Dagster resource for establishing SSH connections and performing remote file operations.

This resource leverages the Paramiko library to provide robust SSH connectivity, including support for key-based and password authentication, tunneling, and SFTP transfers.

Parameters:

  • remote_host (str) – The hostname or IP address of the remote server to connect to.
  • remote_port (Optional[int]) – The SSH port on the remote host. Defaults to standard SSH port 22.
  • username (Optional[str]) – The username for SSH authentication. If not provided, defaults to the current system user.
  • password (Optional[str]) – The password for SSH authentication. Not recommended for production use; prefer key-based authentication.
  • key_file (Optional[str]) – Path to the SSH private key file for authentication.
  • key_string (Optional[str]) – SSH private key as a string for authentication.
  • timeout (int, optional) – Connection timeout in seconds. Defaults to 10.
  • keepalive_interval (int, optional) – Interval for sending SSH keepalive packets. (Defaults to 30 seconds.)
  • compress (bool, optional) – Whether to compress the SSH transport stream. Defaults to True.
  • no_host_key_check (bool, optional) – Disable host key verification.
  • allow_host_key_change (bool, optional) – Allow connections to hosts with changed host keys. (Defaults to False.)

Example:

Creating an SSH resource with key-based authentication:

ssh_resource = SSHResource(
    remote_host="example.com",
    username="myuser", key_file="/path/to/private/key"
)

Creating an SSH resource with password authentication:

ssh_resource = SSHResource(
    remote_host="example.com",
    username="myuser",
    password="my_secure_password"
)

Using the resource to transfer a file:

local_file = ssh_resource.sftp_get("/remote/path/file.txt", "/local/path/file.txt")
dagster_ssh.ssh_resource ResourceDefinition [source]
beta

This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.

A Dagster resource factory for creating SSHResource instances.

This function converts Dagster resource context configuration into an SSHResource that can be used for remote SSH connections and file operations.

Parameters: init_context (InitResourceContext) – The Dagster resource initialization context containing configuration parameters.Returns: A configured SSH resource ready for use in Dagster pipelines.Return type: SSHResource Example:

Configuring the SSH resource in a Dagster pipeline:

from dagster import Definitions, job, op
from dagster_ssh import ssh_resource

@op
def transfer_files(ssh):
    ssh.sftp_get("/remote/file", "/local/file")

@job
def my_ssh_job():
    transfer_files(ssh=ssh_resource.configured({
        "remote_host": "example.com",
        "username": "myuser",
        "key_file": "/path/to/private/key"
    }))

defs = Definitions(jobs=[my_ssh_job])