Changelog#

1.6.6 (core) / 0.22.6 (libraries)#

New#

Dagster officially supports Python 3.12.
dagster-polars has been added as an integration. Thanks @danielgafni!
[dagster-dbt] @dbt_assets now supports loading projects with semantic models.
[dagster-dbt] @dbt_assets now supports loading projects with model versions.
[dagster-dbt] get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!
[dagster-duckdb] The Dagster DuckDB integration supports DuckDB version 0.10.0.
[UPath I/O manager] If a non-partitioned asset is updated to have partitions, the file containing the non-partitioned asset data will be deleted when the partitioned asset is materialized, rather than raising an error.

Bugfixes#

Fixed an issue where creating a backfill of assets with dynamic partitions and a backfill policy would sometimes fail with an exception.
Fixed an issue with the type annotations on the @asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!
[ui] On the asset graph, nodes are slightly wider allowing more text to be displayed, and group names are no longer truncated.
[ui] Fixed an issue where the groups in the asset graph would not update after an asset was switched between groups.
[dagster-k8s] Fixed an issue where setting the security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!

Experimental#

Observable source assets can now yield ObserveResults with no data_version.
You can now include FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.
[ui] In Dagster Cloud, a new feature flag allows you to enable an overhauled asset overview page with a high-level stakeholder view of the asset’s health, properties, and column schema.

Documentation#

Updated docs to reflect newly-added support for Python 3.12.

Dagster Cloud#

[kubernetes] Fixed an issue where the Kubernetes agent would sometimes leave dangling kubernetes services if the agent was interrupted during the middle of being terminated.

1.6.5 (core) / 0.22.5 (libraries)#

New#

Within a backfill or within auto-materialize, when submitting runs for partitions of the same assets, runs are now submitted in lexicographical order of partition key, instead of in an unpredictable order.
[dagster-k8s] Include k8s pod debug info in run worker failure messages.
[dagster-dbt] Events emitted by DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.

Bugfixes#

A previous change prevented asset backfills from grouping multiple assets into the same run when using BackfillPolicies under certain conditions. While the backfills would still execute in the proper order, this could lead to more individual runs than necessary. This has been fixed.
[dagster-k8s] Fixed an issue introduced in the 1.6.4 release where upgrading the Helm chart without upgrading the Dagster version used by user code caused failures in jobs using the k8s_job_executor.
[instigator-tick-logs] Fixed an issue where invoking context.log.exception in a sensor or schedule did not properly capture exception information.
[asset-checks] Fixed an issue where additional dependencies for dbt tests modeled as Dagster asset checks were not properly being deduplicated.
[dagster-dbt] Fixed an issue where dbt model, seed, or snapshot names with periods were not supported.

Experimental#

@observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.
[auto-materialize] A new AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.
[Global op/asset concurrency] Ops and assets now take run priority into account when claiming global op/asset concurrency slots.

Documentation#

Fixed an error in our asset checks docs. Thanks @vaharoni!
Fixed an error in our Dagster Pipes Kubernetes docs. Thanks @cameronmartin!
Fixed an issue on the Hello Dagster! guide that prevented it from loading.
Add specific capabilities of the Airflow integration to the Airflow integration page.
Re-arranged sections in the I/O manager concept page to make info about using I/O versus resources more prominent.

0.9.6#

New

Added ADLS2 storage plugin for Spark DataFrame (Thanks @sd2k!)
Added feature in the Dagit Playground to automatically remove extra configuration that does not conform to a pipeline’s config schema.
[Dagster-Celery/Celery-K8s/Celery-Docker] Added Celery worker names and pods to the logs for each step execution

Community contributions

Re-enabled dagster-azure integration tests in dagster-databricks tests (Thanks @sd2k!)
Moved dict_without_keys from dagster-pandas into dagster.utils (Thanks @DavidKatz-il)
Moved Dask DataFrame read/to options under read/to keys (Thanks @kinghuang)

Bugfixes

Fixed helper for importing data from GCS paths into Bigquery (Thanks @grabangomb (https://github.com/grabangomb)!)
Postgres event storage now waits to open a thread to watch runs until it is needed

Experimental

Added version computation function for DagsterTypeLoader. (Actual versioning will be supported in 0.10.0)
Added version attribute to solid and SolidDefinition. (Actual versioning will be supported in 0.10.0)

0.9.5#

New

UI improvements to the backfill partition selector
Enabled sorting of steps by failure in the partition run matrix in Dagit

Bugfixes

[dagstermill] fixes an issue with output notebooks and s3 storage
[dagster_celery] bug fixed in pythonpath calculation (thanks @enima2648!)
[dagster_pandas] marked create_structured_dataframe_type and ConstraintWithMetadata as experimental APIs
[dagster_k8s] reduced default job backoff limit to 0

Docs

Various docs site improvements

0.9.4#

Breaking Changes

When using the configured API on a solid or composite solid, a new solid name must be provided.
The image used by the K8sScheduler to launch scheduled executions is now specified under the “scheduler” section of the Helm chart (previously under “pipeline_run” section).

New

Added an experimental mode that speeds up interactions in dagit by launching a gRPC server on startup for each repository location in your workspace. To enable it, add the following to your dagster.yaml:


opt_in:
  local_servers: true

Intermediate Storage and System Storage now default to the first provided storage definition when no configuration is provided. Previously, it would be necessary to provide a run config for storage whenever providing custom storage definitions, even if that storage required no run configuration. Now, if the first provided storage definition requires no run configuration, the system will default to using it.
Added a timezone picker to Dagit, and made all timestamps timezone-aware
Added solid_config to hook context which provides the access to the config schema variable of the corresponding solid.
Hooks can be directly set on PipelineDefinition or @pipeline, e.g. @pipeline(hook_defs={hook_a}). It will apply the hooks on every single solid instance within the pipeline.
Added Partitions tab for partitioned pipelines, with new backfill selector.

0.9.3#

Breaking Changes

Removed deprecated --env flag from CLI
The --host CLI param has been renamed to --grpc_host to avoid conflict with the dagit --host param.

New

Descriptions for solid inputs and outputs will now be inferred from doc blocks if available (thanks @AndersonReyes !)
Various documentation improvements (thanks @jeriscc !)
Load inputs from pyspark dataframes (thanks @davidkatz-il !)
Added step-level run history for partitioned schedules on the schedule view
Added great_expectations integration, through the dagster_ge library. Example usage is under a new example, called ge_example, and documentation for the library can be found under the libraries section of the api docs.
PythonObjectDagsterType can now take a tuple of types as well as a single type, more closely mirroring isinstance and allowing Union types to be represented in Dagster.
The configured API can now be used on all definition types (including CompositeDefinition). Example usage has been updated in the configuration documentation.
Updated Helm chart to include auto-generated user code configmap in user code deployment by default

Bugfixes

Databricks now checks intermediate storage instead of system storage
Fixes a bug where applying hooks on a pipeline with composite solids would flatten the top-level solids. Now applying hooks on pipelines or composite solids means attaching hooks to every single solid instance within the pipeline or the composite solid.
Fixes the GraphQL playground hosted by dagit
Fixes a bug where K8s CronJobs were stopped unnecessarily during schedule reconciliation

Experimental

New dagster-k8s/config tag that lets users pass in custom configuration to the Kubernetes Job, Job metadata, JobSpec, PodSpec, and PodTemplateSpec metadata.

This allows users to specify settings like eviction policy annotations and node affinities.
Example:


  @solid(
    tags = {
      'dagster-k8s/config': {
        'container_config': {
          'resources': {
            'requests': { 'cpu': '250m', 'memory': '64Mi' },
            'limits': { 'cpu': '500m', 'memory': '2560Mi' },
          }
        },
        'pod_template_spec_metadata': {
          'annotations': { "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"}
        },
        'pod_spec_config': {
          'affinity': {
            'nodeAffinity': {
              'requiredDuringSchedulingIgnoredDuringExecution': {
                'nodeSelectorTerms': [{
                  'matchExpressions': [{
                    'key': 'beta.kubernetes.io/os', 'operator': 'In', 'values': ['windows', 'linux'],
                  }]
                }]
              }
            }
          }
        },
      },
    },
  )
  def my_solid(context):
    context.log.info('running')

0.9.2#

Breaking Changes

The --env flag no longer works for the pipeline launch or pipeline execute commands. Use --config instead.
The pipeline execute command no longer accepts the --workspace argument. To execute pipelines in a workspace, use pipeline launch instead.

New

Added ResourceDefinition.mock_resource helper for magic mocking resources. Example usage can be found here
Remove the row_count metadata entry from the Dask DataFrame type check (thanks @kinghuang!)
Add orient to the config options when materializing a Dask DataFrame to json (thanks @kinghuang!)

Bugfixes

Fixed a bug where applying configured to a solid definition would overwrite inputs from run config.
Fixed a bug where pipeline tags would not apply to solid subsets.
Improved error messages for repository-loading errors in CLI commands.
Fixed a bug where pipeline execution error messages were not being surfaced in Dagit.

You are viewing an unreleased or outdated version of the documentation

Changelog#

1.6.6 (core) / 0.22.6 (libraries)#

New#

Bugfixes#

Experimental#

Documentation#

Dagster Cloud#

1.6.5 (core) / 0.22.5 (libraries)#

New#

Bugfixes#

Experimental#

Documentation#

0.9.6#

0.9.5#

0.9.4#

0.9.3#

0.9.2#