Changelog#

1.6.6 (core) / 0.22.6 (libraries)#

New#

• Dagster officially supports Python 3.12.
• dagster-polars has been added as an integration. Thanks @danielgafni!
• [dagster-dbt] @dbt_assets now supports loading projects with semantic models.
• [dagster-dbt] @dbt_assets now supports loading projects with model versions.
• [dagster-dbt] get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!
• [dagster-duckdb] The Dagster DuckDB integration supports DuckDB version 0.10.0.
• [UPath I/O manager] If a non-partitioned asset is updated to have partitions, the file containing the non-partitioned asset data will be deleted when the partitioned asset is materialized, rather than raising an error.

Bugfixes#

• Fixed an issue where creating a backfill of assets with dynamic partitions and a backfill policy would sometimes fail with an exception.
• Fixed an issue with the type annotations on the @asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!
• [ui] On the asset graph, nodes are slightly wider allowing more text to be displayed, and group names are no longer truncated.
• [ui] Fixed an issue where the groups in the asset graph would not update after an asset was switched between groups.
• [dagster-k8s] Fixed an issue where setting the security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!

Experimental#

• Observable source assets can now yield ObserveResults with no data_version.
• You can now include FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.
• [ui] In Dagster Cloud, a new feature flag allows you to enable an overhauled asset overview page with a high-level stakeholder view of the asset’s health, properties, and column schema.

Documentation#

• Updated docs to reflect newly-added support for Python 3.12.

Dagster Cloud#

• [kubernetes] Fixed an issue where the Kubernetes agent would sometimes leave dangling kubernetes services if the agent was interrupted during the middle of being terminated.

1.6.5 (core) / 0.22.5 (libraries)#

New#

• Within a backfill or within auto-materialize, when submitting runs for partitions of the same assets, runs are now submitted in lexicographical order of partition key, instead of in an unpredictable order.
• [dagster-k8s] Include k8s pod debug info in run worker failure messages.
• [dagster-dbt] Events emitted by DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.

Bugfixes#

• A previous change prevented asset backfills from grouping multiple assets into the same run when using BackfillPolicies under certain conditions. While the backfills would still execute in the proper order, this could lead to more individual runs than necessary. This has been fixed.
• [dagster-k8s] Fixed an issue introduced in the 1.6.4 release where upgrading the Helm chart without upgrading the Dagster version used by user code caused failures in jobs using the k8s_job_executor.
• [instigator-tick-logs] Fixed an issue where invoking context.log.exception in a sensor or schedule did not properly capture exception information.
• [asset-checks] Fixed an issue where additional dependencies for dbt tests modeled as Dagster asset checks were not properly being deduplicated.
• [dagster-dbt] Fixed an issue where dbt model, seed, or snapshot names with periods were not supported.

Experimental#

• @observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.
• [auto-materialize] A new AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.
• [Global op/asset concurrency] Ops and assets now take run priority into account when claiming global op/asset concurrency slots.

Documentation#

• Fixed an error in our asset checks docs. Thanks @vaharoni!
• Fixed an error in our Dagster Pipes Kubernetes docs. Thanks @cameronmartin!
• Fixed an issue on the Hello Dagster! guide that prevented it from loading.
• Add specific capabilities of the Airflow integration to the Airflow integration page.
• Re-arranged sections in the I/O manager concept page to make info about using I/O versus resources more prominent.

0.6.5#

Breaking

• The interface for type checks has changed. Previously the type_check_fn on a custom type was required to return None (=passed) or else raise Failure (=failed). Now, a type_check_fn may return True/False to indicate success/failure in the ordinary case, or else return a TypeCheck. The newsuccess field on TypeCheck now indicates success/failure. This obviates the need for the typecheck_metadata_fn, which has been removed.
• Executions of individual composite solids (e.g. in test) now produce a CompositeSolidExecutionResult rather than a SolidExecutionResult.
• dagster.core.storage.sqlite_run_storage.SqliteRunStorage has moved to dagster.core.storage.runs.SqliteRunStorage. Any persisted dagster.yaml files should be updated with the new classpath.
• is_secret has been removed from Field. It was not being used to any effect.
• The environmentType and configTypes fields have been removed from the dagster-graphql Pipeline type. The configDefinition field on SolidDefinition has been renamed to configField.

Bugfix

• PresetDefinition.from_files is now guaranteed to give identical results across all Python minor versions.
• Nested composite solids with no config, but with config mapping functions, now behave as expected.
• The dagster-airflow DagsterKubernetesPodOperator has been fixed.
• Dagit is more robust to changes in repositories.
• Improvements to Dagit interface.

New

• dagster_pyspark now supports remote execution on EMR with the @pyspark_solid decorator.

Nits

• Documentation has been improved.
• The top level config field features in the dagster.yaml will no longer have any effect.
• Third-party dependencies have been relaxed to reduce the risk of version conflicts.

0.6.4#

• Scheduler errors are now visible in Dagit
• Run termination button no longer persists past execution completion
• Fixes run termination for multiprocess execution
• Fixes run termination on Windows
• dagit no longer prematurely returns control to terminal on Windows
• raise_on_error is now available on the execute_solid test utility
• check_dagster_type added as a utility to help test type checks on custom types
• Improved support in the type system for Set and Tuple types
• Allow composite solids with config mapping to expose an empty config schema
• Simplified graphql API arguments to single-step re-execution to use retryRunId, stepKeys execution parameters instead of a reexecutionConfig input object
• Fixes missing step-level stdout/stderr from dagster CLI

0.6.3#

• Adds a type_check parameter to PythonObjectType, as_dagster_type, and @as_dagster_type to enable custom type checks in place of default isinstance checks. See documentation here: https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/types.html#custom-type-checks

• Improved the type inference experience by automatically wrapping bare python types as dagster types.

• Reworked our tutorial (now with more compelling/scary breakfast cereal examples) and public API documentation. See the new tutorial here: https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/index.html

• New solids explorer in Dagit allows you to browse and search for solids used across the repository.

• Enabled solid dependency selection in the Dagit search filter.

  • To select a solid and its upstream dependencies, search +{solid_name}.
  • To select a solid and its downstream dependents, search {solid_name}+.
  • For both search +{solid_name}+.

• Added a terminate button in Dagit to terminate an active run.

• Added an --output flag to dagster-graphql CLI.

• Added confirmation step for dagster run wipe and dagster schedule wipe commands (Thanks @shahvineet98).

• Fixed a wrong title in the dagster-snowflake library README (Thanks @Step2Web).

0.6.2#

• Changed composition functions @pipeline and @composite_solid to automatically give solids aliases with an incrementing integer suffix when there are conflicts. This removes to the need to manually alias solid definitions that are used multiple times.
• Add dagster schedule wipe command to delete all schedules and remove all schedule cron jobs
• execute_solid test util now works on composite solids.
• Docs and example improvements: https://dagster.readthedocs.io/
• Added --remote flag to dagster-graphql for querying remote Dagit servers.
• Fixed issue with duplicate run tag autocomplete suggestions in Dagit (#1839)
• Fixed Windows 10 / py3.6+ bug causing pipeline execution failures

0.6.1#

• Fixed an issue where Dagster public images tagged latest on Docker Hub were erroneously published with an older version of Dagster (#1814)
• Fixed an issue where the most recent scheduled run was not displayed in Dagit (#1815)
• Fixed a bug with the dagster schedule start --start-all command (#1812)
• Added a new scheduler command to restart a schedule: dagster schedule restart. Also added a flag to restart all running schedules: dagster schedule restart --restart-all-running.