dagster-polars has been added as an integration. Thanks @danielgafni!
[dagster-dbt] @dbt_assets now supports loading projects with semantic models.
[dagster-dbt] @dbt_assets now supports loading projects with model versions.
[dagster-dbt] get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!
[dagster-duckdb] The Dagster DuckDB integration supports DuckDB version 0.10.0.
[UPath I/O manager] If a non-partitioned asset is updated to have partitions, the file containing the non-partitioned asset data will be deleted when the partitioned asset is materialized, rather than raising an error.
Fixed an issue where creating a backfill of assets with dynamic partitions and a backfill policy would sometimes fail with an exception.
Fixed an issue with the type annotations on the @asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!
[ui] On the asset graph, nodes are slightly wider allowing more text to be displayed, and group names are no longer truncated.
[ui] Fixed an issue where the groups in the asset graph would not update after an asset was switched between groups.
[dagster-k8s] Fixed an issue where setting the security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!
Observable source assets can now yield ObserveResults with no data_version.
You can now include FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.
[ui] In Dagster Cloud, a new feature flag allows you to enable an overhauled asset overview page with a high-level stakeholder view of the asset’s health, properties, and column schema.
[kubernetes] Fixed an issue where the Kubernetes agent would sometimes leave dangling kubernetes services if the agent was interrupted during the middle of being terminated.
Within a backfill or within auto-materialize, when submitting runs for partitions of the same assets, runs are now submitted in lexicographical order of partition key, instead of in an unpredictable order.
[dagster-k8s] Include k8s pod debug info in run worker failure messages.
[dagster-dbt] Events emitted by DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.
A previous change prevented asset backfills from grouping multiple assets into the same run when using BackfillPolicies under certain conditions. While the backfills would still execute in the proper order, this could lead to more individual runs than necessary. This has been fixed.
[dagster-k8s] Fixed an issue introduced in the 1.6.4 release where upgrading the Helm chart without upgrading the Dagster version used by user code caused failures in jobs using the k8s_job_executor.
[instigator-tick-logs] Fixed an issue where invoking context.log.exception in a sensor or schedule did not properly capture exception information.
[asset-checks] Fixed an issue where additional dependencies for dbt tests modeled as Dagster asset checks were not properly being deduplicated.
[dagster-dbt] Fixed an issue where dbt model, seed, or snapshot names with periods were not supported.
@observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.
[auto-materialize] A new AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.
[Global op/asset concurrency] Ops and assets now take run priority into account when claiming global op/asset concurrency slots.
Dagit now allows re-executing arbitrary step subset via step selector syntax, regardless of
whether the previous pipeline failed or not.
Added a search filter for the root Assets page
Adds tooltip explanations for disabled run actions
The last output of the cron job command created by the scheduler is now stored in a file. A new
dagster schedule logs {schedule_name} command will show the log file for a given schedule. This
helps uncover errors like missing environment variables and import errors.
The Dagit schedule page will now show inconsistency errors between schedule state and the cron
tab that were previously only displayed by the dagster schedule debug command. As before, these
errors can be resolve using dagster schedule up
Bugfix
Fixes an issue with config schema validation on Arrays
Fixes an issue with initializing K8sRunLauncher when configured via dagster.yaml
Fixes a race condition in Airflow injection logic that happens when multiple Operators try to
create PipelineRun entries simultaneously.
Fixed an issue with schedules that had invalid config not logging the appropriate error.
dagster pipeline backfill command no longer takes a mode flag. Instead, it uses the mode
specified on the PartitionSetDefinition. Similarly, the runs created from the backfill also use
the solid_subset specified on the PartitionSetDefinition
BugFix
Fixes a bug where using solid subsets when launching pipeline runs would fail config validation.
(dagster-gcp) allow multiple "bq_solid_for_queries" solids to co-exist in a pipeline
Improve scheduler state reconciliation with dagster-cron scheduler. dagster schedule debug
command will display issues related to missing crob jobs, extraneous cron jobs, and duplicate cron
jobs. Running dagster schedule up will fix any issues.
New
The dagster-airflow package now supports loading Airflow dags without depending on initialized
Airflow db
Improvements to the longitudinal partitioned schedule view, including live updates, run filtering,
and better default states.
Added user warning for dagster library packages that are out of sync with the core dagster
package.
Fixed an issue with strict snapshot ID matching when loading historical snapshots, which caused
errors on the Runs page when viewing historical runs.
Fixed an issue where dagster_celery had introduced a spurious dependency on dagster_k8s
(#2435)
Fixed an issue where our Airflow, Celery, and Dask integrations required S3 or GCS storage and
prevented use of filesystem storage. Filesystem storage is now also permitted, to enable use of
these integrations with distributed filesystems like NFS (#2436).