dagster-polars has been added as an integration. Thanks @danielgafni!@dbt_assets now supports loading projects with semantic models.@dbt_assets now supports loading projects with model versions.get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!@asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!ObserveResults with no data_version.FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.k8s_job_executor.context.log.exception in a sensor or schedule did not properly capture exception information.@observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.[dagstermill] You can now have more precise IO control over the output notebooks by specifying output_notebook_name in define_dagstermill_solid and providing your own IO manager via "output_notebook_io_manager" resource key.
We've deprecated output_notebook argument in define_dagstermill_solid in favor of output_notebook_name.
Previously, the output notebook functionality requires “file_manager“ resource and result in a FileHandle output. Now, when specifying output_notebook_name, it requires "output_notebook_io_manager" resource and results in a bytes output.
You can now customize your own "output_notebook_io_manager" by extending OutputNotebookIOManager. A built-in local_output_notebook_io_manager is provided for handling local output notebook materialization.
See detailed migration guide in https://github.com/dagster-io/dagster/pull/4490.
Dagit fonts have been updated.
context.log.info("foo %s", "bar") would not get formatted as expected.QueuedRunCoordinator’s tag_concurrency_limits to not be respected in some casestags argument of the @graph decorator or GraphDefinition constructor. These tags will be set on any runs of jobs are built from invoking to_job on the graph.k8s_job_executor or celery_k8s_job_executor. Use the key image inside the container_config block of the k8s solid tag.jobs argument. Each RunRequest emitted from a multi-job sensor’s evaluation function must specify a job_name.KubernetesRunLauncher image pull policy is now configurable in a separate field (thanks @yamrzou!).dagster-github package is now usable for GitHub Enterprise users (thanks @metinsenturk!) A hostname can now be provided via config to the dagster-github resource with the key github_hostname:execute_pipeline( github_pipeline, {'resources': {'github': {'config': { "github_app_id": os.getenv('GITHUB_APP_ID'), "github_app_private_rsa_key": os.getenv('GITHUB_PRIVATE_KEY'), "github_installation_id": os.getenv('GITHUB_INSTALLATION_ID'), "github_hostname": os.getenv('GITHUB_HOSTNAME'), }}}} )
pipeline_failure_sensor and run_status_sensor queries. To take advantage of these performance gains, run a schema migration with the CLI command: dagster instance migrate.DockerRunLauncher would raise an exception when no networks were specified in its configuration.dagster-slack has migrated off of deprecated slackclient (deprecated) and now uses [slack_sdk](https://slack.dev/python-slack-sdk/v3-migration/).OpDefinition, the replacement for SolidDefinition which is the type produced by the @op decorator, is now part of the public API.daily_partitioned_config, hourly_partitioned_config, weekly_partitioned_config, and monthly_partitioned_config now accept an end_offset parameter, which allows extending the set of partitions so that the last partition ends after the current time.Previously in Dagit, when a repository location had an error when reloaded, the user could end up on an empty page with no context about the error. Now, we immediately show a dialog with the error and stack trace, with a button to try reloading the location again when the error is fixed.
Dagster is now compatible with Python’s logging module. In your config YAML file, you can configure log handlers and formatters that apply to the entire Dagster instance. Configuration instructions and examples detailed in the docs: https://docs.dagster.io/concepts/logging/python-logging
[helm] The timeout of database statements sent to the Dagster instance can now be configured using .dagit.dbStatementTimeout.
The QueuedRunCoordinator now supports setting separate limits for each unique value with a certain key. In the below example, 5 runs with the tag (backfill: first) could run concurrently with 5 other runs with the tag (backfill: second).
run_coordinator: module: dagster.core.run_coordinator class: QueuedRunCoordinator config: tag_concurrency_limits: - key: backfill value: applyLimitPerUniqueValue: True limit: 5
.migrate.enabled=True.Added instance on RunStatusSensorContext for accessing the Dagster Instance from within the
run status sensors.
The inputs of a Dagstermill solid now are loaded the same way all other inputs are loaded in the framework. This allows rerunning output notebooks with properly loaded inputs outside Dagster context. Previously, the IO handling depended on temporary marshal directory.
Previously, the Dagit CLI could not target a bare graph in a file, like so:
from dagster import op, graph @op def my_op(): pass @graph def my_graph(): my_op()
This has been remedied. Now, a file foo.py containing just a graph can be targeted by the dagit
CLI: dagit -f foo.py.
When a solid, pipeline, schedule, etc. description or event metadata entry contains a markdown-formatted table, that table is now rendered in Dagit with better spacing between elements.
The hacker-news example now includes instructions on how to deploy the repository in a Kubernetes cluster using the Dagster Helm chart.
[dagster-dbt] The dbt_cli_resource now supports the dbt source snapshot-freshness command
(thanks @emilyhawkins-drizly!)
[helm] Labels are now configurable on user code deployments.
Bugfixes
EventMetadata.asset and EventMetadata.pipeline_run in
AssetMaterialization metadata. (Thanks @ymrzkrrs and @drewsonne!)Breaking Changes
fs_io_manager, which allows
data to be passed out of the Jupyter process boundary.Community Contributions
Documentation
objects.inv is available at http://docs.dagster.io/objects.inv for other projects to link.execute_solid has been removed from the testing (https://docs.dagster.io/concepts/testing)
section. Direct invocation is recommended for testing solids.gcsfs as a dependency.create_databricks_job_solid now includes an example of how to use it.context.log.info() and other similar functions now fully respect the python logging API. Concretely, log statements of the form context.log.error(“something %s happened!”, “bad”) will now work as expected, and you are allowed to add things to the “extra” field to be consumed by downstream loggers: context.log.info("foo", extra={"some":"metadata"}).config_from_files, config_from_pkg_resources, and config_from_yaml_strings have been added for constructing run config from yaml files and strings.DockerRunLauncher can now be configured to launch runs that are connected to more than one network, by configuring the networks key.env_from and volume_mounts are now properly applied to the corresponding Kubernetes run worker and job pods.end_mlflow_run_on_pipeline_finished hook now no longer errors whenever invoked.context.log calls are now not allowed. context.log.info("msg", foo="hi") should be rewritten as context.log.info("msg", extra={"foo":"hi"}).AssetMaterialization. Previously, it would still yield an AssetMaterialization where the path is a temp file path that won't exist after the notebook execution.InputContext and OutputContext now each has an asset_key that returns the asset key that was provided to the corresponding InputDefinition or OutputDefinition.