dagster-polars has been added as an integration. Thanks @danielgafni!@dbt_assets now supports loading projects with semantic models.@dbt_assets now supports loading projects with model versions.get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!@asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!ObserveResults with no data_version.FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.k8s_job_executor.context.log.exception in a sensor or schedule did not properly capture exception information.@observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.New
configured API makes it easy to create configured versions of resources.Materialization event type in favor of the new AssetMaterialization event type,
which requires the asset_key parameter. Solids yielding Materialization events will continue
to work as before, though the Materialization event will be removed in a future release.intermediate_storage_defs argument to ModeDefinition, which accepts a
list of IntermediateStorageDefinitions, e.g. s3_plus_default_intermediate_storage_defs.
As before, the default includes an in-memory intermediate and a local filesystem intermediate
storage.system_storage_defs argument to ModeDefinition in favor of
intermediate_storage_defs. system_storage_defs will be removed in 0.10.0 at the earliest.@intermediate_storage decorator, which makes it easy to define intermediate
storages.s3_file_manager and local_file_manager resources to replace the file managers
that previously lived inside system storages. The airline demo has been updated to include
an example of how to do this:
https://github.com/dagster-io/dagster/blob/0.8.8/examples/airline_demo/airline_demo/solids.py#L171.Bugfixes
default_value config on a field now works as expected. #2725Breaking Changes
dagster and dagit CLI commands no longer add the working directory to the
PYTHONPATH when resolving modules, which may break some imports. Explicitly installed python
packages can be specified in workspaces using the python_package workspace yaml config option.
The python_module config option is deprecated and will be removed in a future release.New
--path-prefix to the dagit CLI. #2073date_partition_range util function now accepts an optional inclusive boolean argument. By default, the function does not return include the partition for which the end time of the date range is greater than the current time. If inclusive=True, then the list of partitions returned will include the extra partition.MultiDependency or fan-in inputs will now only cause the solid step to skip if all of the
fanned-in inputs upstream outputs were skippedBugfixes
input_hydration_config argumentsalias on a solid output will produce a useful error message (thanks @iKintosh!)daily_schedule) for certain workspace.yaml formatsBreaking Changes
dagster-celery module has been broken apart to manage dependencies more coherently. There
are now three modules: dagster-celery, dagster-celery-k8s, and dagster-celery-docker.dagster-celery worker start command now takes a required -A parameter
which must point to the app.py file within the appropriate module. E.g if you are using the
celery_k8s_job_executor then you must use the -A dagster_celery_k8s.app option when using the
celery or dagster-celery cli tools. Similar for the celery_docker_executor:
-A dagster_celery_docker.app must be used.input_hydration_config and output_materialization_config decorators to
dagster_type_ and dagster_type_materializer respectively. Renamed DagsterType's
input_hydration_config and output_materialization_config arguments to loader and materializer respectively.New
New pipeline scoped runs tab in Dagit
Add the following Dask Job Queue clusters: moab, sge, lsf, slurm, oar (thanks @DavidKatz-il!)
K8s resource-requirements for run coordinator pods can be specified using the dagster-k8s/ resource_requirements tag on pipeline definitions:
@pipeline( tags={ 'dagster-k8s/resource_requirements': { 'requests': {'cpu': '250m', 'memory': '64Mi'}, 'limits': {'cpu': '500m', 'memory': '2560Mi'}, } }, ) def foo_bar_pipeline():
Added better error messaging in dagit for partition set and schedule configuration errors
An initial version of the CeleryDockerExecutor was added (thanks @mrdrprofuroboros!). The celery workers will launch tasks in docker containers.
Experimental: Great Expectations integration is currently under development in the new library dagster-ge. Example usage can be found here
Breaking Changes
Engine and ExecutorConfig have been deleted in favor of Executor. Instead of the @executor decorator decorating a function that returns an ExecutorConfig it should now decorate a function that returns an Executor.New
dict can be used as an alias for Permissive() within a config schema declaration.StringSource in the S3ComputeLogManager configuration schema to support using environment variables in the configuration (Thanks @mrdrprofuroboros!)Bugfixes
$DAGSTER_HOME environment variable is not an absolute path (Thanks @AndersonReyes!)staging_prefix in the DatabricksPySparkStepLauncher configuration to be an absolute path (Thanks @sd2k!)input_hydration_config (Thanks @joeyfreund!)Bugfix
New
dagster asset wipe <asset_key>