apache-airflow-providers-common-sql

变更日志¶

1.26.0¶

特性¶

feat: integrate 'polars' in 'get_df', 'get_df_by_chunks' (#48875)

杂项¶

remove superfluous else block (#49199)

1.25.0¶

特性¶

Add requires_result_fetch Configuration to SQLExecuteQueryOperator (#46997)

1.24.1¶

Bug 修复¶

Fix get_hook() got an unexpected keyword argument 'hook_params' in SQLExecuteQueryTrigger (#47728)
Do not allow semicolon in partition_clause (#48098)

杂项¶

AIP-72: Handle Custom XCom Backend on Task SDK (#47339)
AIP-72: Implement short circuit and branch operators (#46584)

1.24.0¶

特性¶

Add deferred pagination mode to GenericTransfer (#44809)

Bug 修复¶

fIx deprecation warnings in common.sql (#47169)
Add missing 'methodtools' dependency of the 'common-sql' provider (#47148)

杂项¶

Upgrade flit to 3.11.0 (#46938)

1.23.0¶

注意

此版本没有代码更改。发布此版本是因为之前的版本因打包问题被撤回。

1.22.0¶

特性¶

Introduce notion of dialects in DbApiHook (#41327)

Bug 修复¶

Fix DbApiHook.insert_rows logging incorrect number of rows commited (#45789)
Fix escaping of special characters or reserved words as column names in dialects of common sql provider (#45640)

杂项¶

Add run_after column to DagRun model (#45732)
Adding type ignore to sql hook (#46163)
Added ADR document describing why the notion of dialects is introduced (#45456)
Bring back mypy checks for new-structure providers (#45815)
Remove obsolete pandas specfication for pre-python 3.9 (#45399)

1.21.0¶

注意

此版本的 provider 仅适用于 Airflow 2.9+，如 Apache Airflow provider 支持策略中所述。

警告

所有已弃用的类、参数和特性已从 Common SQL Provider 包中移除。可能存在非常旧的 Provider 版本依赖于 common.sql 的私有函数，导致用户在不知情的情况下围绕这些私有函数创建了自定义代码。这个问题早已修复，并且可能会影响使用非常旧的 Provider 版本的用户。我们不认为这是一项破坏性更改，但出于可见性考虑，我们在此明确列出这些更改。引入了以下破坏性更改：

Hooks
- 从 DbApiHook 中移除 _make_serializable 方法。请改用 _make_common_data_structure。

特性¶

Added output_processor parameter to SQLQueryOperator and fixed bug with return_single_query_results handler when None is passed as split_statements (#44781)
Moved common SQL handler methods of common-sql-provider into dedicated module (#43747)

杂项¶

Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)
Fix static checks in common SQL hooks (#44930)
Added documentation for the SqlThresholdCheckOperator (#44787)
Remove Provider Deprecations in Common SQL (#44645)
Update DAG example links in multiple providers documents (#44034)

1.20.0¶

特性¶

Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook (#41916)

1.19.0¶

特性¶

Feature: Added fast_executemany parameter to insert_rows of DbApiHook (#43357)
Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator (#42891)
Allow capture of replication lag in SqlSensor (#43107)

杂项¶

fix mypy failure (#43206)

1.18.0¶

特性¶

feat(providers/common/sql): add warning to connection setter (#42736)

Bug 修复¶

FIX: Only pass connection to sqlalchemy engine in JdbcHook (#42705)

1.17.1¶

Bug 修复¶

fix(providers/common/sql): add dummy connection setter for backward compatibility (#42490)
Changed type hinting for handler function (#42275)

1.17.0¶

特性¶

注意

DB Hook 中的连接现在被缓存，以避免解析 extras 中的属性时多次查找。

Generalize caching of connection in DbApiHook to improve performance (#40751)

杂项¶

feat: log client db messages for provider postgres (#40171)
remove deprecated soft_fail from providers (#41710)

1.16.0¶

注意

此版本的 provider 仅适用于 Airflow 2.8+，如 Apache Airflow provider 支持策略中所述。

Bug 修复¶

fix: rm deprecated import (#41461)

杂项¶

Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)

1.15.0¶

特性¶

Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669)

1.14.2¶

Bug 修复¶

FIX: DbApiHook.insert_rows unnecessarily restarting connections (#40615)

杂项¶

Enable enforcing pydocstyle rule D213 in ruff. (#40448)

1.14.1¶

杂项¶

implement per-provider tests with lowest-direct dependency resolution (#39946)
Update pandas minimum requirement for Python 3.12 (#40272)
standardizes template fields for 'BaseSQLOperator' and adds 'database' as a templated field (#39826)

1.14.0¶

特性¶

Add 'parameters' as template field for SqlSensor (#39588)

Bug 修复¶

DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is formatted correctly (#39690)

杂项¶

refactor: The executemany parameter of insert_rows should not be deprecated as for some hooks we don't want to enable a system-wide supports_executemany parameter, that way we can also keep using it in dedicated situations (#39630)
Faster 'airflow_version' imports (#39552)
Simplify 'airflow_version' imports (#39497)
Add typing for SqlSensor (#39773)

1.13.0¶

注意

此版本的 provider 仅适用于 Airflow 2.7+，如 Apache Airflow provider 支持策略中所述。

特性¶

Add 'sqlalchemy_url' property to 'DbApiHook' class (#38871)
Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715)

Bug 修复¶

Fix 'DbApiHook.insert_rows' when 'rows' is a generator (#38972)
Fix 'update-common-sql-api-stubs' pre-commit check (#38915)

杂项¶

Bump minimum Airflow version in providers to Airflow 2.7.0 (#39240)
openlineage, snowflake: do not run external queries for Snowflake (#39113)

1.12.0¶

特性¶

Add hook_params to template_fields for BaseSQLOperator-related Operators (#38724)
Make 'placeholder' of DbApiHook configurable in UI (#38528)

杂项¶

Undeprecating 'DBApiHookForTests._make_common_data_structure' (#38573)

1.11.1¶

Bug 修复¶

Make 'executemany' keyword arguments only in 'DbApiHook.insert_rows' (#37840)
Limit 'pandas' to '<2.2' (#37748)

1.11.0¶

特性¶

Enhancement: Performance enhancement for insert_rows method DbApiHook with fast executemany + SAP Hana support (#37246)

Bug 修复¶

Fix SQLThresholdCheckOperator error on falsey vals (#37150)

杂项¶

feat: Switch all class, functions, methods deprecations to decorators (#36876)
Add more-itertools as dependency of common-sql (#37359)

1.10.1¶

杂项¶

Set min pandas dependency to 1.2.5 for all providers and airflow (#36698)

1.10.0¶

Make "placeholder" of ODBC configurable in UI (#36000)

Bug 修复¶

Return common data structure in DBApi derived classes
SQLCheckOperator fails if returns dict with any False values (#36273)

1.9.0¶

注意

此版本的 provider 仅适用于 Airflow 2.6+，如 Apache Airflow provider 支持策略中所述。

杂项¶

Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)
Add Architecture Decision Record for common.sql introduction (#36015)

1.8.1¶

杂项¶

Add '_make_serializable' method which other SQL operators can overrides when result from cursor is not JSON-serializable (#32319)
Remove backcompat inheritance for DbApiHook (#35754)

1.8.0¶

注意

此版本的 provider 仅适用于 Airflow 2.5+，如 Apache Airflow provider 支持策略中所述。

Bug 修复¶

fix(providers/sql): respect soft_fail argument when exception is raised (#34199)

杂项¶

Bump min airflow version of providers (#34728)
Use 'airflow.exceptions.AirflowException' in providers (#34511)

1.7.2¶

Bug 修复¶

Fix BigQueryValueCheckOperator deferrable mode optimisation (#34018)

杂项¶

Refactor regex in providers (#33898)

1.7.1¶

杂项¶

Refactor: Better percentage formatting (#33595)
Refactor: Simplify code in smaller providers (#33234)

1.7.0¶

特性¶

Add a new parameter to SQL operators to specify conn id field (#30784)

1.6.2¶

杂项¶

Make SQLExecute Query signature consistent with other SQL operators (#32974)
Get rid of Python2 numeric relics (#33050)

1.6.1¶

Bug 修复¶

Fix local OpenLineage import in 'SQLExecuteQueryOperator'. (#32400)

杂项¶

Add default port to Openlineage authority method. (#32828)
Add more accurate typing for DbApiHook.run method (#31846)

1.6.0¶

特性¶

openlineage, common.sql: provide OL SQL parser as internal OpenLineage provider API (#31398)

杂项¶

Pass SQLAlchemy engine to construct information schema query. (#32371)

1.5.2¶

杂项¶

Remove Python 3.7 support (#30963)

1.5.1¶

杂项¶

Bring back min-airflow-version for preinstalled providers (#31469)

1.5.0 (YANKED)¶

警告

此版本已被 撤回 (yanked)，原因：此版本可能导致旧版 airflow 的无限制安装，从而引发运行时错误。

注意

此版本的 provider 仅适用于 Airflow 2.4+，如 Apache Airflow provider 支持策略中所述。

特性¶

Add conditional output processing in SQL operators (#31136)

杂项¶

Remove noisy log from SQL table check (#31037)

1.4.0¶

特性¶

添加选项以在日志中显示 'SQLExecuteQueryOperator' 的输出 (#29954)

杂项¶

修复 Common SQL provider 的 Python API 文档格式 (#29863)

1.3.4¶

错误修复¶

当 do_xcom_push=False 时不处理输出 (#29599)

1.3.3¶

错误修复¶

处理 Exasol cursor 的非兼容行为 (#28744)

1.3.2¶

错误修复¶

修复最近 isort 版本引入的 isort 问题 (#28434)
修复 Common SQL operators 的模板渲染 (#28202)
在 SQLExecuteQueryOperator 中遵从 hook setting 处理 split_statements (#28635)

杂项¶

澄清更新的 DbApiHook 的 docstrings (#27966)

1.3.1¶

此版本修复了重构通用部分时在 common.sql operator 中引入的一些错误

SQLExecuteQueryOperator 中的 _process_output 方法现在具有一致的语义和类型，它还可以修改从 SQLExecuteQueryOperator 派生的 operators 中的返回值（并存储在 XCom 中）。
所有返回结果的描述都作为 DBApiHook 中的 descriptions 属性存储
游标的最后描述（无论是否返回单查询结果值）现在通过 DBApiHook 中的 last_description 属性公开。

操作员缺乏一致性导致 1.3.0 被撤下（yanked）—— 不应使用 1.3.0 —— 如果您安装了 1.3.0，请升级到 1.3.1。

错误修复¶

恢复在 common.sql 中已删除（但仍在使用）的方法 (#27843)
修复重构时在 Databricks SQL operator 中引入的错误 (#27854)
将 common.sql provider 版本提升至 1.3.1 (#27888)
最终修复 SQL Hooks 和 Operators 的行为 (#27912)

1.3.0 (已撤下)¶

警告

此版本已被 撤下 (yanked)，原因是：导致 Google 8.4.0 provider 中的 SQLExecute 无法工作

注意

此 provider 版本仅适用于 Airflow 2.3+，如 Apache Airflow provider 支持策略中所述。

杂项¶

将所有 provider 的最低 Airflow 版本移至 2.3.0 (#27196)

特性¶

添加 SQLExecuteQueryOperator (#25717)
DbApiHook.get_records 和 DbApiHook.get_first 使用 DbApiHook.run (#26944)
DbApiHook 一致的 insert_rows 日志记录 (#26758)

错误修复¶

Common sql bugfixes 和改进 (#26761)
在 SQLCheckOperator.execute 中使用未使用的 SQLCheckOperator.parameters (#27599)

1.2.0¶

特性¶

使 placeholder 样式可配置 (#25939)
为 pre-common-sql providers 提供更好的错误消息 (#26051)

错误修复¶

修复（并测试）postgresql 上的 SQLTableCheckOperator (#25821)
SQLTableCheckOperator 不使用 Pandas (#25822)
在 SQL hook 中放弃分号去除 (#25855)

1.1.0¶

特性¶

使用 ParamSpec 改进 taskflow 类型提示 (#25173)
将所有“旧”SQL operators 移至 common.sql providers (#25350)
废弃 hql parameters 并同步 DBApiHook 方法 APIs (#25299)
将 DbApiHook.run() 方法与覆盖它的方法统一 (#23971)
Common SQLCheckOperators 多项功能更新 (#25164)

错误修复¶

允许 Legacy SqlSensor 使用 common.sql providers (#25293)
修复 fetch_all_handler 和 db-api 测试 (#25430)
对齐 Common SQL provider 标志位置 (#25538)
修复 SQL split string 以包含 ';-less' 语句 (#25713)

1.0.0¶

provider 的初始版本。添加 SQLColumnCheckOperator 和 SQLTableCheckOperator。将 DBApiHook、SQLSensor 和 ConnectorProtocol 移至 provider。