Google¶

Airflow 支持 Google 服务。

所有钩子都基于 airflow.providers.google.common.hooks.base_google.GoogleBaseHook。一些集成也使用 airflow.providers.google.common.hooks.discovery_api。

请参阅 Google Cloud 连接类型文档来配置与 Google 服务的连接。

Google Cloud¶

Airflow 对 Google Cloud 提供了广泛的支持。

注意

你可以通过分析 Google Cloud 示例 DAG 的源代码来学习如何使用 Google Cloud 集成。

服务¶

这些集成允许你在 Google Cloud 中执行各种操作。

传输¶

这些集成允许您从/向 Google Cloud 复制数据。

Google Cloud Storage (GCS) 到 Amazon Simple Storage Service (S3)¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Amazon Simple Storage Service (S3)
运算符指南: Google Cloud Storage 到 Amazon S3 传输运算符
Python API: airflow.providers.amazon.aws.transfers.gcs_to_s3
提供者: apache-airflow-providers-amazon

Amazon Glacier 到 Google Cloud Storage (GCS)¶

源产品文档: Amazon Glacier
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Amazon S3 Glacier 到 GCS
Python API: airflow.providers.amazon.aws.transfers.glacier_to_gcs
提供者: apache-airflow-providers-amazon

Presto 到 Google Cloud Storage (GCS)¶

源产品文档: Presto
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Presto 到 Google Cloud Storage 传输运算符
Python API: airflow.providers.google.cloud.transfers.presto_to_gcs
提供者: apache-airflow-providers-google

Trino 到 Google Cloud Storage (GCS)¶

源产品文档: Trino
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Trino 到 Google Cloud Storage 传输运算符
Python API: airflow.providers.google.cloud.transfers.trino_to_gcs
提供者: apache-airflow-providers-google

通用 SQL 到 Google Cloud Storage (GCS)¶

源产品文档: 通用 SQL
目标产品文档: Google Cloud Storage (GCS)
Python API: airflow.providers.google.cloud.transfers.sql_to_gcs
提供者: apache-airflow-providers-google

Google Cloud Storage (GCS) 到 Google Drive¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Google Drive
运算符指南: Google Cloud Storage 到 Google Drive 传输运算符
Python API: airflow.providers.google.suite.transfers.gcs_to_gdrive
提供者: apache-airflow-providers-google

Google Drive 到 Google Cloud Storage (GCS)¶

源产品文档: Google Drive
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Google Drive 到 Google Cloud Storage 传输运算符
Python API: airflow.providers.google.cloud.transfers.gdrive_to_gcs
提供者: apache-airflow-providers-google

Microsoft Azure FileShare 到 Google Cloud Storage (GCS)¶

源产品文档: Microsoft Azure FileShare
目标产品文档: Google Cloud Storage (GCS)
运算符指南: 将数据从 Azure FileShare 存储传输到 Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.azure_fileshare_to_gcs
提供者: apache-airflow-providers-google

Apache Cassandra 到 Google Cloud Storage (GCS)¶

源产品文档: Apache Cassandra
目标产品文档: Google Cloud Storage (GCS)
Python API: airflow.providers.google.cloud.transfers.cassandra_to_gcs
提供者: apache-airflow-providers-google

Google Calendar 到 Google Cloud Storage (GCS)¶

源产品文档: Google Calendar
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Google Calendar 到 Google Cloud Storage 传输运算符
Python API: airflow.providers.google.cloud.transfers.calendar_to_gcs
提供者: apache-airflow-providers-google

Google 表格到 Google Cloud Storage (GCS)¶

源产品文档: Google 表格
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Google Sheets 到 Google Cloud Storage 传输操作符
Python API: airflow.providers.google.cloud.transfers.sheets_to_gcs
提供者: apache-airflow-providers-google

Amazon Simple Storage Service (S3) 到 Google Cloud Storage (GCS)¶

源产品文档: Amazon Simple Storage Service (S3)
目标产品文档: Google Cloud Storage (GCS)
运算符指南: 从 Amazon S3 传输数据到 Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.s3_to_gcs
提供者: apache-airflow-providers-google

Google Cloud Storage (GCS) 到 SSH 文件传输协议 (SFTP)¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: SSH 文件传输协议 (SFTP)
运算符指南: Google Cloud Storage 传输操作符到 SFTP
Python API: airflow.providers.google.cloud.transfers.gcs_to_sftp
提供者: apache-airflow-providers-google

PostgreSQL 到 Google Cloud Storage (GCS)¶

源产品文档: PostgreSQL
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Postgres 到 Google Cloud Storage 操作符
Python API: airflow.providers.google.cloud.transfers.postgres_to_gcs
提供者: apache-airflow-providers-google

Google BigQuery 到通用 SQL¶

源产品文档: Google BigQuery
目标产品文档: 通用 SQL
Python API: airflow.providers.google.cloud.transfers.bigquery_to_sql
提供者: apache-airflow-providers-google

Google BigQuery 到 MySQL¶

源产品文档: Google BigQuery
目标产品文档: MySQL
运算符指南: Google Cloud BigQuery 传输操作符到 MySQL
Python API: airflow.providers.google.cloud.transfers.bigquery_to_mysql
提供者: apache-airflow-providers-google

Google BigQuery 到 Microsoft SQL Server (MSSQL)¶

源产品文档: Google BigQuery
目标产品文档: Microsoft SQL Server (MSSQL)
运算符指南: Google Cloud BigQuery 传输操作符到 Microsoft SQL Server
Python API: airflow.providers.google.cloud.transfers.bigquery_to_mssql
提供者: apache-airflow-providers-google

Google BigQuery 到 PostgreSQL¶

源产品文档: Google BigQuery
目标产品文档: PostgreSQL
运算符指南: Google Cloud BigQuery 传输操作符到 Postgres
Python API: airflow.providers.google.cloud.transfers.bigquery_to_postgres
提供者: apache-airflow-providers-google

Google Cloud Storage (GCS) 到 Google BigQuery¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Google BigQuery
运算符指南: Google Cloud Storage 传输操作符到 BigQuery
Python API: airflow.providers.google.cloud.transfers.gcs_to_bigquery
提供者: apache-airflow-providers-google

Google Cloud Storage (GCS) 到 Google Cloud Storage (GCS)¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Google Cloud Storage (GCS)
运算符指南: 在 Google Cloud Storage 中传输数据
Python API: airflow.providers.google.cloud.transfers.gcs_to_gcs
提供者: apache-airflow-providers-google

Facebook 广告到 Google Cloud Storage (GCS)¶

源产品文档: Facebook 广告
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Facebook 广告到 GCS 操作符
Python API: airflow.providers.google.cloud.transfers.facebook_ads_to_gcs
提供者: apache-airflow-providers-google

SSH 文件传输协议 (SFTP) 到 Google Cloud Storage (GCS)¶

源产品文档: SSH 文件传输协议 (SFTP)
目标产品文档: Google Cloud Storage (GCS)
运算符指南: SFTP 到 Google Cloud Storage 传输操作符
Python API: airflow.providers.google.cloud.transfers.sftp_to_gcs
提供者: apache-airflow-providers-google

Microsoft Azure Data Lake Storage 到 Google Cloud Storage (GCS)¶

源产品文档: Microsoft Azure Data Lake Storage
目标产品文档: Google Cloud Storage (GCS)
Python API: airflow.providers.google.cloud.transfers.adls_to_gcs
提供者: apache-airflow-providers-google

Google BigQuery 到 Google BigQuery¶

源产品文档: Google BigQuery
目标产品文档: Google BigQuery
运算符指南: Google Cloud BigQuery 传输操作符到 BigQuery
Python API: airflow.providers.google.cloud.transfers.bigquery_to_bigquery
提供者: apache-airflow-providers-google

MySQL 到 Google Cloud Storage (GCS)¶

源产品文档: MySQL
目标产品文档: Google Cloud Storage (GCS)
运算符指南: MySQL 到 Google Cloud Storage 操作符
Python API: airflow.providers.google.cloud.transfers.mysql_to_gcs
提供者: apache-airflow-providers-google

Oracle 到 Google Cloud Storage (GCS)¶

源产品文档: Oracle
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Oracle 到 Google Cloud Storage 操作符
Python API: airflow.providers.google.cloud.transfers.oracle_to_gcs
提供者: apache-airflow-providers-google

Google Cloud Storage (GCS) 到 Google 表格¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Google 表格
运算符指南: Google Cloud Storage 到 Google Sheets 传输操作符
Python API: airflow.providers.google.suite.transfers.gcs_to_sheets
提供者: apache-airflow-providers-google

本地到 Google Cloud Storage (GCS)¶

目标产品文档: Google Cloud Storage (GCS)
运算符指南: 从本地文件系统上传数据到 Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.local_to_gcs
提供者: apache-airflow-providers-google

Google BigQuery 到 Google Cloud Storage (GCS)¶

源产品文档: Google BigQuery
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Google Cloud BigQuery 传输操作符到 Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.bigquery_to_gcs
提供者: apache-airflow-providers-google

Google Cloud Storage (GCS) 到本地¶

源产品文档: Google Cloud Storage (GCS)
运算符指南: 从 Google Cloud Storage 下载数据到本地文件系统
Python API: airflow.providers.google.cloud.transfers.gcs_to_local
提供者: apache-airflow-providers-google

Salesforce 到 Google Cloud Storage (GCS)¶

源产品文档: Salesforce
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Salesforce 到 GCS 操作符
Python API: airflow.providers.google.cloud.transfers.salesforce_to_gcs
提供者: apache-airflow-providers-google

Google 广告到 Google Cloud Storage (GCS)¶

源产品文档: Google 广告
目标产品文档: Google Cloud Storage (GCS)
Python API: airflow.providers.google.ads.transfers.ads_to_gcs
提供者: apache-airflow-providers-google

Microsoft SQL Server (MSSQL) 到 Google Cloud Storage (GCS)¶

源产品文档: Microsoft SQL Server (MSSQL)
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Microsoft SQL Server 到 Google Cloud Storage 操作符
Python API: airflow.providers.google.cloud.transfers.mssql_to_gcs
提供者: apache-airflow-providers-google

Microsoft Azure Blob Storage 到 Google Cloud Storage (GCS)¶

源产品文档: Microsoft Azure Blob Storage
目标产品文档: Google Cloud Storage (GCS)
运算符指南: Azure Blob Storage 到 Google Cloud Storage (GCS) 传输操作符
Python API: airflow.providers.google.cloud.transfers.azure_blob_to_gcs
提供者: apache-airflow-providers-google

Google Cloud Storage (GCS) 到 Presto¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Presto
运算符指南: Google Cloud Storage 到 Presto 传输操作符
Python API: airflow.providers.presto.transfers.gcs_to_presto
提供者: apache-airflow-providers-presto

Google Cloud Storage (GCS) 到 Samba¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Samba
运算符指南: Google Cloud Storage 传输操作符到 Samba
Python API: airflow.providers.samba.transfers.gcs_to_samba
提供者: apache-airflow-providers-samba

Google Cloud Storage (GCS) 到 Snowflake¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Snowflake
Python API: airflow.providers.snowflake.transfers.copy_into_snowflake
提供者: apache-airflow-providers-snowflake

Google Cloud Storage (GCS) 到 Trino¶

源产品文档: Google Cloud Storage (GCS)
目标产品文档: Trino
运算符指南: Google Cloud Storage 到 Trino 传输操作符
Python API: airflow.providers.trino.transfers.gcs_to_trino
提供者: apache-airflow-providers-trino

Google Marketing Platform¶

注意

您可以通过分析示例 DAG 的源代码来了解如何使用 Google Marketing Platform 集成。

Google 广告¶

操作符: airflow.providers.google.ads.operators.ads.
钩子: airflow.providers.google.ads.hooks.ads.
指南: Google 广告操作符.
提供者: apache-airflow-providers-google
产品文档: Google 广告

Google Analytics (GA4)¶

操作符: airflow.providers.google.marketing_platform.operators.analytics_admin.
钩子: airflow.providers.google.marketing_platform.hooks.analytics_admin.
指南: Google Analytics (GA4) 管理操作符.
提供者: apache-airflow-providers-google
产品文档: Google Analytics (GA4)

Google Display&Video 360¶

操作符: airflow.providers.google.marketing_platform.operators.display_video.
钩子: airflow.providers.google.marketing_platform.hooks.display_video.
传感器: airflow.providers.google.marketing_platform.sensors.display_video.
指南: Google Display & Video 360 操作符.
提供者: apache-airflow-providers-google
产品文档: Google Display&Video 360

Google Search Ads 360¶

操作符: airflow.providers.google.marketing_platform.operators.search_ads.
钩子: airflow.providers.google.marketing_platform.hooks.search_ads.
指南: Google Search Ads 操作符.
提供者: apache-airflow-providers-google
产品文档: Google Search Ads 360

其他 Google 操作符和钩子¶

Google¶

钩子: airflow.providers.google.common.hooks.base_google.
提供者: apache-airflow-providers-google
产品文档: Google

Google 日历¶

钩子: airflow.providers.google.suite.hooks.calendar.
提供者: apache-airflow-providers-google
产品文档: Google Calendar

Google Cloud Batch¶

操作符: airflow.providers.google.cloud.operators.cloud_batch.
钩子: airflow.providers.google.cloud.hooks.cloud_batch.
指南: Google Cloud Batch 操作符.
提供者: apache-airflow-providers-google
产品文档: Google Cloud Batch

Google Cloud Composer¶

操作符: airflow.providers.google.cloud.operators.cloud_composer.
钩子: airflow.providers.google.cloud.hooks.cloud_composer.
传感器: airflow.providers.google.cloud.sensors.cloud_composer.
指南: Google Cloud Composer 操作符.
提供者: apache-airflow-providers-google
产品文档: Google Cloud Composer

Google Cloud Dataform¶

操作符: airflow.providers.google.cloud.operators.dataform.
钩子: airflow.providers.google.cloud.hooks.dataform.
传感器: airflow.providers.google.cloud.sensors.dataform.
指南: Google Dataform 操作符.
提供者: apache-airflow-providers-google
产品文档: Google Cloud Dataform

Google Cloud Run¶

操作符: airflow.providers.google.cloud.operators.cloud_run.
钩子: airflow.providers.google.cloud.hooks.cloud_run.
指南: Google Cloud Run 操作符.
提供者: apache-airflow-providers-google
产品文档: Google Cloud Run

Google Discovery API¶

钩子: airflow.providers.google.common.hooks.discovery_api.
提供者: apache-airflow-providers-google
产品文档: Google Discovery API

Google Drive¶

钩子: airflow.providers.google.suite.hooks.drive.
传感器: airflow.providers.google.suite.sensors.drive.
提供者: apache-airflow-providers-google
产品文档: Google Drive

Google LevelDB¶

操作符: airflow.providers.google.leveldb.operators.leveldb.
钩子: airflow.providers.google.leveldb.hooks.leveldb.
指南: Google LevelDB 操作符.
提供者: apache-airflow-providers-google
产品文档: Google LevelDB

Google 表格¶

操作符: airflow.providers.google.suite.operators.sheets.
钩子: airflow.providers.google.suite.hooks.sheets.
指南: Google Sheets 操作符.
提供者: apache-airflow-providers-google
产品文档: Google 表格

Google¶

Google Cloud¶

服务¶

Google AutoML¶

Google BigQuery¶

Google BigQuery Data Transfer Service¶

Google Bigtable¶

Google Campaign Manager¶

Google Cloud AlloyDB¶

Google Cloud Build¶

Google Cloud Common¶

Google Cloud Data Loss Prevention (DLP)¶

Google Cloud Firestore¶

Google Cloud Functions¶

Google Cloud Key Management Service (KMS)¶

Google Cloud Life Sciences¶

Google Cloud Memorystore¶

Google Cloud Natural Language¶

Google Cloud OS Login¶

Google Cloud Pub/Sub¶

Google Cloud Secret Manager¶

Google Cloud Spanner¶

Google Cloud Speech-to-Text¶

Google Cloud SQL¶

Google Cloud Stackdriver¶

Google Cloud Storage (GCS)¶

Google Cloud Storage Transfer Service¶

Google Cloud Tasks¶

Google Cloud Text-to-Speech¶

Google Cloud Translation¶

Google Cloud Video Intelligence¶

Google Cloud Vision¶

Google Cloud Workflows¶

Google Compute Engine¶

Google Data Catalog¶

Google Data Fusion¶

Google Dataflow¶

Google Dataplex¶

Google Dataprep¶

Google Dataproc¶

Google Dataproc Metastore¶

Google Datastore¶

Google Deployment Manager¶

Google Kubernetes Engine¶

Google Looker¶

Google Machine Learning Engine¶

Google Vertex AI¶

传输¶

Google Cloud Storage (GCS) 到 Amazon Simple Storage Service (S3)¶

Amazon Glacier 到 Google Cloud Storage (GCS)¶

Presto 到 Google Cloud Storage (GCS)¶

Trino 到 Google Cloud Storage (GCS)¶

通用 SQL 到 Google Cloud Storage (GCS)¶

Google Cloud Storage (GCS) 到 Google Drive¶

Google Drive 到 Google Cloud Storage (GCS)¶

Microsoft Azure FileShare 到 Google Cloud Storage (GCS)¶

Apache Cassandra 到 Google Cloud Storage (GCS)¶

Google Calendar 到 Google Cloud Storage (GCS)¶

Google 表格到 Google Cloud Storage (GCS)¶

Amazon Simple Storage Service (S3) 到 Google Cloud Storage (GCS)¶

Google Cloud Storage (GCS) 到 SSH 文件传输协议 (SFTP)¶

PostgreSQL 到 Google Cloud Storage (GCS)¶

Google BigQuery 到通用 SQL¶

Google BigQuery 到 MySQL¶

Google BigQuery 到 Microsoft SQL Server (MSSQL)¶

Google BigQuery 到 PostgreSQL¶

Google Cloud Storage (GCS) 到 Google BigQuery¶

Google Cloud Storage (GCS) 到 Google Cloud Storage (GCS)¶

Facebook 广告到 Google Cloud Storage (GCS)¶

SSH 文件传输协议 (SFTP) 到 Google Cloud Storage (GCS)¶

Microsoft Azure Data Lake Storage 到 Google Cloud Storage (GCS)¶

Google BigQuery 到 Google BigQuery¶

MySQL 到 Google Cloud Storage (GCS)¶

Oracle 到 Google Cloud Storage (GCS)¶

Google Cloud Storage (GCS) 到 Google 表格¶

本地到 Google Cloud Storage (GCS)¶

Google BigQuery 到 Google Cloud Storage (GCS)¶

Google Cloud Storage (GCS) 到本地¶

Salesforce 到 Google Cloud Storage (GCS)¶

Google 广告到 Google Cloud Storage (GCS)¶