Data Sources and Sinks
Data Sources
Currently Available Sources
gcp-airflow-foundations supports ingesting data from the following sources:
Google Cloud Storage (including loading from Parquet)
SFTP
Oracle (using Dataflow)
MySQL (using Dataflow)
Salesforce
Facebook Ads
Sources in the Making
Google Ads
Snapchat
The Trade Desk
LinkedIn Marketing
TikTok
Twitter
Amazon DSP
CM360 & DV360
Spotify Ads
Pinterest
Data Sinks
gcp-airflow-foundations currently supports ingesting data only to BigQuery.
Data Transformation
gcp-airflow-foundations is an ingestion framework (EL part of ELT), but it supports triggering commoen transformation framweworks post ingestion
Transormations can be scheduled to run using post ingestion task dependecies