GCS
This page guides you through the process of setting up the GCS source connector. This connector supports loading multiple CSV files (non compressed) from a GCS directory. The conntector will check for all files ending in .csv, even nested files.
info
Cloud storage may incur egress costs. Egress refers to data that is transferred out of the cloud storage system, such as when you download files or access them from a different location. For more information, see the Google Cloud Storage pricing guide.
Prerequisites
- JSON credentials for the service account that has access to GCS. For more details check instructions
- GCS bucket
- Path to file(s)
Set up Source
Create a Service Account
First, you need to select existing or create a new project in the Google Cloud Console:
- Sign in to the Google Account.
- Go to the Service Accounts page.
- Click
Create service account. - Create a JSON key file for the service user. The contents of this file will be provided as the
service_accountin the UI.
Grant permisison to GCS
Use the service account ID from above, grant read access to your target bucket. Click here for more details.
Set up the source in Airbyte UI
- Paste the service account JSON key to
service_account - Enter your GCS bucket name to
gcs_bucket - Enter path to your file(s) to
gcs_path
Changelog
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.3.4 | 2024-01-11 | 34158 | Fix issue in stream reader for document file type parser |
| 0.3.3 | 2023-12-06 | 33187 | Bump CDK version to hide source-defined primary key |
| 0.3.2 | 2023-11-16 | 32608 | Improve document file type parser |
| 0.3.1 | 2023-11-13 | 32357 | Improve spec schema |
| 0.3.0 | 2023-10-11 | 31212 | Migrated to file based CDK |
| 0.2.0 | 2023-06-26 | 27725 | License Update: Elv2 |
| 0.1.0 | 2023-02-16 | 23186 | New Source: GCS |