Google

Introduction

The Google technology stack is supported in Hop through a number of plugins. We briefly touch upon them below.

Pipeline Transforms

• BigQuery Input (Beam)

• BigQuery Output (Beam)

• Bigtable Input (Beam)

• Bigtable Output (Beam)

• GCP Pub/Sub Publisher (Beam)

• GCP Pub/Sub Subscriber (Beam)

VFS

Apache VFS Support in Hop allows you to directly read from a multitude of file systems and protocols, including Google:

• Google Drive: read and write data directly from and to Google Drive files and folders.

• Google Cloud Storage: read and write data directly from and to files and folders in Google Cloud Storage buckets

Beam vs Google Cloud

When executing your pipeline using a Beam runner which is NOT DataFlow, make sure to pass the default Google cloud project ID by running:

gcloud config set project <project-id>

This affects Google Cloud specific APIs like BigQuery, Pub/Sub and others.