cancel
Showing results for 
Search instead for 
Did you mean: 
JoeM
Community Manager
Community Manager

Introduction

As a part of Incorta's open data delivery focus, ensuring that your data is not subject to 'vendor lock-in' is a core part of the platform. This is evident with capabilities like data destinations, where you can not only ingest Incorta but also push it to other applications, 3rd party BI tools, and cloud platforms like Google BigQuery.

What you need to know before reading this article

The data destination of Google BigQuery was added in 2023.7. Any earlier versions of Incorta will not have the destination available.

JoeM_0-1692026849730.png

Let's Go

The process of pushing to BigQuery will follow the below steps:

  1. Configuring Incorta to write to Google BigQuery
  2. Setting up the data destination
  3. Pushing data to BigQuery

Configuring Incorta to write to Google BigQuery

First, we must ensure that Google BigQuery will allow Incorta to write to the platform. 

  1. In Incorta, go to the 'Data' Tab and select '+New.'
  2. Select 'Add a data destination,' then select 'BigQuery.'
  3. In the destination setup, there will be a service account. Copy this service account.JoeM_0-1692024295797.png
  4. Go to https://console.cloud.google.com/, and ensure the correct project is selected.
  5. Go to IAM & Admin and select 'Grant Access.' Note that you must have proper project permissions to assign roles to other users. If you do not see this service, contact your cloud admin to request the below steps.
  6. Grant the roles of 'BigQuery Data Editor' and 'BigQuery Job User' to the Incorta service principal.

JoeM_1-1692024540273.png

Setting up the data destination

  1. Go to the BigQuery service and copy the project name to write to:JoeM_1-1692024780872.png

     

    Spoiler
    Sometimes the project name and the BigQuery project name in the explorer differ. Use the project name in the explorer. 
  2. Return to Incorta and enter the project name into the data destination.JoeM_2-1692024932163.png
  3. Test and save the destination.

Pushing Data to BigQuery

It's time to assign what schemas will be pushed to Google BigQuery. 

  1. Go to the 'Schema' and open a schema to push to BigQuery.
  2. In the schema settings (cog), select 'Set a Data Destination.'
  3. Select the destination that was just created in the data tab.
  4. Next, enter a name for a target schema name. This schema name will be written as a dataset in BigQuery. You may have multiple schemas writing into a single BigQuery dataset.
  5. Option: Add a prefix that should be applied to all tables created in BigQuery.
  6. When viewing the schemas list, you'll see a BigQuery icon denoting which schemas will push to BigQuery.
  7. Now incremental and full loads will complete incremental and full loads into BigQuery, respectively. 

       JoeM_3-1692025324416.png

Performance Tips

  • Consider making the tables in a schema non-optimized to skip the post-load phase.
  • If relevant, allocate less memory to the analytics service in CMC and give the rest to the loader service since we would not need to use Incorta dashboards.

Bonus Tips

  • Time-to-time BigQuery will have rate limiting in place. Incorta will automatically attempt to load a table 3x before failing.
  • If a table fails to load, you can only load an individual table to a destination by invoking ingestion into Incorta. 
  • You can also consider passing business schemas to BigQuery by creating a MV in schemas that queries a view. 
  • Inspect load jobs in the BigQuery console by selecting the project history.
  • If a table schema has changed between loads, a full load to BigQuery will be required. 

Related Material

 

Best Practices Index
Best Practices

Just here to browse knowledge? This might help!

Contributors
Version history
Last update:
‎12-07-2023 12:11 PM
Updated by: