GCP Profile

Click Browse to select a predefined GCP profile. The profile contains the credential information required to connect to BigQuery in the GCP Project.

UDR Type

Type of UDR to populate the target BigQuery table.

Project ID

Specify the reference ID of the project. This is a mandatory field as the workflow will be invalid if you try to save the configuration without specifying the Project ID.

Dataset

Select the target dataset.

Data Table

Select the target table name. Only the tables found within the selected dataset can be listed and chosen.

Batch Status Table

A separate table to keep track of the transaction ID to allow the agent to better provide transaction safety in the event that the workflow aborts. A transaction ID is unique to each batch process, the agent will use the transaction ID to identify UDRs that belong to the batch when the data insertion into the table was interrupted.

See the example below on creating a Batch Status table:

Transaction ID Column

Select the column that is designated for the transaction ID from the Batch Status Table and Data Table. The transaction ID has to be present in both Data and Batch Status table with the same name, for the transaction ID to be available for selection.

Batch Size

The amount of UDR to be inserted into the target data table for each batch process. This corresponds to the rows per insert request into the target Data table. The default value is 500 records per batch transaction.

Concurrent Requests

The amount of connections the agent will open to the target data table to insert the UDRs. The default value is 1 connection.

Refresh

Updates the table with all the columns from the selected table.

If rows already exist in the table, the refresh operation preserves the configuration for all rows with a corresponding column. Thus, if a table has been extended with a new column, the old column configurations are left untouched and the new column appears when  Refresh is selected.

Field Name

Displays a list of all columns for the selected Data table, except the Transaction ID column.

Field Type

Displays the BigQuery data type for each column as declared in the BigQuery table.

UDR Field

Allows you to map the Field Name of the Data table with the UDR Field from the UDR Type configured from the Target tab. Only valid UDR Field types will be available for selection in the Column Assignment dialog based on the Field Type of the BigQuery Column. You can refer to the BigQuery to valid UDR data types mappings in the table below.

BIGNUMERIC

byte, int, short, long, float, double, bigint, bigdec

BOOLEAN

boolean

DATE

date

DATETIME

date

TIME

date

TIMESTAMP

date

FLOAT64

byte, int, short, long, float, double

NUMERIC

byte, int, short, long, float, double

GEOGRAPHY

string

STRING

string

INT64

byte, int, short, long