Parquet Profile Configuration Schema

The Schema tab is the primary configuration in the Parquet Profile. This tab allows the author to specify a Parquet Schema. This schema will be used for different purposes by the Parquet Encoder and Paruet Decoder agents.

• Parquet Encoder Agent - The Parquet Encoder agent will generate a Parquet document that conforms to the specified schema. Not only will the data conform to the schema, but the schema itself is included in the Parquet document.

• Parquet Decoder Agent - When the Parquet Decoder processes a Parquet document, only the columns included in the specified schema will be included. For example:

  • Consider a document with columns A, B, C, and D.

  • Assume that the schema in the Parquet Profile only specifies columns A and D.

  • The generated ParquetDecoderUDRs will include only fields A and D in the payload map.

Note that the Parquet Profile (and hence the schema) are required for Parquet Encoder agents and optional for Parquet Decoders.

Open

The Parquet profile's Schema tab with an example of a defined Schema. You will have to write the Schema for your desired functions.

Defining the Parquet Schema

To be able to define a Schema, it is useful to have knowledge about primitives, nested groups, repetition levels, and logical types, as described below:

Example Parquet Schema