Page Comparison

The Schema tab is the primary configuration in the Parquet Profile. This tab allows the author to specify a Parquet Schema. This schema will be used for different purposes by the Parquet Encoder and Paruet Decoder agents.

• Parquet Encoder Agent - The Parquet Encoder agent will generate a Parquet document that conforms to the specified schema. Not only will the data conform to the schema, but the schema itself is included in the Parquet document.
  
  
• Parquet Decoder Agent - When the Parquet Decoder processes a Parquet document, only the columns included in the specified schema will be included. For example:
  • Consider a document with columns A, B, C, and D.
  • Assume that the schema in the Parquet Profile only specifies columns A and D.
  • The generated ParquetDecoderUDRs will include only fields A and D in the payload map.

Note that the Parquet Profile (and hence the schema) are required for Parquet Encoder agents and optional for Parquet Decoders.