The Parquet Decoder agent receives Parquet data from file collectors in bytearray format, converts the data into ParquetDecoderUDRs, using one UDR per record, and routes those UDRs forward into the workflow.
Field | Description |
|---|---|
payload | Fields passed for the row as an APL map (map: string -> any). The keys for the map are column names from the Parquet document, while values are dependent on the data type. The structure of the payload is specified by the corresponding schema field described below. |
| schema | APL representation of the Parquet schema for the message. Encoded as an APL map that maps the top-level field names to ParquetType objects that describes the field (for example, type, required/optional). The structure of the payload field (described above) conforms to this schema. |
schemaname | Name of top-level schema for the encapsulated message. |
Add Comment