This chapter describes the Avro support that has been added to the Ultra Format Definition Language (UFDL). This addition enables you to compile Avro definitions, and to encode data into, and decode it from the Avro format. A schema in Avro is represented in JSON. You can implement non-protocol data issues in an Analysis agent.
Overview
In you manage Avro parsing in UFDL by applying the avro
_block
construct.
avro_block { <avro json schema> };
The output of an Avro encoder represents a single Avro record, not a complete Avro data file. To deserialize these records using a third party tool, you must add the correct Avro header to the records based on the description of Avro provided in the link below.
The full description of the Avro language can be found at: https://avro.apache.org/docs/current/spec.html.
The Avro Types
The Avro JSON schema can be defined with any of the following types:
Note!
There are limitations to the implementation of the following elements. Refer to the the section below, Limitations, for further information.
Primitive Type | Notes |
---|---|
null | Implemented as type byte |
boolean | Value must be true or false . |
int | 32-bit signed |
long | 64-bit signed |
float | Single precision (32-bit) IEEE 754 floating-point number |
double | Double precision (64-bit) IEEE 754 floating-point number |
bytes | Sequence of 8-bit unsigned bytes |
string | Unicode character sequence |
Complex Type | Notes |
---|---|
record | Supports the attributes:
|
enum | Supports the attributes: name , namespace, doc (optional), aliases (optional), symbols . |
array | Supports the attribute: items |
map | Supports the attribute: values |
union | Implemented as a record with all types set to optional. If two values are set, both are encoded. If no values are set, none are encoded. To specify a union value, you must use |
fixed | Supports the attributes: |
Note!
Use the doc
attribute to add a comment.
Limitations
The following limitations apply for the Avro support in:
- Line and Column are not fully implemented.
- No string values are permitted in APL for
enum
type. - Encoding the Avro object container file is not supported.
aliases
is not fully supported.default
is not fully supported.- Cross references to other Ultra definitions are not supported.
array of arrays
is not correctly implemented. E garray of array of x
is implemented witharray of y
, wherey
is a record with onevalue
field. Thisvalue
field is anarray of x
.map of x
is implemented via anarray of y
, wherey
is a record with two fields:key
andvalue
, wherevalue
isx
.
An Avro Format Example
To encode an Arvo data file, a format definition is included in the Ultra avro
_block
block in the Ultra format.
Example - The xml_schema Ultra Code Block
avro_block { { "namespace": "example.avro", "type": "record", "name": "User3", "fields": [ {"name": "name", "type": { "name": "FullName2", "type": "record", "fields": [ {"name": "firstName", "type": "string"}, {"name": "lastName", "type": "string"} ] } }, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]}, {"name": "favorite_fotball_team", "type": { "name": "teams", "type": "enum", "namespace": "example.avro.teams", "symbols": ["Djurgården", "Hammarby", "Malmö", "Göteborg"] } }, {"name": "ipAddresses", "type": { "type": "array", "items": [ { "name": "ipv4Address", "type": "fixed", "size": 4 }, { "name": "ipv6Address", "type": "fixed", "size": 16 } ] } }, {"name": "favoriteFoodList", "type": { "name": "favoriteFood", "type": "record", "fields": [ {"name": "dish", "type": "string"}, {"name": "next", "type": ["null", "favoriteFood"]} ] } }, {"name": "salary", "type": "long"}, {"name": "myFixed", "type": { "name": "myfixed", "type":"fixed", "size": 4 }}, {"name": "myFloat", "type": "float"}, {"name": "myDouble", "type": "double"}, {"name": "rootUsers", "type": { "type": "map", "values": { "name": "RootUsers", "type": "record", "fields": [ {"name": "rootUser", "type": "string"}, {"name": "privileges", "type": "int"} ] } } }, {"name": "maps", "type": { "type": "array", "items": { "type": "map", "values": { "name": "Map2", "type": "record", "fields": [ {"name": "favoriteUser", "type": "string"}, {"name": "favoriteNumber", "type": "int"} ] } } } } ] } };