Versions Compared

Old Version 5

changes.mady.by.user Jenni Wallin

Saved on

compared with

New Version Current

changes.mady.by.user Jenni Wallin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This page describes the Avro support for Ultra Format Definition Language (UFDL). This functionality enables you to compile Avro definitions, to encode data into, and from the Avro format. A schema in Avro is represented in JSON. You can implement non-protocol data issues in an Analysis agent. 

Note
title

Note!

It is also possible to encode/decode Avro messages using schema retrieved from an external schema registry. See Decoder Agent Configuration or Encoder Agent for more information.

...

Avro parsing in UFDL is managed by applying the avro_block construct.

Code Block
languagetext
themeEclipse
avro_block {
<avro json schema>
 
};

The output from an Avro encoder represents a single Avro record, not a complete Avro data file. To de-serialize these records using a third-party tool, add the correct Avro header to the records based on the description of Avro provided in the link below.

...

The Avro JSON schema can be defined with any of the following types:

Note
title

Note!

There are limitations to the implementation of the following elements. Refer to the the section below, Limitations, for further information.

Primitive Type

Notes

null

Implemented as type byte.

boolean

Value must be true or false.

int

32-bit signed.

long

64-bit signed.

float

Single precision (32-bit) IEEE 754 floating-point number.

double

Double precision (64-bit) IEEE 754 floating-point number.

bytes

Sequence of 8-bit unsigned bytes.

string

Unicode character sequence.

Complex Type

Notes

record

Supports the attributes: name, namespace, doc (optional), aliases (optional), fields.

The fields attribute is an array of listing fields and supports the attributes: name, doc (optional), type, default, order (optional), aliases (optional).

enum

Supports the attributes: name, namespace, doc (optional), aliases (optional), symbols.

array

Supports the attribute: items

map

Supports the attribute: values

union

Implemented as a record with all types set to optional. If two values are set, both are encoded. If no values are set, none are encoded.

To specify a union value, you must use udrCreate.

fixed

Supports the attributes: name, namespace, aliases (optional), size

Note
title

Note!

Use the doc attribute to add a comment.

Limitations

The following limitations apply for the Avro support:

  • Line and Column are not fully implemented.

  • No string values are permitted in APL for the enum type.

  • Encoding the Avro object container file is not supported.

  • aliases are not fully supported.

  • default is not fully supported.

  • Cross references to other Ultra definitions are not supported.

  • array of arrays is not correctly implemented. For example, array of array of x is implemented with array of y, where y is a record with one value field. This value field is an array of x.

  • map of x is implemented via an array of y, where y is a record with two fields: key and value, where value is x.

Avro Format Example

To encode an Arvo data file, a format definition is included in the Ultra avro_block in the Ultra format.

title
Info

Example - The xml_schema Ultra Code Block

Code Block
languagetextthemeEclipse
avro_block {
{
"namespace": "example.avro",
 "type": "record",
 "name": "User3",
 "fields": [
     {"name": "name", "type": {
             "name": "FullName2",
             "type": "record",
             "fields": [
                {"name": "firstName", "type": "string"},
                {"name": "lastName", "type": "string"}
                ]
             }
           
     },
     {"name": "favorite_number",  "type": ["int",    "null"]},
     {"name": "favorite_color",   "type": ["string", "null"]},
     {"name": "favorite_fotball_team", "type": {
         "name": "teams",
         "type": "enum",
         "namespace": "example.avro.teams",
         "symbols": ["Djurgården", "Hammarby", "Malmö", "Göteborg"]
         }
     },
     {"name": "ipAddresses", "type": {
         "type": "array",
         "items": [
            {
                 "name": "ipv4Address",
                 "type": "fixed",
                 "size": 4
             },
             {
                 "name": "ipv6Address",
                 "type": "fixed",
                 "size": 16
              }
           ]
         }
      },
      {"name": "favoriteFoodList", "type": {
          "name": "favoriteFood",
          "type": "record",
          "fields": [
            {"name": "dish", "type": "string"},
            {"name": "next", "type": ["null", "favoriteFood"]}
          ]
          }
      },
      {"name": "salary", "type": "long"},
      {"name": "myFixed", "type": { "name": "myfixed", "type":"fixed", "size": 4 }},
      {"name": "myFloat", "type": "float"},
      {"name": "myDouble", "type": "double"},
      {"name": "rootUsers", "type": {
              "type": "map",
              "values": {
                "name": "RootUsers",
                "type": "record",
                "fields": [
                  {"name": "rootUser", "type": "string"},
                  {"name": "privileges", "type": "int"}
                ]
              }
          }
          },
          {"name": "maps", "type": {
              "type": "array",
              "items": {
                "type": "map",
                "values": {
                    "name": "Map2",
                    "type": "record",
                    "fields": [
                        {"name": "favoriteUser", "type": "string"},
                        {"name": "favoriteNumber", "type": "int"}
                    ]
                }
             }
          }
          }
 ]
}
};