Handling of Data Formats (UFDL)

Owned by Jenni Wallin
Last updated: May 05, 2023

The Usage Engine Ultra formatting subsystem manages transcription of incoming data into internal records, and internal records into outgoing data structures valid for the downstream systems receiving data. The internal records are referred to as UDRs (usage data records) and can be processed by Usage Engine agents, such as the Analysis Agent and Aggregation Agent.  

Ultra is configured with a language called the Ultra Format Definition Language (UFDL). There are 6 building blocks of UFDL: 

  • External formats - describes the way data is structured physically. This information is used by Ultra when decoding and encoding the data to include real-time requests and answer messages. 
  • Internal formats - describes a UDR (fields and types) that can be accessed from a Usage Engine agent 
  • In maps - describes how to map the information of an external format to an internal format (for a decoder) 
  • Out maps - describes how to map the information of an internal format to an external format (for an encoder) 
  • Decoders – a declaration of a decoder that converts an external format to an internal format 
  • Encoders - a declaration of an encoder that converts an internal format to an external format 

UFDL handles conversion of data formats

Example below of a workflow that collects data from local disk, decodes it into an internal UDR, then validates the data in an Analysis node. The validated data is either sent to the Error Correction System (ECS), or to be Encoded into an external format before forwarding. 

Example workflow

Ultra includes support for, and has been deployed using, numerous different physical formats. Direct support is included for any format described in ASN.1 (BER and PER), and XML Schema (XML). Ultra also contains a specification language to describe other physical formats, such as proprietary tagged records, and fixed size records. Among other formats, this language has been used to provide support for AMA, INAMA, TAP2, TAP3, EMI, and a number of proprietary equipment vendor formats. 

Important to note is that, based on format specifications, Usage Engine generates code for decoding and encoding functionality automatically. This means that there is no interpretation, with associated performance penalties, performed at runtime. 

External sequential format