1. Format Management Overview

Owned by Jenni Wallin
Last updated: Apr 28, 2023 by Former user (Deleted)
3 min read

is capable of handling various Usage Detail Record (UDR) formats provided that their structure is known. Vendor UDR formats are often complex and hard to work with in agents that will view or manipulate data. Therefore, the software converts the external UDR formats to an internal representation, and vice versa when UDRs are leaving the system.

In order to make this conversion, both internal and external format definitions have to be defined. When the definitions exist, there must be a mapping description that maps each external field of interest to an internal field.

Ultra does not require a one-to-one relationship between external and internal field names. Normally, only fields of interest or those that are needed when leaving the system are declared in the internal format definition. Often the internal definition holds additional fields that are initially empty and then populated on the way through a workflow. The software does not constrain how these fields are used.

Example - Format management

In this example, the external field E1C is not needed and not mapped at all. Internally IFA and IFB are manipulated. IFE is given a value, probably depending on values defined in IFC and IFD during the UDRs way through the workflow. At the exit, the two original fields are mapped into E2A and E2B plus the new (IFE) into E2C.



The software introduces a language called Ultra Format Definition Language (UFDL) that is used when describing UDR formats and mappings.

Apart from describing external formats, internal formats and mappings, the UFDL requires separate decoder and encoder specifications. A decoder or encoder refers to one or several mappings that are used for the actual translation.

Format Storage

In order to operate on UDRs, there must be a compiled version (Java class) of the internal format definition available in the Code or Ultra server. Such Java classes are introduced into the servers in two different ways:


  1. Format definitions entered in the Ultra Format Editor are compiled and inserted in the Ultra server when saved.
     

  2. Precompiled format definitions (for instance; Radius, NetFlow, and SNMP protocols) are inserted in the Code server during package commit.

A format can be recompiled while workflows using it are running. In this case, the changes to the format will take effect the next time the workflow is activated.

All Java classes generated by the Ultra server are kept in the system. This is to make UDRs, for instance, stored in ECS, using the old format version reprocessable. For further information, see 5. Ultra Format Converter.

Note!

Saving a format in the Ultra Format Editor may take some time, since the system may have to re-validate and regenerate many dependent configurations.

If an Ultra format is moved to another directory or is renamed, the configurations using the format will become invalid.

MZ Tagged Formats

There are two built-in external formats that can be used between workflows, typically by Inter Workflow agents; MZ Tagged Format and MZ Tagged (compressed). Using the compressed format will reduce the size of the UDRs significantly. However, since compression will require more CPU, you should consider the trade off between I/O and CPU when choosing encoder. No Decoder or Encoder needs to be defined in Ultra to handle these formats. 

The MZ Tagged formats contain header data that is based on the time zone settings of the host on which it was generated. This is important to consider when you compare output from identical workflows that are running on hosts with different time zone settings. The binary data may differ due to the header content.

Decoding of an UDR

The Ultra engine is designed to decode only the necessary parts of an Usage Detail Record, UDR, to keep workflow performance as high as possible. By default (that is, provided that Full Decode is not selected), the decoder will only decode enough to calculate the size of the UDRs. The UDR content will be decoded as the field values are accessed. Hence, if there is a decoding error, this may not be discovered by the decoder however by the agent accessing a specific field.


Note!

Full decoding of complicated nested UDR structures can be time consuming. In most cases the performance cost of this option is however small.