/
Glossary

Glossary

576a2e8d-1dc6-4166-88ce-fe1eebec9f82.png

Term or Acronym

Definition

Term or Acronym

Definition

Bounded stream

A bounded stream is a unidirectional Glossary | Stream that processes finite input units, for example, complete files or a fixed amount of rows in a database.

Example

Collecting .csv files via SFTP and only setting a "committed" checkpoint for a particular input file once it is guaranteed an entire file has been successfully processed.

Collector

 

A collector is a Glossary | Function that collects data from a Glossary | Source.

Collection

 

The process of Usage Engine acquiring data from a Glossary | Source.

Examples

  • Collecting .csv files from an S3 bucket by acting as an HTTPS client

  • Collecting RESTful requests by acting as an HTTPS server

Client mode

Client mode is a communication mode where https://infozone.atlassian.net/wiki/spaces/DAZ/pages/7798857/Glossary#Collection initiate outgoing requests.

Examples

  • Usage Engine acting as an HTTPS client.

  • Usage Engine acting as an SFTP client.

  • Usage Engine acting as a JDBC client.

Custom references

Custom references refer to an optional feature that is used with SAP-related functions. When enabled it allows you to specify one or more typeCode and id values in the relevant usage records. 

Decode

Decoding is the process of parsing input from a Glossary | Source and converting it into Glossary | Record in a format that can be sent across a https://infozone.atlassian.net/wiki/spaces/DAZ/pages/7798857/Glossary#Route.

Destination

Destination is an endpoint to which data is sent from a Glossary | Stream.

Encode

Encoding is the process of formatting a Glossary | Record into a format that can be understood by a Glossary | Destination.

Example

Creating a JSON string from a Glossary | Record.

Endpoint

An endpoint is an interface used by the service to an external system or resource.

Examples

  • S3 bucket

  • SFTP server

  • HTTP server

  • HTTP client

  • Database instance

Flush

The flush operation forces the stored/buffered (aggregated) data to be passed to the next function in the stream.

Forwarder

A forwarder is a Glossary | Function that sends data to a given Glossary | Destination.

Forwarding

The process of sending data to a Glossary | Destination.

Examples

  • Sending JSON records to a billing system via HTTPS

  • Storing .csv files in an S3 bucket

Function

A function is a processing step in a stream that receives input, applies defined logic, and produces output. Functions can be stateless (acting only on the current input) or stateful (maintaining information across multiple inputs). In JSON import/export and backend APIs, a function is called a node.

Example - Function behavior

  • Filtering fields from records

  • Renaming fields in records

  • Aggregating multiple records.

Function input

A function input is where a function receives data from another function or from an external system. In the UI, function inputs are shown as circular markers on the left (incoming) side of a function icon.

Function output

A function output is where a function sends its processed data to another function or to an external system. In the UI, function outputs are shown as circular markers on the right (outgoing) side of a function icon. A function can have one or multiple outputs, depending on its type and configuration.

HTTP response headers

This is an option found within the HTTP client collector and processor function, intended to support the sending out of response headers in the network stream. The users can enter a custom value in the result header key and associated result key to configure the target web service. 

Link (edge)

A link connects two functions in a stream so that data can flow from one to the other. In the UI, it is shown as a line connecting an output on one function to an input on another. In JSON import and export and backend APIs, a link is called an edge.

Notification group

 A notification group is an email list to which automated messages regarding erroneous events related to streams are being sent. The content of the error messages includes stream execution errors, missing files, and related events. 

Payload

Payload is the content of the record.

Record

A record is a unit of data that can be processed by a Glossary | Function or multiple functions and sent across https://infozone.atlassian.net/wiki/spaces/DAZ/pages/7798857/Glossary#Route.

Examples

  • A decoded line in a .csv file.

  • A decoded JSON payload in an HTTP request.

  • A decoded row in a relational database.

  • A decoded Kinesis message.

RegEx (Regular expression)

The use of regular expressions implies the creation of custom schemes to which various operations can be executed. 

Route

A route is a link between two Glossary | Function, across which data flows.

Example

A Glossary | Function exposing two outgoing https://infozone.atlassian.net/wiki/spaces/DAZ/pages/7798857/Glossary#Route. For example, a function can send Glossary | Record that are either successfully processed or erroneous records.

Session

A representation of aggregated records that is stored before being flushed out. 

Solution

A solution is a set of Glossary | Stream that together solve a particular customer problem.

Examples

  • Streams required to perform billing mediation for a cloud biller on incoming usage records.

  • Streams required to acquire the product catalog data for a cloud biller.

Source

A source is an Glossary | Endpoint from which data is collected into a Glossary | Stream using a given protocol and data format.

Stream

A stream is a graphical representation of data flowing from a Glossary | Source, through a series of Glossary | Function, to a Glossary | Destination.

Example

 A stream collecting .csv files from an Amazon S3 source and forwarding them to a destination database using JDBC.

Time to live (TTL) 

Time to live is a mechanism that limits the lifespan of data in an environment. It is used in functions like Aggregation to express the date and time when a record expires. 

Timeout

Timeout is a method that allows you to set a timer for the execution of a certain function. Timeouts enable more efficient usage of limited resources. The idea is that in when a system must wait for something to happen, rather than waiting indefinitely, the waiting will be aborted after the timeout period has elapsed.

Transaction

A transaction is a unit of data that is being processed inside a stream. The definition of a completed transaction is when the stream processes it without any errors. The transactions are by design limited to their specified data sources.

A transaction can refer to a file that is being processed in a stream or a result set from a query. 

Data correction also creates transactions for reference which are considered to be separate processes. 

For more information, see Transaction safety.

Transactions per second (TPS)

Transactions per second is a metric showing the number of transactions per second (tps) that can be processed by a given web service. This value can be calculated using this formula:

Tps value for a function = Total number of output events handled by the function / Duration of the stream