03. Batch Interface Agents MZ 9

Optional offline interfaces for file based collection and distribution. The table is split between generic interfaces and vendor specific interfaces.

General Availability

Package

Type

Included in Subscription base license

Rel.

Detailed Information
ADLS2 File AgentsOfflineY9.0.0.0Collects and forwards files from and to ADLS (Azure Data Lake Storage) Gen2.

Collection:
Uses the Standard File Collection Functions

Forwarding:
Uses the Standard File Forwarding Functions.

Amazon (AWS) S3 (Collection & Forwarding)OfflineY9.0.0.0
Collects and forwards files from and to Amazon S3 buckets. 

Collection:
Uses the Standard File Collection Functions - with the exception of "Collection Strategies", which is not supported. 

Forwarding:
Uses the Standard File Forwarding Functions.

Apache Parquet AgentsOfflineY9.0.0.0Enables handling of data encoded and compressed with Apache Parquet. The Parquet Decoder Agent converts rows from Parquet documents into UDRs. The Parquet Encoder Agent converts UDRs into Parquet format.

Disk Collection

Offline

Y

9.0.0.0

Scans and collects files from local file system.
Uses the Standard File Collection Functions
.

Disk Forwarding

Offline

Y9.0.0.0
Forwards files to local file system. 
Uses the Standard File Forwarding Functions

Email Batch Collector Agent

OfflineY9.0.0.0Collects emails (including attachments) from email servers using the IMAP or POP3 protocols. Implements the Jakarta Mail standard https://eclipse-ee4j.github.io/mail/. The collection criterion is read or unread emails, and a subject can optionally be filtered on. After collection, the emails can be deleted on the email server, or set to read or flagged.

Encryption Agent

OfflineY9.0.0.0Masks or unmasks complete files using keys and algorithms. Keys can be read from key store (JCEKS type only), or generated directly. Supported algorithms: AES-128 or AES-256. Empty files can be handled or ignored. 
FTAM Ericsson IOGOfflineY9.1.0.0Collets files from Ericsson IOGs via FTAM Interface Services. The FTAM Service must be running on a host in the network. TCP connection to the switch is required, alternatively a third-party RFC 1006 (ISO Transport Service on top of the TCP - Supported from 7.0.0.0) to X.25 converter can be used.
FTAM/Siemens EWSDOfflineY9.1.0.0Collets files from Siemens EWSD switches via FTAM Interface Services. The FTAM Service must be running on a host in the network. TCP connection to the switch is required, alternatively a third-party RFC 1006 (ISO Transport Service on top of the TCP - Supported from 7.0.0.0) to X.25 converter can be used.

FTP Collection

Offline

Y

9.0.0.0

Collects files from remote file system using the standard FTP (RFC 959) protocol. Remote file systems can be of type Unix, Windows NT or VAX/VMS. Uses the Standard File Collection Functions. Allows for collection to be tried from a list of hosts, with a configurable number of retries and retry intervals.

 

FTP Forwarding

Offline

Y

9.0.0.0

Forwards files to remote file system using the standard FTP (RFC 959) protocol. Remote file systems can be of type Unix, Windows NT or VAX/VMS. Supports IPv4 and IPv6 environments. Uses the Standard File Forwarding Functions. Optionally, a local backlog directory for temporary storage can be configured where files are stored (configurable number of files or bytes) before moved to the final destination. Allows for forwarding to be tried from a list of hosts.
FTP(/SFTP) Nokia DX200OfflineY9.1.0.0Collects data files from DX200 network elements, using the FTP or SFTP protocol. The agent reads the storage control file, and registers successfully collected data in the transaction control file.
FTP/Nokia NMSCOfflineY9.1.0.0Collects data files from Nokia SMS/MMS Billing Center Interfaces, using the FTP protocol. The agent reads the storage control file, and registers successfully collected data in the transaction control file.
FTP/Siemens EWSDOfflineY9.2.0.0

The FTP/EWSD agent enables collection of cyclic files from Siemens EWSD switches according to specifications:

FTPS Collection

Offline

Y

9.0.0.0

Collects files from remote file system using the standard FTP (RFC 959) protocol, with the addition of Transport Layer Security (TLS) and Secure Sockets Layer (SSL) cryptographic protocols. Remote file systems can be of type Unix, Windows NT or VAX/VMS. Supports IPv4 and IPv6 environments. 
Uses the Standard File Collection Functions. Allows for collection to be tried from a list of hosts, with a configurable number of retries and retry intervals.

FTPS Forwarding

Offline

Y

9.0.0.0

Forwards files to remote file system using the standard FTP (RFC 959) protocol, , with the addition of Transport Layer Security (TLS) and Secure Sockets Layer (SSL) cryptographic protocols. Remote file systems can be of type Unix, Windows NT or VAX/VMS. Supports IPv4 and IPv6 environments. Uses the Standard File Forwarding Functions. Optionally, a local backlog directory for temporary storage can be configured where files are stored (configurable number of files or bytes) before moved to the final destination. Allows for forwarding to be tried from a list of hosts.

HDFS (Collection & Forwarding)

Offline

Y

9.0.0.0

Collects and forwards files from and to Hadoop Distributed File System (HDFS). The agents comply to distributions from hadoop.apache.org; open source versions of Hortonworks 2.0 and Cloudera 4.x distributions of Hadoop 2.X. Supports Kerberos authentication.

The agents can easily be extended by the user if a different distribution is used. 

Collection:
Uses the Standard File Collection Functions

Forwarding:
Uses the Standard File Forwarding Functions.

HTTP Collection

Offline

Y

9.0.0.0

Collects the files via HTTP/HTPPS protocols originating from either a single URL or, optionally, all linked-to URLs found in an HTML-formatted document (Anchor HREF attributes) can be included. Duplication check can be activated, with a configurable number of days to keep the references.
Netia ExtensionsOfflineN9.2.0.0
Customer specific features.
Python Batch CollectionOfflineY9.2.0.0
Collect data from sources supporting Python Batch API.
Python Batch ProcessingOfflineY9.2.0.0
Processes batches of UDRs using Python.

SCP Collection

Offline

Y

9.0.0.0

Collects files using the SCP protocol over SSH2. Uses the Standard File Collection Functions. Allows for collection to be tried from a list of hosts, optionally with key exchange. Offers an additional level of security options:

  • Cipher mode supported algorithms:

3des-cbc, 3des-ctr, blowfish-cbc, aes128-cbc, aes192-cbc, aes256-cbc, aes128-ctr, aes192-ctr, aes256-ctr, arcfour, arcfour128, arcfour256

  • HMac type encryption methods:

hmac-sha1, hmac-sha1-96, hmac-sha1-etm@openssh.com, hmac-md5, hmac-md5-96, hmac-md5-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-96, hmac-sha2-512-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-etm@openssh.com

SCP Forwarding

Offline

Y

9.0.0.0

Forwards data using the SCP protocol over SSH2. Uses the Standard File Forwarding Functions. Optionally, a local backlog directory for temporary storage can be configured where files are stored (configurable number of files or bytes) before moved to the final destination. Allows for forwarding to be tried from a list of hosts, optionally with key exchange. Offers an additional level of security options:

  • Cipher mode supported algorithms:

3des-cbc, 3des-ctr, blowfish-cbc, aes128-cbc, aes192-cbc, aes256-cbc, aes128-ctr, aes192-ctr, aes256-ctr, arcfour, arcfour128, arcfour256

  • HMac type encryption methods:

hmac-sha1, hmac-sha1-96, hmac-sha1-etm@openssh.com, hmac-md5, hmac-md5-96, hmac-md5-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-96, hmac-sha2-512-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-etm@openssh.com

SFTP Collection

Offline

Y

9.0.0.0

Collects files using the SFTP protocol over SSH2. Supports IPv4 and IPv6 environments. Uses the Standard File Collection Functions. Allows for collection to be tried from a list of hosts, optionally with key exchange. Offers an additional level of security options:

  • Cipher mode supported algorithms:

3des-cbc, 3des-ctr, blowfish-cbc, aes128-cbc, aes192-cbc, aes256-cbc, aes128-ctr, aes192-ctr, aes256-ctr, arcfour, arcfour128, arcfour256

  • HMac type encryption methods:

hmac-sha1, hmac-sha1-96, hmac-sha1-etm@openssh.com, hmac-md5, hmac-md5-96, hmac-md5-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-96, hmac-sha2-512-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-etm@openssh.com

SFTP Forwarding

Offline

Y

9.0.0.0

Forwards data using the SCP protocol over SSH2. Supports IPv4 and IPv6 environments. Uses the Standard File Forwarding Functions. Optionally, a local backlog directory for temporary storage can be configured where files are stored (configurable number of files or bytes) before moved to the final destination. Allows for forwarding to be tried from a list of hosts, optionally with key exchange. Offers an additional level of security options:

  • Cipher mode supported algorithms:

3des-cbc, 3des-ctr, blowfish-cbc, aes128-cbc, aes192-cbc, aes256-cbc, aes128-ctr, aes192-ctr, aes256-ctr, arcfour, arcfour128, arcfour256

  • HMac type encryption methods:

hmac-sha1, hmac-sha1-96, hmac-sha1-etm@openssh.com, hmac-md5, hmac-md5-96, hmac-md5-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-96, hmac-sha2-512-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-etm@openssh.com

Standard File Collection Functions

CategoryDescription

Collection Strategies:

Additional collection criteria that can be added to the basic. Available for Disk, FTP, SFTP, SCP, ADLS, HDFS Collection Agents.
Control File Strategy

Enables collection of only specific files listed in a control file.

Multi Directory Collection Strategy

Enables collection from multiple directories listed in a control file.

APL Collection StrategyEnables APL configuration for definition of files to collect.
Duplicate Filter Collection StrategyRestricts already collected files from being collected again.
File name

Name of the source files on the remote host. Regular expressions according to Java syntax applies.

Before collection functions

Temporary moving of files to other directory, optionally with suffix.

After collection functions

Indicates if collected files are to be moved, renamed, left (used for test typically) or removed.

Route FileReferenceUDR

Used if the workflow contains an SQL Loader Agent that shall insert the collected batch into a database. Available for Disk, FTP, SFTP, FTPS, HDFS, ADLS and S3 Collection Agents

File name sequence

Applicable for files with sequence in their filename pattern. The workflow will ensure that the files are collected in sequence.

NOTE! Only available for batch versions of file collection agents.

Sort order

Decides if a particular processing order of files matching the filename pattern applies.

Standard File Forwarding Functions

CategoryDescription
Filename TemplatePatterns for naming files forwarded.
Input type

Besides the regular bytearray as input, the forwarding agent can also accept an object containing a list of UDRs from a preceding Analysis or Aggregation agent. This allows for full control over the contents of the forwarded file.

Directory

Where to forward the files. Optionally, a directory may be created if non-existent.

Comment: During processing, a forwarding file is appended to in a temporary directory. This files is moved to it's final destination when the file is complete. For remote hosts (FTP etc) this temporary directory can reside on the local host.

Compression

States if files are to be compressed (gzip).

After treatment

Executes a binary/script when a file is forwarded.

Produce empty files

If no data is produced for the file forwarded for a particular input file, an empty file can optionally be produced.



Limited Availability

Package

Type

Included in Subscription base license

Rel.

Description
Excel Encoder AgentOfflineY9.0.0.0Converts (Encodes) UDRs into Excel files.
Excel Decoder AgentOfflineY9.0.0.0
Converts (Decodes) Excel files into UDRs. The complete file can be decoded, or a selection of tabs and cells. 
GCP Big Query ForwardingOfflineY9.0.0.0Google Cloud Platform Big Query Batch Forwarding.
GCP Storage AgentsOfflineY9.0.0.0File collection and forwarding agents for Google Cloud Platform.
SAP-CC Batch AgentOfflineY9.0.0.0

Forwards data synchronously to SAP Convergent Charging (CC). For instance:

  • Charging requests
  • Batch acquisition operation requests for chargeable items
  • Event based advice of charge

Allows for forwarding to be tried from a list of hosts. Also, a debug mode convenient for testing is a configurable option.

SAP CC supported versions:

  • Version 4.0, 4.1, 5.0
  • Version 2020, 2021, 2022