03. Batch Interface Agents PE 4
- Sandra Dusic
Optional offline interfaces for file based collection and distribution. The table is split between generic interfaces and vendor specific interfaces.
General Availability
Package | Type | Included in Subscription base license | Release | Detailed Information |
|---|---|---|---|---|
ADLS2 File Agents | Offline | Y | 4.2.0 | Collects and forwards files from and to ADLS (Azure Data Lake Storage) Gen2. Collection: Forwarding: |
Amazon (AWS) S3 (Collection & Forwarding) | Offline | Y | 4.2.0 | Collects and forwards files from and to Amazon S3 buckets. Collection: Forwarding: |
Apache Parquet Agents | Offline | Y | 4.2.0 | Enables handling of data encoded and compressed with Apache Parquet. The Parquet Decoder Agent converts rows from Parquet documents into UDRs. The Parquet Encoder Agent converts UDRs into Parquet format. |
Disk Collection | Offline | Y | 4.2.0 | Scans and collects files from local file system. |
Disk Forwarding | Offline | Y | 4.2.0 | Forwards files to local file system. |
Email Batch Collector Agent | Offline | Y | 4.2.0 | Collects emails (including attachments) from email servers using the IMAP or POP3 protocols. Implements the Jakarta Mail standard https://eclipse-ee4j.github.io/mail/. The collection criterion is read or unread emails, and a subject can optionally be filtered on. After collection, the emails can be deleted on the email server, or set to read or flagged. |
Encryption Agent | Offline | Y | 4.2.0 | Masks or unmasks complete files using keys and algorithms. Keys can be read from key store (JCEKS type only), or generated directly. Supported algorithms: AES-128 or AES-256. Empty files can be handled or ignored. |
FTAM Ericsson IOG | Offline | Y | 4.2.0 | Collets files from Ericsson IOGs via FTAM Interface Services. The FTAM Service must be running on a host in the network. TCP connection to the switch is required, alternatively a third-party RFC 1006 (ISO Transport Service on top of the TCP)Â to X.25 converter can be used. |
FTAM/Siemens EWSD | Offline | Y | 4.2.0 | Collets files from Siemens EWSD switches via FTAM Interface Services. The FTAM Service must be running on a host in the network. TCP connection to the switch is required, alternatively a third-party RFC 1006 (ISO Transport Service on top of the TCP)Â to X.25 converter can be used. |
FTP Collection | Offline | Y | 4.2.0 | Collects files from remote file system using the standard FTP (RFC 959) protocol. Remote file systems can be of type Unix, Windows NT or VAX/VMS. Uses the Standard File Collection Functions. Allows for collection to be tried from a list of hosts, with a configurable number of retries and retry intervals. |
FTP Forwarding | Offline | Y | 4.2.0 | Forwards files to remote file system using the standard FTP (RFC 959) protocol. Remote file systems can be of type Unix, Windows NT or VAX/VMS. Supports IPv4 and IPv6 environments. Uses the Standard File Forwarding Functions. Optionally, a local backlog directory for temporary storage can be configured where files are stored (configurable number of files or bytes) before moved to the final destination. Allows for forwarding to be tried from a list of hosts. |
FTP/Nokia DX200 | Offline | Y | 4.2.0 | Collets files from Nokia DX200 switches via FTAM Interface Services. The FTAM Service must be running on a host in the network. TCP connection to the switch is required, alternatively a third-party RFC 1006 (ISO Transport Service on top of the TCP)Â to X.25 converter can be used. |
FTP/Nokia NMSC | Offline | Y | 4.2.0 | Collects data files from Nokia SMS/MMS Billing Center Interfaces, using the FTP protocol. The agent reads the storage control file, and registers successfully collected data in the transaction control file. |
FTP/Siemens EWSD | Offline | Y | 4.2.0 | Collets files from Siemens EWSD switches via FTAM Interface Services. The FTAM Service must be running on a host in the network. TCP connection to the switch is required, alternatively a third-party RFC 1006 (ISO Transport Service on top of the TCP)Â to X.25 converter can be used. |
FTPS Collection | Offline | Y | 4.2.0 | Collects files from remote file system using the standard FTP (RFC 959) protocol, with the addition of Transport Layer Security (TLS) and Secure Sockets Layer (SSL) cryptographic protocols. Remote file systems can be of type Unix, Windows NT or VAX/VMS. Supports IPv4 and IPv6 environments. |
FTPS Forwarding | Offline | Y | 4.2.0 | Forwards files to remote file system using the standard FTP (RFC 959) protocol, , with the addition of Transport Layer Security (TLS) and Secure Sockets Layer (SSL) cryptographic protocols. Remote file systems can be of type Unix, Windows NT or VAX/VMS. Supports IPv4 and IPv6 environments. Uses the Standard File Forwarding Functions. Optionally, a local backlog directory for temporary storage can be configured where files are stored (configurable number of files or bytes) before moved to the final destination. Allows for forwarding to be tried from a list of hosts. |
HDFS (Collection & Forwarding) | Offline | Y | 4.2.0 | Collects and forwards files from and to Hadoop Distributed File System (HDFS). The agents comply to distributions from hadoop.apache.org; open source versions of Hortonworks 2.0 and Cloudera 4.x distributions of Hadoop 2.X. Supports Kerberos authentication. The agents can easily be extended by the user if a different distribution is used. Collection: Forwarding: |
HTTP Collection | Offline | Y | 4.2.0 | Collects the files via HTTP/HTPPS protocols originating from either a single URL or, optionally, all linked-to URLs found in an HTML-formatted document (Anchor HREF attributes) can be included. Duplication check can be activated, with a configurable number of days to keep the references. |
Netia Extensions | Offline | N | 4.2.0 | Customer specific features. |
Python Batch Collection | Offline | Y | 4.2.0 | Collects from sources supporting batch Python APIs. |
Python Batch Processing | Offline | Y | 4.2.0 | Processes UDR using Python. A set of out-of-the-box functions and functions blocks are available. |
SCP Collection | Offline | Y | 4.2.0 | Collects files using the SCP protocol over SSH2. Uses the Standard File Collection Functions. Allows for collection to be tried from a list of hosts, optionally with key exchange. Offers an additional level of security options:
3des-cbc, 3des-ctr, blowfish-cbc, aes128-cbc, aes192-cbc, aes256-cbc, aes128-ctr, aes192-ctr, aes256-ctr, arcfour, arcfour128, arcfour256
hmac-sha1, hmac-sha1-96, hmac-sha1-etm@openssh.com, hmac-md5, hmac-md5-96, hmac-md5-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-96, hmac-sha2-512-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-etm@openssh.com |
SCP Forwarding | Offline | Y | 4.2.0 | Forwards data using the SCP protocol over SSH2. Uses the Standard File Forwarding Functions. Optionally, a local backlog directory for temporary storage can be configured where files are stored (configurable number of files or bytes) before moved to the final destination. Allows for forwarding to be tried from a list of hosts, optionally with key exchange. Offers an additional level of security options:
3des-cbc, 3des-ctr, blowfish-cbc, aes128-cbc, aes192-cbc, aes256-cbc, aes128-ctr, aes192-ctr, aes256-ctr, arcfour, arcfour128, arcfour256
hmac-sha1, hmac-sha1-96, hmac-sha1-etm@openssh.com, hmac-md5, hmac-md5-96, hmac-md5-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-96, hmac-sha2-512-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-etm@openssh.com |
SFTP Collection | Offline | Y | 4.2.0 | Collects files using the SFTP protocol over SSH2. Supports IPv4 and IPv6 environments. Uses the Standard File Collection Functions. Allows for collection to be tried from a list of hosts, optionally with key exchange. Offers an additional level of security options:
3des-cbc, 3des-ctr, blowfish-cbc, aes128-cbc, aes192-cbc, aes256-cbc, aes128-ctr, aes192-ctr, aes256-ctr, arcfour, arcfour128, arcfour256
hmac-sha1, hmac-sha1-96, hmac-sha1-etm@openssh.com, hmac-md5, hmac-md5-96, hmac-md5-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-96, hmac-sha2-512-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-etm@openssh.com |
SFTP Forwarding | Offline | Y | 4.2.0 | Forwards data using the SCP protocol over SSH2. Supports IPv4 and IPv6 environments. Uses the Standard File Forwarding Functions. Optionally, a local backlog directory for temporary storage can be configured where files are stored (configurable number of files or bytes) before moved to the final destination. Allows for forwarding to be tried from a list of hosts, optionally with key exchange. Offers an additional level of security options:
3des-cbc, 3des-ctr, blowfish-cbc, aes128-cbc, aes192-cbc, aes256-cbc, aes128-ctr, aes192-ctr, aes256-ctr, arcfour, arcfour128, arcfour256
hmac-sha1, hmac-sha1-96, hmac-sha1-etm@openssh.com, hmac-md5, hmac-md5-96, hmac-md5-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-96, hmac-sha2-512-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-etm@openssh.com |
Standard File Collection Functions
Category | Description |
|---|---|
Collection Strategies: | Additional collection criteria that can be added to the basic. Available for Disk, FTP, SFTP, SCP, ADLS, HDFS Collection Agents. |
Control File Strategy | Enables collection of only specific files listed in a control file. |
Multi Directory Collection Strategy | Enables collection from multiple directories listed in a control file. |
APL Collection Strategy | Enables APL configuration for definition of files to collect. |
Duplicate Filter Collection Strategy | Restricts already collected files from being collected again. |
File name | Name of the source files on the remote host. Regular expressions according to Java syntax applies. |
Before collection functions | Temporary moving of files to other directory, optionally with suffix. |
After collection functions | Indicates if collected files are to be moved, renamed, left (used for test typically) or removed. |
Route FileReferenceUDR | Used if the workflow contains an SQL Loader Agent that shall insert the collected batch into a database. Available for Disk, FTP, SFTP, FTPS, HDFS, ADLS and S3 Collection Agents |
File name sequence | Applicable for files with sequence in their filename pattern. The workflow will ensure that the files are collected in sequence. NOTE! Only available for batch versions of file collection agents. |
Sort order | Decides if a particular processing order of files matching the filename pattern applies. |
Standard File Forwarding Functions
Category | Description |
|---|---|
Filename Template | Patterns for naming files forwarded. |
Input type | Besides the regular bytearray as input, the forwarding agent can also accept an object containing a list of UDRs from a preceding Analysis or Aggregation agent. This allows for full control over the contents of the forwarded file. |
Directory | Where to forward the files. Optionally, a directory may be created if non-existent. Comment: During processing, a forwarding file is appended to in a temporary directory. This files is moved to it's final destination when the file is complete. For remote hosts (FTP etc) this temporary directory can reside on the local host. |
Compression | States if files are to be compressed (gzip). |
After treatment | Executes a binary/script when a file is forwarded. |
Produce empty files | If no data is produced for the file forwarded for a particular input file, an empty file can optionally be produced. |
Limited Availability
Package | Type | Included in Subscription base licens | Rel. | Description |
|---|---|---|---|---|
Excel Encoder Agent | Offline | Y | 4.2.0 | Converts (Encodes) UDRs into Excel files. |
Excel Decoder Agent | Offline | Y | 4.2.0 | Converts (Decodes) Excel files into UDRs. The complete file can be decoded, or a selection of tabs and cells. |
GCP Big Query Forwarding | Offline | Y | 4.2.0 | Google Cloud Platform Big Query Batch Forwarding. |
GCP Storage Agents | Offline | Y | 4.2.0 | File collection and forwarding agents for Google Cloud Platform. |
SAP-CC Offline Batch Agent for Synchronous Charging | Offline | N | 4.2.0 | Forwards data synchronously to SAP Convergent Charging (CC). For instance:
Allows for forwarding to be tried from a list of hosts. Also, a debug mode convenient for testing is a configurable option. SAP CC supported versions:
|