Online collection
Online Collection allows you to create secure endpoints for connecting and sending events from one or more external applications to your Online collection storage space. It is designed to efficiently handle high-frequency streams of small events, converting them into micro-batches for further processing in Usage Engine.
A single online collection send endpoint can handle up to 500,000 events per second, 2,000 requests per second, or 5 MB per second of data (whichever limit is reached first). For example, with an average event size of 2,500 bytes, you can send 2,000 requests per second, reaching the 5 MB/s throughput limit.
Once the data is stored, you can retrieve it into your solution using the Online collector function.
Note!
More details about rate limits can be found in the Online collection API docs:
EU Production -https://api.digitalroute.io/
US Production - https://api.us.digitalroute.io/
If you need to handle larger data batches or files, such as payloads exceeding 1 MB per request, we recommend using an S3 bucket and collecting data with theAmazon S3 collector for greater efficiency.
Caution!
There is a small risk that data ingested by Online collection may be duplicated, for example, in rare cases such as a system restart. To ensure data integrity, we recommend using the Deduplicate function after pulling data from Online collection to remove any potential duplicates.
This section has the following subsections: