This page describes what you need to consider when designing and executing your Usage metering streams.
Recommendations on Stream Execution
The processing of meters relies on stream executions. If a stream containing a Meter processor connected to a set of meters is not executed, the meters linked to that stream will not be reset or flushed, see Creating Meter Types for more information about Reset schedule and Flushing below for more information about reset and flush.
To ensure that meters are reset and get flushed, it is recommended to schedule the stream with a frequency higher than the meters are reset according to the meter types' Reset schedule. For example, if you have configured the meter type to reset the meters once a day, then the stream linked to your meter type should run at least once a day or more. If you only run your stream every second day, the new period of the meter will cover two days instead of one. The reason is that when the Meter processor calculates the new period end, it will pick the nearest future timestamp based on the periodicity of the meter type.
Flushing
Flush is when the result of the aggregated data in the Meter Processor is sent to downstream systems. When this happens, the values of the meters are also reset to their initial value, for example 0, to empty the bucket and prepare for the next metering period.
Based on the settings configured in the Meter Processor, a stream might flush the meters at the beginning of its execution, at the end of the stream, or both.
When a stream starts in a batch stream scenario, the Meter Processor will start by scanning the meter storage for meters whose period end has passed, performing a reset scan. If any expired meters are found, they will be flushed and reset, starting a new metering period. If a stream is configured to flush by Period end, a reset scan will also be performed at the end of the stream execution, but if a stream is configured to flush by Stream execution then all meters in your storage linked to that meter processor will be reset and flushed regardless of whether the meters have expired or not.
When you have a stream in a real-time scenario, the reset scan will execute once every minute to check for any meters that should be flushed. This means that meters can be flushed once every minute. A Meter processor running in a real-time scenario can only be configured with a flush by Period end.
Realtime vs Batch Streams
Batch streams are streams that start, process a set of data and end.
Real-time streams are streams that are started and then constantly running, receiving data and processing it as it arrives. Streams using
Data Correction for Usage Metering
If the incoming usage data records are missing any required fields (i.e. userId or value), or if no matching meter is found for them based on the guiding rules, or ir the data has records with timestamps before the Active from: date or after the Active to: dates, they will be sent to Data Correction so you can correct them or adjust any conditions to find matching meters and then reprocess them.