Problems related to submission of a Spark application are logged in the Platform log. Errors that occur after submission of a Spark application, i e runtime errors, are logged in the the Spark environment. Error information related to Kafka and Zookeeper services can be found in the SC-logs for the respective service.
Runtime Errors
Cluster
Runtime errors that occur in the cluster are logged in SPARK_HOME/logs.
Spark Application
Runtime errors that occur in the Spark application when it is running are logged in the file SPARK_HOME/work/driver-<number>-<number>/stderr.
Runtime errors on the executor level are logged in the file SPARK_HOME/work/app-<number>-<number>/stderr
You can also access these logs from the Spark Master Web UI:
Click a Worker id under Running Drivers.
Spark UI - Master
Click stderr under Logs.
Spark UI - Worker
KPI Processing Accumulators
When a Spark batch has finished processing, a set of accumulators are logged in the file SPARK_HOME/work/driver-<number>/stdout. These accumulators serve as a summary of what has been collected and calculated within the batch.
The following accumulators are logged:
Accumulator | Description |
|---|---|
CalculatedKPIs | This accumulator includes |
DiscardedKPIs | This accumulator is incremented by one for each calculated KPI that belongs to a previously closed period. |
FailedMetricCalculations | This accumulator is incremented by one for each metric calculation that fails, e g due to invalid data in the input records. If there are several nodes in the node tree(s) that contain the metric, one input record may affect several metric calculations. |
FailedKPICalculations | This accumulator is incremented by one for a KPI calculation that fails due to undefined metrics in the KPI expression. In order for the accumulator to be incremented, the following conditions must apply: - The period for the KPI ends during the Spark batch. - The KPI expression uses multiple metrics and one or more of these are undefined. |
GeneratedKPIOutputs | This accumulator is incremented by one for each successfully calculated and delivered KPI. |
MissingExpressionForInputType | This accumulator is increased by one for each input record that does not match a |
Example - Counters in stdout
The example below indicates that 20 input records failed to match both a metric and dimension expression in the service model.============= SPARK BATCH: 2023-10-19 12:35:20:0 ===============
CalculatedKPIs = 222
GeneratedKPIOutputs = 200
MissingExpressionForInputType = 20
DiscardedKPIs = 0
FailedMetricCalculations = 0
FailedKPICalculations = 0
You can also access these accumulators from the Spark Master Web UI:
Click a Worker id under Running Drivers.
Click stdout under Logs.
Note!
The accumulators are logged using log4j, meaning that the configured log level will decide whether or not the accumulators will be logged. The log level is specified in submit.sh by assigning log4j_setting and supply --conf spark.driver.extraJavaOptions=$log4j_setting. The default log level in Spark is WARNING and the log level for the accumulators is INFO.
Note!
It is possible to log the accumulators to a separate log file by adding the log4j.properties
log4j.appender.accumulatorlog=org.apache.log4j.RollingFileAppender
log4j.appender.accumulatorlog.File=accumulators.log
log4j.appender.accumulatorlog.layout=org.apache.log4j.PatternLayout
log4j.appender.accumulatorlog.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.logger.com.digitalroute.mz.spark.StreamOperations$=INFO, accumulatorlog
log4j.additivity.com.digitalroute.mz.spark.StreamOperations$=false