Logging

Owned by Kevin Wu
Last updated: Oct 20, 2023 by Former user (Deleted)
3 min read

Problems related to submission of a Spark application are logged in the Platform log. Errors that occur after submission of a Spark application, i e runtime errors, are logged in the the Spark environment. Error information related to Kafka and Zookeeper services can be found in the SC-logs for the respective service.

Runtime Errors

Cluster

Runtime errors that occur in the cluster are logged in SPARK_HOME/logs.

Spark Application

Runtime errors that occur in the Spark application when it is running are logged in the file SPARK_HOME/work/driver-<number>-<number>/stderr.

Runtime errors on the executor level are logged in the file SPARK_HOME/work/app-<number>-<number>/stderr

You can also access these logs from the Spark Master Web UI:

  1. Click a Worker id under Running Drivers.


    Spark UI - Master


  2. Click stderr under Logs.


    Spark UI - Worker

KPI Processing Accumulators

When a Spark batch has finished processing, a set of accumulators are logged in the file  SPARK_HOME/work/driver-<number>/stdout. These accumulators serve as a  summary of what has been collected and calculated within the batch.  

The following accumulators are logged:

Accumulator

Description

Accumulator

Description

CalculatedKPIs

This accumulator includes GeneratedKPIOutputs and calculated KPIs that are not closed yet.

DiscardedKPIs

This accumulator is incremented by one for each calculated KPI that belongs to a previously closed period.

FailedMetricCalculations

This accumulator is incremented by one for each metric calculation that fails, e g due to invalid data in the input records. If there are several nodes in the node tree(s) that contain the metric, one input record may affect several metric calculations.

FailedKPICalculations

This accumulator is incremented by one for a KPI calculation that fails due to undefined metrics in the KPI expression. In order for the accumulator to be incremented, the following conditions  must apply:

- The period for the KPI ends during the Spark batch.

- The KPI expression uses multiple metrics and one or more of these are undefined.

GeneratedKPIOutputs

This accumulator is incremented by one for each successfully calculated and delivered KPI.

MissingExpressionForInputType

This accumulator is increased by one for each input record that does not match a metric and a dimension object in the service model.



Example - Counters in stdout

The example below indicates that 20 input records failed to match both a metric and dimension expression in the service model.
============= SPARK BATCH: 2023-10-19 12:35:20:0 =============== CalculatedKPIs = 222 GeneratedKPIOutputs = 200 MissingExpressionForInputType = 20 DiscardedKPIs = 0 FailedMetricCalculations = 0 FailedKPICalculations = 0

 

 

You can also access these accumulators from the Spark Master Web UI:

  1. Click a Worker id under Running Drivers.

  2. Click stdout under Logs.

Note!

The accumulators are logged using log4j, meaning that the configured log level will decide whether or not the accumulators will be logged. The log level is specified in submit.sh by assigning log4j_setting and supply --conf spark.driver.extraJavaOptions=$log4j_setting. The default log level in Spark is WARNING and the log level for the accumulators is INFO. 



Note!

It is possible to log the accumulators to a separate log file by adding the log4j.properties
log4j.appender.accumulatorlog=org.apache.log4j.RollingFileAppender
log4j.appender.accumulatorlog.File=accumulators.log
log4j.appender.accumulatorlog.layout=org.apache.log4j.PatternLayout
log4j.appender.accumulatorlog.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.logger.com.digitalroute.mz.spark.StreamOperations$=INFO, accumulatorlog
log4j.additivity.com.digitalroute.mz.spark.StreamOperations$=false



Â