monitoring spark 3.0 with Prometheus in K8s

[marcoverl]

Hi,
i had a look at https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-2/ but something is not clear:
according to https://databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 adding kubernetes annotations as spark.kubernetes.driver.annotation.prometheus.io/scrape=true -c spark.kubernetes.driver.annotation.prometheus.io/path=/metrics/prometheus/ -c spark.kubernetes.driver.annotation.prometheus.io/port=4040 -c spark.kubernetes.driver.service.annotation.prometheus.io/scrape=true -c spark.kubernetes.driver.service.annotation.prometheus.io/path=/metrics/executors/prometheus/ -c spark.kubernetes.driver.service.annotation.prometheus.io/port=4040 …
should be enough to allow Prometheus to discover pods/services to monitor.
Do you agree?
I don't like much the idea to manually add labels to the driver service, also because is not possible to do this automatically via spark configuration as can be done for annotations.

[dzlab]

Kubernetes annotations are different from spark configuration and are not used for same thing. Prometheus will use those annotations to discover and scrape the metrics from the endpoint exposed by spark service.