Open Data Hub Monitoring

Project Description

Open Data Hub collects logs, metrics, and traces from all data pipelines running in the Kubernetes cluster and from the cluster itself.
This project focuses on building one or more Grafana dashboards to monitor critical insights and setting up meaningful alarms.
It also includes validating the observed data for presence and correctness and possibly creating a small observation intelligence service where raw data is insufficient.

Skills Needed

• Grafana
• Tempo
• Prometheus
• Elasticsearch
• PostgreSQL
• Data analysis

Difficulty Level

Medium

Project Size

175 hours

Mentors

• Matteo Roggia
• Clemens Zagler

Expected Outcomes

• One or more Grafana dashboards for Open Data Hub monitoring
• Meaningful alerting rules and alarms
• Documentation of observed SLIs (Service Level Indicators)
• Optional observation intelligence service for enriched insights