refactor: remove telegraf service and switch to direct influxdb write

README.md

@@ -34,18 +34,17 @@ All services share a bridge network named `datalink` and rely on the admin token
 
 ## System overview
 
-The compose stack deploys nine cooperating containers:
+The compose stack deploys eight cooperating containers:
 
 1. **InfluxDB 3** – Time-series database seeded with a tiny example dataset.
 2. **InfluxDB 3 Explorer** – Web UI for browsing and querying telemetry.
-3. **Telegraf** – Reads the importer’s line protocol output and forwards metrics to InfluxDB.
-4. **Grafana** – Pre-provisioned dashboards that visualise the stored telemetry.  Load your own dashboard provisioning files into `installer/grafana/dashboards/`.
-5. **Sandbox** - *Under active development.* Connecting InfluxDB3 with LLM for natural language queries and analysis.
-6. **Slack bot d.b.a. Lappy** – Optional automation/notification bot for race ops.
-7. **Lap analysis app** – *Under active development.* Dash-based location data visualiser and lap timer. (Useful if GPS data is available.) 
-8. **Startup data loader** – Seeds the database on boot with sample CAN frames.
-9. **File uploader** – Streams uploaded CSV logs into InfluxDB using the shared DBC file.
-10. **Data downloader** - Scans InfluxDB periodically, visual SQL query builder, and CSV export service.
+3. **Grafana** – Pre-provisioned dashboards that visualise the stored telemetry.  Load your own dashboard provisioning files into `installer/grafana/dashboards/`.
+4. **Sandbox** - *Under active development.* Connecting InfluxDB3 with LLM for natural language queries and analysis.
+5. **Slack bot d.b.a. Lappy** – Optional automation/notification bot for race ops.
+6. **Lap analysis app** – *Under active development.* Dash-based location data visualiser and lap timer. (Useful if GPS data is available.) 
+7. **Startup data loader** – Seeds the database on boot with sample CAN frames.
+8. **File uploader** – Streams uploaded CSV logs into InfluxDB using the shared DBC file.
+9. **Data downloader** - Scans InfluxDB periodically, visual SQL query builder, and CSV export service.
 
 Detailed documentation for each service is available in `docs/containers/`.
 
@@ -80,7 +79,7 @@ https://github.com/Western-Formula-Racing/daq-2023
 
 We also want to acknowledge the open-source tools and libraries that make this project possible. Key components include:
 * Docker / Docker Compose for containerisation
-* InfluxDB 3 and Telegraf for time-series storage and ingestion
+* InfluxDB 3 for time-series storage
 * Grafana for visualisation
 * Python open-source packages (NumPy, Pandas, Requests, etc.) used throughout the stack
 
@@ -97,4 +96,4 @@ If you’re interested in our team’s broader engineering projects, here are so
 
 
 ## License
-AGPL-3.0 License. See LICENSE file for details.
+AGPL-3.0 License. See LICENSE file for details.

dev-utils/ci/stack-smoke-test.sh

@@ -57,7 +57,6 @@ trap 'cleanup $?' EXIT
 ENABLED_SERVICES=(
   influxdb3
   influxdb3-explorer
-  telegraf
   grafana
   frontend
   lap-detector

docs/containers/influxdb2-legacy-build.md

@@ -17,4 +17,4 @@ docker run -d --name influxdb2 \
   influxdb:2
 ```
 
-For current deployments use the Docker Compose stack under `installer/`, which provisions InfluxDB 3 along with Grafana, Telegraf, and the rest of the telemetry tooling.
+For current deployments use the Docker Compose stack under `installer/`, which provisions InfluxDB 3 along with Grafana and the rest of the telemetry tooling.

docs/containers/influxdb3.md

@@ -31,4 +31,4 @@ Data is stored in the `influxdb3-data` Docker volume. Removing the volume (`dock
 ## Related services
 
 - **Startup data loader** seeds the bucket with the example dataset on first run.
-- **Grafana**, **telegraf**, **file-uploader**, and **slackbot** authenticate using `INFLUXDB_ADMIN_TOKEN`.
+- **Grafana**, **file-uploader**, and **slackbot** authenticate using `INFLUXDB_ADMIN_TOKEN`.

docs/containers/startup-data-loader.md

@@ -6,15 +6,14 @@ The startup data loader seeds InfluxDB 3 with a small, deterministic dataset on
 
 - Loads CSV files from `/data` (mounted from `installer/startup-data-loader/data/`; copy `2024-01-01-00-00-00.csv.md` to a `.csv` file for the bundled sample).
 - Uses `example.dbc` to decode CAN frames into human-readable metrics.
-- Generates line protocol for Telegraf and optionally writes directly to InfluxDB.
+- Writes decoded metrics directly to InfluxDB.
 
 ## Environment variables
 
 | Variable | Description | Default |
 | --- | --- | --- |
-| `INFLUXDB_TOKEN` | Token used for direct writes when `BACKFILL=1`. | `dev-influxdb-admin-token` |
+| `INFLUXDB_TOKEN` | Token used for direct writes. | `dev-influxdb-admin-token` |
 | `INFLUXDB_URL` | Target InfluxDB endpoint. | `http://influxdb3:8181` |
-| `BACKFILL` | Set to `1` to stream directly into InfluxDB; otherwise only emits line protocol. | `1` |
 
 ## Extending the dataset
 
@@ -26,4 +25,4 @@ The startup data loader seeds InfluxDB 3 with a small, deterministic dataset on
 
 - Logs are available via `docker compose logs -f startup-data-loader`.
 - Progress is tracked in `/app/load_data_progress.json` inside the container.
-- The importer supports resuming partially processed files; remove the progress file to force a clean run.
+- The importer supports resuming partially processed files; remove the progress file to force a clean run.

docs/docker-compose.md

@@ -5,10 +5,10 @@ The `installer/docker-compose.yml` file orchestrates the complete DAQ telemetry
 ## High-level architecture
 
 ```text
-┌────────────┐      ┌──────────────────┐      ┌────────────┐
-│ Startup    │      │  Telegraf        │      │ InfluxDB 3 │
-│ data loader├─────▶│  (line protocol) ├─────▶│ + Explorer │
-└────────────┘      └──────────────────┘      └────────────┘
+┌────────────┐                                ┌────────────┐
+│ Startup    │                                │ InfluxDB 3 │
+│ data loader├───────────────────────────────▶│ + Explorer │
+└────────────┘                                └────────────┘
        │                                           │
        │                                           ▼
        │                               ┌─────────────────────┐
@@ -29,7 +29,6 @@ All containers join the `datalink` bridge network, enabling them to communicate
 | `influxdb3-data` | `influxdb3` | Persists InfluxDB 3 metadata and stored telemetry. |
 | `influxdb3-explorer-db` | `influxdb3-explorer` | Keeps explorer UI preferences. |
 | `grafana-storage` | `grafana` | Stores dashboards, plugins, and Grafana state. |
-| `telegraf-data` | `telegraf`, `startup-data-loader` | Shared folder for generated line protocol and importer progress. |
 
 Remove volumes with `docker compose down -v` if you need a clean slate.
 
@@ -66,4 +65,4 @@ docker compose logs -f startup-data-loader
 docker compose exec influxdb3 /bin/sh
 ```
 
-For detailed service documentation, browse the files under [`docs/containers/`](containers/).
+For detailed service documentation, browse the files under [`docs/containers/`](containers/).

installer/README.md

@@ -67,7 +67,6 @@ All secrets and tokens are defined in `.env`. The defaults provided in `.env.exa
 | `influxdb3` | `9000` (mapped to `8181` internally) | Core time-series database. Initialised with the admin token from `.env`. |
 | `influxdb3-explorer` | `8888` | Lightweight UI for browsing data in InfluxDB 3. |
 | `data-downloader` | `3000` | Periodically downloads CAN CSV archives from the DAQ server. Visual SQL query builder included. |
-| `telegraf` | n/a | Collects CAN metrics produced by the importer and forwards them to InfluxDB. |
 | `grafana` | `8087` | Visualises telemetry with pre-provisioned dashboards. |
 | `slackbot` | n/a | Socket-mode Slack bot for notifications and automation (optional). |
 | `lap-detector` | `8050` | Dash-based lap analysis web application. |
@@ -82,16 +81,15 @@ All secrets and tokens are defined in `.env`. The defaults provided in `.env.exa
 ## Observability
 
 - Grafana dashboards are provisioned automatically from `grafana/dashboards/` and use the datasource in `grafana/provisioning/datasources/`.
-- Telegraf writes processed metrics to `/var/lib/telegraf/can_metrics.out` before forwarding them to InfluxDB. Inspect this file inside the container for debugging (`docker compose exec telegraf tail -f /var/lib/telegraf/can_metrics.out`).
 
 ## Troubleshooting tips
 
 - **Service fails to connect to InfluxDB** – Confirm the token in `.env` matches `influxdb3-admin-token.json`. Regenerate the volumes with `docker compose down -v` if you rotate credentials.
-- **Re-import sample data** – Remove the `telegraf-data` volume and rerun the stack.
+- **Re-import sample data** – Run `docker compose down -v` and restart the stack to re-trigger the data loader.
 - **Slack services are optional** – Leave Slack variables empty or set `ENABLE_SLACK=false` to skip starting the bot during development.
 
 ## Next steps
 
 - Replace the example dataset and `example.dbc` file with production equivalents once you are ready to ingest real telemetry.
 - Update the Grafana dashboards under `grafana/dashboards/` to match your data model.
-- Review each service’s README in its respective directory for implementation details.
+- Review each service’s README in its respective directory for implementation details.

installer/docker-compose.yml

@@ -6,7 +6,6 @@ volumes:
   grafana-storage:
   influxdb3-data:
   influxdb3-explorer-db:
-  telegraf-data:
 
 services:
   influxdb3:
@@ -57,24 +56,6 @@ services:
     depends_on:
       - influxdb3
 
-  telegraf:
-    image: telegraf:1.30
-    container_name: telegraf
-    restart: unless-stopped
-    depends_on:
-      - influxdb3
-    environment:
-      INFLUXDB_TOKEN: "${INFLUXDB_ADMIN_TOKEN:-apiv3_dev-influxdb-admin-token}"
-      INFLUXDB_URL: "${INFLUXDB_URL:-http://influxdb3:8181}"
-    volumes:
-      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
-      - telegraf-data:/var/lib/telegraf
-    networks:
-      - datalink
-    command: >
-      sh -c "touch /var/lib/telegraf/can_metrics.out &&
-             telegraf --config /etc/telegraf/telegraf.conf"
-
   grafana:
     image: grafana/grafana
     container_name: grafana
@@ -148,15 +129,14 @@ services:
     environment:
       INFLUXDB_TOKEN: "${INFLUXDB_ADMIN_TOKEN:-apiv3_dev-influxdb-admin-token}"
       INFLUXDB_URL: "${INFLUXDB_URL:-http://influxdb3:8181}"
-      BACKFILL: "1"
+      # Removed BACKFILL env var as it's now the default/only mode
       CSV_RESTART_INTERVAL: "${CSV_RESTART_INTERVAL:-10}"
       # Fixed container path, host path remains flexible
       DBC_FILE_PATH: "/installer/example.dbc"
     volumes:
       # Host can name DBC file anything they want
       - ./${DBC_FILE_PATH:-example.dbc}:/installer/example.dbc:ro
       - ./startup-data-loader/data:/data:ro
-      - telegraf-data:/var/lib/telegraf
     working_dir: /app
     networks:
       - datalink
@@ -165,9 +145,7 @@ services:
     command: >
       sh -c "echo 'Waiting additional 5 seconds for InfluxDB 3 to stabilize...' &&
              sleep 5 &&
-             echo 'Creating telegraf input file...' &&
-             python load_data.py &&
-             echo 'Telegraf input file created successfully'"
+             python load_data.py"
 
 
   file-uploader:
@@ -245,4 +223,4 @@ services:
       - datalink
     depends_on:
       data-downloader-api:
-        condition: service_started
+        condition: service_started

installer/startup-data-loader/Dockerfile

@@ -15,11 +15,8 @@ RUN pip install --no-cache-dir -r requirements.txt
 # Keep the build context small by only copying the files needed to run the loader.
 COPY load_data.py requirements.txt ./
 
-# Create Telegraf output directory
-RUN mkdir -p /var/lib/telegraf
-
 # Make the script executable
 RUN chmod +x load_data.py
 
-# Run the Telegraf/direct-write loader
-CMD ["python", "load_data.py"]
+# Run the direct-write loader
+CMD ["python", "load_data.py"]

installer/startup-data-loader/README.md

@@ -7,7 +7,7 @@ This container pre-loads InfluxDB 3 with small CAN datasets whenever the compose
 1. Waits for InfluxDB 3 to pass its health check.
 2. Reads CSV files from the mounted `/data` directory (copy `2024-01-01-00-00-00.csv.md` to `2024-01-01-00-00-00.csv` for the sample dataset).
 3. Uses the shared `/installer/example.dbc` file (or the path specified by `DBC_FILE_PATH`) to decode each CAN frame into human-readable signals.
-4. Writes the decoded metrics to InfluxDB 3 (`WFR25` bucket, `WFR` organisation) and emits line protocol for Telegraf.
+4. Writes the decoded metrics directly to InfluxDB 3 (`WFR25` bucket, `WFR` organisation).
 5. Exits once all files finish processing.
 
 ## CSV format
@@ -22,9 +22,8 @@ relative_ms,protocol,can_id,byte0,byte1,byte2,byte3,byte4,byte5,byte6,byte7
 
 | Variable | Description |
 | --- | --- |
-| `INFLUXDB_TOKEN` | Token used for direct writes when `BACKFILL=1` (injected from `.env`). |
+| `INFLUXDB_TOKEN` | Token used for direct writes (injected from `.env`). |
 | `INFLUXDB_URL` | URL for the InfluxDB 3 instance (defaults to `http://influxdb3:8181`). |
-| `BACKFILL` | Set to `1` to stream directly into InfluxDB; set to `0` to only generate line protocol for Telegraf. |
 | `CSV_RESTART_INTERVAL` | Number of CSV files to process before the loader re-execs itself (defaults to `10`; set to `0` to disable). |
 
 ## Adding real data
@@ -41,4 +40,4 @@ Check progress with:
 docker compose logs -f startup-data-loader
 ```
 
-The loader also tracks its state in `load_data_progress.json` inside the container so that it can resume large imports after interruptions.
+The loader also tracks its state in `load_data_progress.json` inside the container so that it can resume large imports after interruptions.