Prometheus service stops

[robleed]

Hello All,

I am running a Prometheus server as the database collector, it seems to be running fine for a few days and then it fails/stops. This is what is being displayed. Was wondering if anyone has encountered this issue:

● prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Sun 2024-09-22 10:25:37 EDT; 1 day 2h ago
Process: 1378 ExecStart=/usr/local/bin/prometheus --web.enable-remote-write-receiver --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/console>
Main PID: 1378 (code=killed, signal=KILL)

Sep 22 10:14:01 uspilvlpcsmap01.ey.net prometheus[1378]: ts=2024-09-22T14:14:01.153Z caller=db.go:1074 level=error component=tsdb msg="compaction failed" err="mkdir /var/lib/prometheus/01J8D1A45MRESTMQZQJEWCN8KY.tmp-for->
Sep 22 10:15:01 uspilvlpcsmap01.ey.net prometheus[1378]: ts=2024-09-22T14:15:01.164Z caller=db.go:1074 level=error component=tsdb msg="compaction failed" err="mkdir /var/lib/prometheus/01J8D1BYS6V1Z1VFNPCAAG84RQ.tmp-for->
Sep 22 10:16:01 uspilvlpcsmap01.ey.net prometheus[1378]: ts=2024-09-22T14:16:01.177Z caller=db.go:1074 level=error component=tsdb msg="compaction failed" err="mkdir /var/lib/prometheus/01J8D1DSCGR2X2EWV0WESECRP0.tmp-for->
Sep 22 10:17:01 uspilvlpcsmap01.ey.net prometheus[1378]: ts=2024-09-22T14:17:01.186Z caller=db.go:1074 level=error component=tsdb msg="compaction failed" err="mkdir /var/lib/prometheus/01J8D1FKZX53FAKYQM7ZHG7YC1.tmp-for->
Sep 22 10:18:01 uspilvlpcsmap01.ey.net prometheus[1378]: ts=2024-09-22T14:18:01.194Z caller=db.go:1074 level=error component=tsdb msg="compaction failed" err="mkdir /var/lib/prometheus/01J8D1HEK5V11M7XW0DWMCYKDX.tmp-for->
Sep 22 10:19:01 uspilvlpcsmap01.ey.net prometheus[1378]: ts=2024-09-22T14:19:01.207Z caller=db.go:1074 level=error component=tsdb msg="compaction failed" err="mkdir /var/lib/prometheus/01J8D1K96FNTQFAN07KAMGFS0K.tmp-for->
Sep 22 10:20:01 uspilvlpcsmap01.ey.net prometheus[1378]: ts=2024-09-22T14:20:01.224Z caller=db.go:1074 level=error component=tsdb msg="compaction failed" err="mkdir /var/lib/prometheus/01J8D1N3SZP0DXSE8ZF5TT2NJQ.tmp-for->
Sep 22 10:21:01 uspilvlpcsmap01.ey.net prometheus[1378]: ts=2024-09-22T14:21:01.234Z caller=db.go:1074 level=error component=tsdb msg="compaction failed" err="mkdir /var/lib/prometheus/01J8D1PYDCC8NYGG4D70DC494N.tmp-for->
Sep 22 10:25:37 uspilvlpcsmap01.ey.net systemd[1]: prometheus.service: Main process exited, code=killed, status=9/KILL
Sep 22 10:25:37 uspilvlpcsmap01.ey.net systemd[1]: prometheus.service: Failed with result 'signal'.

I am searching the internet for -- prometheus.service: Failed with result 'signal'.
I am not finding anything to help me resolve this issue.
I am able to restart the prometheus service and it starts without any issues.
If anyone encountered this can you please let me know how you resolved it.