Development environment for RelMonService

.gitignore

@@ -115,3 +115,6 @@ relmons/*
 
 # pid file
 *.pid
+
+# Docker deployment
+deploy/mongo/data

deploy/README.md

@@ -0,0 +1,37 @@
+# Deploy a development version
+
+> [!IMPORTANT]
+> There are several prerequisites you must fulfill before deploying a development version of this tool:
+
+- Use a development runtime environment reachable inside the CERN general purpose network as the HTCondor jobs need to be able to send callback signals.
+- Have a CERN account with access to the [lxplus](https://lxplusdoc.web.cern.ch/) service or to a VM node located inside the CERN general purpose network.
+- Have an X.509 certificate to authenticate to the CMS Web Services configured in your CERN account. Please follow these guidelines to request one if required [Personal certificates from CERN](https://twiki.cern.ch/twiki/bin/view/CMSPublic/PersonalCertificate) - [Basic requirements for using the Grid](https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookStartingGrid#BasicGrid).
+- Have `docker` and `docker compose` available in your development environment. If you do not have it already, you could install by following the guidelines at: [Install Docker Engine](https://docs.docker.com/engine/install/).
+- Have `python3.11` installed for your development environment. If you do not have it already, you could install it via [pyenv](https://github.com/pyenv/pyenv).
+- Have `node` installed for your development environment. It is recommended to use the latest LTS version, nevertheless any version `node > 16` should work. If you do not have it already, you could install it via [nvm](https://github.com/nvm-sh/nvm).
+- Have a copy of this repository, fork it.
+
+After you fulfill these prerequisites, follow the next steps:
+
+1. Clone the forked repository you created before
+2. Edit the file `deploy/env.sh` and complete the `TODO` items available there.
+3. Source the environment variables via `source deploy/env.sh`.
+4. Edit the file `deploy/nginx/conf.d/proxy.conf` and complete the `TODO` items available there.
+5. Deploy the application's components via `docker compose -f deploy/compose.yml up -d`.
+
+At this stage, you have deployed the database and a reverse proxy to forward and authenticate the requests. Next, let's deploy the RelMonService2 application.
+
+6. Create a virtual environment, activate the environment and install the required dependencies: `python3.11 -m venv venv && source ./venv/bin/activate && pip install -r requirements.txt`
+7. Install `node` dependencies: `cd frontend/ && npm i && cd ..`
+
+To conclude, deploy the RelMon Report page. This web server will be used to render the generated reports available via SQLite files. To ease this step, deploy an Apache server via the [CERN Web Services](https://webservices-portal.web.cern.ch/) portal using WebEOS site.
+
+8. Set the category of this site as `Test`.
+9. Create a folder inside the `/eos` personal filesystem for the `SERVICE_ACCOUNT_USERNAME` account. This will be the path you will set in the `WEB_LOCATION_PATH` variable.
+10. Complete the required steps to share the folder with the web server - [Details](https://cernbox.docs.cern.ch/advanced/web-pages/personal_website_content/).
+11. Enable `.htaccess` files for the web server.
+12. Copy all the content available in the folder `report_website/` to `${WEB_LOCATION_PATH}/`. Do not forget to copy the `.htaccess` file too.
+
+With this steps, the web application should have been deployed successfully and you're ready to continue with your development tasks!
+
+13. Start the development application via: `./relmonsvc.sh dev`. Press `Ctrl+C` to stop the services.

deploy/compose.yml

@@ -0,0 +1,22 @@
+name: relmonservice2
+
+services:
+  nginx:
+    image: nginx:2.26.3-alpine3.20-perl@sha256:ecf827c698f16db476cdf5ebd094881b7fde88514a1cbf0d11780bdd8120041f
+    volumes:
+      - "${PWD}/deploy/nginx/conf.d:/etc/nginx/conf.d:ro"
+      - "${PWD}/report_website:/var/www/html:ro"
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    ports:
+      - "10000:10000"
+  mongodb:
+    image: mongodb/mongodb-community-server:7.0.7-ubi9@sha256:0ab7391ec61a618ff2d9b999146812e0c1c29631cdc86bc096c60c77ba9e2cfb
+    user: "${UID}:${GID}"
+    environment:
+      - "MONGODB_INITDB_ROOT_USERNAME=${MONGO_DB_USER}"
+      - "MONGODB_INITDB_ROOT_PASSWORD=${MONGO_DB_PASSWORD}"
+    volumes:
+      - "${PWD}/deploy/mongo/data:/data/db"
+    ports:
+      - "${MONGO_DB_PORT}:27017"

deploy/env.sh

@@ -0,0 +1,78 @@
+# !/bin/bash
+# This script sets the required environment variables for
+# deploying the application. Source its content!
+
+# RelMonService2 endpoints
+export CALLBACK_URL="http://$(hostname):10000/relmonservice/api/update"
+export SERVICE_URL="http://$(hostname):10000/relmonservice"
+
+# RelMon Reports page
+# TODO: Set the web server url for related to the WebEOS site
+# deployed via CERN Web Services. This URL will be something like
+# e.g: relmon-mytest.web.cern.ch
+export REPORTS_URL=""
+
+# Submission host to access HTCondor
+# In case you have issues opening the SSH session, change
+# this property to a specific node inside the lxplus8 pool.
+export SUBMISSION_HOST="lxplus8.cern.ch"
+
+# Default CMSSW release for picking the RelMon generation module.
+export CMSSW_RELEASE='CMSSW_14_1_0_pre7'
+
+# The following directory is relative to $HOME for the
+# user used in the $SUBMISSION_HOST. Make sure the directory exists!
+# e.g: export REMOTE_DIRECTORY="test/relmonservice/jobs/"
+# TODO: Create the folder and assign the relative path.
+export REMOTE_DIRECTORY=""
+
+# Service account to access the submission host
+# TODO: Set the username for opening the SSH session.
+export SERVICE_ACCOUNT_USERNAME=""
+
+# Service account password, provide the real one only if
+# your runtime environment does not have Kerberos enabled.
+export SERVICE_ACCOUNT_PASSWORD="NotRequiredToSet"
+
+# Absolute path to a folder to store the reports
+# This is the folder you will link on the WebEOS site for rendering the
+# reports.
+# This folder must be reachable from the submission host and the HTCondor
+# pool running the job so it can copy the file directly.
+# e.g: export WEB_LOCATION_PATH="/eos/user/u/user/test/relmonservice/reports"
+#
+# TODO: Set the WebEOS site path
+export WEB_LOCATION_PATH=""
+
+# MongoDB configuration
+export MONGO_DB_HOST="127.0.0.1"
+
+# TODO: Set the MongoDB port to be exported in the host scope!
+export MONGO_DB_PORT=""
+
+# TODO: Set the MongoDB administrator user
+export MONGO_DB_USER=""
+
+# TODO: Set its password
+export MONGO_DB_PASSWORD=""
+
+# Flask configuration for session cookies
+export SECRET_KEY="$(openssl rand -base64 64 | sed 's#/#!#g')"
+
+# The following is used to perform callbacks, set a real value
+# in case an authentication middleware is set for your 
+# development deployment.
+
+# Related to the RelmonService2 deployment.
+export CLIENT_ID="NotRequiredToSet"
+
+# Credentials for requesting a token.
+export CALLBACK_CLIENT_ID="NotRequiredToSet"
+export CALLBACK_CLIENT_SECRET="NotRequiredToSet"
+export DISABLE_CALLBACK_CREDENTIALS="True"
+
+# The following is required by Docker Compose
+# to properly match the running host user so that host
+# volumes access is granted properly
+export UID=$(id -u)
+export GID=$(id -g)

deploy/mongo/data/.keep

@@ -0,0 +1,3 @@
+// This is just a dummy file to instruct Git to include
+// this folder. The purpose of this subfolder is storing the
+// data of your MongoDB deployment.

deploy/nginx/conf.d/proxy.conf

@@ -0,0 +1,21 @@
+server {
+    listen 10000;
+    root /var/www/html;
+
+    # RelMonService 2
+    location /relmonservice {
+        # Authentication headers
+        proxy_set_header Adfs-Group "cms-pdmv-serv";
+        proxy_set_header Adfs-Login ""; # TODO: Set your username
+        proxy_set_header Adfs-Fullname ""; # TODO: Set your fullname
+        proxy_set_header Adfs-Firstname ""; # TODO: Set your first name
+        proxy_set_header Adfs-Lastname ""; # TODO: Set your last name
+        proxy_set_header Adfs-Email ""; # TODO: Set your email address
+
+        # The following is a reference to the localhost scope
+        # in the host machine.
+        proxy_pass http://host.docker.internal:8000/;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+    }
+}

environment.py

@@ -44,6 +44,16 @@
     CALLBACK_CLIENT_ID (str): Client ID for CLI integration application.
     CALLBACK_CLIENT_SECRET (str): Client secret for CLI integration application.
     CMSSW_RELEASE (str): cms-sw version to use for generating the monitoring report.
+    HTCONDOR_CAF_POOL (bool): If this environment variable is provided,
+        RelMon batch jobs will be configured to run inside the dedicated pool CMS CAF.
+        Otherwise, they will run in the public shared pool.
+    FILE_CREATOR_GIT_SOURCE (str): RelMonService2 source code to load inside the
+        HTCondor batch jobs.
+    FILE_CREATOR_GIT_BRANCH (str): Branch to use from `FILE_CREATOR_GIT_SOURCE`.
+    _CMSSW_CUSTOM_REPO (str): This is an optional setting, allows users to
+        compile the RelMon module from a custom source instead of using `cmssw` releases.
+    _CMSSW_CUSTOM_BRANCH (str): If `_CMSSW_CUSTOM_REPO` is set, this is the branch
+        for taking the code from.
 """
 import os
 import inspect
@@ -78,12 +88,27 @@
 CLIENT_ID: str = os.getenv("CLIENT_ID", "")
 CALLBACK_CLIENT_ID: str = os.getenv("CALLBACK_CLIENT_ID", "")
 CALLBACK_CLIENT_SECRET: str = os.getenv("CALLBACK_CLIENT_SECRET", "")
+DISABLE_CALLBACK_CREDENTIALS = bool(os.getenv("DISABLE_CALLBACK_CREDENTIALS"))
+
+# HTCondor submission pool
+HTCONDOR_CAF_POOL = bool(os.getenv("HTCONDOR_CAF_POOL"))
+HTCONDOR_MODULE = "lxbatch/tzero" if HTCONDOR_CAF_POOL else "lxbatch/share"
+
+# Repository source for the remote execution in HTCondor.
+FILE_CREATOR_GIT_SOURCE: str = os.getenv("FILE_CREATOR_GIT_SOURCE", "https://github.com/cms-PdmV/relmonservice2.git")
+FILE_CREATOR_GIT_BRANCH: str = os.getenv("FILE_CREATOR_GIT_BRANCH", "master")
+
+# Custom `cmssw` sources for development
+# Use this to override the RelMon module from a custom source
+# instead of using a release
+_CMSSW_CUSTOM_REPO: str = os.getenv("CMSSW_CUSTOM_REPO", "")
+_CMSSW_CUSTOM_BRANCH: str = os.getenv("CMSSW_CUSTOM_BRANCH", "")
 
 # Check that all environment variables are provided
 missing_environment_variables: dict[str, str] = {
     k: v
     for k, v in globals().items()
-    if not k.startswith("__")
+    if not k.startswith("_")
     and not inspect.ismodule(v)
     and not isinstance(v, bool)
     and not v

local/controller.py

@@ -11,14 +11,18 @@
 import json
 from multiprocessing import Manager
 from mongodb_database import Database
-from local.ssh_executor import SSHExecutor
+from core_lib.utils.ssh_executor import SSHExecutor
 from local.relmon import RelMon
 from local.file_creator import FileCreator
 from local.email_sender import EmailSender
 from environment import (
+    SUBMISSION_HOST,
+    SERVICE_ACCOUNT_USERNAME,
+    SERVICE_ACCOUNT_PASSWORD,
     SERVICE_URL,
     REPORTS_URL,
     REMOTE_DIRECTORY,
+    HTCONDOR_MODULE
 )
 
 
@@ -54,7 +58,11 @@ def set_config(self):
         if self.remote_directory[-1] == "/":
             self.remote_directory = self.remote_directory[:-1]
 
-        self.ssh_executor = SSHExecutor()
+        self.ssh_executor = SSHExecutor(
+            host=SUBMISSION_HOST,
+            username=SERVICE_ACCOUNT_USERNAME,
+            password=SERVICE_ACCOUNT_PASSWORD
+        )
         self.file_creator = FileCreator()
         self.email_sender = EmailSender()
         self.service_url = SERVICE_URL
@@ -183,8 +191,7 @@ def rename_relmon_reports(self, relmon_id, new_name):
         """
         Rename relmon reports file
         """
-        ssh_executor = SSHExecutor()
-        ssh_executor.execute_command(
+        self.ssh_executor.execute_command(
             [
                 "cd %s" % (self.file_creator.web_location),
                 "EXISTING_REPORT=$(ls -1 %s*.sqlite | head -n 1)" % (relmon_id),
@@ -327,12 +334,12 @@ def __submit_to_condor(self, relmon, database):
             # Run condor_submit
             # Submission happens through lxplus as condor is not available on website machine
             # It is easier to ssh to lxplus than set up condor locally
-            stdout, stderr = self.ssh_executor.execute_command(
+            stdout, stderr, _ = self.ssh_executor.execute_command(
                 [
                     "cd %s" % (remote_relmon_directory),
                     "voms-proxy-init -voms cms --valid 24:00 --out $(pwd)/proxy.txt",
-                    "module load lxbatch/tzero && condor_submit RELMON_%s.sub"
-                    % (relmon_id),
+                    "module load %s && condor_submit RELMON_%s.sub"
+                    % (HTCONDOR_MODULE, relmon_id),
                 ]
             )
             # Parse result of condor_submit
@@ -369,9 +376,9 @@ def __check_if_running(self, relmon, database):
         self.logger.info(
             "Will check if %s is running in HTCondor, id: %s", relmon, relmon_condor_id
         )
-        stdout, stderr = self.ssh_executor.execute_command(
-            "module load lxbatch/tzero && condor_q -af:h ClusterId JobStatus | "
-            "grep %s" % (relmon_condor_id)
+        stdout, stderr, _ = self.ssh_executor.execute_command(
+            "module load %s && condor_q -af:h ClusterId JobStatus | "
+            "grep %s" % (HTCONDOR_MODULE, relmon_condor_id)
         )
         new_condor_status = "<unknown>"
         if stdout and not stderr:
@@ -506,7 +513,7 @@ def __terminate_relmon(self, relmon):
         condor_id = relmon.get_condor_id()
         if condor_id > 0:
             self.ssh_executor.execute_command(
-                "module load lxbatch/tzero && condor_rm %s" % (condor_id)
+                "module load %s && condor_rm %s" % (HTCONDOR_MODULE, condor_id)
             )
         else:
             self.logger.info(