Skip to content

fix end2end testing #444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nicHoch merged 3 commits into from
Nov 24, 2025
Merged

fix end2end testing #444

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
854f277
Add step to show installed Python packages
nicHoch Nov 24, 2025
544688d
fix back to typo needs to be addressed in its own issue
nicHoch Nov 24, 2025
a9f095e
fixes after review
nicHoch Nov 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/end2end_run.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ jobs:
python -m pip install --upgrade pip
python -m pip install -e .[dev]
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: show environment
run: python -m pip freeze
- name: end2end test with pytest
run: |
pytest -p no:warnings --doctest-rst -m "end2end" --pyargs stixcore --remote-data=any
4 changes: 2 additions & 2 deletions docs/pipelineconfiguration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ SOLMOC TM data is synced from pub026 via rsync by cron every 15min.
.. code-block::

# sync TM data from GFTS to local "wait for processing" folder
*/15 * * * * rsync -av stixcore@147.86.8.26:'/home/solmoc/from_edds/tm/*PktTmRaw*.xml' /data/stix/SOLSOC/from_edds/tm/incoming > /dev/null
*/15 * * * * rsync -av stixcore@147.86.8.26:'/home/solmoc/from_edds/tm/*PktTmRaw*.xml' /data/stix/SOLSOC/from_edds/tm/incomming > /dev/null

The very latest SPICE kernels are required for accurate pointing information of the satellite.
We may receive TM files via GFTS that contain data for times that are not available in the latest SPICE kernels, as the latest SPICE kernels have not yet been updated and delivered.
Expand All @@ -53,7 +53,7 @@ Therefore, a stage and delay step is added via cron (every 30 min) that only pub
.. code-block::

# move the TM data from the wait into the processing folder - that will trigger the pipeline to "get started": the wait period is meanwhile short
*/30 * * * * find /data/stix/SOLSOC/from_edds/tm/incoming/ -type f -mmin +2 -exec rsync -a {} /data/stix/SOLSOC/from_edds/tm/processing/ \;
*/30 * * * * find /data/stix/SOLSOC/from_edds/tm/incomming/ -type f -mmin +2 -exec rsync -a {} /data/stix/SOLSOC/from_edds/tm/processing/ \;

Sync STIX-CONF repo in all used instances
*****************************************
Expand Down
2 changes: 1 addition & 1 deletion stixcore/util/scripts/end2end_testing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ def end2end_pipeline(indir, fitsdir):
splits=1,
outdir=datapath,
socdir=Path(
"/data/stix/SOLSOC/from_edds/tm/incoming/",
"/data/stix/SOLSOC/from_edds/tm/incomming/",
),
)

Expand Down
2 changes: 1 addition & 1 deletion stixcore/util/scripts/find_unprocessed_tm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
TM_REGEX = re.compile(r".*PktTmRaw.*.xml$")
logging_dir = Path(CONFIG.get("Pipeline", "log_dir"))
tm_dir = Path(CONFIG.get("Paths", "tm_archive"))
tm_dir = Path("/data/stix/SOLSOC/from_edds/tm/incoming")
tm_dir = Path("/data/stix/SOLSOC/from_edds/tm/incomming")
list_of_log_files = logging_dir.glob("*.xml.out")
latest_log_file = (
logging_dir / "TM_BatchRequest.PktTmRaw.SOL.0.2023.305.15.33.13.852.bBWz@2023.352.21.00.01.039.1.xml.out"
Expand Down