[OpenSTEF 4.0] Memory profiling and efficiency improvements

[MvLieshout]

As
I want to
So that I

🌍 Background

Important

Changes should be done by forking OpenSTEF 4.0.0 tracking branch (release/v4.0.0) and then make a PR back onto release/v4.0.0 branch.
Please read the contributing guide before you start.

memray-flamegraph-liander_2024_benchmark_xgboost_gblinear.py.27754.html

Ideas for improvements:

• Check html writing efficiency, (fig.write_html(path, include_plotlyjs='cdn') # Use CDN instead of embedding)
• Check if deepcopies can be replaced by copies.

❗Priority (What if we don't do this?/Are there any deadlines? etc.)

Definition of Done:

✅ Acceptance criteria

• [ ]

📄 Documentation criteria:

Interfaces with other products and services

Update relevant confluence pages

🧪 Test criteria:

Unit tests written and passed

⌛ Dependencies:

N/A

🚀 Releasing:

N/A

Other information: