Visualisations

.github/workflows/run-tests.yml

@@ -1,7 +1,7 @@
 # This workflow installs required Python dependencies and then runs the available tests.
 # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
 
-name: Run Acceptance and Unit tests
+name: Run Acceptance and Unit tests (with and without visualisation dependencies)
 
 on:
   pull_request:
@@ -21,10 +21,18 @@ jobs:
       uses: actions/setup-python@v3
       with:
         python-version: "3.10"  # Only the oldest supported Python version is included here
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip   # upgrade pip to latest version
-        pip install .                         # install pyproject.toml dependencies - excludes optional dependencies (such as visualisation)
-    - name: Run tests
+
+    - name: Install dependencies (without extra visualisation dependencies)
       run: |
-        python run_tests.py
+        python -m pip install --upgrade pip           # upgrade pip to latest version
+        pip install .                                 # no additional [visualization] dependencies
+
+    - name: Run tests (without visualisation dependencies)
+      run:  python run_tests.py
+
+    - name: Install extra visualisation dependencies
+      run:  pip install ".[visualization]"            # extra [visualization] dependencies in pyproject.toml
+
+    - name: Run Tests (with visualisation dependencies)
+      run:  python run_tests.py
+

.gitignore

@@ -6,6 +6,10 @@ __pycache__/
 # C extensions
 *.so
 
+# VPP (Visual Paradigm) files
+*.vpp.bak_*
+*.vpp.lck
+
 # Distribution / packaging
 .Python
 build/
@@ -35,6 +39,9 @@ results/
 *.manifest
 *.spec
 
+# Ignore pyenv Pipfile 
+Pipfile
+
 # Installer logs
 pip-log.txt
 pip-delete-this-directory.txt
@@ -152,4 +159,7 @@ cython_debug/
 #  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
+.idea/
+
+# json documents for graphs
+json/

CONTRIBUTING.md

@@ -33,7 +33,6 @@ When reporting a defect, be precise and concise in your description. Write in wa
 Note that all information in the issue tracker is public. *Do not include any confidential information there*.
 
 Be sure to add information about:
-
 - The applicable version(s) of RobotMBT (use `pip list` and check for `robotframework-mbt`)
 - Your Robot Framework version (use `pip list` and check for `robotframework`)
 - Your Python version (check using `python --version`)
@@ -114,3 +113,117 @@ Researchers have suggested that longer lines are better suited for cases when th
   - Information that is useful for analysing failed tests is logged at debug-level.
 
 - Be careful not to make assumptions in what you log: Recheck log statements if your changes affect the context in which the code is run, and only report about what you know to be true.
+
+### Creating new graphs
+
+Extending the functionality of the visualizer with new graph types can result in better insights into created tests. The visualizer makes use of an abstract graph class that makes it easy to create new graph types.
+
+To create a new graph type, create an instance of `robotmbt/visualise/graphs/AbstractGraph`, instantiating the abstract methods. Please place the graph under `robotmbt/visualise/graphs/`.
+
+**NOTE**: when manually altering the `networkx` field, ensure its IDs remain as a serializable and hashable type when the constructor finishes.
+
+As an example, we show the implementation of the scenario graph below. In this graph type, nodes represent scenarios encountered in exploration, and edges show the flow between these scenarios.
+It does not enable tooltips.
+
+```python
+class ScenarioGraph(AbstractGraph[ScenarioInfo, None]):
+    @staticmethod
+    def select_node_info(trace: list[tuple[ScenarioInfo, StateInfo]], index: int) -> ScenarioInfo:
+        return trace[index][0]
+
+    @staticmethod
+    def select_edge_info(pair: tuple[ScenarioInfo, StateInfo]) -> None:
+        return None
+
+    @staticmethod
+    def create_node_description(trace: list[tuple[ScenarioInfo, StateInfo]], index: int) -> str:
+        return ''
+
+    @staticmethod
+    def create_node_label(info: ScenarioInfo) -> str:
+        return info.name
+
+    @staticmethod
+    def create_edge_label(info: None) -> str:
+        return ''
+
+    @staticmethod
+    def get_legend_info_final_trace_node() -> str:
+        return "Executed Scenario (in final trace)"
+
+    @staticmethod
+    def get_legend_info_other_node() -> str:
+        return "Executed Scenario (backtracked)"
+
+    @staticmethod
+    def get_legend_info_final_trace_edge() -> str:
+        return "Execution Flow (final trace)"
+
+    @staticmethod
+    def get_legend_info_other_edge() -> str:
+        return "Execution Flow (backtracked)"
+
+    @staticmethod
+    def get_tooltip_name() -> str:
+        return ""
+```
+
+Once you have created a new Graph class, you can direct the visualizer to use it when your type is selected. 
+Simply add your class to the `GRAPHS` dictionary in `robotmbt/visualise/visualiser.py` like the others. For our example:
+
+```python
+GRAPHS = {
+    ...
+    'scenario': ScenarioGraph,
+    ...
+}
+```
+
+Now, when selecting your graph type (in our example `Treat this test suite Model-based  graph=scenario`), your graph will get constructed!
+
+
+## Development Tips
+### Python virtual environment
+Installing the proper virtual environment can be done with the default `python -m venv ./.venv` command built into python. However, if you have another version of python on your system, this might break dependencies.
+
+#### Pipenv+Pyenv (verified on Windows and Linux)
+For the optimal experience (at least on Linux), we suggest installing the following packages:
+- [`pyenv`](https://github.com/pyenv/pyenv) (Linux/Mac) or [`pyenv-win`](https://github.com/pyenv-win/pyenv-win) (Windows)
+- [`pipenv`](https://github.com/pypa/pipenv) 
+
+Then, you can install a python virtual environment with:
+
+```bash
+pipenv --python <python_version>
+```
+..where the python version can be found in the `pyproject.toml`. For example, for 3.10: `pipenv --python 3.10`.
+
+You might need to manually make the folder `.venv` by doing `mkdir .venv`.
+
+You can verify if the installation went correctly with:
+```bash
+pipenv check
+```
+This should return `Passed!`
+
+Errors related to minor versions (for example `3.10.0rc2` != `3.10.0`) can be ignored.
+
+Now activate the virtual environment by running 
+```bash 
+pipenv shell
+```
+
+..and you should have a virtual env! If you run
+```bash
+python --version
+```
+..while in your virtual environment, it should show the `<python_version>` from before.
+
+
+### Installing dependencies
+***NOTE: making sure that you are in the virtual environment***. 
+
+It is recommended that you also include the optional dependencies for visualisation, e.g.:
+```bash
+pip install ".[visualization]"
+```

README.md

@@ -36,7 +36,7 @@ The recommended installation method is using [pip](http://pip-installer.org)
 
 After installation include `robotmbt` as library in your robot file to get access to the new functionality. To run your test suite model-based, use the __Treat this test suite model-based__ keyword as suite setup. Check the _How to model_ section to learn how to make your scenarios suitable for running model-based.
 
-```
+```robotframework
 *** Settings ***
 Library        robotmbt
 Suite Setup    Treat this test suite model-based
@@ -50,7 +50,8 @@ Modelling can be done directly from [Robot framework](https://robotframework.org
 
 Consider these two scenarios:
 
-```
+```robotframework
+*** Test Cases ***
 Buying a postcard
     When you buy a new postcard
     then you have a blank postcard
@@ -63,7 +64,8 @@ Preparing for a birthday party
 
 Mapping the dependencies between scenarios is done by annotating the steps with modelling info. Modelling info is added to the documentation of the step as shown below. Regular documentation can still be added, as long as `*model info*` starts on a new line and a white line is included after the last `:OUT:` expression.
 
-```
+```robotframework
+*** Keywords ***
 you buy a new postcard
     [Documentation]    *model info*
     ...    :IN: None
@@ -117,7 +119,8 @@ All example scenarios naturally contain data. This information is embedded in th
 
 #### Step argument modifiers
 
-```
+```robotframework
+*** Test Cases ***
 Personalising a birthday card
     Given there is a birthday card
     when Johan writes their name on the birthday card
@@ -126,7 +129,8 @@ Personalising a birthday card
 
 The above scenario uses the name `Johan` to create a concrete example. But now suppose that from a testing perspective `Johan` and `Frederique` are part of the same equivalence class. Then the step `Frederique writes their name on the birthday card` would yield an equally valid scenario. This can be achieved by adding a modifier (`:MOD:`) to the model info of the step. The format of a modifier is a Robot argument to which you assign a list of options. The modifier updates the argument value to a randomly chosen value from the specified options.
 
-```
+```robotframework
+*** Keywords ***
 ${person} writes their name on the birthday card
     [Documentation]    *model info*
     ...    :MOD: ${person}= [Johan, Frederique]
@@ -138,7 +142,8 @@ ${person} writes their name on the birthday card
 
 When constructing examples, they often express relations between multiple actors, where each actor can appear in multiple steps. This makes it important to know how modifiers behave when there are multiple modifiers in a scenario.
 
-```
+```robotframework
+*** Test Cases ***
 Addressing a birthday card
    Given Tannaz is having their birthday
    and Johan has a birthday card
@@ -148,7 +153,8 @@ Addressing a birthday card
 
 Have a look at the when-step above. We will assume the model already contains a domain term with two properties: `birthday.celebrant = Tannaz` and `birthday.guests = [Johan, Frederique]`.
 
-```
+```robotframework
+*** Keywords ***
 ${sender} writes the address of ${receiver} on the birthday card
     [Documentation]    *model info*
     ...    :MOD: ${sender}= birthday.guests
@@ -175,10 +181,12 @@ It is not possible to add new options to an existing example value. Any constrai
 
 It is possible for a step to keep the same options. The special `.*` notation lets you keep the available options as-is. Preceding steps must then supply the possible options. Some steps can, or must, deal with multiple independent sets of options that must not be mixed, because the expected results should differ. Suppose you have a set of valid and invalid passwords. You might be reluctant to include the superset of these as options to an authentication step. Instead, you can use `:MOD: ${password}= .*` as the modifier for that step. Like in the when-step for this scenario:
 
-```
-Given 'secret' is too weak a password
-When user tries to update their password to 'secret'
-then the password is rejected
+```robotframework
+*** Test Cases ***
+Reject password
+    Given 'secret' is too weak a password
+    When user tries to update their password to 'secret'
+    then the password is rejected
 ```
 
 In a then-step, modifiers behave slightly different. In then-steps no new option constraints are accepted for an argument. Its value must already have been determined during the given- and when-steps. In other words, regardless of the actual modifier, the expression behaves as if it were `.*`. The exception to this is when a then-step signals the first use of a new example value. In that case the argument value from the original scenario text is used.
@@ -193,12 +201,45 @@ For now, variable data considers strict equivalence classes only. This means tha
 
 By default, trace generation is random. The random seed used for the trace is logged by _Treat this test suite model-based_. This seed can be used to rerun the same trace, if no external random factors influence the test run. To activate the seed, pass it as argument:
 
-```
+```robotframework
 Treat this test suite model-based    seed=eag-etou-cxi-leamv-jsi
 ```
 
 Using `seed=new` will force generation of a new reusable seed and is identical to omitting the seed argument. To completely bypass seed generation and use the system's random source, use `seed=None`. This has even more variation but does not produce a reusable seed.
 
+
+### Graphs
+
+By default, no graphs are generated for test-runs. For development purposes, having a visual representation of the test-suite you are working on can be very useful. To have robotmbt generate a graph, ensure you have installed the optional dependencies (`pip install .[visualization]`) and pass the type as an argument:
+
+```robotframework
+Treat this test suite Model-based  graph=<type>
+```
+
+Here, `<type>` can be any of the supported graph types, which can be seen in `robotmbt/visualise/visualiser.py`.
+
+Once the test suite has run, a graph will be included in the test's log, under the suite's `Treat this test suite Model-based` setup header.
+
+#### JSON exporting
+
+It is possible to extract the exploration data after the library has found a covering trace. To enable this feature, set the following argument to true:
+
+```robotframework
+Treat this test suite Model-based  export_graph_data=<directory>
+```
+
+A JSON file named after the test suite will be created containing said information.
+
+#### JSON importing
+
+It is possible to skip running the exploration step and produce a graph (e.g. of another type) from previously exported data.
+
+```robotframework
+Treat this test suite Model-based  graph=<type>  import_graph_data=<directory and file_name>.json
+```
+
+A graph will be created from the imported data.
+
 ### Option management
 
 If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite.

atest/resources/helpers/__init__.py

@@ -0,0 +1,29 @@
+# BSD 3-Clause License
+#
+# Copyright (c) 2026, T.B. Dubbeling,  J. Foederer, T.S. Kas, D.R. Osinga, D.F. Serra e Silva, J.C. Willegers
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this
+#    list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+#    this list of conditions and the following disclaimer in the documentation
+#    and/or other materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its
+#    contributors may be used to endorse or promote products derived from
+#    this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.