Skip to content

Conversation

@jonas-becker
Copy link
Collaborator

vllm required the correct model name to be set, which was falsely defaulted to gpt-3.5 before.

Also fixes the logprobs with vllm

Removed the status check in the config to cause fewer problems with differently working backends

@jonas-becker jonas-becker self-assigned this Jun 2, 2025
@jonas-becker jonas-becker requested a review from lkaesberg June 2, 2025 14:58
@github-actions
Copy link
Contributor

github-actions bot commented Jun 2, 2025

Coverage Report

Name                                                        Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------------------------
mallm/__init__.py                                               1      0      0      0   100%
mallm/agents/__init__.py                                        0      0      0      0   100%
mallm/agents/agent.py                                          90     46     20      3    45%
mallm/agents/draftProposer.py                                  11      1      2      1    85%
mallm/agents/judge.py                                          59     44     20      1    20%
mallm/agents/panelist.py                                       11      6      2      0    38%
mallm/coordinator.py                                          163     73     54      7    52%
mallm/decision_protocol/__init__.py                             0      0      0      0   100%
mallm/decision_protocol/approval_voting.py                     47     31     10      0    28%
mallm/decision_protocol/consensus.py                           52     11     10      1    74%
mallm/decision_protocol/consensus_voting.py                    74     56     20      0    19%
mallm/decision_protocol/cumulative_voting.py                   57     39     18      0    24%
mallm/decision_protocol/protocol.py                           150    117     60      1    16%
mallm/decision_protocol/ranked_voting.py                       45     30     12      0    26%
mallm/decision_protocol/simple_voting.py                       47     31     10      0    28%
mallm/decision_protocol/summary.py                             29     14      4      0    45%
mallm/discourse_policy/__init__.py                              0      0      0      0   100%
mallm/discourse_policy/collective_refinement.py                61     46     14      1    21%
mallm/discourse_policy/debate.py                               85     69     34      1    14%
mallm/discourse_policy/memory.py                               20      7      2      1    64%
mallm/discourse_policy/policy.py                               92     71     30      1    18%
mallm/discourse_policy/relay.py                                22      9      2      1    58%
mallm/discourse_policy/report.py                               26     13      4      1    47%
mallm/evaluation/__init__.py                                    0      0      0      0   100%
mallm/evaluation/evaluator.py                                 220    185    122      1    11%
mallm/evaluation/metrics/__init__.py                            0      0      0      0   100%
mallm/evaluation/metrics/bertscore.py                           9      2      0      0    78%
mallm/evaluation/metrics/bleu.py                                9      2      0      0    78%
mallm/evaluation/metrics/ifeval.py                             90     59     18      0    29%
mallm/evaluation/metrics/meteor.py                             20      9      2      0    50%
mallm/evaluation/metrics/metric.py                             10      1      0      0    90%
mallm/evaluation/metrics/qa.py                                 52     27     14      0    38%
mallm/evaluation/metrics/rouge.py                              10      3      0      0    70%
mallm/evaluation/plotting/__init__.py                           0      0      0      0   100%
mallm/evaluation/plotting/plots.py                            139    122     22      1    11%
mallm/models/Chat.py                                           76     47     22      0    30%
mallm/models/__init__.py                                        2      0      0      0   100%
mallm/models/discussion/CriticalResponseGenerator.py           44     30     12      0    25%
mallm/models/discussion/FreeTextResponseGenerator.py           63     41     14      0    29%
mallm/models/discussion/ReasoningResponseGenerator.py          46     29     10      0    30%
mallm/models/discussion/ResponseGenerator.py                  151     96     40      1    29%
mallm/models/discussion/SimpleResponseGenerator.py             55     37     14      0    26%
mallm/models/discussion/SplitFreeTextResponseGenerator.py      36     24      4      0    30%
mallm/models/discussion/__init__.py                             0      0      0      0   100%
mallm/models/personas/ExpertGenerator.py                       31     21      8      0    26%
mallm/models/personas/IPIPPersonaGenerator.py                  49     40     20      0    13%
mallm/models/personas/MockGenerator.py                          9      0      0      0   100%
mallm/models/personas/NoPersonaGenerator.py                    11      2      0      0    82%
mallm/models/personas/PersonaGenerator.py                       9      1      0      0    89%
mallm/models/personas/__init__.py                               0      0      0      0   100%
mallm/scheduler.py                                            306    306     52      0     0%
mallm/scripts/__init__.py                                       0      0      0      0   100%
mallm/scripts/batch_mallm.py                                  106    106     24      0     0%
mallm/utils/CustomFormatter.py                                 10     10      0      0     0%
mallm/utils/__init__.py                                         0      0      0      0   100%
mallm/utils/config.py                                          97     42     30      1    44%
mallm/utils/dicts.py                                           29      0      0      0   100%
mallm/utils/enums.py                                           10      0      0      0   100%
mallm/utils/evaluation/__init__.py                              0      0      0      0   100%
mallm/utils/evaluation/instructions.py                        698    398    144      0    36%
mallm/utils/evaluation/instructions_registry.py                19      5      4      0    61%
mallm/utils/evaluation/instructions_util.py                    61     40     12      0    29%
mallm/utils/functions.py                                       12     12      6      0     0%
mallm/utils/task_instructions.py                                1      0      0      0   100%
mallm/utils/types.py                                           91     17     14      0    70%
mallm/utils/utils.py                                           35     35      0      0     0%
---------------------------------------------------------------------------------------------
TOTAL                                                        3758   2463    936     24    29%

@jonas-becker jonas-becker merged commit 2dc5897 into main Jun 2, 2025
@lkaesberg lkaesberg deleted the vllm-compatibility branch June 2, 2025 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants