There is no automatic evaluator

[liuliuyaonuli]

How is the webvoyager dataset in the code used the method of manual evaluation, and I don't have the automatic evaluation method in the code It seems that the principle is different from that of webvoyager, can the author explain, thank you

[deepak-akkil]

We did not include an automatic evaluator from webvoyager and in the paper we report manual evaluation results. In practice, the annotators watched the video of the system executing the task and the final answer, and annotated each task pass or fail.

The main reason for not including webvoyagers automatic evalutor is that (while many papers have used to the automated evaluator from webvoyager), it is really not that great nor rigorously studied to our taste. In our repeated evaluation of it, it was wrong roughly 15% of the time, which is really not reliable enough.

We do have an automatic evaluation harness (that can make use of URL , text using LLM as a judge approach and JS script based evaluations) that can be expanded to cover different automatic evaluation techniques.