Could you please explain which script files should be executed sequentially for each stage?

[JINSUBY]

Thank you for your excellent contributions to the open-source community.

While training the model, I noticed that there are many shell scripts provided for training.
According to the paper, the training process consists of three stages.
Could you please explain which script files should be executed sequentially for each stage?

Thank you very much for your time and help.

[zhangyl4]

You can use https://github.com/SooLab/EyeWO/blob/master/scripts/estp/livebase_it.sh to reproduce Stage-1 training.

Scripts named https://github.com/SooLab/EyeWO/blob/master/scripts/estp/beacon_livel_h_stage* correspond to training with compression enabled. Here, x and x_5 indicate one epoch of training.

For better understanding:

• One epoch is used to understand ask high-resolution frame tokens.
• Another epoch is used to insert high-resolution frame tokens at uncertain positions.

If your goal is simple reproduction, you can directly start from Stage-2 training.
In practice, this does not lead to a significant performance drop.

By the way, using DINOv2 features can slightly improve performance on ESTP, but it significantly degrades out-of-domain performance (e.g., StreamQA).
You can enable this option if needed by setting:

--add_type fusion \
--add_vision_pretrained facebook/dinov2-large

[JINSUBY]

Thank you very much for the kind and detailed explanation. I find this work very interesting and have been looking into it closely, and I have a few questions.

1. In scripts/estp/beacon_livel_h_stage2_5_livebase_cqa.sh, add_random_high_res_ratio is set to "0", while in beacon_livel_h_stage3_livebase.sh, add_random_high_res_ratio is set to "high".
   *Shouldn’t add_random_high_res_ratio be set to "high" only in the _5.sh stage?

2. Based on Table 6 of the paper (https://openreview.net/pdf?id=2X0Fy0jqPa), I inferred that the training pipeline in the code consists of the following four steps. Could you please let me know whether my understanding is correct, or if I have missed anything?
   (1) beacon_livel_h_stage1.sh
   Since stream_loss_weight is set to 0, this stage performs only a warm-up to improve answer quality.
   The training data used is the Ego4D caption dataset.
   (2) beacon_livel_h_stage2_livebase.sh
   Since stream_loss_weight is set to 1, the model is fine-tuned jointly with the LM loss and the streaming loss.
   The training data consists of ESTP-IT SQA and CQA, and high-resolution features are not used.
   (3) beacon_livel_h_stage3_livebase.sh
   With stream_loss_weight = 1, the model is fine-tuned jointly with the LM loss and the streaming loss.
   The training data is ESTP-IT SQA, and high-resolution features are used.
   (4) beacon_livel_h_stage3.5_livebase_cqa.sh
   With stream_loss_weight = 1, the model is fine-tuned jointly with the LM loss and the streaming loss.
   The training data is ESTP-IT CQA, and high-resolution features are used.

3. During inference, frame_token_interval_threshold needs to be provided as an argument.
   For reproducibility, would it be appropriate to use the value 0.725, or is there a different recommended setting (e.g. None)?

[zhangyl4]

1. About add_random_high_res_ratio

You are right. This is from a historical version of the script, and the setting is indeed inconsistent.
add_random_high_res_ratio = "high" was intended to be used only in the _5.sh stage. Some older scripts were not fully cleaned up and may contain such issues.

2. About the training pipeline inferred from Table 6

Your understanding is largely correct conceptually, but there is an practical difference compared to what is described in the original paper. Although the paper mentions joint training with Ego4D caption data, in later iterations we switched to a more efficient setup:

• Instead of re-training jointly from scratch with Ego4D captions,
• We initialize from a VideoLLM-Online model already trained on Ego4D captions, and then apply LoRA fine-tuning for the ESTP setting.

In practice, you can directly refer to livebase_it.sh, where:

• stream_loss_weight is set to 1
• The model is initialized from the Ego4D-caption-trained VideoLLM-Online checkpoint

This change was mainly for iteration efficiency, and empirically we did not observe degradation compared to explicit joint training.

3. About frame_token_interval_threshold during inference

The purpose of interval-based training is exactly to avoid relying on manually tuned inference-time hyperparameters.
So for reproducibility, it is recommended to set

• frame_token_interval_threshold = None

That said:

• Using a balanced value like 0.5 is also reasonable.
• If you want the model to over-respond, you can increase it.
• If you want the model to under-respond, you can decrease it.