Rita does not detect new logs after the first import (v5.0.8)

[Mohammad-Mirasadollahi]

Hi

The first time I use the following command, RITA analyzes all my data and shows it to me. There's no problem in this part. But when I want to run this command from the second time onwards, it always tells me that "[!] all files were previously imported." However, new data is created in my log path, and I have checked that the log files are being updated, but RITA keeps giving me this error.

Command: rita import --database=mydb6 --logs=/opt/zeek/logs/current --rolling

Error: "[!] all files were previously imported."

Is this a bug, or am I doing something wrong? Because I didn't have this issue in the previous version, but this version gives me this message.

[nebriv]

Also having this issue

[kstjo]

Also have this issue. RITA v5.0.8 on Ubuntu 24.04.01 LTS.

[kstjo]

My problem exhibiting these symptoms was fixed by transferring the Zeek log directories for each day, then having RITA import each of these directories. It worked despite all the files having the same names in each directory.

[koffierombouts]

When executing rita import --database=mydb6 --logs=/opt/zeek/logs/current --rolling, you call a startup script located in /usr/local/bin/rita. This script reads all the input parameters and prepares the docker compose file. This is also the same script as rita.sh, but is called when you execute the command.

Here you can see the path argument is prepared, the provided logs are mounted to /tmp/zeek_logs.

    if [ -d "$ABS_PATH" ]; then
        DOCKER_ARGS+=("--volume" "$ABS_PATH:/tmp/zeek_logs")
    elif [ -f "$ABS_PATH" ]; then
        PARENT_PATH=$(dirname "$ABS_PATH")
        DOCKER_ARGS+=("--volume" "$PARENT_PATH:/tmp/zeek_logs")
    else
        echo "logs do not exist: $ABS_PATH"
        exit 1
    fi

Further in the script the docker compose is started. The bind mount is also done here with the provided arguments.

$SUDO docker compose -f ${COMPOSE_FILE} run ${DOCKER_ARGS[@]} rita "${RITA_ARGS[@]}"

RITA checks if files have already been loaded in ./database/metadb.go. The function checkFileHashes (wich does not check any hashes) checks if a certain file path has already been used in the called dataset. If it has already been used, it wont be used again. And since your logs are bindmounted to /tmp/zeek_logs, if they are not unique logs, than RITA will think they have already been used.

I solved it by adjusting the RITA startup script to not use a hardcoded bind mount file path. Instead it uses the provided filepath and mounts this inside the container. It is not entirely clear to me why @kstjo solution works, because as far as i know the log path does not matter. It always gets converted to /tmp/zeek_logs.

So to round it up, in my eyes the solution is to either change the startup script. Or all logfiles have to have a unique name.