New feat : add a tools for species management

[alexbelgium]

This adds a new tab to the tools allowing to have an overview of all detected species

Objective : easily review anomalous species (low number of observations ; high number of observations but low max confidence, high number of observations but low threshold) and delete all linked sqlite lines and local files if needed. For species that we consider as actual observations, we can confirm them in the "confirmed file" that is created upon usage

Features :

• Shows : common name, scientific name, occurrences, Max confidence, Threshold, If confirmed, If excluded, If whitelisted
• Allow to delete all occurences of a species in the database, and all linked files
• Specific elements :
  • Single php script
  • Sortable fields
  • Allow to set/unset whitelist and exclusion list
  • Improved mini-charts with duration selector
  • Lazy load of longer elements using existing API (threshold and disk count shell scripts)

Implement : php + using existing python scripts

[Nachtzuster]

Interesting - I'll see if I find time to look into this one further

[jasondostal]

Really cool!

[Nachtzuster]

Looks pretty good.
One thing you probably are not aware of is that I've been working to move away from using common name and instead using scientific name. (Common names can change if somebody contributes improved translations or even switching between models)

It would be nice if new code is 'common name agnostic' as it were.

[Nachtzuster]

BTW Future idea: it would be nice if we somehow logged the "Excluded as below Species Occurrence Frequency Threshold" detections and marked those species as such on this page

[alexbelgium]

Hi, I've done the modifications and tested it as functional on my system, thanks for the recommendations.

Regarding your proposal, the easiest might be to :

• Save in a separate txt file, like discarded_observations.txt
• With the structure following the relevant fields here : "sci name_com name;last detection date;number of occurrence;max detection confidence"

That way individual observations are not saved which could greatly bloat the system, but there is still a trace. What do you think of this proposal. The best would be to keep the one with max confidence just in case for user review but I fear this introduces a whole new level of complexity

[Nachtzuster]

Tested a bit, found a couple of issues:

• images/check.svg is missing
• the 'Probability' calculation still relies on common name
• deleting does not seem to delete the files, only the DB entries

[alexbelgium]

Thanks for the review.

• images/check.svg => added
• "the 'Probability' calculation still relies on common name" => adapted
• "deleting does not seem to delete the files, only the DB entries" => this is strange, I had tested it to work on my system. Have you confirmed with a specific species name that I could test ?

root@db21ed7f-birdnet-pi:/mnt/NAS/Birdnet# find . -type f -name "grand-gravelot" | wc -l
14
// Performs species deletion from the tool
root@db21ed7f-birdnet-pi:/mnt/NAS/Birdnet# find . -type f -name "grand-gravelot" | wc -l
0

[Nachtzuster]

It took a bit of debugging, but the unlink() call actually fails; (it returns false)
I guess that is why play.php deletes files with sudo rm maybe?

It is probably a good idea not to continue with deleting the records from the DB in that case too.

[alexbelgium]

Thanks, I've now implemented the sudo rm with the same logic as play.php, and modified the code to still show the deletion message using common name as it should be more user friendly, while keeping the deletion logic based on sci name

[alexbelgium]

Thanks for the review, I've made some changes but wanted a double check on :

• logic for deleted files counting
• usefulness of under_base to avoid deleting files through manually encoded url

[Nachtzuster]

Thanks for the review, I've made some changes but wanted a double check on :

* logic for deleted files counting

* usefulness of under_base to avoid deleting files through manually encoded url

Not sure if this is a question, but my take is this:

• No preference on which method to use with deletion
• I do not see how a malicious url might 'jail-break' the deletion. (Since the code is looking up files though the DB, right?)

[alexbelgium]

Thanks for the review, I've made some changes but wanted a double check on :

* logic for deleted files counting

* usefulness of under_base to avoid deleting files through manually encoded url

Not sure if this is a question, but my take is this:

• No preference on which method to use with deletion
• I do not see how a malicious url might 'jail-break' the deletion. (Since the code is looking up files though the DB, right?)

Thanks. For 2., it was to prevent someone deleting data by typing an url directly in the address bar instead of using the script, but I can remove it. For me then it will be ready to merge!

[Nachtzuster]

Thanks. For 2., it was to prevent someone deleting data by typing an url directly in the address bar instead of using the script, (...)

I understand that, I just struggle to see via which code path that is possible - but I did write the code, so I might be overlooking something, if you can confirm either way, that would be great.

edit: if you can confirm there is no code path, then please remove

[alexbelgium]

Well it would be possible to delete any species with species_tools.php?delete=<Sci_Name> but indeed the user would still need to be authenticated. I'll remove the code

[Nachtzuster]

Thanks!