Add process to check for major drops in data between updates

[andrewtavis]

Terms

• I have searched open and closed feature requests
• I agree to follow Scribe-Server's Code of Conduct

Description

Based on scribe-org/Scribe-Data#68, we need to keep in mind that there will be cases that a property on Wikidata will change such that there will be a large drop in data. In the referenced issue, Portuguese verbs are using a non-standard past perfect PID that could be combined with the more widely used one at some point.

This issue would look into ways of diffing the current data coverage against the new data coming in, which could be as simple as total keys and total non-null values of keys of sub-objects. We could then discuss a viable cutoff, and trigger some kind of warning or a Scribe-Data issue if it's too low 😊

Contribution

Would be happy to discuss! Could also help implement, but might be better if others get to this eventually as I'm a long way off on Go :)

[andrewtavis]

Even just an email or a Matrix bot with a summary of the changes with some coverage metrics would be great 😊

[daveads]

cool i will take a look at this... @andrewtavis

[wkyoshida]

Hey @daveads - thank you for the interest in this issue!
Just a quick FYI though that this might be a little ways away, since this is likely dependent on having the CLI for Scribe-Data polished up (i.e. the ongoing GSoC project) and then likely at least v1 of Scribe-Server implemented as well. But once we get there, we can definitely have you take this on!

[daveads]

Hey @daveads - thank you for the interest in this issue! Just a quick FYI though that this might be a little ways away, since this is likely dependent on having the CLI for Scribe-Data polished up (i.e. the ongoing GSoC project) and then likely at least v1 of Scribe-Server implemented as well. But once we get there, we can definitely have you take this on!

oh okay

[andrewtavis]

Blocking this issue until such a time when the API is released :)