It seems that if there are any modded spells in the inventory when trying to research spells or wands, the UI vanishes. At one point I think I saw it go flying off the screen for some reason, as if the position of the UI got messed up.
My expectation would be for the UI to function as expected and only allow interactions with vanilla spells, ignoring the existence of any modded ones.
This is the mod list I was using at the time of seeing the bug:
