Emotions that set multipliers do so when the emotion is added, and then the multiplier is removed under certain other conditions not directly related to losing the Emotion. This is seen most clearly when form changing, which resets Emotion.
For now, it can work to simply reset the multiplier when changing Emotion to one that does not have a multiplier. However, this could be incorrect in the future when modders can set multipliers through scripting.