The tax is calculated from rows with tax class, if i submit a credit memo with adjustment fee or adjustment refund it will result in incorrect tax amount.
This is usually not a big problem but in some cases we use the tax amount to calculate tax percent to send to a payment provider and it may cause problems in reports that are ground for bookkeeping.
This may be a couple of possible solutions:
https://gist.github.com/Schrank/896561c841f564efb003c87441e90a4e
https://magento.stackexchange.com/questions/93393/is-it-possible-to-refund-tax-on-a-partial-order-refund
Will need to be a subject of testing.