Villager prices are going up!

Villagers are tripping again. I just started a SkyBlock world and I got one villager there. I did hit him by accident while trying to break his Lectern and he was pretty upset and raised his prices up, so I traded with him and got another Villager later on, on my Skyblock of course.

They both were got very very happy at my trades until when I got a third villager, then they started raising their prices again for no reason at all. I didn’t hit them aynmore, I traded with them and they still keep raising their prices up.

They used to trade 24 Paper for 1 Emerald, now it’s 41 paper, which isn’t bad but still. Good thing I have an automatic Sugar Cane Farm, but I would like to know what’s causing them to do this cause it’s hard to keep up.

Could it be because I’m trading with them too much? Or do I have to spend my Emeralds with villagers too? Even if I AFK for the night, will their prices go down or up?