Fishing not working 100% of the time

Sometimes when I catch something and I play the minigame, the thingy will stop in green but nothing will happen and the bobber just remains in the water. It happens like once and then I switch to something else on my hotbar, and when I try again I catch something, but then on the third try it again gets bugged so I have to constantly keep changing.