Coral totem loot pool not working

I have fished with coral totem twice and it doesn’t give me anything apart from normal fish. The first time I thought it was because I used the coral totem plus the frog shrine but now I’ve just used another coral totem by itself and got nothing but cod and pufferfish.