Here's a biological solution to your whale problem, of course it's up to you if you take it
Due to genetic deficiencies, the majority of the whales body is capable for living centuries or more, however their lungs have some problem and can only last for 50 years or so. The LARGE majority of the species dies when the lungs are fully depleted, however a select few that stayed near the surface and the shore more than most of their kind as this process was happening managed to get some small plant life growing on their backs, and as they age and grow bigger this small plant life eventually grows with them to become full fledged ecosystems. You could go two ways with the instinctual-ness of this process for the whales. One could be that their brain knows that they need to get plant life on their back to live longer, and so many stay close to the coast where they are incredibly likely to be poached, explaining the low frequency of island whales. Another could be that it's a blip in nature that was never supposed to happen, but because of some strange property of their skin it could. The few 1-3 that actually managed it did so by mistake. Perhaps they were sick, their lungs were giving out faster than normal, and so they were hugging the shore, away from the rest of their pod (or whatever a family of whales is called). Sort of like how cats go and hide when they die, except in this case when they went to "hide" they unwittingly found a way to live longer. If you choose that option I recommend against the poaching idea, as if both were the case it seems more likely that there would be 0 island whales. I'm not entirely sure how smart whales are IRL, but perhaps the little ones you said sometimes swim around the island ones recognize it as one of them that has cheated death, and so they're following it to either try to figure out how it did it or, if they're REALLY smart, they're hoping some of the island whale's plant life will drop seeds that the little ones can catch. You could maintain the exceptionally low frequency of island whales because unless the seeds the little ones catch are the smallest of grasses, their smaller size wont be able to handle the root system. For instance, if they manage to catch a tree seed when they're that size the tree roots will be more parasitic than symbiotic and will overpower and kill them.
It's a symbiotic, unnatural relationship, in which the whale's skin can somehow filter the seawater and provide the plants with fresh water, while the plants provide them with continued oxygen from their photosynthesis after their lungs give out. This idea would not allow them to submerge post-lung collapse (perhaps the old lung cavity is where all the roots are burrowing into them) or their gardens would die from the salt water exposure and lack of sunlight and the whales would suffocate.
There's just a few ideas for you to play with and mix and match. Hope it helps!