The simple answer would be that the skin tastes awful!
The reason why some fruits are meant to be eaten by animals is because plants need a way to disperse their seeds. So plants have evolved in such a way that the flresh of their fruits are tasty and juicy to attract animals to eat them. When that happens, the seeds within the fruits are dispersed to new locations when they are dropped by the animals or when they pass out in the faeces of the animals. The skin which was never meant to be eaten anyway, did not “evolve” to become nice tasting. Also, the skin also serves as a protective covering for the seeds until they dispersed. Hence, the skin of most fruits tend to be tough and have an unpleasant taste!
The bad taste of the skin may mean that there are compounds in it that are not safe to eat. Plants have learned to include chemicals that make animals ill in their leaves, stems and fruit, all designed to stop animals munching on things. Bitter tastes are associated with some poisons. These poisons might not be enough to kill someone who eats them, but might be enough to make them feel quite ill if they eat the skin.
Other plants are trying to protect their seeds as Daphne says – the seeds might have to survive the winter before they can grown into new fruit trees so they need to be tough.