Many on the right assert that elitism is an approach to social problems that recognizes inherent differences in individuals. Elites belong in leadership positions where their natural talents can be used to best benefit society. Most people are not cut out for responsible positions within the social apparatus, according to their argument.
Understood in this narrow sense, I do not find elitism dangerous as an abstract analysis. Indeed, there are a vast variety of competencies inherent in people, whether through their choosing to develop them or whether they come "naturally" (whatever you think that means). That some should gravitate to a place where their talents are best used is not a problem; it is a core purpose around which we associate.
The problems enter in when a mere measurement of talent distribution is expanded into an individual or group identity. Without elitist pretensions, there is no need for a purposeful elevation of the more competent over the less. There is no need for institutional structures that maintain elite predominance. Why go to great lengths to stress differentiation between non-elite and elite if those differences are obvious?
In other words, it appears that elites are elite due to their ability to render some sort of service to others. But over time, elites come to be served by others. This happens because, instead of the elite status being a matter of demonstration and service, it turns into a status existing in and of itself. If the elite status cannot be commonly seen, then it must be imposed. Hence, institutional structures like royal families, aristocratic classes, and executive professional networks maintaining exclusive access to power. The elite become an identity, not a competency.
The importance of coercive structure to elevate these elites cannot be underemphasized. Societies are narrower than the humans they comprise. They select for qualities, talents, and characteristics they value based on their imperfect understandings at a given time. For societies to develop, they cannot simply perpetuate the same patterns for which they select; they must broaden their appreciation for underutilized talents, unappreciated qualities. The elite, in order to maintain their position as a matter of identity, must arrest this progress as a matter of preserving their status. Service to society is once again hampered.
I can imagine elite apologists saying that certain individuals are more valuable to society than others. For whatever reason, their talents are rarer. The loss to society of an improperly elevated talent is worth the danger of codified supremacy. The values informing this distinction between individuals are arbitrary but inherent in the social body. But this views the danger of elitism only in terms of its social consequences. It does not speak to the consequences to the so-called elite individual.
Talent within the self is not alone sufficient. It must be developed and actionable in order to useful. After all, if elites are distinguished by their usefulness to society, then their talents must be realized or the elite status is illegitimate. In a very real sense, the only legitimate use for a concept of elite is the service by the elite to the net benefit of society.
If one's sense of identity comes from the opinion that one is elite as a matter of what one can do, and not what one does do, it can hamper this striving to develop the talent. It can invert the pattern of service and squander the talent through demanding that others serve the elite. This then becomes a mere power relationship and, as earlier mentioned, will require recognition by society through coercive means in the end.
I'd argue that egalitarianism is not the argument that everybody is equal in talents. Instead, egalitarianism is the argument that what constitutes virtue is service, not identity, and that human potential is the basis for moral equality. It is through the kinetic that the potential is demonstrated and work is performed, if accomplishing work is the point in the first place.
What counts as "talent" is after all a normative construct. It isn't important at all, in the end, whether everybody has the same capabilities; what is important is that we understand genuine service, and that we cultivate a society that sees value in service to others so that potential is realized wherever it lay and not be squandered by mere institutional momentum.
The egalitarian approach has perhaps one construct on top of this: that perhaps potentials of import are not so easily perceived by us mortals, and therefore the safe bet is to value all instead of directly ordering the social body to select for the obviously desirable talents. Rousseau may have been correct that institutions corrupt man, but it seems more important to me that they may promote the development of individual talents based solely on their value to institutions. Obviously, human potential is broader than the society can integrate at a given moment. We can have faith in people, or we can have faith in leaders - this is the insight of the anarchist.