Colonisation as Legitimate Collective Enterprise?

One of the key issues I encountered during this reading pertains to apparent distinction between different forms of arbitrariness, and the argument Nagel is making about which forms of arbitrariness we must work to eliminate, and which forms we are not required to.
I'm going to try and sketch his reasoning on this front and then try and relate it to a later discussion about inter state relations and state legitimacy.
On page 128 Nagel observes: "What is interesting and somewhat surprising about this condition is that such co-membership is itself arbitrary, so and arbitrary distinction is responsible for the scope of the presumption against arbitrariness." He is referring here to an earlier section in which he points out that Rawls is concerned not with arbitrariness per se, but rather arbitrariness amongst "fellow participants in a collective enterprise of coercively imposed legal and political institutions."
This argument is very troubling to me, as one of the most endearing aspects of Rawls is his recognition of the need for a theory of justice that corrects for many of the distributional inequalities that result directly from the natural lottery, or other purely luck based events. As much as it pains me to say it, Rawls' domestic focus undermines the strength of his argument in this case, as the universality of arbitrariness is one of its key features.
Nonetheless, Nagel goes on to offer a potential explanation for this split. He argues that the moral weight of arbitrary distinctions arises from: "a special involvement of agency or the will that is inseparable from membership in a political society." He goes on to elaborate on this point, explaining that he is referring to the dual role of citizens not only as subjects but also as "those in whose name authority is exercised."
At first glance this seems fine and well, so long as one assumes Nagel is referring to democracy and the role of the people in policy making. Troublingly, he goes on to make clear that is not the case on page 129: "This request for justification has moral weight even if we have in practice no choice but to live under the existing regime. The reason is that its requirements claim our active cooperation, and this cannot be legitimately done without justification--otherwise it is pure coercion." So if not democracy, what is he appealing to then? In a footnote on the same page (I know, I hate me too) he explains the following: "requires a broad interpretation of what it is for a society to be governed in the name of its members." That sounds okay, but he continues to explain that even colonial powers purport to rule not through force alone, but additionally are: "providing and enforcing a system of law that those subject to it are expected to uphold as participants, and which is intended to serve their interests even if they are not its legislators."
This is problematic for all of the obvious reasons. I highly doubt most of the colonised people would agree with Nagel here, and his argument is on difficult footing even if we consider the rationalisations of the colonial powers. He is right to say that states often felt compelled to do more than simply claim the right of the sword, but the justifications they did turn to are no less acceptable. White supremacy, sexism, imperialism, rabid pursuit of profit and glory, forced mass proselytisation, forced settlement, and so on are some of the justifications given by colonial powers, none of which seem acceptable to me. Nonetheless Nagel seems to (explicitly, in fact) accept the legitimacy of such states or forms of power. In fact, he concludes: "the path from anarchy to justice must go through injustice."