The question is this: why should it be that America's decisive turn to overseas empire in 1898 involved a dramatic worsening of the political status of African Americans and the explicit exporting policies of white supremacy to the new colonies, while the far more ambitious imperial grab after 1945 was not only non-colonial, but actually necessitated some concessions to African American struggles? Part of the strategic rationale for pursuing a non-colonial empire was discussed in Neil Smith's work on Isaiah Bowman, 'Roosevelt's Geographer', some of which I outline here. But there is another issue here.
This was Truman's problem. He was a racist from a segregated state who never abandoned his disdain for social equality, notwithstanding the pragmatic concessions he was willing to make in his political career (he supported anti-lynching laws throughout his time as Senator, for example - a stance that later helped win him the Vice Presidency). His close advisor George Kennan distrusted African Americans and Jews, and his anticommunism was bound up with his racism since, as he saw it in his 'Long Telegram', the Russian Revolution had overthrown "the Westernized upper crust" of the Tsarist elite, and revealed a population Orientalised by "a century-long contact with Asiatic hordes". As to the American polity, he had written in 1938 that the US should be turned into a "benevolent despotism" of upper class white males, excluding women, immigrants and blacks from the franchise. Truman chose segregationists and racists for his cabinet: his choices for Secretary of State were all racists in one way or another. James F Byrne was an explicit segregationist. Dean Acheson was, behind his smooth patrician veneer, of the same ilk. He disparaged Africans, Latin Americans and Asians, and was an open supporter of white supremacy in southern Africa. He reflected the attitudes of his class: "If you truly had a democracy and did what the people wanted," he once argued, "you’d go wrong every time". On top of that, the legislature was dominated by southern white politicians. These were not all open advocates of racist violence, as Senator Theodore Bilbo was, but most shared his basic outlook.
Even so, Truman wanted to create a broad, multiracial, anticommunist coalition domestically that excluded only the radical Left (which obviously had a vastly superior position on race), while allowing him to conserve the racial hierarchy with some modifications. To do so he had to somehow neutralise the threat from the radical left represented by Henry Wallace (who was successfully vilified as a communist agent) and also overcome the militantly left-wing and pro-Soviet African American leadership, exemplified by people like the Fabian socialist W E B Du Bois, and the New York City council communist Ben Davis. It turned out that anticommunism was quite useful for conserving segregation and keeping black people in their place. First of all, the attacks on the more racially inclusive and radical CIO unions strengthened the racist unions and helped drive African Americans away from the organised working class. This partially accounts for the later strength of more militant-sounding but sectarian groups such as the Nation of Islam. Secondly, the attacks on communism helped domesticate the extant African American organisations. The NAACP was witch-hunted, Du Bois driven from its leadership, and the organisation essentially reduced its criticisms to the meek claim that America's stance on race relations at home harmed its struggle against communism. Thirdly, the turn to the Cold War was coterminous with a sharp rise in violence against African Americans, and a wave of repression that was justified in part on the grounds that African Americans would have been perfectly happy were it not for the subventions of Stalin's local auxiliaries. The red-scares were of course not completely successful in mitigating such struggles since, in order to neutralise progressive opposition and replace popular front liberalism with vital centre liberalism, it was necessary to accomodate some of the demands of African Americans and indicate that desegregation was an ultimate goal of the US elite.
Globally, Truman also had to find a way to win over newly decolonised societies and particularly the indigenous elites without ceasing his embrace of European colonial powers. His administration did not, any more than Eisenhower or Kennedy administrations, favour 'premature independence' as it was called. The line was that the US supported self-rule for the colonies - eventually. This, of course, was also the line of many colonial powers themselves, including Great Britain. And here, anticommunism was also enormously useful. Firstly, it dramatically reduced the scope for anti-colonial criticism within the United States itself, associating it with communism. As late as 1960, the Reader's Digest pleaded with readers "Don't Decry Colonialism", complaining that "Communist mouthings about colonialism not only have poisoned public opinion in neutral nations but have affected some Americans", and assuring everyone that the system was being gradually and beneficially relinquished before the natives were even ready. Secondly, just as southern politicians justified repression of African American struggles on the grounds that the insurgency was the product of red agitation, so colonial systems and white power structures in southern Africa and elsewhere could justify their existence on the grounds that opposition was Soviet inspired. This was also true of US corporations overseas. In Robert Vitalis' book American Kingdom, we see ARAMCO defending its Jim Crow system in the Dhahran oil camps by describing Arab strikes against the system as simply following the communist party line. Thirdly, of course, America's attempts to frustrate anticolonial movements in Korea and Vietnam were justified as attempts to frustrate Soviet expansionism, with Ho Chi Minh and Kim Il Sung reduced to mere ciphers for the machinery of 'totalitarianism'. Official documents, such as NSC 68, recognised decolonization as a distinctive dilemma for the US, describing it as an increasing "absence of order among nations" that was becoming "less and less tolerable". And where anticolonial rebellions broke out against American allies, the US officials understood well what was at stake. America's representative in Nairobi knew that the Mau Mau struggle was motivated by the violent and exploitative racial hierarchy perpetrated by settlers. But such understanding was repressed in favour of explanations referring to what was and what was not alleged to be in the interests of "International Communism" as opposed to the "Free World" (of colonies and dictatorships).
The interpretation of its global hegemonic struggles as a battle with 'totalitarianism', the antagonistic negative ideograph of American democracy, was ratified by the exceptionalist discourse in which America, uniquely, was possessed by a messianic goal of exporting democracy. Like all exceptionalist discourses, American exceptionalism highlights real discongruities only to overstate them, while sidestepping more significant continuities. As Daniel Rodgers points out, exceptionalism involves more than just difference: it contrasts "one’s own nation’s distinctiveness to every other people’s sameness – to general laws and conditions governing everything but the special case at hand." You can still see this in operation today, even in critical histories. Thomas Borstelmann, one of the better writers on the Cold War and the colour line, spends a great deal of time demonstrating that for the American ruling class there was no tradition of recognising a right of self-government for non-white people - but then finally falls back on the assertion that on those occasions in which American policymakers did seek to undermine European colonialism, as with Truman's sanctions against the Dutch in Indonesia, it was partially the result of a "traditional political principle" arising from 1776 and all that. To service that myth, while also dealing honestly with the facts, Borstelmann simply gainsays himself. Mary L Dudziak similarly bases her assessment of 'Cold War Civil Rights' on the idea that America was intent on ensuring that democracy was appealling to those new nations emerging from decolonization. That doesn't exhaust her analysis, which is frequently insightful, but it does foreground what is supposedly taken to be unique about America. The precedents for African Americans appealing to American democratic traditions to support their political claims are outlined by Dudziak, including the stance of Du Bois and the NAACP during WWI (although I think it notable that such tendencies are most pronounced during authoritarian crackdowns and wars). Dudziak also recites the Myrdalian position that the conflict between racism and democracy is a distinctively American problem, and that America's experiences in WWII traumatically exposed that conflict. Myrdal was, for his part, confident that a distinctive "American creed" could overcome the atavistic cultural throwback that he took racism to be.
This exceptionalist language triumphed after WWII. It had its precedents, but leading American thinkers and statesmen were more apt in previous years to draw on European heritage than to reject it. Woodrow Wilson appealled to Teutonic bloodlines to explain the success of American statebuilding. The early basis of the 'special relationship' between Great Britain and the United States, after decades of wars and tensions, was precisely the doctrine of Anglo-Saxon racial solidarity (in his 'Iron Curtain' speech, Churchill would substitute for Anglo-Saxon, "the English-speaking world"). After WWII, however, it was suddenly extremely useful for the US to be seen as different from those nasty old European powers. And thus, a process of historical forgetting was introduced. A caesura, separating not only America from Old Europe, but also the America of the past from America post-1945, was energetically introduced. The official doctrine of colour-blindness also assured blindness toward America's previous obsession with race, its explicitly imperialist doctrines, its official promotion of eugenics and racial 'science', and so on. Suddenly, slavery and racial genocide was epiphenomenal, or an unfortunate aberration from the sacred canons of constitutional liberty. Just as soon, America's colonial experiment was treated as aberrant. The language of empire was expunged, official discourse decontaminated, and the long-standing association in American thinking of liberty and democracy with 'whiteness' suppressed. That could not have happened were it not for the propagation of militant anticommunism.