The Free ‘Brown’ Slaveholders

Born a slave in 1790, William Ellison purchased his freedom when he was twenty six, and by the time of his death in 1861 he had acquired a fortune. In the numbers of slaves he owned he ranked among the top 1 percent of all slaveholders. He also became a large landowner and planter. Compared with the mean wealth of whites in the South, Ellison’s was fifteen times greater. That did not make him the richest free man of color, for there were a half dozen richer in Louisiana, but he was probably the richest who started life as a slave. On the backs of slaves he built his fortune and founded his freedom.

Ellison lived his entire free life in the small aristocratic village of Stateburg, South Carolina, a community of great planters and plantations in the High Hills of the Santee about a hundred miles inland from Charleston. One measure of his status was that a pew was granted for him and his family in 1824 by the vestry and wardens of the Holy Cross Episcopal Church. That made him the only person of color privileged to worship on the same level as the white gentry, though he was seated to their rear. All other colored parishioners, free or slave, sat in the gallery upstairs. Another symbol of rising status was the house he bought in 1835. This had belonged to the illustrious Stephen D. Miller, former governor and recently retired senator of the state. In this house, incidentally, Miller’s daughter Mary Boykin Chesnut, the famous diarist, grew up. Ellison named it “Wisdom Hall.” It was underneath this house that the Ellison papers, on which the two books under review are based, were discovered just a century afterward. More about them later.

Ellison was not attracted to Stateburg by its aristocratic society, from which he was excluded by his origins and color. He was drawn there by the richest market available for the products he made and sold—the cotton gin and its hundreds of delicate parts. He had learned the trade as a slave apprenticed to a skilled craftsman by his master, who was probably his father and may have been preparing him for the freedom he later bought with overtime earnings. His skills and reliability in making and servicing gins were in great demand at the time, in the midst of the Piedmont cotton boom, and few minions in the court of King Cotton were as indispensable as the gin master. His shop and his growing force of slave workers had all the business they could handle. In four years he made more money than most southern whites did in a lifetime.

Vital as were the skills of his craft, he knew that his reputation for “respectability” was just as essential, and he worked as hard at that as he did at his trade. Everything depended on white perceptions. The very year he moved to Stateburg extensive plans for a slave insurrection were uncovered at nearby Camden, and a few years later the greatest plot of all, that of Denmark Vesey, another freed man of color, was discovered at Charleston. Free coloreds were under special and constant suspicion. Ellison bought and freed his wife and daughter so that his three sons were born free. In 1820, “endeavoring to preserve a good character,” he put aside his slave name “April” and petitioned to change it to “William,” that of his former master. “April Ellison,” as his biographers say, “lived a wrenching irony.” The surest evidence of “respectability” in a slave society was the ownership and correct management of slaves. Having rescued himself and his family from slavery, he sought to ensure their freedom by buying more and more slaves.

Slaves and land and money were not enough—not for those of his color and origins. Racial subordination was as strictly required of free as of enslaved Afro-Americans, and no person of color was permitted to strike a white person who struck him or to commit any act or gesture that might be construed as “insolence.” No free person of color of whatever wealth or status could afford to risk an act that might be so interpreted. A careless word or boast, lack of sufficient deference, failure to lower the voice or raise a hat, ever so slight a breach in the etiquette of subordination could lead to dire consequences. Compliance could become as instinctive among the free as among slaves.

But in the conduct of business with whites as extensive as that of William Ellison, it must have demanded excruciating restraint and self-discipline. A slight error in a bill, the wrong tone in bargaining, an awkwardness or untimeliness in the settlement of an account, a fault of workmanship—any of these could have destroyed his hard-won “respectability.” But Ellison knew the rules and seems never to have faltered—so long as the rules were not changed. The psychological costs can only be imagined. His acceptance by whites owed something to his physician and white “guardian,” required of all free coloreds by law. This was Dr. W. W. Anderson, whose eighteenth-century mansion, now a sort of private museum still in the family, stands not far from the house of Ellison and displays among its treasures a few products of his old gin shop.

Antebellum race relations were further complicated by distinctions of color entirely within the Afro-American population, even within the small free population. William Ellison would never have called himself a “black,” as in current usage. In his own words he was “a freed yellow man.” He would have found “brown” or “mulatto” or “free person of color” acceptable, but never “black.” None of his friends or associates was black, free or not. All the members of his family and their spouses and relatives were brown, and all of his slaves were black. No evidence suggests that he showed the slightest racial affinity for his slaves or that he regarded the plantation as an extended family or a halfway house to freedom. He certainly never offered to any of them the hard way to freedom his master had opened to him. He employed a slave catcher once and deliberately sold off slave girls, presumably for capital to buy more male slaves and land. Local tradition had it that his slaves were “the district’s worst fed and worst clothed.” That would be consistent with his character and his frugality and less dangerous to his reputation for “respectability” than indulgence. His biographers, however, find no evidence that he was either more mild or more harsh than white slaveholders or, for that matter, more troubled in his conscience.

All four of Ellison’s children, one daughter and three sons, married into the “brown aristocracy” of Charleston at or near the top level of that exclusive society. They were a working aristocracy, skilled tradesmen for the most part, with property in slaves and substantial realestate holdings. They had over the centuries made a separate world for themselves between free whites and black slaves. The entering wedge had come with the original white settlers from Barbados, who tolerated miscegenation and a brown caste to mediate between them and their slaves. This brown elite identified with slaves no more than did white slaveholders and they excluded free blacks from their ranks. They numbered only about five hundred, or some 3 percent of the city’s Afro-American population, but together they owned all the slaves and three-quarters of the real estate in the hands of free Negroes. With these assets at their disposal they could choose whether or not to live in the slave South, but some who tried the North found Charleston, for all its limitations, preferable by far. Charleston was home. A time was approaching, however, when they would despair of Charleston too, and of the country as well.

It is for the insight they provide, not only on the brown elite of Charleston and the Ellison family but on slavery, race, and freedom in America that the deteriorating letters discovered under the former home of William Ellison in 1935 are so significant. In the words of their editors, Michael P. Johnson and James L. Roark, “They are the only extant collection of a sustained correspondence between members of a free Afro-American family in the later antebellum South.” When we recall that, in addition to the four million slaves, a quarter of a million free Afro-Americans then lived in the South, and consider the critical place they occupied in a slave society based on race, the uniqueness and significance of this collection are all the more astonishing. Even so the collection is quite small—thirty-two complete letters dated between 1848 and December 1860, with all but six written during the last year, and five written during and after the Civil War. Despite their deteriorated condition all but three have been fully deciphered, elaborately annotated, and scrupulously edited under the title No Chariot Let Down.

Had these letters fallen in other hands they would most likely have remained, even after deciphering and publishing, an archival curiosity full of unsolved puzzles. In the hands of two such master craftsmen as Michael Johnson and James Roark, both experienced historians with important books on this period,^* the puzzles became a challenge that they have handsomely met with a remarkably fine work of creative scholarship in Black Masters: A Free Family of Color in the Old South. Deciphering the crumbling letters was merely a start. The real work lay in following up obscure clues in thousands of documents in files of deeds, wills, tax returns, manuscript census returns, remotely connected correspondence of whites, diaries, parish registers, medical records, country graveyards, court records, and on and on.

In view of their anomalous status and insecurity, the free colored elite often sought to avoid public notice, leave meager records, and retreat into privacy. Compared with the white aristocracy they were obscure people anyway and naturally attracted no such public attention as their betters did. In spite of the wealth of solid information unearthed, the complete absence of crucial pieces of evidence forced the authors “to ask questions we cannot fully answer, to consider possibilities, and to imagine what was likely.” With their keen sense of the possible and the probable and their instinct for the right question and the best source, Johnson and Roark have opened to view “a South quite different from the one witnessed by white planters and black slaves.” Different also, they might have added, from that familiar to the three-quarters of southern whites who owned no slaves.

Among the many valuable contributions made by Black Masters, the most striking is the disclosure of a crisis heretofore completely overlooked by historians. This was the crisis of enslavement that threatened and in many cases suppressed the freedom of free people of color in Charleston and South Carolina during the hot and pestilential summer of 1860. It was the summer following the John Brown raid on Harpers Ferry and preceding the secession crisis. In the mounting national tension local authorities began in the spring to enforce slave laws long neglected, a campaign that reached a high level of intensity in August.

One of the laws was that of 1820 prohibiting manumission after that date and requiring documentary proof of one’s freedom or that of one’s mother before 1820. As a result of lax enforcement and reliance on the word of white friends or “respectable” people of color, many who were legally free neglected to obtain the required document. Enforcement of the old law also caught many without such claims living in de facto freedom on the indulgence, illegal manumission, or neglect by owners. All Afro-Americans of Charleston lacking legal proof of their freedom were required to purchase and wear a slave badge under penalty of being sold to the highest bidder into slavery. Another old law suddenly revived was that requiring every free man of color to have a white “guardian” duly registered.

In addition to these and other forgotten laws a petition was being circulated for a new law to sell into slavery any free Negro “considered worthless.” Distinctions between brown and black and between freedom and slavery, on which the free mulatto aristocracy had built its world, began to blur, and its members were pushed back toward slavery. Families sat at night awaiting a knock on the door by the police and the loss of freedom itself. Reaching to much later times for an analogy, the authors say the crisis of enslavement at its height “amounted to an antebellum Krystalnacht.”

For free persons of color to cross the line into slavery meant, among other things, instant loss of control over their lives and loss of all property. For a woman it meant that all her children also became slaves and, with “a sickening domino effect,” all her descendants. The foundations of family integrity were undermined. All contracts in which they were involved were rendered void, and any legal suit which they may have begun was ended. A slave without a master must have one, and the tragic victims of this sudden enslavement faced life under complete control of an unknown master.

In the ensuing panic, scores took flight as emigrants to free states. Though they were deeply shaken in their faith in the future and even subject to mob violence and police harassment, the members of the free brown elite, most of them with legal proof of their status in hand, stuck it out. They found it impossible to liquidate extensive property holdings in a glutted market. William Ellison, Jr., one of the South’s largest colored slaveholders, sent his children to an abolitionist boarding school in Philadelphia, a dangerous connection for him to make and one against his proslavery sentiments, but safer for the children even though he had to attest falsely that they were slaves in order to get them out of Charleston. He and the other members of the Ellison family and their prosperous friends hunkered down in mortal fear that worse was to come. William Jr. expected no “miracle by having a Chariot let down to convey us away.”

The enslavement crisis was precipitated by white workingmen of Charleston who were motivated by a mixture of class and racial animosity. Upper-class white friends of the besieged mulattos generally capitulated, though police and mob persecution stirred enough white sympathy to curb it somewhat. In the election preceding secession, however, the workingmen’s anti-Negro candidate James Eason outpolled the fire-eating secessionist Robert Barnwell Rhett, Jr. Once in the legislature, Eason presented a bill that would have shut down every enterprise in the state operated by a free colored craftsman, including William Ellison’s gin shop. This bill confirmed the worst fears of the brown elite, but their white aristocrat friends saved them by sidetracking it with the clinching argument that free Negroes of Charleston alone owned more than a million-and-a-half dollars in taxable wealth of which “more than three hundred thousand are in slaves.”

After the secession of South Carolina, however, the remaining free Negroes were convinced that the new nation had no place for them save that of slaves. An estimated 780 of them had emigrated before secession, two hundred of them to Philadelphia in November. About two thousand more left during the first month and a half of 1861, mainly the poor and propertyless. Now the soberest and wealthiest were at last persuaded that emigration was an urgent necessity. But where and how to go? They considered Canada, Haiti, Liberia, and the northern states, and sent emissaries to scout them. Reports were discouraging everywhere, and few of the mulatto elite were among the refugees. Tax records show that 92 percent of them remained in 1862. The wealthier they were, the less likely to leave, faced as they were with disposing of property at giveaway prices. How could the Ellison family instantly dispose of eighty slaves, nine hundred acres of valuable land, and many other properties overnight? How could they even advertise a sale without proclaiming bankruptcy, not to mention disloyalty? And what could gin makers do in the North anyway? They were trapped by their property and their trade and finally locked in by the war. As Johnson and Roark put it, “they mortgaged their freedom to their wealth.”

William Ellison died in December 1861, but his heirs carried on his example of patriotic support of the war. Far from being slackers or hoarders, they became model Confederates with excellent war records. Fulfilling every obligation imposed, they converted from cotton and gin-making to food crops. Ineligible for military service by law, one member of the family broke through the color barrier to become “an honorary white man” and a Confederate soldier—unmistakable confirmation of the family’s political sympathies.

The Ellisons even prospered at their patriotic food trade, making fabulous profits on paper—Confederate paper. They escaped molestation by Confederates and, with luck, by Federal troops as well. Collapse of the Confederacy and the emancipation of slaves plummeted their fortunes in the same general plunge of values common among whites. While, their estate declined to one-tenth of its antebellum value, they were still described as “rich” by the leading creditrating agency of the time—rich in relation to the postwar poverty of the South. They remained in Stateburg and established a general store that thrived (again relatively) on a white, not black, clientele. With their white clients they joined the Democratic Club, shunned Reconstruction causes, and toward their freed hired hands and renters “exhibited no more benevolence than their father had shown toward his slaves.”

But the cherished and dearly bought status of “free person of color” had vanished with slavery, and the Ellisons were now only southern Negroes. They continued to receive protection from the violence that lurked outside the community. Within a few years, however, the family began to disintegrate with quarrels and deaths, and the children scattered east, west, and into Canada, leaving behind only the crumbling letters underneath “Wisdom Hall.”