This sentence is an interesting one. On one hand it seems like it should provoke joy in the reader because not every "person of color" was subjugated by way of slavery. However, even today it's clear that the term "person of color" is a gloss over term seemingly encompassing everyone who is non-white. As evidenced by the last two paragraphs under the section titled United States territorial era it's clear that even in the late 1700s/early 1800s the distinction was made between Black, Biracial, and White. This contradicts the majority of claims that people continue to make today regarding the so-called one-drop rule.
Additionally, earlier in this wikipedia page the settlement of Filipinos was listed as an event that began to occur in the 1760s. Although we know that Asians immigrants were a part of early America and have experienced many hardships. I unaware of their specific experiences in the general South or New Orleans specifically. Either way I wish the term "people of color" in this context was more defined.