In Chapter 4 of Amos Fortune, what does Mr. Richardson mean when he tells his wife "we'll be slaves ourselves if we don't keep watch"?

What Mr. Richardson is doing here is referring to the
sorts of ideas that will end up leading to the Revolutionary War.  This conversation is
taking place at a time when the colonies are still part of Great
Britain.

When the colonists later started rebelling against
Great Britain, they liked to say that the relationship between themselves and the
British was one of slave to master.  They liked to say that they needed to cast off the
chains of slavery and become free.  What Mr. Richardson is saying here is that he and
the rest of the white colonists will themselves become slaves to the British if they are
not careful.

So, what Richardson means is that white
colonists will be enslaved by the British if they are not
careful.