Charles Darwin was a descendant of two members of the "Birmingham Lunar Society" His maternal grandfather was Josiah Wedgwood whose pottery company still exists, his paternal grandfather was Erasmus Darwin, considered at the time to be the best doctor in England. Other members included James Watt. All adult members of his family were anti-slavery. Josiah Wedgwood produced high grade ceramic medallions with an image of a chained slave and "Am I not a man and a brother?" in thousands, many of which were exported to the newly formed USA.
The Wedgwood family have been real liberals ever since the 1700s. That's English liberals, not American.
Charles Darwin may have had a paternalistic attitude to non-whites, just as Abraham Lincoln did, but that is a far cry from the racism of the Ku Klux Klan, which comes from the same area of the USA where young Earth creationism is strongest.
Darwin used the word "race" quite a lot in his books, but if you read the context, what he really means is "species". These days we distinguish between these words.