Thousands of emigrants, many of them slaveholders, flocked in to take advantage and the settler population on the stolen lands skyrocketed