Few say growing diversity makes their country a better place to live