Remainers and leavers deeply divided over immigration