Fewer than half of voters who have changed parties since 2018 say they follow politics ‘most of the time’