"The Labour Party of today has fits of horrors of the very thought of somebody like me might saying that they bought in white Australia. But I believe they did."