Several high-profile business leaders and far-left politicians said Rupert Lowe’s remarks were anti-British, he said he “couldn’t give a shit” what they think