Prime minister fights to stop Cabinet exodus