Diehards remain, but most just don’t want a byelection that would propel the disgraced former PM back into the limelight