When the tick labels and
QCPAxisRect's margins are both set to auto, you get the effect you describe as undesirable. You can either set the format and precission as Bennj suggests, or you can generate the ticks yourself (meither of these methods will get you rid of the axis rocking, but both can improve your user experience considerably if your situation is favourable). Or you can set one or more margins to fixed value (something similar to your original proposal of labels of constant length), which brings a little risk of your labels' dimensions exceeding the preset margin.
These are the methods you are looking for if you want to set fixed margin.
QCPAxisRect::setMargins (const QMargins &margins)
QCPAxisRect::setMinimumMargins (const QMargins &margins)
QCPAxisRect::setAutoMargins (QCP::MarginSides sides)