Hi Regnmand,
If you leave the two 6V6 sockets empty, the Plate voltage on the two EL84s will be somewhat higher and your output impedance will also change. You probably should re-bias to compensate for the higher Plate voltage, and if you want you can re-match the impedance by selecting the next lowest impedance on the selector switch using the same speaker cab, or using a speaker cab with twice the impedance of the one that you are using now and leave the impedance selector as is. If you do decide to pull two tubes, I would not change anything else. You might prefer the tone with the slight impedance mismatch, or you might like it better when matched.
Having said that, IMO your best bet would be to just leave the amp alone and use it with the selector on the EL84 side and be done with it. If you put in new EL84s you should check the bias.