This is a known limitation of the App Designer and the developers might consider it for a future release.
However there are a couple of work-arounds to achieve this functionality.
1. Force the buttons to resize so they remain centered, by doing one of the following:
a. Make the buttons wider, so they are left aligned with the labels, and right aligned with the edit fields, see App_a.mlapp that is attached.
b. Create a non-visible edit field behind the button, that is of the same size as the button. Note: you only need to do this for one of the buttons, not all. Since they are aligned, the other buttons will behave like the one that has the edit field behind it. See App_b.mlapp , and especially the component named "app.NonVisibleEditfield".
2. Create 2 non-visible edit fields to the left and right of one of the buttons. Note, this only needs to be one for one of the buttons, not all. See App_c.mlapp, components "app.NonVisible1" and "app.NonVisible2". This would make the buttons remain with the same size and they would align to the center of the panel.
All the apps that I have mentioned are included in the zip folder that I have attached to this answer. App.mlapp is the application that recreates the issue.
Hope this helps