It all boils down to the fact that we understand the “flatness” of our phone screens. Faux 3d elements and real-world textures mentally clash with that flatness creating some dissonance.
By going flat the idea was that the interface is what it is a bunch of pixels displayed on a flat surface. While accurate and modern, people understood something was missing. Some playfulness of actually “pushing” a button down instead of touching a flat surface.