Start with the icing: place the caramels and water in a double boiler. Heat and stir until smooth and creamy. Cool.
Place butter in a mixing bowl, add sugar and mix at a low speed to a smooth paste. Mix at a medium speed while adding the caramel mixture. Set aside.
Make the cake: butter an 8-inch square cake pan and line with a square of parchment paper, then butter parchment.
Sift together flour, baking powder, baking soda, and salt.
Beat butter and sugar in a large bowl with an electric mixer at medium speed until pale and fluffy, then beat in vanilla. Add eggs 1 at a time, beating well after each addition. At low speed, beat in buttermilk until just combined (mixture may look curdled). Add flour mixture in 3 batches, mixing until each addition is just incorporated.
Spread batter evenly in cake pan, then rap pan on counter several times to eliminate air bubbles. Bake until golden and a wooden pick inserted in center of cake comes out clean, 35 to 40 minutes. Cool in pan on a rack 10 minutes, then run a knife around edge of pan. Invert onto rack and discard parchment, then cool completely, about 1 hour.