Peach and Pecan Topping: Melt butter in medium saucepan over medium heat. Add brown sugar; whisk until sugar dissolves, about 1 minute. Pour mixture into 9-inch-diameter cake pan with 2-inch-high sides; spread evenly over bottom of pan (layer will be thin). Arrange pecan halves, side by side with round sides down, in circle around outer edge of pan bottom. Arrange peach wedges, slightly overlapping, inside circle of pecans, covering pan bottom. Set aside while making cake batter.
Cake: Position rack in center of oven; preheat to 350°F. Combine first 6 ingredients in processor. Blend until nuts are finely ground. Using electric mixer, beat sugar and butter in large bowl until butter is pale in color, about 4 minutes. Whisk eggs and vanilla in small bowl until well blended. Add egg mixture to butter mixture, 1 tablespoon at a time, beating until well blended after each addition. Add flour mixture in 3 additions alternately with milk in 2 additions, beating just until blended after each addition. Drop batter by large spoonfuls atop pecans and peaches in pan; spread evenly and gently with offset spatula or rubber spatula.
Bake cake until golden brown and tester inserted into center comes out clean, about 55 minutes. Transfer cake to rack; cool in pan 25 to 30 minutes (do not cool longer or peach layer may stick to pan). Run small knife around sides of cake to loosen. Place serving platter atop cake pan. Using oven mitts, firmly hold cake pan and platter together and invert cake onto platter. Let cake rest 1 minute, then very slowly lift off pan. If necessary, rearrange any peach wedges or pecans that may have become dislodged. Let cool to room temperature.
Cut cake into wedges. Serve with vanilla ice cream or whipped cream.