Whisk pudding mix with milk until smooth. Stir in chopped bananas; set aside.
In a large bowl, whip cream until stiff peaks are just about to form. Beat in vanilla and sugar until peaks form. Make sure not to over-beat, or cream will become lumpy and butter-like.
Layer half the pound cake cubes, half the pudding, half the whipped cream, and half the crushed vanilla wafers in a trifle bowl. Repeat with the remaining cake, pudding, and whipped cream, and top with the remaining crushed wafers. Chill in the refrigerator until ready to serve.