Meanwhile, saute leeks in olive oil over medium heat in a nonstick pan until slightly golden, about 4 minutes. Add rosemary and stir for an additional minute or so - until you can smell it).
Strain the potatoes when they are tender but do not rinse. Add buttermilk and mash or use a hand blender on low (never high!) to mash them. Add most of the leek mixture and salt and pepper to taste ~ important!
Scoop into a serving dish and sprinkle reserved leek mixture on top.