In a heavy saucepan over medium heat, combine sugar, brown sugar, 1 cup condensed milk and corn syrup. Heat to between 234 and 240 degrees F (112 to 116 degrees C), or until a small amount of syrup dropped into cold water forms a soft ball that flattens when removed from the water and placed on a flat surface. Pour in remaining milk. Heat, without stirring, to 250 to 265 degrees F (121 to 129 degrees C), or until a small amount of syrup dropped into cold water forms a rigid ball. Pour into prepared pan. Let cool before cutting into 1 inch squares.