The formability of hole expansion for punched ultra high strength steel sheets with low ductility was improved by smoothing the sheared edge of the sheet. In the smoothing process, the notch effect of the macroscopic roughness in the fracture face was eliminated using a conical punch. The effects of the smoothing conditions on the limiting expansion ratio were investigated. Setting the angle of the conical punch in the smoothing to the same angle as the fracture face was effective in improving the limiting expansion ratio. The clearance ratio in the punching influenced the limiting expansion ratio, thus the optimum clearance ratio resulted in a maximum limiting expansion ratio after the smoothing. The smoothing of the sheared edge was effective in improving of the formability of hole expansion for ultra high strength steel sheets.