If your phone number's voicemail is forwarded to Google Voice, change your voicemail back to your regular phone number before you delete your Google Voice number.