I was swapping around sim cards and ended up in this weird state where I don’t have any voicemail accounts, but ended up with an indicator. The easiest way is to stop the phone app and clear the data on it from the Android Application Settings under “All”. In the event that it’s the carrier’s system that’s out of sync, you’ll still be stuck here, but in the event that your phone is waiting for a magic sms that will never arrive to clear it, this could be useful.