Your fonts are stored as part of your brand. If you delete your brand, all of the fonts you uploaded will also be deleted.
If you want to delete a specific font, you can do that by clicking the trash can icon next to the font name in your brand manager.