I have good results using Class 10 micro SD cards with 16 gb capacity.
Above 32 gb, you may max out the ability of the current Clip Sport firmware to manage more than 2000 files in the Music database. Maybe this file limit problem is a bigger concern than re-formatting the SDXC memory card?
You could decide to use the Folder mode to navigate your large audio file library on the SD card , But I think the problems with non-linear file sorting (and then re-sorting with external apps) may get too frustrating.
I recently solved my problem of swapping out (and dropping) those tiny micro SD cards by buying more Clip Sports. And I use the player color for genre sorting. 
