I find the Clip Sport player does really well with a wide variety of mp3 encoded files (128-320 kbps).
I had mixed results with m4a files and I decided to abandon this choice to keep me & my Clip Sport happy.
Make sure you are using Clip Sport Firmware version 1.39 (or higher).
The list of Bugs Fixed in FW 1.39 included this comment: “Fixed the long duration m4a file playback”.
What software do you use to prepare your m4a files?
I think iTunes is a reliable choice?