When I plug in my device to the computer to charge it I expect it to do just that.
When I remove it I want it to remember where I was on the playlist and not go back to the main menu and force me to go through Music->Playlists etc again.
When plugged into your computer, it establishes a data connection, which sorta wipes out where you were. The way to do what you want is to get a AC - USB wall charger or tape off the 2 inner pins on the USB plug. The outside ones are the pins that connect the charging circuitry. You can also gently insert the plug into the computer just until those slightly longer pins on the outside make contact.
AC - USB chargers are very reasonably priced, so that may be your best long-term solution. You will have to get a USB - mini USB adaptor though to use it with the Clip and/or Clip+.