Can never eject passport - always "In Use"

I see how to fix it online through steps like deactivating the drive on the backend, disabling the recycling bin, etc. But these solutions are very frustrating to manage. Are other people getting this error often? How do you permanently fix it? What is the culprit? Thanks.