Reply
SanDisk User
Posts: 27
Registered: ‎04-08-2009

Re: Fuze won't find files without filename extension?


@summerlove wrote:
Was there a reason the files didn't have extensions, out of interest?

 

I'm not the OP but I use the same program to sync. Amarok lets you configure exactly how to rename the files when you transfer them: whether or not to add various tags in the file name and/or folder(s), how to deal with spaces, what to do with characters illegal in FAT32 filenames, etc. The file extension is also an option, and it's pretty easy to overlook.

 

 

 

--
She is looking at you ; So maybe you're looking too
Do you want to be her or don't you? ; Of course you do
But would she be you? /* Ladytron - Jet Age */
SanDisk Professor
Posts: 507
Registered: ‎04-24-2008

Re: Fuze won't find files without filename extension?


@Peach wrote:

@summerlove wrote:
Was there a reason the files didn't have extensions, out of interest?

 

I'm not the OP but I use the same program to sync. Amarok lets you configure exactly how to rename the files when you transfer them: whether or not to add various tags in the file name and/or folder(s), how to deal with spaces, what to do with characters illegal in FAT32 filenames, etc. The file extension is also an option, and it's pretty easy to overlook.

 

 

 


Thank you!  Was just interested

SanDisk Guru
Posts: 1,018
Registered: ‎02-05-2009

Re: Fuze won't find files without filename extension?


@Peach wrote:

@summerlove wrote:
Was there a reason the files didn't have extensions, out of interest?

 

I'm not the OP but I use the same program to sync. Amarok lets you configure exactly how to rename the files when you transfer them: whether or not to add various tags in the file name and/or folder(s), how to deal with spaces, what to do with characters illegal in FAT32 filenames, etc. The file extension is also an option, and it's pretty easy to overlook.

 

 

 


Just to be sure I'm understanding you correctly, are you saying that files can be saved in Amarok without file extensions at all?  I always thought that files always needed an extension and that the only time that it doesn't have one is when the extension is hidden.

SanDisk Guru
Posts: 1,265
Registered: ‎04-24-2009

Re: Fuze won't find files without filename extension?


@mags1230 wrote:

@Peach wrote:

@summerlove wrote:
Was there a reason the files didn't have extensions, out of interest?

 

I'm not the OP but I use the same program to sync. Amarok lets you configure exactly how to rename the files when you transfer them: whether or not to add various tags in the file name and/or folder(s), how to deal with spaces, what to do with characters illegal in FAT32 filenames, etc. The file extension is also an option, and it's pretty easy to overlook.

 

 

 


Just to be sure I'm understanding you correctly, are you saying that files can be saved in Amarok without file extensions at all?  I always thought that files always needed an extension and that the only time that it doesn't have one is when the extension is hidden.


 
No.  Theres nothing special about extensions.  You can have 0, 1, 2 ...  

 

SanDisk Guru
Posts: 1,018
Registered: ‎02-05-2009

Re: Fuze won't find files without filename extension?

Thanks for the explanation.
SanDisk Fanatic
Posts: 209
Registered: ‎04-09-2009

Re: Fuze won't find files without filename extension?

[ Edited ]

@saratoga wrote:

@Fenrir wrote:

 

I have to say that I find that a bit strange as all the tags are there in the files, so can someone from Sandisk tell me if the search algorithm only scans for .mp3, .ogg, .flac etc?

 


 
Theres no real good way to implement this.  Without knowing the file tag its extremely difficult to determine what format a file is in, particularly for MP3, which need not have any tags at all.  Not to mention iteratively trying each file parser on every file in your DAP would take hours to refresh the database.

 


 

I don't agree on that. It is possible to determine almost any file type out there by evaluating the first few bytes (their "magic numbers" ). Check this page for a good list of matches. I know it is hard to imagine - especially for windows users (and programmers) Smiley Wink

 

Nevertheless it is not a good idea to use arbitrary file name extensions. And it would certainly take a bit more processing power to analyze the first few bytes on every file. 

Message Edited by ewelot on 10-25-2009 09:05 PM
___________
Fuze 8GB (V01.02.28), Olympus LS-10
SanDisk Guru
Posts: 1,265
Registered: ‎04-24-2009

Re: Fuze won't find files without filename extension?


@ewelot wrote:

@saratoga wrote:

@Fenrir wrote:

 

I have to say that I find that a bit strange as all the tags are there in the files, so can someone from Sandisk tell me if the search algorithm only scans for .mp3, .ogg, .flac etc?

 


 
Theres no real good way to implement this.  Without knowing the file tag its extremely difficult to determine what format a file is in, particularly for MP3, which need not have any tags at all.  Not to mention iteratively trying each file parser on every file in your DAP would take hours to refresh the database.

 


 

I don't agree on that. It is possible to determine almost any file type out there by evaluating the first few bytes (their "magic numbers" ). Check this page for a good list of matches. I know it is hard to imagine - especially for windows users (and programmers) Smiley Wink

 


 
I think you should look more carefully at your link.  The "magic number" is the ID3v2 tag header.  If you tried to use such a naive approach any file without an ID3v2 tag would not show up as an MP3.  Looking at my collection I have more then a few files that method says aren't MP3s but that decode just fine on my Sandisk player.
 
I know its hard to image - especially for people who aren't embedded audio programmers - but assuming every file looks exactly like the example file you have means that your software won't work for anyone but you Smiley Happy

 

SanDisk Fanatic
Posts: 209
Registered: ‎04-09-2009

Re: Fuze won't find files without filename extension?


@saratoga wrote:

@ewelot wrote:

@saratoga wrote:

@Fenrir wrote:

 

I have to say that I find that a bit strange as all the tags are there in the files, so can someone from Sandisk tell me if the search algorithm only scans for .mp3, .ogg, .flac etc?

 


 
Theres no real good way to implement this.  Without knowing the file tag its extremely difficult to determine what format a file is in, particularly for MP3, which need not have any tags at all.  Not to mention iteratively trying each file parser on every file in your DAP would take hours to refresh the database.

 


 

I don't agree on that. It is possible to determine almost any file type out there by evaluating the first few bytes (their "magic numbers" ). Check this page for a good list of matches. I know it is hard to imagine - especially for windows users (and programmers) Smiley Wink

 


 
I think you should look more carefully at your link.  The "magic number" is the ID3v2 tag header.  If you tried to use such a naive approach any file without an ID3v2 tag would not show up as an MP3.  Looking at my collection I have more then a few files that method says aren't MP3s but that decode just fine on my Sandisk player.
 
I know its hard to image - especially for people who aren't embedded audio programmers - but assuming every file looks exactly like the example file you have means that your software won't work for anyone but you Smiley Happy

 


 

I didn't wont to attack any programmer at all. Apologies, if it came out like that (my english is not really good). Honestly, I count you as an experienced programmer, especially from your excellent work on Rockbox!

 

Maybe you missed something from the page I linked too. To my knowledge any mp3 file starts either with an ID3 tag or with a MPEG audio frame. Their "magic numbers" are both mentioned correctly with respect to the usual "MP3" extension. If you know any mp3 file starting with something differently I'd be glad to learn from you.

 

Don't worry too much. All I wanted to say is, that there is another way of determining the correct file type which does not rely on file name extension. But whilst beeing more accurate in a few cases this method is more expensive in terms of computational power. If I would have to vote between faster startup time (shorter database rebuild in our case) or correct file type recognition for oddly named files I would definitely go for the first.

 

___________
Fuze 8GB (V01.02.28), Olympus LS-10
SanDisk Guru
Posts: 1,265
Registered: ‎04-24-2009

Re: Fuze won't find files without filename extension?


@ewelot wrote:

 

Maybe you missed something from the page I linked too. To my knowledge any mp3 file starts either with an ID3 tag or with a MPEG audio frame. Their "magic numbers" are both mentioned correctly with respect to the usual "MP3" extension. If you know any mp3 file starting with something differently I'd be glad to learn from you.

 

Don't worry too much. All I wanted to say is, that there is another way of determining the correct file type which does not rely on file name extension. But whilst beeing more accurate in a few cases this method is more expensive in terms of computational power. If I would have to vote between faster startup time (shorter database rebuild in our case) or correct file type recognition for oddly named files I would definitely go for the first.

 


 
 
I did miss that part, I actually only skimmed the site since the premise was silly.  But even so, your link is incorrect.  It is required that all MP3 frames start with a sync word.  It not required that files start with a frame, so the sync word can be anywhere in the file.  This means that if you simply assume that any file beginning with 11 '1's is an mp3, you will try to decode 1 in 2048 non-MP3s and fail to play all valid mp3s that do not start with a frame.  You can verify this yourself by inserting bytes in front of the sync word on your mp3s.  If your decoder is standard compliant, it will not care.  I think that guy's idea probably works great for his MP3 files, but probably fails for all sorts of stuff floating around out there.  Trying to cheat the standard usually just means you end up with buggy crap.

Anyway, while I understand that you want to be able to say that file extensions aren't important and that you should be able to figure out what a file is without a priori information, there is no real way to do this for MP3 short of parsing through the file, checking every byte to see if its the start of a header, and then parsing the header to see if it encodes a valid MPEG audio frame.  And unfortunately, since the MPEG frame header is quite short, you will more then likely encounter random files with what look like headers that are not, in fact, MP3 files with surprising frequency, so I do not recommend this approach.  Its simply not accurate nor is it entirely standard.  The MPEG people assumed that you would know in advance what was an MP3 file, and so did not include a way to figure it out. 
 

 

 

 

SanDisk Fanatic
Posts: 209
Registered: ‎04-09-2009

Re: Fuze won't find files without filename extension?

[ Edited ]

I agree with you on many points (especially about false negative cases). Still I'd like to comment few things.

Noone is saying that "any file beginning with 11 '1's is an mp3". The author of the linked page just states that files starting with hex signature "FF Fx" usualy represent" MPEG, MP, MP3 MPEG audio file frame synch pattern". It doesn't go into detail about what "x" means but provides a link to the frame description.

 

From this description it is pretty easy to deduce that any audio frame of an mp3 (MPEG Version 1 Layer III) starts with 15 known bits (representing either "FF FA" or "FF FB" ). The false positive rate is therefore 1 in 32768 files if you assume that all files start with random bits. But they don't! The vast majority of files uses well defined bit sequences ("magic numbers" ) to distinguish their file types on a lower level - at least on the filesystems of the computers I'm working with. Therefore I expect the number of false positives to be much lower - by several orders of magnitudes.

I'm currently running a test to get a real number on a file space used by several users running many apps on different OS'es.

 

Note that false positives are even much more unlikely with mp3 files starting with ID3 tags or ogg or flac files due to a longer unique hex signature.

Message Edited by ewelot on 10-27-2009 04:59 PM
___________
Fuze 8GB (V01.02.28), Olympus LS-10