Three things: first, I assume that when you said that FAT uses 32 bytes per entry, you were talking about the file records in the
directory table, not the FAT entries, but it wasn't entirely clear.
There were several versions of
FAT, most of which were named for the size of the file allocation table entries: for example, the original FAT8 (which was used in a product called 'Standalone Disk BASIC-86' before MS-DOS was developed, for use with single-sided 5 1/4" disks) used only 8 bits, FAT12 (used for floppy disks in MS-DOS) used 12, etc. However, these related to the blocks that held the file
contents, while the information about the files themselves (name, attributes, etc.) were in a separate table. I don't know offhand how large the available space will be on a MegaDrive's secondary storage, but I presume that the smaller maximum file offset (24 bits -> 16,777,215 sectors/clusters) is reasonable for your hardware.
Second, as has already been mentioned, having a hardcoded, fixed-length extension was a bad idea for DOS, and it's still a bad idea (actually, I would argue that file extensions are a bad idea in general). Even if you are enforcing file types, using the extension is a bad way to do it; on the one hand, for a system like this one you probably won't have 17576 file types (26^3, the number of permutations of three case-insensitive letters from the Latin alphabet), so it would be too many bits to represent fixed types, while at the same time, it is too few to really give a useful file extension for some things (e.g., HTML, which infamously lead to people using the '.htm' extension for years). If you are going to have typed files, use a one-byte type marker for the file entries and have a separate table mapping to known file types, that way you can either skip the extensions altogether, or have the file system add them automagically.
Third, if you are going to that much trouble in setting up compressed filenames, just bite the bullet and use a bitwise
Huffman encoding with a fixed table seeded on the letter frequencies for your native language (e.g., ETAOINSHRDLU...0123456789-_ for English). It wouldn't be consistent in the filename length (assuming 26 single-case letters + 10 digits + underscore + hyphen + space + period = 40 code points for a maximum required bit width of 6, with an 8-byte field width, it would vary between 18 and 64 characters), but it could be compressed and extracted with just bitwise operations and so would be reasonably fast.