-*-mode: Text; fill-column: 79-*- * What is this? MPLE is a library for reading and writing MP3 files on Sony Network Walkman devices that support MP3 playback. The files are run through a simple obscuring algorithm and stripped of tagging information; this library recreates enough of the obscuring to make the player happy, although there are a few "magic" bytes unaccounted for. It's called MPLE because the tagging of the table-of-contents files (pblist*.dat) is WMPLESYS, and the name kinda evolved from there. WMMP might be a better name since that's the tag used for the obscured MP3 files. Do note that this is largely developer-oriented; there are some command-line tools, a fairly clunky GUI tool, and a Gnome VFS module, but these should be considered at best examples of how to use the library functions as opposed to actual end-user applications. In particular, the VFS module has a bug I've not been able to trace whereby it will get stuck in a loop, and you'll have to kill Nautilus in order to interrupt it. * Supported Devices I own a NW-S23, so that's guaranteed to work. I have had some assistance from the owner of a NW-E55; going by Sony's software update for the MP3FileManager program, it appears that if the E55 works, then this code should also support the E53, E73, E75, E95, and S21. If you have one of these devices, please test this code with it and let me know if it's ok. If you've got some other Network Walkman device and would like me to figure out how it stores files, email me (my address is at the end of the file). The NW-E99 uses v1.2 of the MP3FileManager program. I do not know at present if this is compatible with the code I've written; from experimenting with the program, it looks like it outputs the same files and formats. If you have one of these devices, please try out the code and let me know if it works. The NW-E103, E105, E107, E403, E405, E407, E503, E505 and E507, and any other devices which use the v2.0 MP3FileManager program, will NOT work with this code. I've done some work on reverse-engineering the storage format used by these devices and again, it's an obfuscated MP3 file, but I have yet to figure out how the obfuscation mask (a 32-bit value which is XOR'd with the MP3 data) is obtained. * API There are two major groups in the API: an application-focused one and a filesystem-focused one. The former is intended for e.g. writing standalone tools for manipulating the device contents, while the latter provides a file open/read/write/stat/seek/tell/close interface akin to the STDIO f* functions. The latter is still under heavy development; in particular, only basic read and write modes are supported, and seeking while writing is probably a bad idea as there may be unwritten data buffered in the API. However, it's good enough to support a fairly rudimentary GVFS module which I hope to eventually include as part of the codebase. The filesystem code transparently strips out or inserts ID3 tagging as the data is passing through. Since the device can only hold Artist and Title info, this may cause some data loss during round trips. However, the code works surprisingly well and is modelled after the fread/fwrite libc calls so it should be easy enough to integrate it into anything that already uses such calls. There is a potential for confusion between track number and track position in, e.g. a folder. Any function that takes a tracknum parameter is using the tracknumber used to create the conversion matrix to uniquely identify the file, not the position in the folder. A pos parameter, on the other hand, is referring to the position within a folder. For folders, pos is the order of the folders since there is no folder equivalent to the track number. All mple_folder_/mple_track_ functions implicitly read in the device's state (if it's not already available) and call mple_dev_sync after making changes. You can switch this behaviour off by setting device->autosync to false. Currently there's a little too much exposure of the internals outside the library code. Ideally things like the device, folder and track structures should be opaque types. so far: DEVICE mple_device *mple_dev_init( char *path ) - init the device structure and return a pointer to it. path is the mountpoint of the device (/media/NW-S23 on Fedora Core, for example) mple_dev_free( mple_device *device ) - free the device structure mple_dev_sync( mple_device *device ) - write the in-memory structure out to the device. has limited rollback capability in the event of failure. mple_get_media_serial_number( mple_device *device ) - read the media serial number off the device; unless your device's media serial number ends in 0xff, you'll need this. If HAL is available, the device serial number will be retrieved from that (requires that the device structure's device path or mount point be set); if the device isn't blank, the serial number will be retrieved from an existing pblist file (requires that the device be mounted and the mount point be set); failing either of these conditions, this function will attempt to read the serial number out of the FAT header (requires that the device's device path be set). The last method will fail if you do not have read access to the raw device (which you probably don't, unless you're root) FOLDER mple_get_folder( mple *device, pblist_folder *folder ) - locate folder on device and return its index number mple_add_folder( mple *device, pblist_folder *folder, guint16 pos ) - add folder to device at position. pos = 0 means append to end of folderlist mple_del_folder( mple *device, guint16 pos ) - delete folder at pos. DOES NOT CHECK IF FOLDER IS EMPTY. mple_mv_folder( mple *device, guint16 oldpos, guint16 newpos ) - move folder from oldpos to newpos in the folderlist mple_ren_folder( mple *device, guint16 pos, pblist_folder *folder ) - rename the folder at pos to the name in folder mple_add_to_folder( mple_device *, guint16 folder, pblist_track *track, guint16 tracknum, guint16 pos ) - add track to device's folder'th folder, at position pos, using tracknum as its identifying number, and track as the metadata. Does not write the track itself, just updates the folderlist TRACK mple_add_track mple_del_track mple_mv_track mple_ren_folder FILESYSTEM mple_open mple_close mple_read mple_write mple_seek mple_tell mple_stat MISC mple_get_metadata_NAME/FILE mple_build_conv_array utf8/utf16be conversion functions * Memory Leaks I've gone over this logically, with Electic Fence, with dmalloc, and with valgrind. Unfortunately, the GList functions allocate a chunk of memory the first time a list is set up, and this is never explicitly freed. This is intentional, according to the GLib people. This is the tail-end of a dmalloc run showing the "lost" memory. 1112822812: 190: total-size count source 1112822812: 190: 2048 1 ra=0x83ed2d 1112822812: 190: 52 1 ra=0x83e797 1112822812: 190: 29 1 ra=0x84c5b6 1112822812: 190: 20 1 ra=0x83f563 1112822812: 190: 2149 4 Total of 4 You can use a suppression file with valgrind's memcheck tool to ignore this. If HAL is used, there appears to be similar loss via dbus; I've poked at it a bit and included it in the valgrind suppression file but it's entirely possible I'm simply missing something. * Helping Out As mentioned in the "Supported Devices" section, this should work with about a half-dozen devices as it stands - specifically, if you use v1.1 of Sony's MP3 File Manager tool with your device, this code should work. If you use any other version then here's how you can help: 1. Open a command window (CMD), change drive to the device (e.g. enter "F:" at the "C:\...>" prompt) and enter "DIR". This should give a directory of your empty device, one part of which is the device serial number. Copy this number as it's used in the obfuscation for the v1.1 code and I suspect may also be involved in the v2.0 code. 2. Completely clear down the device. If you have Sonic Stage installed, there's an "Initialize Device" option that will basically reformat your player. This is probably the best option. Using Windows to format the drive may not be the best idea as Sony has a different idea about the drive geometry to Windows (at least, for my player, anyway). 3. Install the MP3 File Manager tool onto the device. If there isn't a version of MP3 File Manager for your device, use Sonic Stage, but my preference is for MP3 File Manager. 4. Use MP3 File Manager to copy an MP3 to the device. Please use a small file, and preferably one that is not covered by copyright. 5. Make an archive of the entire device, preferably using something that can create Unix tar files, but I'll accept zip files. 6. Repeat steps 1 to 5. 7. After you've repeated step 5 (i.e. you've made a second archive), copy the same MP3 file onto the device again and make a third archive. 8. One last step: clear the device again, and load it up with plenty music. Then create a zip file of just the metadata files, i.e. don't include any of the .OMA files - just the .DAT files. You should now have four archives, an MP3 file, and a device serial number. Email this lot to me and I'll see what I can do with it. Explanation for the curious: The two clean installs allow me to identify files that change between one drive format and another. This exposed some timestamping on the v1.1 code, for example. It also allows me to see what remains static between formats; this information is probably software-version or device-specific. The addition of the same file twice allows me to identify changes that happen based on the file number; in the v1.1 code, this affects the obfuscation algorithm. The final step is to try and untangle the relationships between the various metadata files on the device - I don't need the music tracks for this, just the metadata. * Who did this? Waider NW-E55 sample data and testing from Yann CANTIN