Skip to content

Native gvfs backend for MTP devices

Hi again! It’s been a while since I’ve had something to write about, and it’s filesystems again, but with less April Fool’s.

What is MTP?

MTP is a standardised protocol that was originally designed to allow a PC to effectively manage the contents of a media player device – specifically, audio, video and image files. It is, in turn, based on an older specification called PTP (Picture Transfer Protocol) that was designed for use with Cameras. Note that neither of these uses cases has to do with managing the contents of an arbitrary filesystem. Of course, you can read more about MTP on wikipedia.

Why should I care?

Well, most people didn’t need to care about MTP for a long time – the chances were pretty good that their media player device didn’t use MTP (it either used USB Mass Storage or was an Apple device with its own crazy protocol), and their camera had a reasonable chance of using USB Mass Storage, and in the worst case you could always eject the memory card and use a reader with it.

However, since Android 3.0 (Honeycomb), Android devices have stopped using USB Mass Storage for PC connectivity, and switched to MTP. Now wait, you say, why would you use MTP to manage the contents of an arbitrary filesystem – a very good question. The primary reason is that USB Mass Storage is a block level protocol, and consequently operates below the filesystem layer. This means that it can’t be used to share a filesystem between the phone/tablet and the PC – only one device can read/write at a time. In older Android devices, this meant having a separate partition or memory card that was inaccessible to the phone while the PC was using it. But, from Honeycomb onward, Google wanted to have a more unified filesystem on the device, and not have to worry about ensuring there was a storage area that could be unmounted at random times. MTP may be an ill-fitting choice, but it’s the only standardised protocol which offers the key required feature – that being that you can have both the phone and PC use the filesystem at the same time.

This is posible because MTP treats files as the atomic data unit, rather than blocks. So you read and write whole files, and nothing smaller. At that point, the PC interacts with the filesystem pretty much like any other application running on the phone.

Ok, so I have to care about MTP to access files on my phone. Why’s that worth talking about?

Well, here’s the kicker. Your shiny new Android phone uses MTP, and there are a plethora of applications and components for Linux that notionally can manage MTP devices. Unfortunately, they are all limited in various ways, that make them pretty much unusable. The most common flaw is that the tools use an MTP library call that attempts to enumerate the entire device filesystem in one go. This causes the initial connection to be very slow, and on the newest Android devices, it flat out doesn’t work, as the phone end will timeout the operation before it completes. This ends up taking out every single tool on the market (including: mtpfs, gmtp, banshee, rhythmbox, gvfs gphoto2 backend) except for one: go-mtpfs.

go-mtpfs?

go-mtpfs is a recent creation of a Google employee, who was aware of the timeout behaviour of android devices, and so they wrote a FUSE filesystem (much like the original mtpfs) but which did on-demand file enumeration, as each directory is loaded and queried (which is how the Windows MTP implementation works). The end result is quick connections, and a tool that can talk to Android devices well.

So we’re done?

Well, no, not quite. Although go-mtpfs uses MTP the right way, it can’t avoid the horrible impedence mismatch between MTP’s file-atomic model and a traditionally filesystem where you can do random I/O within a file (ie: open, seek, read, write, close, etc). MTP only allows you to download a complete file, or upload one – you can’t even move a file between locations on the device through MTP (you have to download it, delete it, then upload it to the new location). Of course, FUSE doesn’t care that you can’t provide normal filesystem semantics, so you have to improvise. In this case, that means that any read/write operations requires go-mtpfs to do an elaborate and fragile dance to download a file, modify it, and upload it again, and so on. This causes simple file operations to behave strangely, and things can go very bad if you use a tmpfs for /tmp and try to access a very large file.

And so: gvfsd-mtp

And finally we reach the meat of this post. gvfs is the virtual filesystem layer that’s used by Gtk+ based desktop environments (GNOME 3, Unity, XFCE, etc). It happens to allow backends to implement a much higher level API than FUSE, and this API happens to explicitly offer ‘pull’ and ‘push’ operations (download and upload respectively). As such, it’s possible to meaningfully map functionality without jumping through crazy hoops.

So, I’ve been working for the last few weeks on a native mtp backend for gvfs. It’s heavily based on the existing gphoto2 backend, but only attempts to implement the operations that MTP can cleanly accomplish.

The end result is a filesystem like view that you can effectively browse through nautilus, and that you can download files from and upload files to with a reasonable hope of success. It’s not seamless as gvfs/nautilus do not do anything very special when you attempt an unsupported operation – so trying to just open a file will probably fail (although some GNOME apps like evince or file-roller seem to be able to detect the behaviour and will automatically download the file and open the temporary copy), but it’s functional and reliable.

Another thing I did was to avoid caching any metadata that I could avoid caching. All other MTP implementations tend to save directory listings after getting them and don’t go back to the device. But the device can be adding/removing files all the time, so it’s important to be able to get a fresh listing when you need it. To achieve that, I took advantage of the fact that gvfs lets you define ‘display names’ for files – which can be completely different from the reported filename. So I directly mapped the MTP object IDs (numbers) to the reported filename and made the real filename appear as the display name – so the view in Nautilus looks completely normal but the gvfs URI might be something like mtp://[usb:001,015]/65537/1/128/2445

This general behaviour is actually very similar to how Windows presents MTP devices – they appear as normal looking drives and folders and files in Windows Explorer but you can only really download and upload files – it doesn’t pretend to offer normal read/write operations on the files.

Ok, where do I get it?

Here!. I’m continuing to work on it – I should be able to provide thumbnails from the MTP metadata, and I’d like to implement a way to copy directories back and forth (gvfs doesn’t implement recursive push/pull for you – you have to do it yourself), but it’s otherwise very usable – it was proper plug/unplug detection and I modified the gphoto2 backend to not grab mtp devices like it used to. Eventually I’d really like to get it upstream (especially as you can’t build gvfs backends outside of the gvfs source tree) but there’s a fair way to go yet.

Enjoy!

{ 39 } Comments

  1. Delio | 8th August 2012 at 02:02 | Permalink

    Hello,
    It’s nice to have mtp support in gvfs, but I still don’t understand why google choose to use that in Honeycomb.
    I think that using a samba server over usbnet would have been far more powerfull. And in fact I tested that on my phone with android 2.3 and the transfers are faster with samba+usbnet than with usb mass storage (but I have not compared with mtp)

  2. davidz | 8th August 2012 at 07:02 | Permalink

    Hey, this is cool stuff… does it support PTP as well? If so, when it’s merged into GVfs master, I would suggest making just disabling the gphoto2 volume monitor / backend since 99% of all devices the last five years are either PTP or MTP and thus probably better driven by your backend than libgphoto2 [1]. When you open a bug for merging it, please add me to the Cc (I wrote the GVfs libgphoto2 backend).

    [1] : libgphoto2 is a problematic library; for example it hangs if you have a stale NFS mount…

  3. Philip Langdale | 8th August 2012 at 07:08 | Permalink

    Delio,

    Yeah, I forgot to write about it, but cifs-over-usb-networking is more performant and gives you normal filesystem semantics. Unfortunately, I’m sure the reason they don’t do it is that Windows doesn’t provide any way to make such a device appear in a sane place in the UI. Even though windows supports all the individual pieces out-of-the-box, it can’t magically give you a device icon in Windows Explorer. I’ve thought about whether you could cut the network out of the middle and do cifs over usb directly, but that would be a non-standard protocol, so it wouldn’t be usable anywhere. :-)

  4. Philip Langdale | 8th August 2012 at 07:10 | Permalink

    Hi David,

    No, it won’t support PTP devices – to do that I’d need to switch to libptp (which libmtp actually uses internally). That’s probably not a hard thing to do as the API wrapping is pretty much 1:1. We’d lose the MTP specific parts around managing media metadata but I was never going to expose those, and the image thumbnail stuff is in PTP as well. I’ll look into it.

    Update: Wow – the libptp story is messier than I thought. libmtp reuses a forked copy of libptp from gphoto2, with significant additions, and libmtp provides a fair amount of convenience over libptp. Upshot being that I’d basically need to copy the forked libptp into the backend along with chunks of libmtp that do the useful for me. Ultimately, it would probably end up looking more like a forked copy of libmtp with the mtp initialization parts disabled. (And yes, I’ve confirmed that libmtp can’t talk to my ptp cameras)

  5. pvagner | 9th August 2012 at 00:53 | Permalink

    Hello,
    This looks verry cool!
    I have came accross this because ubuntu 12.04 uses libgphoto2 by default and I can confirm it crawls entire folder structure with my Samsung Galaxy S II and then eventually fails while trying to copy a file.
    Is there a simple guide on how to try out this on an ubuntu 12.04 64 bit?
    What I would like to achieve is is copying files and folders with ability to preserve their timestamps.

  6. pvagner | 9th August 2012 at 01:45 | Permalink

    ummm, I have installed most required packages but this most likelly requires newer or modified libmtp than the one provided in ubuntu repos. I am afraid I am unable to get this working easily.

  7. Julian Andres Klode | 9th August 2012 at 07:46 | Permalink

    Given that libmtp (some git after 1.1.3) takes minutes (really) to establish a connection (mostly to run mtp-detect) to my Galaxy Nexus running JB, I don’t see how this is going to help.

    Then there are many other problems with mtp, like ignoring directory and file names for music, causing it to loose music, as it (phone/libmtp) stores them at /sdcard/Music/Artist – Title.ogg, which obviously misses the album, so you can’t have two different versions of a song (this test was done in rhythmbox, I’m not sure whether that’s a bug in rhythmbox, libmtp, or the phone).

  8. Philip Langdale | 9th August 2012 at 07:59 | Permalink

    Julian, that’s because the tool is doing the full enumeration that I described. As long as you do incremental enumeration, it’s fine – which I do.

    I think rhythmbox uses the music specific part of the API which don’t report filenames and paths like the generic file API does.

  9. Philip Langdale | 9th August 2012 at 08:01 | Permalink

    Pvagner, I develop on 12.04 so I know it works fine with the normal libmtp in 12.04.

  10. mgreene | 9th August 2012 at 10:08 | Permalink

    What would one have to do to fix any of the mentioned applications (e.g. Rhythmbox) to perform on-demand file enumeration? What would be the mtp library call(s) to avoid?

  11. Julian Andres Klode | 9th August 2012 at 10:16 | Permalink

    OK, thanks, that seems correct. Some points on gvfsd-mtp:

    * All file names are numerical

    * The device ID seems to change (my first connection was mtp://[usb:002,013], then it was mtp://[usb:002,014])

    * Copying a movie in a format not understood by Android (which is intentionally crippled to not understand common video formats, despite having the codecs in the hardware) to Movies does not work (recognized .mp4 movies work however)

    * Viewing photos or watching films from the device does not seem to work. It seems they need some more seekable access to files; for example, eog fails with: “Failed to open input stream for file”)
    * Also, as soon as the phone turns the screen off (which it does automatically),

    I was running in a strange (??) way, though; by combining the system’s gvfsd with a gvfsd-mtp and gvfs-mtp-volume-monitor built from your git.

  12. Philip Langdale | 9th August 2012 at 10:27 | Permalink

    Julian,

    You’re running it the way I run it. I assume you worked out how to place the .mount, .monitor and dbus service files.

    * Filenames: Yes, I described this in my blog post. This is how I avoid maintaining a cache – the filename itself lets me do direct lookups against libmtp (which are all by ID). This does make it pretty much unusable in any gvfs client that doesn’t use display names, and also makes the fuse bridge useless (but the fuse bridge requires open/read/write/close to work anyway, so it wasn’t going to do anything for you)

    * Device ID: Yes, and this is how the gphoto2 backend works too. The second number increases by one every time you plug the device in.

    * Copying a movie doesn’t work? How are you copying it? With Nautilus? The movie probably doesn’t work in the built-in player, but the file should copy fine. I’ve just tested here by copying some avi files to Movies.

    * Yeah, I touched on this. The app needs to understand gvfs in a particular way that I haven’t researched. So evince or file-roller know they have to download the file to /tmp to access it, but totem or eog don’t know, and try to do open/read/close on the URI, which will fail. You have to copy the file locally first – this is a fundamental limitation of mtp – it’s only a question of who does the copying.

    * Your sentence about the screen got cut off. I’m guessing you were going to say that you lost access to the storage area after the phone screen turned off (and the phone locked). For me, this doesn’t happen. I have to unlock the phone to initially see the storage area, but after that the access remains good until the phone is unplugged. It may time out eventually but that’s measured in 10s of minutes or hours. It’s a phone-side behaviour. Perhaps your phone is more aggressive in locking you out again. What model? I’ve tested with a Galaxy Nexus, Galaxy Note, and Nexus 7.

    Thanks for the feedback!

  13. Philip Langdale | 9th August 2012 at 10:31 | Permalink

    mgreene,

    Don’t use LIBMTP_Get_Filelisting or LIBMTP_Get_Filelisting_WithCallback. Instead, you’d use LIBMTP_Get_Files_And_Folders on the root directory, and then call it again on each sub directory as you go.

    Ideally you’d do this lazily, so only load the directory when you need it (which is how my backend works). But Rhythmbox’s code probably assumes that the full tree is loaded up front and changing that would be tedious. You could instead write a function that walks the whole tree using LIBMTP_Get_Files_And_Folders and emulates the external behaviour of LIBMTP_Get_Filelisting.

  14. Julian Andres Klode | 9th August 2012 at 13:22 | Permalink

    Copying movie: OK, I now noticed that the reason was that the file already existed in the directory. The error message “Error while uploading entity.” is not that helpful in that case.

    Screen: Yes, it seems to work with the screen turned off as well. Might just be some temporary problem.

    File names: How about encoding the real filename as well? Instead of using just the ID, use ID – Real Name, and then just ignore the part after the ID. This makes it work better with Evince, which otherwise might just just a number as the title.

    Streaming: It should be possible to support streaming in MTP as well, without seeking, much like it’s done with many HTTP streams. You basically start to download the file via MTP, but if the user aborts the streaming, you just let the rest of the download continue and wait for it to finish (or cancel it, but not all devices seem to be happy with that according to libmtp documentation). Might be better than having it copy that to a local file first.

    I also tried to copy a file from the device by using gvfs-copy, but apparently this does not work correctly and causes gvfsd-mtp to enter some bad state (the volume is unmounted and it can’t be remounted without reconnecting the Galaxy Nexus).

  15. Philip Langdale | 9th August 2012 at 13:58 | Permalink

    Julian,

    Copying movie: Yeah, the error message handling in libmtp is pretty terrible. It swallows the specific error codes generated by libptp and just gives you a binary success/fail. In this particular case, the error from libptp is useless, so it wouldn’t help. I have pushed a change to at least print the error message to stderr, but it’s not suitable to use in the user facing error. In general, I have more to do to try and beat error reporting into shape.

    gvfs-copy: That was a bug – I was assumming there would always be a progress callback – which there isn’t with the command line tools. It works fine now.

    File names: Maybe. I’m not sure how much benefit there is here. It’s pretty much impossible to use outside of Nautilus. If you’re looking for a bulk copy or sync solution, then mtp just isn’t it. As far as I can tell, rooting the device and using samba over the network is the least bad way to handle that. I’ve also had success using an rsync app.

    Streaming: Yes, you can do streaming without seeking in libmtp but it doesn’t map well to anything in the gvfs backend api, so it doesn’t really help. You might imagine using it with open/read/close and mark the stream as not supporting seeking, but then it’ll just fail for all the applications that can’t handle using the pull api anyway, because they expect to be able to seek in normal files. And you’ve now got to worry about temporary file location, size, lifetime, and the fact that you can’t cancel without b0rking up the connection (one of my devices can survive a cancellation).

    Ultimately, I’d rather not pretend it can do any more than it really can; none of the other tools have succeeded in this, and I don’t see how I can do any better.

  16. Dan Nicholson | 13th August 2012 at 11:12 | Permalink

    You’re awesome! I’ve been wanting this forever and always had it as a project I’d like to play with but never found time. I very much agree with your decisions about how to implement this, especially about not getting too cute abstracting real file system semantics on top of MTP. MTP is not perfect, but definitely serves a purpose.

    I’d suggest pinging the libmtp list with any issues. I follow it and development is pretty pleasant. Both the libmtp maintainer (Linus Walleij) and gphoto2/libptp maintainer (Marcus Meissner) are responsive there.

  17. Jeremy | 15th August 2012 at 21:38 | Permalink

    I can’t get this installed on Ubuntu 12.04. Yes, I’ve tried:
    installed gtk-doc-tools; autogen
    installed libdbus-1-dev; autogen
    make then gives:
    gvfsafpvolume.c: In function ‘create_directory_get_filedir_parms_cb’:
    gvfsafpvolume.c:1085:3: error: dereferencing pointer to incomplete type

    Past that, I code too much from 9 to 5 to dig into this at midnight. Anyone know what I’m missing or doing wrong? Or is it just a broken build, and I need to wait a few days?

    I’m completely stoked to get mtp access to my GNex, and if this works well enough, buy myself a shiny new Nexus 7.

    Thanks, regardless!

  18. Philip Langdale | 15th August 2012 at 21:56 | Permalink

    I don’t understand it.

    So I build with –disable-afp

  19. nocrack | 30th August 2012 at 07:11 | Permalink

    Sorry for the dumb question but what am I supposed to do with your link to github ?
    Can you explain what is needed to be done to get this working on Ubuntu 12.04 x86_64 ?
    That would be awesome!

    Thanks!!

  20. Stefan Glasenhardt | 9th October 2012 at 13:18 | Permalink

    Hi Philip,

    I’ve updated the code tree to version 1.12.3 (Only source and translation files) to e.g. fix the compilation problem with the file “gvfsafpvolume.c” (You only have to add “&” in front of the variable “info” in line 1085 of the file).

    You can download the patch-file at the following URL:

    http://ubuntuone.com/0Zvv74awsfOnd48rNuWaV2

    I’m currently thinking to upload my self-build packages to my PPA. Do you think the code is robust enough for daily use?

    Greetings Stefan

  21. Philip Langdale | 9th October 2012 at 13:23 | Permalink

    The recommended way of updating it to 1.12.3 would be to merge the mtp-1.12.1 branch into a new branch based off the 1.12.3 tag, but that’s getting into git shenanigans.

    The code is certainly robust enough for daily use, and if you’re in a position to do PPA builds, I think that would be very beneficial for other users.

    Thanks!

  22. Stefan Glasenhardt | 10th October 2012 at 05:17 | Permalink

    Just one question:

    Did you implement copying of folder structures? Every time i’m trying to copy something to the device there is an error message. Copying a folder from the device crashes the connection.

  23. Philip Langdale | 10th October 2012 at 06:19 | Permalink

    No. Copying of folder structures has to be done by the application (ie: nautilus), but it doesn’t understand what to do. I can’t secretly copy all the files if the app is only expecting one, as it can’t show correct progress information. But it shouldn’t crash either. It should just create an empty folder.

  24. Stefan Glasenhardt | 11th October 2012 at 11:27 | Permalink

    > But it shouldn’t crash either. It should just create an empty folder.
    It is not a real crash. An empty folder is created and the Nautilus window and also the connection closes. Your solution is far from perfect but works at least better than the “build-in” implementation.

  25. Philip Langdale | 11th October 2012 at 13:36 | Permalink

    I’ve checked in a fix for the crash, so copying from the device now “works”. I have a WIP fix for copying to the device, but it’s not quite right at the momemnt.

  26. Adam Baxter | 27th November 2012 at 13:36 | Permalink

    When running autogen.sh, I’m getting a version that’s built with:
    MTP support: no
    What parameters do I need to pass?

  27. Odin Hørthe Omdal | 10th January 2013 at 22:20 | Permalink

    So… Upstreamed yet? :D

  28. Philip Langdale | 11th January 2013 at 21:10 | Permalink

    It is now!

  29. Michael | 28th January 2013 at 11:31 | Permalink

    Hello,
    which way do i have to use, to get the files from the device with the right name?
    I see your point:
    “I took advantage of the fact that gvfs lets you define ‘display names’ for files – which can be completely different from the reported filename”

    Regards.

  30. Han-Wen | 6th February 2013 at 23:10 | Permalink

    I’ve just rewritten parts of go-mtpfs; I thought you might be interested in it. It now uses android MTP extensions to avoid the upload/download dance but edit files in place.

    See https://github.com/hanwen/go-mtpfs/blob/master/fs/android.go for more information.

  31. Philip Langdale | 7th February 2013 at 07:27 | Permalink

    That’s fascinating! Are these extensions documented anywhere?

    I’ll need to expose them through libmtp first.

    Thanks!

  32. Han-Wen | 17th February 2013 at 04:08 | Permalink

    USTL!

    Have a look at git://git.omapzoom.org/platform/frameworks/av.git , under media/mtp/MtpServer.cpp.

  33. David Díaz | 2nd May 2013 at 23:46 | Permalink

    Hi,

    Not sure if this is the correct place to report an issue.
    When copying files from an Android device (Samsung SII) to Ubuntu 13.04 the timestamp of the files created on local is not preserved. They take the actual timestamp.

    May I file a bug at bugzilla.gnome.com?.
    Regards.

  34. Philip Langdale | 3rd May 2013 at 12:43 | Permalink

    I don’t consider that a bug. It seems desirable to preserve date/time information in this situation.

    But yes, bugs should be filed in the gnome bugzilla.

  35. David Díaz | 7th May 2013 at 23:35 | Permalink

    As you stated, I meant the time stamp of the original files is not preserved when copying from phone to PC. I also consider desirable to preserve the time and date of the original photo file.

    I have file the bug:
    https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/1175947

    Hope it will be resolved.

    Thank you very much,
    Regards

  36. lester crsm | 21st November 2013 at 16:26 | Permalink

    Not sure where this is going but from a users point of view
    With images
    I am using mint xfce distro(15) and if using gvfsd-mtp my galaxy S3 mounts ok but there are no thumbnails loaded and both gthumb and ristresso fail to show thumbnails and take forever to load one image.
    If I use go-mtpfs the thumbnails load reasonably quickly in thunar and also in ristresso and gthumb and can open them in gimp. I’ve added go-mtfs as a custom action to thunar to make it easier. for everything but images get i/o error with go-mtpfs
    pdf’s open directly from the volume with gvfsd-mtp only.
    files and documents needing to be copied do not keep there names etc.
    can’t play music directly
    You state at the beginning that USB mass storage is somehow deficient, but MTP is presently not fit for purpose to replace it. (just waiting for my S3 4.3 upgrade to explode)

  37. Philip Langdale | 22nd November 2013 at 08:37 | Permalink

    The main thing you’re suffering from is the lack of support for the direct i/o extensions in Samsung’s mtp stack. With those, you’d get working previews. You could install cyanogenmod but that’s obviously a major step.

  38. Bob | 28th November 2013 at 20:54 | Permalink

    Does this work on FreeBSD?

  39. Philip Langdale | 28th November 2013 at 21:35 | Permalink

    In theory it should. libusb has a FreeBSD backend and gvfs/GNOME work there, but I’ve never tried it.

{ 2 } Trackbacks

  1. [...] already have fuse/fuse-libs and gvfs-fuse installed. Googling around I see there’s work on a MTP interface to GVFS but it’s not ready [...]

  2. GNOME 3.8 is underway | ewgeny | 26th March 2013 at 03:00 | Permalink

    [...] usage. One the features I have discovered already is the integrated MTP support (via libmtp) in gvfs. It allows you an out of the box connection, transfer and managing of you files with new Android [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *