As Opera Software's lead QA for Opera Mail, I get a lot of opportunities to analyze the communication between IMAP clients and servers. Gmail is by far the quirkiest and most troublesome IMAP implementation I've seen to date. Gmail's own documentation describes how clients must work around the deficiencies built into their implementation. I understand that Gmail's IMAP implementation is still in beta (a Google norm), but some of the basic design choices are at the heart of the problem, which leads me to believe they won't be changing. These decisions are anti-IMAP (even the IMAP spec. author finds Gmail's IMAP disappointing) and cause Opera users headaches. Here are some of the specific problems:
- Gmail's labeling system could integrate marvelously with IMAP clients if only it used IMAP keywords. Instead, IMAP mailboxes are used to represent labels. All messages (sent and received) are always available in the "Gmail/[All Mail]" mailbox, so any time a message is labeled, a duplicate message is added to the label's IMAP mailbox. IMAP clients then receive several copies of the same message, none of which integrate with the client-side labeling system. If Gmail had instead used IMAP keywords, only one message would be needed and integration would be seamless.
- Gmail doesn't allow users to change mailbox subscriptions. Unsubscribing from label mailboxes is a sensible work-around for the above design deficiency, but it's inexplicably not allowed.
- Gmail doesn't handle standard IMAP flags, such as "\Deleted", "\Answered", and "\Recent". For instance, instead of intelligent handling of "\Deleted" flags in the web interface, users are required to use a variety of work-arounds to mimic the functionality already built into the IMAP protocol.
In addition to the unfortunate design decisions listed above, Gmail's IMAP implementation has a number of bugs:
- Gmail encodes message headers containing Slavic characters (at least) incorrectly when clients download only message headers. Headers are encoded correctly if a client downloads message bodies and headers together.
- Gmail sometimes sends malformed BODYSTRUCTURE responses (often used to determine the number and size of attachments to a message before it's downloaded).
To their credit, Gmail has already fixed another bug (in the STATUS command response) we reported to them. I hope they rethink their design decisions and fix the remaining bugs soon, enabling a seamless web interface and IMAP client experience. Otherwise, the burden falls on IMAP clients to implement hackish work-arounds for Gmail, rather than relying on open standards.
Special thanks to Arjan van Leeuwen for investigating many of these issues.
Note: Comments with a light blue background were made by the site owner.
Re: Gmail's Buggy IMAP Implementation
and thats why I hate google. google is only a search-engine for me and thats all o.O
No surprise
Google just doesn't give a damn about web standards. They completely ignore XHTML (bot doesn't accept it plus AdSense, Analytics, Maps and other scripts break in XML mode), their websites are like in '90s — IE and Netscape (Firefox) only and none of their websites even resemble proper HTML (gosh, by now even Microsoft cleaned up their pages!)
Re: Gmail's Buggy IMAP Implementation
I totally agree with you. But we must at least give them credit for trying.
Re: Gmail's Buggy IMAP Implementation
Hey, thats why they call it beta. I'm generally happy with it, however it screws up apple mail :(
Re: Gmail's Buggy IMAP Implementation
Actually, I like the Label-as-mailbox implementation. It does make sense to me and I can see why they chose that path. No need to support the delete flag. Deleting it from a Label / imap folder means removing the label, and if I want to really delete it, I'll move the message to the gmail/trash folder.
Re: Gmail's Buggy IMAP Implementation
It may make sense to you since you took the time to understand what Gmail is doing, but it certainly isn't normal IMAP server behavior. What you do is work-around Gmail's implementation rather than use the functionality built into your IMAP client.
Re: Gmail's Buggy IMAP Implementation
Yep, that really isn't very smart. My ~3GB gmail inbox uses up more than 10GB on my harddrive, because I like to use labels. Very, very annoying!
Re: Gmail's Buggy IMAP Implementation
Yeah this really messes with clients like Apple Mail. Makes things a bit weird to use.
-Owl
Re: Gmail's Buggy IMAP Implementation
I abandoned Gmail for this very reason after only using it for a few days. The way Gmail's web interface for mail 'labelling' works is really weird and trying to get it working with Apple Mail was a hellish experience. Google make some great applications but in this case, the implementation falls short of accepted behaviour in a most disappointing way. I hope you guys manage to get Google to listen..!
Re: Gmail's Buggy IMAP Implementation
I understand the rationale behind not using IMAP keywords: every IMAP client supports folders, but support for keywords is patchy, at best (and filtering 20,000 messages based on keyword can be excruciatingly slow in some clients).
I've worked around the fact that I get duplicate messages in [Gmail]/All Mail by forcing a folder path prefix and prepending that to all of my labels; Mail.app really doesn't handle the sheer amount of e-mail I have in All Mail very well at all—the snag is that I lose IMAP access to GMail's spam folder. The problem Google has is that it's possible to archive messages, which stores them _only_ in All Mail, which keywords wouldn't help with (though really there should be an “Archive” folder (with a matching view in the web interface) that showed messages that had been archived and didn't have any labels applied to it.
I have to say, though, I actually prefer treating labels as folders, because that's how I work with GMail: if I stored all of my mail in a single mailbox and just used labels/keywords, every mail client I use regularly (Mail.app, Thunderbird, mutt) would suffer greatly performance-wise—at the very least when starting up. They're just not designed for that many messages in a single folder (it'd be lovely if they did deal with it, but I don't hold out much hope).
All of this could be solved with an Archive folder, settings to control whether labels should be exposed as keywords or as folders, and the ability to unsubscribe from All Mail—none of it's beyond the bounds of possibility.
Re: Gmail's Buggy IMAP Implementation
Gmail is just very strange all round. If it weren't for the awesome webmail I would have dumped it by now.
What possible excuse can there be, for example, for sometimes presenting sent mail to IMAP clients in the inbox?! There are all kinds of weird bugs that basically seem to be the result of them shoehorning IMAP/POP access into their email model, rather than using their webmail interface to simply present standard information in the way they require.
Re: Gmail's Buggy IMAP Implementation
I consider myself a techie, but there so many standards I rely on the experts and I really thought Google was the expert. But today I learn that IMAP could easily support GMail's labels!!! It's possibly true that many clients do not support keywords (Opera possibly being the notable exception and most GMail like client). On the otherhand, if GMail implemented keywords how long would it have taken all the open-source clients to implement fixes for keyword support? Isn't that the beauty of open-source?
Re: Gmail's Buggy IMAP Implementation
Tim, I didn't know you had a website except your profile on My Opera. I must have stupidly overlooked the link you had in your profile the times I've been there. Stupid me, but it looks like, my friend, you've been fireballed.
I ran into this problem trying to set up my gmail account on Opera and Mail.app both. What I ended up doing is forwarding the email from my gmail account to my personal one and just setting up a send only account so I can reply with the gmail address. I never knew what was going wrong until you explained it here. Interesting, indeed.
Re: Gmail's Buggy IMAP Implementation
I normally don't do comments that only say "I agree", but in this case I hope a high number of comments lead to Google considering your points. Google: Please include mailbox unsubscription! And for everything else: I couldn't agree more!
Gernot.
Re: Gmail's Buggy IMAP Implementation
Very few implementations support IMAP keywords and when they do it's limited. Thunderbird if I recall, can support up to 5 keywords. Apple Mail doesn't support them AFAIK.
Most e-mail programs are folder-based. With IMAP keywords, you'd lose the ability to store messages in folders unless you had an unusual e-mail client that represented keywords as folders. Gmail itself is the only one I know of that does this; it's this very impedance mismatch that they are trying to solve.
If Gmail had subscription support, you could unsubscribe from the "Gmail/[All Mail]" mailbox, which fixes the duplicate messages. Keywords are useful, I just don't think they're the solution to this particular problem.
Re: Gmail's Buggy IMAP Implementation
Keyword support in Thunderbird 2 was actually completely reworked - the limit of 5 keywords is gone, as is the absurd server-side naming ("label-1", "label-2", etc.). It's really pretty solid. In combination with a decent IMAP server (e.g. Dovecot) and saved searches based upon keyword you can put together a pretty nice "tag-as-folder" system ala Gmail. Was decent server-side keyword support more widespread, Thunderbird could easily be extended to provide magic keyword folders by default.
Re: Gmail's Buggy IMAP Implementation
FWIW, Thunderbird (2.0.0.9, at least) is still using $labelN for the built-in labels.
Re: Gmail's Buggy IMAP Implementation
Yes, I believe that is a necessary concession to backward compatibility, and for IMAP servers that do not support arbitrary user-defined flags. New tags use an IMAP flag based on the tag name ("Call Back" is marked with a flag named call_back); it is not necessary to use the $labelN format.
The good news is that Gimap allows arbitrary flags, so this is supported. The bad news is that there is no Gmail web counterpart. (This is where the two keyword concepts do not intersect - a better solution would be to use Gmail labels for user-defined flags, and allow us to choose which labels map to flags, and which to folders, in Gimap.)
Re: Gmail's Buggy IMAP Implementation
...If you delete the first five default tags you can say goodbye to the $labelN forever. As the other poster says, they're only there for compatibility, but can easily be scrubbed.
"BETA"
I guess that's why it's "BETA". Speaking of which, we're coming up on the 4th anniversary of the GMail BETA period. Will there be cake?
Re: Gmail's Buggy IMAP Implementation
I have to agree with those who support Gmail labels as folders. It makes sense, and is supported in most IMAP clients, unlike the keywords or user-defined flags.
There are also some factual errors in the blog:
Not really an error, but incomplete:
The \Deleted flag is supported in the "All Mail", "Trash" and "Spam" folders, and the latter two support EXPUNGE. The \Deleted flag behavour doesn't bother me, but is non-standard.
My biggest complaint is the lack of subscription support. It is possible, as noted above, to work around it by using a server directory prefix, but that should not be necessary.
The header encoding problem is understandable, but the delay in getting it fixed is inexplicable.
Re: Gmail's Buggy IMAP Implementation
Thank you for the additional information. Gmail's documentation says they don't support the \Answered flag, so I took that for granted. It's good to hear that a BODYSTRUCTURE bug was fixed, though there were at least two issues we reported. Do you have any more information on the fix they did?
Re: Gmail's Buggy IMAP Implementation
They didn't support \Answered when they first launched Gimap, but they fixed it early. They also added support for the $Forwarded flag.
The BODYSTRUCTURE response problem I knew about was omitting the extra parameters on a multipart entry. The lack of a BOUNDARY value prevent Windows Mobile users from seeing HTML messages, and caused Thunderbird to download full messages, including attachments, when opening a message.
Another bug, not mentioned before:
They are still returning a short value in response to RFC822.SIZE, which can lead to truncated messages. (They are off by one byte for each header line, apparently not counting two characters for CR/LF.)
Re: Gmail's Buggy IMAP Implementation
There were two problems we found:
Re: Gmail's Buggy IMAP Implementation
I have not seen that one. The only time I have seen a sub-type of RFC822, is for an attachment of type Message/RFC822, as it should be. But, there is a bug here that I had not previously noticed.
RFC3501 requires:
A body type of type MESSAGE and subtype RFC822 contains,
immediately after the basic fields, the envelope structure,
body structure, and size in text lines of the encapsulated
message.
Google doesn't provide any information about the contents of the attached message. This may be related to the overall Gmail implementaton, which treats message/rfc822 attachments as plain text.
Re: Gmail's Buggy IMAP Implementation
I had forgotten about another BODYSTRUCTURE response bug: Gimap reports the correct byte size for text parts, but zero lines. I don't know of any clients that rely on the line count, but there may be some.
Re: Gmail's Buggy IMAP Implementation
That's why I switched back to POP account in Apple Mail 3 days ago. I couldn't keep any longer those multiple messages, multiple drafts saved in Gmail's Inbox when I compose mail in Apple Mail... When I downloaded all mail again to my POP acount, I had to spend several hours cleening up those multiple emails.
Re: Gmail's Buggy IMAP Implementation
Gmail's IMAP has been the most double-edged sword in my computing life in memory: awesome and awful. I laugh out loud when I see articles that make it sound simple to set up Apple Mail with Gmail and IMAP. I pity da foo ...
Re: Gmail's Buggy IMAP Implementation
What I hate most about the using Mail.app to read my Gmail ( as opposed to the browser ) is having to logon to Gmail via the browser once a week to delete the messages in the All folder that I already deleted from my Inbox. And then since Gmail created a "Deleted Messages" folder to represent the folder in Mail.app I also have to delete the messages in there. Then I have to go to the Gmail trash label and empty the trash. AAARRRGHHHHH!
Re: Gmail's Buggy IMAP Implementation
what I don't like about Apple mail + Gmail IMAP is that when I delete a message, it automatically gets deleted instead of archived :(
I have do manually drag it in to "all mail" folder, which is hard to remember.