Transferring Email From Gmail/Google Apps to Dovecot With Larch

eric — Wed, 09 Dec 2009 12:15:56 +0000

As regular readers of this blog know, I am in the process of trying to back up Google Apps accounts to Dovecot. Well I have finally found my solution. Not only does it work, but its in Ruby.

First thing that you’ll need to do is grab yourself a copy of Larch. I did this simply by typing and it installed everything nicely, but click the link to the repository on Github if it doesn’t work for you.

$ sudo gem install larch

Immediately I dove in and ran it. And immediately I hit my first problem:

$ larch --from imaps://imap.gmail.com --from-user user@gmail.com --from-pass FILTERED --to-pass me@myserver.com --to-pass FILTERED --all
... Removed for brevity ...
NameError: uninitialized constant Larch::IMAP::OpenSSL (cannot recover)

The fix for this issue is that SSL needs to be available (or installed as a library) to Ruby. Since I have Ruby built from source, it was just as easy as what’s below. I have removed the output for readability.

$ cd /usr/local/src/ruby-1.8.6/ext/openssl
$ ruby extconf.rb
$ make
$ sudo make install

Now that we have SSL installed, let’s try it again.

[Dec 07 16:54:06] [info] user1@googleappsdomain.com@mail.mydomain.com: Net::IMAP::NoResponseError: Mailbox doesn't allow inferior mailboxes (will retry)
[Dec 07 16:54:09] [fatal] Net::IMAP::NoResponseError: Mailbox doesn't allow inferior mailboxes (giving up)
[Dec 07 16:54:09] [info] 69 message(s) copied, 0 failed, 0 untouched out of 69 total

Whoops, another stopping point. If you are receiving the above error, then you need to make sure that your dovecot installation is set up for maildir. What this error is saying is that your installation doesn’t support folder that are folders and mailboxes at the same time. If your current dovecot installation is setup for mbox, you will need to convert it. That is beyond the scope of this post, but add the code below in the /etc/dovecot.conf if it’s not already there.

mail_location = maildir:/var/spool/mail/vhosts/%d/%u/Maildir
namespace private {
separator = /
prefix = ""
inbox = yes
}

I have been happily and successfully running Larch ever since.

Backing Up Gmail/Google Apps to a Dovecot Server

eric — Thu, 03 Dec 2009 11:00:48 +0000

I have been trying to find a way to copy everything from a Gmail account to a Dovecot mail server. The way I have ended up doing it so far is simply by using Apple Mail (if you regularly read this blog, you’d know that I use a Mac). The steps are as follows:

Create 2 accounts in Apple Mail: Gmail and the Dovecot account
Sync the Gmail account to your local computer
Copy everything to the Dovecot server

This works, but I have to use a slow connection (my home connection) and I have a lot of accounts to do this for, so I would much prefer to script this. The problem is that I have been trying to get this to work with either imapsync or imapcopy. Neither seem to work properly.

According to the documentation, I have the proper version and it should work with GMail. I tried all the supported –authmech1 possibilities. I found an article saying that Google doesn’t accept the word AUTHENTICATE when authenticating, but then imapsync wouldn’t work and it claims it does. So, when trying imapsync, this is what it looked like (some of the security issues have been blocked to protect the innocent):

[root@smtp4 imapsync-1.286]# ./imapsync --host1 imap.gmail.com --ssl1 --user1 user1@gmail.com --password1 MASKED --host2 mail.server.com --user2 user1@server.com --password2 MASKED
$RCSfile: imapsync,v $ $Revision: 1.286 $ $Date: 2009/07/24 15:53:04 $
Here is a [linux] system (Linux mail.server.com 2.6.18-92.1.18.el5xen #1 SMP Wed Nov 12 09:48:10 EST 2008 x86_64)
with perl 5.8.8
Mail::IMAPClient 3.21
IO::Socket 1.31
IO::Socket::SSL
Digest::MD5 2.36
Digest::HMAC_MD5
Term::ReadKey
Date::Manip
and the module Mail::IMAPClient version used here is 3.21
Command line used:
./imapsync --host1 imap.gmail.com --ssl1 --user1 user1@gmail.com --password1 MASKED --host2 mail.server.com --user2 user1@server.com --password2 MASKED --useheader Message-Id --skipsize --regextrans2 s/\[Gmail\]/Gmail/

My attempt at imapcopy was closer to successful, but still not what I needed. I ran into some Dovecot errors (at least I think that’s what it is) that I am not sure what to do with.

[root@smtp4 imapCopy]# time sh imapCopy.sh imaps://user1%40gmail.com:MASKED@imap.gmail.com imap://user1%40server.com:MASKED@mail.server.com
2009-12-02 18:13:28,811 INFO ImapCopier - Starting
2009-12-02 18:13:28,812 DEBUG ImapCopier - Parameters length:2
2009-12-02 18:13:28,813 DEBUG ImapCopier - opening conections
2009-12-02 18:13:28,816 DEBUG ImapCopier - opening imaps conection to imap.gmail.com
2009-12-02 18:13:32,633 DEBUG ImapCopier - opening imap conection to mail.server.com
2009-12-02 18:13:34,666 DEBUG ImapCopier - Creating target Folder: Adbrite
Exception in thread "main" javax.mail.MessagingException: Unsupported type;
nested exception is:
com.sun.mail.iap.ProtocolException: Unsupported type
at com.sun.mail.imap.IMAPFolder.doCommandIgnoreFailure(IMAPFolder.java:2618)
at com.sun.mail.imap.IMAPFolder.create(IMAPFolder.java:725)
at com.fisbein.joan.model.ImapCopier.copyFolderAndMessages(ImapCopier.java:190)
at com.fisbein.joan.model.ImapCopier.copy(ImapCopier.java:152)
at com.fisbein.joan.model.ImapCopier.main(ImapCopier.java:35)
Caused by: com.sun.mail.iap.ProtocolException: Unsupported type
at com.sun.mail.imap.IMAPFolder$6.doCommand(IMAPFolder.java:745)
at com.sun.mail.imap.IMAPFolder.doProtocolCommand(IMAPFolder.java:2639)
at com.sun.mail.imap.IMAPFolder.doCommandIgnoreFailure(IMAPFolder.java:2611)
... 4 more

So if you have any idea what to do here in a more intelligent manner than I am stuck with, please let me know in the comments. Thanks.

Erics Tech Blog » imap

Transferring Email From Gmail/Google Apps to Dovecot With Larch

Backing Up Gmail/Google Apps to a Dovecot Server