The Autonomous Configuration of mainstream email clients

If you’ve ever configured a mainstream commercial email account (e.g. Gmail Hotmail) in a mail client such as Thunderbird, Outlook or Apple Mail, you would have noticed that all you need to do is to type in the email address, and then it’s done. How cool it is! However, if you set up an email account hosted on your own mail server, what they do at best is that they guess the server address (for me@abc.com, they put smtp.abc.com for SMTP server, etc) Nothing more than that. If you want to use a different address for email server, you’ll have to type it in manually.?So there we have the auto configuration.

For different clients we have different configuration mechanisms. I’ll cover some mainstream email clients: Mozilla Thunderbird, Microsoft Outlook, and Apple Mail.app. Most of the other clients follow the same criteria as at least one of those above.

SRV Records

Defined by RFC 6186, the SRV Records was accepted by most of the email clients.

 

Mozilla Thunderbird

They have five steps to determine the email server:

    1. Check the local configuration file.
    2. Check for autoconfig.example.com, an server hosted by the email provider.
    3. Inquiry ISPDB, a free-to-use central database maintained by?Mozilla Messaging.
    4. Try to use SRV Records as described above.
    5. Guessing.

 

 

Given the email address “fred@example.com”, Thunderbird first checks <http://autoconfig.example.com/mail/config-v1.1.xml?emailaddress=fred@example.com> and then <http://example.com/.well-known/autoconfig/mail/config-v1.1.xml>

The definition of the configuration file can be found here. But for the sake of convenience I’ll post an example here.



<clientConfig version="1.1">
  <emailProvider id="freenet.de">
    <domain>freenet.de</domain>
    <displayName>Freenet Mail</displayName>
    <displayShortName>Freenet</displayShortName>
    <incomingServer type="imap">
      <hostname>imap.freenet.de</hostname>
      <port>993</port>
      <socketType>SSL</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <incomingServer type="imap">
      <hostname>imap.freenet.de</hostname>
      <port>143</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <incomingServer type="pop3">
      <hostname>pop.freenet.de</hostname>
      <port>995</port>
      <socketType>SSL</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <incomingServer type="pop3">
      <hostname>pop.freenet.de</hostname>
      <port>110</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <outgoingServer type="smtp">
      <hostname>smtp.freenet.de</hostname>
      <port>587</port>
      <socketType>SSL</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </outgoingServer>
    <outgoingServer type="smtp">
      <hostname>smtp.freenet.de</hostname>
      <port>587</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </outgoingServer>
    <documentation url="http://kundenservice.freenet.de/hilfe/email/programme/config/index.html">
      <descr lang="de">Allgemeine Beschreibung der Einstellungen</descr>
      <descr lang="en">Generic settings page</descr>
    </documentation>
    <documentation url="http://kundenservice.freenet.de/hilfe/email/programme/config/thunderbird/imap-thunderbird/imap/index.html">
      <descr lang="de">TB 2.0 IMAP-Einstellungen</descr>
      <descr lang="en">TB 2.0 IMAP settings</descr>
    </documentation>
  </emailProvider>
</clientConfig>

Microsoft Outlook

Outlook performs an HTTP POST request to https://autodiscover.emaildomain.com/autodiscover/autodiscover.xml with the following content:



  
    http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a
    fred@emaildomain.com
  

There is some decent documentation in this MS TechNet article on how to format the response. To very quickly cut to the chase, you can find my example here with the minimum required to set up a simple email account. You will notice that the is blank. I’ll get to that in a minute:?https://ns2.samhostuk.net/autodiscover/autodiscover.xml

Remember that this POST request will?go via HTTPs and not HTTP. You don’t want to have different SSL certificates for each email domain you are hosting, each matching autodiscover.emaildomain.com (also considering that Apache cannot have name based virtual hosts with different SSL certs). There is a very elegant way to solve this:

Outlook first does a DNS lookup for a SRV record at _autodiscover._tcp.emaildomain.com?to find the address of the autodiscover server. This record should have the following format:

0 0 443 ssl.mailprovider.com

This means that instead of looking for the XML file at https://autodiscover.emaildomain.com/autodiscover/autodiscover.xml, Outlook will now look at https://ssl.mailprovider.com/autodiscover/autodiscover.xml. If you are running bind, the entry in your zone file will look somewhat like this:

_autodiscover._tcp      SRV       0 0 443 ns2.samhostuk.net.

If you want to test your record before going through the pain of firing up your Outlook client, run something like:

#> dig +short -t SRV _autodiscover._tcp.mydomain.com

This also means you do not have to set up an autodiscover subdomain for each email domain you host. You still however have to add this SRV record to each email domain’s DNS zone.

Getting the user’s email address in the response XML

I have found no documentation on valid tokens you can use in the response XML such as the %EMAILADDRESS% token honored by Thunderbird. This means it’s up to you to populate the element on the server side. Fortunately, the client just posted it to you, so all you have to do is extract it from the POST data and print it into the output XML. PHP is my language of choice, but you can implement a similar solution in your preferred server side scripting language.

First, you need to tell Apache that .xml files should be sent through the PHP interpreter. Locate the directive for your SSL host (on my Ubuntu install it’s in /etc/apache/sites-available/default-ssl) and add the following line:

AddType application/x-httpd-php .php .php3 .php4 .php5 .xml

Save your config and reload Apache.

Then, edit your autodiscover.xml file and add some magic. Here is what mine looks like (I am a believer that for this simple search, a preg_match is quicker than a full XML parse):



  
    
      myLANsite
    
    
      email
      settings
      
        IMAP
        ns2.samhostuk.net
        993
        off
        off
        on
        on
        
      
      
        SMTP
        ns2.samhostuk.net
        465
        off
        off
        on
        on
        
      
    
  

To test your script, save a copy of the Autodiscover XML Request above into a local file called req.xml?and then execute the below command to execute the request and validate the response:

cat req.xml | lwp-request -m POST https://autodiscover.mymaildomain.com/autodiscover/autodiscover.xml

If your _autodiscover._tcp SRV record and the autodiscover.xml file are correctly in place, Outlook users will be able to set up their email accounts absolutely hassle free.

Apple Mail

The Mail App works differently from Outlook at Thunderbird. It does not?perform any lookups to discover settings. It’s got the built-in presets for providers such as Hotmail and Gmail, but makes not attempt to look up settings for any other mail domains.

TheOdd1sOut and my deskmate who took his own life

Today I found this amazing channel on Youtube: TheOdd1sOut

I guess it means the odd ones could stand out in a crowd of people. Indeed it is, otherwise the odd Youtuber who made this somewhat childish channel wouldn’t get over 4 million subscribers.

When I say childish and odd, I mean these:

This guy told us a story about how he kept drawing comics regardless of the fact that he had only got 100 subscribers for his first 100 comics.

That remind me of my deskmate in the 9th grade, Zhao. He is a little bit fat with a relatively long hair, and he loves drawing. He draws wonderful Japanese Comics, and the way he uses his pencil is just like magic. He sits just next to me, and I always tutor him maths. We were really good friends as we were both weird, and we could still be friends today if he didn’t leave us in that dark May, right before the middle school graduation test, an exam designed to determine if we can go to high school and what high school we can go to.

I can’t really find out why he did that to himself. Actually I didn’t even notice he’s gone at first – I thought he’s just sick or something.

One week after that, he still didn’t come back. Then one of my fellows told me: “You know what? Zhao committed suicide!” I’ll never forget the casualness and the excitement of witnessing something unusual in his tone.

“What? I don’t believe it. You must be joking.”

“No, I’m serious!”said him with no seriousness at all, “I just saw his parents in teachers‘ office, talking about that!”

NO.” I exclaimed, running to the office with a traumatizing regret.  However, the doors were locked.

All my teachers refuse to talk about him. He wasn’t on the news, wasn’t mentioned in the school radio, can’t be found anywhere. My classmates stopped talking about him shortly after that. He just evaporated into the air, with no one noticing anything wrong. Two weeks later, one of my teachers finally said something like “Weak mentality” “Deepest regret” “Showed no filiality to his parents” “Don’t be affected or letting him disturbing your test prep”. I wasn’t really listening. “I guess he’s a little bit odd.” my teacher said at last. And then it’s just like nothing happened.

Maybe it’s the exam he “failed”, in which he’s only 1 credit from full; maybe it’s the conflict between he and his parents, most likely to be about drawing comics; maybe it has something to do with his girlfriend. The reason is not important for now, but only if he could hold on for a while, if the teachers and parents weren’t applying so much pressure on him, if we had a more favorable social environment…

Then he could be on his way of becoming the new TheOdd1sOut, drawing as much as he like.

And we could still be friends.

My story at the 2016 VEX World Championship

After we won the Asian-Pacific Championship, I continued to update my robot, preparing for the world championship. I added some important features in the program, including a menu system that configure the robot intuitively, and an auto-calibrate system that adjust the parameters in the runtime. I also amended the ball collecting mechanism, so that the robot could collect and shoot the balls much more quickly. The new system is much more accurate and rapid – by the time we left for the Worlds, the robot is capable of shooting 4 balls in 1second, each one of them dropping into the basket with 100% accuracy.

Our school sent 3 teams, one of which was from International Department. I was the captain of the teams. We set off on a chill, bright Tuesday morning with robots in giant boxes. After 12 hours of flight, we landed in Chicago O’Hare airport, and took a 6-hour bus ride to Louisville, where our competition would be hosted. It was a long journey, so I was wondering why we didn’t just fly directly to Louisville, as our hotel was right next to the airport. My complaint turned into approbation shortly afterwards when I saw the scenery outside the window. Unlike the gray, oppressive sky in China, it was blue and clear in the US. Exquisite farmer houses stood on the roadside, immense number of wind turbine dotted the background.

When we finally got to the hotel, another thing impressed me – the door card was actually an exclusive souvenir with VEX Worlds written on it.

The next morning when we step into the Kentucky Expedition Center, we found the competition more like a party, or a team talent showcase. There I saw people made a lot of things not for the competition but for fun. For us, though, anytime in the robotics lab was precious, so I spent all of that to improve my system.

A few hours later we went to the freedom hall where the opening ceremony would be held. That is the center of the party – people sit on their seats, cheering and applauding, bouncing giant balls up and down on the stand. They intentionally created a lot of haze in the hall. I do understand that it was to create a Tyndall effect so that the light beams would be visible in the air, but what I was thinking at that time was:’Chinese run from haze while Americans run for haze. How ironic it is!’

Teams from different countries walked the stage one by one, waving their nation flags. Surprisingly, there’s even a Syrian team, financed by an international charity institute. When the host introduced them, everyone in the hall stood up and applauded for them. Then started the match.

It turned out that our robot’s performance was much better than I expected. What concerned me, though, was the arrangement within our team. Only three members out of five would be allowed to stand beside the field during match; other people would need to seat on the bleachers, outside the restricted zone and away from the game. Why it matters? Well, since the match was significant, there will be TV broadcast during match. Those who stand within the restricted zone will have a much higher odd of being on the TV. There’s no argue that the driver, John, should be in, but the selection of the other two people became a problem. According to my plan, it should be myself and another international department guy, Anthony, who speaks fluent English. I wished him in because I valued the significance of good communication, which is the prerequisite of a successful game. I need to be in because I knew the robot best, so that I could configure the system before match, and fix the problem if there’s any.

However, driver John didn’t approve the plan. He insisted that he want Tim, a friend of him back in the primary school. I didn’t even see him very much before our departure. How he got into our team, I didn’t know: it’s just another teachers’ arrangement. What I do know is that he wasn’t even capable of using a screwdriver.

“No way.” I protested.

“Tim was my primary school partner. We work best together.” John argued.

It was totally bullshit: Tim didn’t even come to the lab once a month, so how could he practice for cooperation with John? And as far as I know, they rarely contact each other ever since their graduation from primary school. I could only assume they or their parents were having some secret deals. I was about to say “I’m the captain”, but I suddenly realized that it was a very bad time for conflicts. Thinking that I could configure the robot before matches begin, I said:”Okay, it’s up to you this time, but Anthony, the international department guy, must be in.” I was confident about the quality of the robot, since I’ve checked every component of it. And even if it did stopped working during match, I could still rush into the field, switching someone down temporarily. As for Tim, he was smart enough to load balls and carry the robot. I completely forgot the broadcast thing at that moment.

Then started the competition. I configured the robot, calibrated the sensors, handed it to my teammates, and then stand on the bleachers to gesture them what to do. We went for the alliances before each match, talking about technical issues and strategies. This job was mainly done by Anthony, because back at the time my English was far from fluent. I did met tons of friendly guys who are willing to listen to my crappy English, though. We also played against the Syrian team, who prayed in their pit before matches. How devout they are!

We were so strong that literally no team could beat us in the first 9 Qualifying Matches, but there was one extremely strong team playing against us in the last match. Then I came up with the idea that could save us from potential failure: Anthony and I persuaded them to be our alliance in the finals. Therefore, no matter who wins the game, the other one would also be in the finals. They ended up sandbagging us, because if my team won, we could rank first in our subdivision and do the alliance selection before anyone else.

The next day, I woke up early in the morning, knowing that this is the day when the finals would be held. Stepping outside the hotel, I saw an amazingly magnificent scenery: the dawn cloud was red, yellow and blue; I could really see a giant fireball rising on the horizon. The sky was clear with clouds as white as pearls. Light frost covered the trees, making them look fantastic. You may think this is common, but I couldn’t see such common things back in China; the sky was always grayish and hazy, and the horizon was obscured by high-rising skyscrapers.A good omen, I said to myself.

The finals was much more intense the qualifying matches. Matches followed matches with no rests in-between. Soon our motors started to lose power because of the heat. Fortunately we got help again: another Chinese team got beaten, so they want us to revenge. They gave us a bucket of dry ice, using which we could cool down the motors effectively.

We went through Quarters, Semifinals with no match lost. Then we went into the finals, winning the first and second match with ease. Or it was just I thought we won the second one. The referee received a note from somewhere, and dropped it angrily on the ground. After the match ended, we could see our balls significantly more than the opponents’, and both of the alliances did the Elevation, an important scoring action at the end of the game that earns us 50 points. The action must be preformed inside the scoring zone, a triangle at the corner of the field. The referee noticed that we had a tiny little bit of the robot stretched out of the zone in the midair. Maybe he was racist, maybe he preferred our opponent alliance which was consisted of three US teams instead of a mixture of US, Canada and China team, or he thought the fact that we didn’t lose a single match was abnormal. But for whatever reason, he decided to make it “an invalid action.”

“The rules clearly states that the robot must be in the scoring zone.” Said the referee indifferently.

“But we are! It’s just a ribbon!”  exclaimed Anthony angrily. “The purpose of the rule is to designate the position of the robot at the end of the match, not to offer you an weak excuse to subtract our score!”

“WhydonUcheouthatotheopponts’? Iswaretheirnobetter.” A member of our Canadian alliance team shouted so quickly that I could hardly recognize.

Nothing could be changed, though. But it doesn’t matter, because we still have the third match. We screwed them. They had less balls than us, and their elevation failed dramatically: the robot tipped over at the last second of the game. We even scored after the Elevation! It was such a surprise that all of the spectators cheers for us. It wasn’t something good for the match, though. The elevation scoring zone was far away from the goal, so shooting there would have a much smaller odd of scoring. Still, we did it, not for scoring, but to show how cool we are, so cool that we could win the game without referee’s help.

Perhaps impressed by our coolness, a team who made alliance with us earlier gave us a nice shirt of themselves, and intended to take a photo with us. I kept that shirt in my closet even today.

Then we went into the Freedom Hall for the Round Robin. Five champions of each subdivisions would compete against each other one by one for the qualification of the high school finals. We got loads of messages from teams that was formally our alliance, most of which wishing us good luck. Nothing noteworthy, but I do remember the high atmosphere in the hall. Everybody worked an entire year on their robot, and now they want to see what the best ones looks like. We won three out of four games, ranking first among five alliances. Then we won the high school finals, 2-1. The game was much intenser than the subdivision finals: For more than once we nearly lose the entire game. Fortunately, we made it to the end.

I still remember the excitement when I found my antagonist failed their Elevation in the very final match: some mechanical errors made their mechanism malfunction. We stared at their failed robots, mouth wide open, knowing that we just won the game – they can’t beat us with 50 points short. The commentator said: “So the red alliance… they just know that they win the world championship… For the blue, consistency and repeatability is everything in engineering…” Then a huge firework exploded just beside my feet. I was a little bit surprised by the sound, but I didn’t care. Then we rushed to the stage for the cup, the medal and the large certificate. A fun fact: we lose the certificate somewhere out of excitement, perhaps on the bus, or in a restaurant, so the certificate we had right now was actually a fake one, printed by ourselves. We did keep our cup and medals safe and sound; maybe we just didn’t value the huge piece of paper that much.

(I wasn’t on this photo because I was 1 meter behind them in the shadow)

Now thinking retrospectively, I still figure it the most inspiring experience in my lifetime. I’ve never worked with so many intelligent people before, and I think it’s a glory to play against the essence of them. It is this time that made me realized the engineering doesn’t have to be nerdy; It could be cool sometimes. And it is also this experience that intrigued me to think about the option of going to US for college. Is there anything better than working with the smartest and coolest people in the world?

Continue reading “My story at the 2016 VEX World Championship”