x

fastmail.xml (387721B)

---

 <?xml version='1.0' encoding='utf-8'?>
 <feed xmlns='http://www.w3.org/2005/Atom' xml:base='https://www.fastmail.com'>
     <title>Fastmail Blog</title>
     <subtitle>Blog posts from the Fastmail team</subtitle>
     <updated>2025-03-21T00:00:01Z</updated>
     <id>https://www.fastmail.com</id>
     <link rel='alternate' type='text/html' hreflang='en' href='https://www.fastmail.com/blog/' />
     <link rel='self' type='application/atom+xml' href='https://www.fastmail.com/blog/feed.xml' />
 	<rights>&#169; 2025 Fastmail Pty. Ltd. All rights reserved</rights><entry>
             <title>Not OK, Cupid</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/not-ok-cupid/' />
 			<id>https://www.fastmail.com/blog/not-ok-cupid/</id>
 			<updated>2025-03-21T00:00:01Z</updated><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;I don’t usually like to call out the bad behaviour of specific companies, but the egregious mis-design and lack of acknowledging it justify this case.&lt;/p&gt;&lt;h2 id=&quot;welcome-to-ok-cupid&quot; tabindex=&quot;-1&quot;&gt;Welcome to OkCupid&lt;/h2&gt;&lt;p&gt;A couple of weeks ago, I started seeing many “Welcome to OkCupid” emails, both on my personal address and a couple of related addresses, but also to multiple Fastmail official contact addresses — legal, partnerships, press, etc. Specifically, this list included &lt;code&gt;trash@brong.net&lt;/code&gt; — an address that has never been used to send or receive email and appears in precisely one place — &lt;a href=&quot;https://www.fastmail.com/blog/a-tangled-path-of-workarounds/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;an article on our blog&lt;/a&gt;! It seems quite clear that somebody scraped our website and used the addresses to sign up. I’m aware of at least 10 addresses, but there are likely others that either go to someone else or addresses that no longer exist.&lt;/p&gt;&lt;p&gt;It didn’t stop there, though. I’ve been getting tons of “someone likes you”, “you have an intro,” and even an “IMPORTANT: We removed your photo on OkCupid.” email saying that inappropriate content was posted to “our” account!&lt;/p&gt;&lt;h2 id=&quot;the-real-world-consequences-of-poor-email-validation&quot; tabindex=&quot;-1&quot;&gt;The real-world consequences of poor email validation&lt;/h2&gt;&lt;p&gt;This isn’t just an inconvenience — it has real security implications. Websites that fail to properly validate email ownership can be exploited for malicious purposes. Attackers can use unverified sign-ups to flood inboxes, making it easier to hide critical emails among the noise — something we’ve discussed our own experience of in our post on &lt;a href=&quot;https://www.fastmail.com/blog/when-two-factor-authentication-is-not-enough/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;2FA vulnerabilities&lt;/a&gt;. There are established &lt;a href=&quot;https://www.m3aawg.org/sites/default/files/document/M3AAWG_Senders_BCP_Ver3-2015-02.pdf&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;best practices&lt;/a&gt; (PDF) for handling email sign-ups responsibly, practices that OkCupid is failing to follow.&lt;/p&gt;&lt;h2 id=&quot;no-way-out&quot; tabindex=&quot;-1&quot;&gt;No way out&lt;/h2&gt;&lt;p&gt;When I tried to unsubscribe using the one-click unsubscribe button in one of the emails, I was met with an error: “Something went wrong, please try again later.”&lt;/p&gt;&lt;p&gt;Curious, I tried to recover a password on one of these accounts (the one with my personal email address) and successfully changed the password. Then, I was asked to confirm my login with a message sent to the number associated with the account. A number I didn’t know. A number that wasn’t mentioned on that page, so I still don’t know anything about it — not even which country it was from.&lt;/p&gt;&lt;p&gt;This raises further security concerns; the attacker could have also caused random recovery numbers to be texted to another poor victim’s phone. Alternatively, they could confirm that my email address is actively monitored, increasing its value for further attacks. Either way, what I couldn’t do was actually close the account.&lt;/p&gt;&lt;h2 id=&quot;whack-a-mole&quot; tabindex=&quot;-1&quot;&gt;Whack-a-mole&lt;/h2&gt;&lt;p&gt;So, I contacted OkCupid’s support. Here’s what they said:&lt;/p&gt;&lt;p&gt;&lt;em&gt;I’ve removed the user from the site and banned the email address to prevent any new accounts from being created. That should resolve the issue, but if you encounter anything like this again in the future, please don’t hesitate to reach out, and we’ll address it right away.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;So, I need to contact support manually for each new email address. This is neither scalable nor acceptable; people don’t have this amount of time.&lt;/p&gt;&lt;p&gt;Furthermore, my email address is now on another random blocklist somewhere on the internet, where I have no control and no way to unblock it. I don’t anticipate wanting to use OkCupid’s service, but if I did in the future, I would have to go through another dance to get the address unlocked again — or more likely, treat that particular email address as soiled and create another one.&lt;/p&gt;&lt;h2 id=&quot;not-ok&quot; tabindex=&quot;-1&quot;&gt;Not OK&lt;/h2&gt;&lt;p&gt;So I say, not OK, OkCupid. Not OK.&lt;/p&gt;&lt;p&gt;The usefulness of email depends on responsible behaviour from all service providers. Companies that engage in shady or outright inappropriate practices make the internet worse for everyone.&lt;/p&gt;&lt;p&gt;OkCupid’s failure to implement even the &lt;a href=&quot;https://en.wikipedia.org/wiki/Opt-in_email#Confirmed_opt-in_(COI)/double_opt-in_(DOI)&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;simplest form&lt;/a&gt; of email validation is unacceptable. Until they address these issues properly (not through the support response provided here), they remain part of the problem, not the solution.&lt;/p&gt;&lt;h2 id=&quot;could-we-have-avoided-this&quot; tabindex=&quot;-1&quot;&gt;Could we have avoided this?&lt;/h2&gt;&lt;p&gt;In this case, we published those addresses online. There’s always a risk of receiving spam when you do that, one could even reasonably say “we were asking for it”. We expected spam. If you want to reduce your risk of being spammed, it helps to not publish your email address on the public web!&lt;/p&gt;&lt;p&gt;What we we didn’t was expect a relatively reputable service being used to facilitate us being spammed.&lt;/p&gt;&lt;p&gt;One great protection is using different address for each different organisation you deal with — that way if your address leaks (or they sell it), you know where the breach happened, and you can more easily block just the problem messages.&lt;/p&gt;&lt;p&gt;Fastmail’s masked email feature is a great way to implement this strategy. Masked emails are designed, particularly when integrated with a password manager, to make it very easy to create new addresses, and track where they are expected to be used.&lt;/p&gt;&lt;p&gt;Being a good internet citizen is one of &lt;a href=&quot;https://www.fastmail.com/company/values/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Fastmail’s core values&lt;/a&gt;. We require verification for sending identities, ensuring that only legitimate users can send from an address they claim they own. This is the level of responsibility every email provider should uphold, and we applaud the others who also do.&lt;/p&gt;</content>
         </entry><entry>
             <title>The evolution of the advanced fee scam</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/the-evolution-of-the-advanced-fee-scam/' />
 			<id>https://www.fastmail.com/blog/the-evolution-of-the-advanced-fee-scam/</id>
 			<updated>2025-02-14T05:00:00Z</updated><author>
 				<name>Aric Archebelle-Smith</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;As one of Fastmail’s customer support agents, part of my job is making sure that our customers are well-informed about rising trends in fraud so that they can be sure to steer clear of them. While our customers tend to be tech-savvy enough to spot the average scam email from a mile away, online scammers grow increasingly more sophisticated every year.&lt;/p&gt;&lt;p&gt;I recently attended the 62nd General Meeting of the &lt;a href=&quot;https://www.m3aawg.org/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Messaging, Malware, Mobile Anti-Abuse Working Group (M3AAWG)&lt;/a&gt; in Toronto. There, I spoke with others in the email and anti-abuse industry about the increase in advanced fee scams they’d observed in the years since the onset of the COVID-19 pandemic.&lt;/p&gt;&lt;p&gt;Advanced fee scams are not a new type of scam, but scammers have begun running a much more sophisticated version of this old-school scam. One that can convince even those who know to be cautious when navigating the internet.&lt;/p&gt;&lt;p&gt;Historically, advanced fee scams involved scammers promising the victim some sort of too-good-to-be-true opportunity or reward. The only catch is that the the victim has to pay a fee before they can receive the promised reward or opportunity. Generally, the scammer claims this fee is just to cover processing fees, background checks, training materials, or some other reasonable sounding expense. They assure the victim that they’ll be reimbursed for this expense down the road. Once the victim pays the fee, the scammer goes silent and the victim realizes that they’ve been conned.&lt;/p&gt;&lt;p&gt;Until recently, advanced fee scams were your garden variety “Nigerian prince” scam that savvy internet users quickly learned to avoid. Someone would offer the victim a large payoff if the victim could just cover the relatively small wire transfer or bank processing fees. For most email users, this type of con was easy to detect and most people knew to watch out for them.&lt;/p&gt;&lt;p&gt;Advanced fee scams have recently evolved to masquerade as hiring and work-from-home opportunities, targeting people who are looking for work in an already highly competitive job market. The scammers will pose as hiring managers or recruiters, and will even go so far as to reach out to victims over legitimate hiring websites, such as LinkedIn.&lt;/p&gt;&lt;p&gt;The victim is led to believe that they are being considered for a job or internship opportunity, but they’ll be asked to pay a fee as part of the hiring process. In some cases, the victim is given a link to the company’s preferred online vendor, where they are told to purchase the items they’ll need for the job. The scammer tells the victim that they’ll be reimbursed for these purchases later. However, the link takes the victim to a fake webstore where the payment is taken, but no goods are ever sent. At this point, the scammer stops responding to the victim.&lt;/p&gt;&lt;p&gt;More frequently, the scammers ask the victim to pay a small fee to cover some other aspect of the hiring process. Generally, the scammer will claim this is an application fee or something similar. Of course, the scammer stops responding to the victim’s messages as soon as they receive the payment.&lt;/p&gt;&lt;p&gt;In some cases, scammers will even conduct actual phone or video interviews with the victim as part of the phony hiring process. There’s no way to know how this data is being used by the attackers without insider knowledge.&lt;/p&gt;&lt;p&gt;This combination of fraudulent hiring and advanced fee scams allows attackers to collect both money and personally identifying information from vulnerable populations.&lt;/p&gt;&lt;p&gt;I recommend the following precautions to avoid becoming the victim of one of these scams:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Confirm the legitimacy of any jobs you are interested in applying for by verifying that the position is listed on the company’s website.&lt;/li&gt; &lt;li&gt;Make sure that any emails you receive from a hiring manager are actually coming from the company’s domain or from the domain of a legitimate staffing agency. Double check that there are no typos or &lt;a href=&quot;https://itservices.wp.st-andrews.ac.uk/2024/03/12/identifying-fraudsters-using-cyrillic-characters&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;look-alike characters&lt;/a&gt; in the domain.&lt;/li&gt; &lt;li&gt;Take the same precautions with any URLs that are shared with you via email or on hiring sites. Scammers can set up convincing look-alike websites, but you can check the URL to verify that you are being directed to the company’s legitimate website.&lt;/li&gt; &lt;li&gt;Even if a message appears to be sent from a company’s actual domain, there’s a chance that the message could be spoofed, meaning the scammer forged the email’s “From” address to make it look like it came from a certain person or company. Chances are that these messages would get flagged as spam, but it’s still a good idea to confirm that a message hasn’t been spoofed by checking the headers of the message. Fastmail makes it easy to view the full headers of a message. Simply click the &lt;strong&gt;Actions&lt;/strong&gt; drop down and select &lt;strong&gt;Show raw message&lt;/strong&gt; to see the full headers of the message and verify that the message passed &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/1500000280461-Sender-authentication&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;sender authentication checks&lt;/a&gt;. If you’re not familiar with how to read email headers, you can always reach out to Fastmail’s friendly and knowledgeable &lt;a href=&quot;https://support.fastmail.com/support/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;support team&lt;/a&gt; to help confirm a message’s legitimacy.&lt;/li&gt; &lt;li&gt;If a job opportunity seems too good to be true, or you’re told that you’ve been accepted for a position almost immediately with little to no interview process, chances are the hiring manager or recruiter that you’re talking with is actually a scammer.&lt;/li&gt; &lt;li&gt;If at any point in the interview process the recruiter asks to stop communicating via email and asks you to contact them on Telegram, WhatsApp, or any other end-to-end encrypted communication platform, they are almost certainly trying to scam you.&lt;/li&gt; &lt;li&gt;If the company requires payment from you for a job opportunity, we ultimately recommend that you do not proceed. It’s extraordinarily rare for a legitimate company to require payment from you for a job opportunity.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;As these scams become more pervasive, it’s crucial that those on the job market educate themselves on the potential scams that are out there. Knowing how to recognize and avoid these fraudulent job listings can ensure you don’t waste your time, lose money, or divulge your personal data to scammers.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 24: Twenty five years of Fastmail</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/twenty-five-years-of-fastmail/' />
 			<id>https://www.fastmail.com/blog/twenty-five-years-of-fastmail/</id>
 			<updated>2024-12-24T00:00:01Z</updated><author>
 				<name>Rob Mueller</name>
 			</author><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the twenty-fourth and final post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/ten-years-of-jmap/&quot;&gt;Dec 23: Ten years of JMAP&lt;/a&gt;. Thanks for reading, see you again next year.&lt;/p&gt;&lt;p&gt;As we conclude this year’s Advent posts, we are reflecting back over 25 years! Fastmail was founded in 1999, to fill a gap which existed at the time — in the space between ISPs, slow and ad-riddled free email services, and clunky, bloated Enterprise systems, there was no professional email service for a small business or sophisticated email user.&lt;/p&gt;&lt;p&gt;So we built one! Fastmail: a slick, professional, web-based email service.&lt;/p&gt;&lt;p&gt;In the 25 years since, we have seen many changes in the email landscape and the world around us. The advent of Gmail and conversations as a standard email model. The rise of encryption focused services like Protonmail (with the &lt;a href=&quot;https://www.fastmail.com/features/security/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;pros and cons&lt;/a&gt; of storing email as opaque, unsearchable blobs). Highly opinionated “reinventions” of email like Hey. And of course the multiple premature announcements that email was dead, to be replaced by the latest new craze.&lt;/p&gt;&lt;p&gt;We were purchased by Opera Software in 2010, but after some changes in Opera’s strategic direction, thankfully a handful of the staff managed to buy the company back in 2013. We then purchased another email service Pobox in 2015, who had been running an email service even longer than us. We have recently finished merging their product into our system; who knew it was going to take so long to integrate everything!&lt;/p&gt;&lt;p&gt;Through all of this we’ve been grateful to have such loyal customers. We regularly hear from customers how much they appreciate the Fastmail service. Our fantastic customer support. The continuous, thoughtful, and well designed improvements to our product. The high performance and reliability of our service. The ongoing commitment to integrity, privacy, and longevity.&lt;/p&gt;&lt;p&gt;The result is that we have a greater than 90% annual renewal rate, and an ongoing stream of new customers from the word of mouth recommendations of existing customers. We have and continue to grow every year in a sustainable and deliberate way. We’re insanely grateful for this. We get to focus on making email better for our customers, to work with and build cool technology — with really smart colleagues. We can solve complex problems, build well-designed solutions, and improve email standards without having to always hustle for the next sale.&lt;/p&gt;&lt;p&gt;It’s an enviable position to be in. Email remains the largest open federated communication network on the internet. Not controlled by a single company. Not part of any walled garden that can change at any time. Through open standards, email allows you to choose the best provider and to move your email where is best for you. As we said in our first post of this series, we will continue to “Make email better”, for our customers and for everyone.&lt;/p&gt;&lt;p&gt;We love our work, and the customers who trust us with their email and make this all possible. So cheers to you, Fastmail’s customers. We get to make email better, the product you use and the ecosystem we all operate in, while having fun and working on interesting problems with great people.&lt;/p&gt;&lt;p&gt;Here’s to another 25 years.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 23: Ten years of JMAP</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/ten-years-of-jmap/' />
 			<id>https://www.fastmail.com/blog/ten-years-of-jmap/</id>
 			<updated>2024-12-23T00:00:01Z</updated><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the twenty-third post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/why-we-use-our-own-hardware/&quot;&gt;Dec 22: Why we use our own hardware at Fastmail&lt;/a&gt;. The final post is &lt;a href=&quot;/blog/twenty-five-years-of-fastmail/&quot;&gt;Dec 24: Twenty five years of Fastmail&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Exactly 10 years ago, we &lt;a href=&quot;/blog/jmap-a-better-way-to-email/&quot;&gt;announced JMAP on our blog&lt;/a&gt;, along with a &lt;a href=&quot;https://youtu.be/8qCSK-aGSBA&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;video by baby-faced Bron and Neil&lt;/a&gt;!&lt;/p&gt;&lt;p&gt;JMAP: A better way to email. We knew it would be a long road, but we’re really glad we did it and created an open standard rather than staying with our own custom protocol.&lt;/p&gt;&lt;h2 id=&quot;some-moments-along-the-way&quot; tabindex=&quot;-1&quot;&gt;Some moments along the way&lt;/h2&gt;&lt;p&gt;We started by workshopping the idea around the industry. I did a &lt;a href=&quot;https://www.youtube.com/watch?v=yyXlUR1hbr4&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;lightning talk at OSCON in 2014&lt;/a&gt;, our first attempt to find developers who could give us feedback on our design. By far the best find was &lt;a href=&quot;https://rjbs.cloud/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Ricardo Signes&lt;/a&gt;, Pobox developer, who I met at a bar on the last day! This led to us acquiring the product and (our main goal) acqui-hiring Rik, who is now one of the company owners, as well as a JMAP enthusiast!&lt;/p&gt;&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;/assets/images/oscon-bron-rik-91PNJqGmRJ-375.webp 375w, /assets/images/oscon-bron-rik-91PNJqGmRJ-750.webp 750w, /assets/images/oscon-bron-rik-91PNJqGmRJ-1154.webp 1154w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;img alt=&quot;Bron and Rik at OSCON in 2014&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;/assets/images/oscon-bron-rik-91PNJqGmRJ-375.png&quot; width=&quot;1154&quot; height=&quot;893&quot; srcset=&quot;/assets/images/oscon-bron-rik-91PNJqGmRJ-375.png 375w, /assets/images/oscon-bron-rik-91PNJqGmRJ-750.png 750w, /assets/images/oscon-bron-rik-91PNJqGmRJ-1154.png 1154w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;/picture&gt;&lt;/p&gt;&lt;p&gt;Neil and I attended &lt;a href=&quot;https://web.archive.org/web/20150908015219/http://inboxlove.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Inbox Love&lt;/a&gt; in the Bay Area in 2014 as well. This gave us a chance to meet some of our technical peers in the big companies, relationships which we have continued to foster over the years. This hasn’t led to everyone dropping everything and implementing our protocols, but it has led to some collaborative design and ongoing conversations, and I believe its has prevented a proliferation of other protocols since people point to JMAP instead of inventing a new thing themselves. We also were told “go to the IETF”, but the IETF seemed big and scary and we didn’t know how, so that took a while.&lt;/p&gt;&lt;p&gt;Instead, we joined &lt;a href=&quot;https://www.calconnect.org/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;CalConnect&lt;/a&gt; and started working on Calendar formats and standards, while promoting JMAP more generally. Eventually we made more contacts in the IETF, and finally in 2017 went to our first IETF meeting in Chicago. At this point, the &lt;a href=&quot;https://datatracker.ietf.org/wg/jmap/history/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;JMAP working group&lt;/a&gt; was born.&lt;/p&gt;&lt;p&gt;In the crucible of the IETF, we made major changes. The authentication was removed. Method names were split into &lt;code&gt;Object/action&lt;/code&gt; and a ton of smaller changes were made. The &lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc8620.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Core&lt;/a&gt; and &lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc8621.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Mail&lt;/a&gt; JMAP specifications were published in 2019, and then we got to work on the rest of the stack.&lt;/p&gt;&lt;p&gt;JMAP &lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc9610.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Contacts&lt;/a&gt; was published just last week, and JMAP &lt;a href=&quot;https://datatracker.ietf.org/doc/draft-ietf-jmap-calendars/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Calendars&lt;/a&gt; is very close to being published. I’m also keen to add &lt;a href=&quot;https://datatracker.ietf.org/doc/draft-ietf-jmap-filenode/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Filenode&lt;/a&gt; support, but we want to get more experience with other filesystem providers before we standardize that (it’s currently based very closely on Fastmail’s own custom Node objects for our filestorage feature).&lt;/p&gt;&lt;h2 id=&quot;what-s-next&quot; tabindex=&quot;-1&quot;&gt;What’s next&lt;/h2&gt;&lt;p&gt;We created JMAP because we could see that without it, the email world was going to become more insular, with the only modern standards for email access being proprietary. With Calendars and Contacts, we’re bringing the same easy-to-use JSON objects under a single protocol.&lt;/p&gt;&lt;p&gt;We started the &lt;a href=&quot;https://makebetter.email/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Make Better Email&lt;/a&gt; conference last year, focused on improving the authentication workflow and also promoting JMAP usage. It’s a very small, invite-only conference where we do deep technical design work on improving interoperability and discoverability between clients and services. It was in Philadelphia last year, London this year, and we expect to be in Philadelphia again next year — likely in mid November after &lt;a href=&quot;https://www.ietf.org/meeting/124/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;IETF 124&lt;/a&gt; so we don’t cross over Halloween. If you think you’d be a useful addition to the meeting, pop us an email via the link at the bottom of the site.&lt;/p&gt;&lt;p&gt;Some work products of the previous conferences have been:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://datatracker.ietf.org/doc/draft-jenkins-oauth-public/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;An OAuth profile for open-protocol clients&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://datatracker.ietf.org/doc/draft-jenkins-emailpush/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;A profile for JMAP push&lt;/a&gt; allowing you get some of the benefits of JMAP’s push capability without having to do a full JMAP implementation&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://datatracker.ietf.org/doc/draft-ietf-mailmaint-autoconfig/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;A specification for auto-discovery of configuration information&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;Over the past year some of us have also been working in the server-to-server space with &lt;a href=&quot;https://datatracker.ietf.org/doc/draft-gondwana-dkim2-motivation/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;an idea&lt;/a&gt; that may wind up replacing or enhancing DKIM.&lt;/p&gt;&lt;p&gt;And finally, next year we will be investing a lot more effort into making the &lt;a href=&quot;https://www.cyrusimap.org/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Cyrus IMAP&lt;/a&gt; server not just a reference implementation for JMAP, but much easier to both develop and run.&lt;/p&gt;&lt;p&gt;In 10 years time, I hope to post about how Cyrus and JMAP have taken over the world, but I’ll also happily settle for them having both improved Fastmail’s product immeasurably, having plenty of happy customers, and continuing to help make email better for everybody through our work.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 22: Why we use our own hardware at Fastmail</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/why-we-use-our-own-hardware/' />
 			<id>https://www.fastmail.com/blog/why-we-use-our-own-hardware/</id>
 			<updated>2024-12-22T00:00:01Z</updated><author>
 				<name>Rob Mueller</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the twenty-second post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/fastmail-in-a-box/&quot;&gt;Dec 21: Fastmail in a box&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/ten-years-of-jmap/&quot;&gt;Dec 23: Ten years of JMAP&lt;/a&gt;.&lt;/p&gt;&lt;h2 id=&quot;why-we-use-our-own-hardware&quot; tabindex=&quot;-1&quot;&gt;Why we use our own hardware&lt;/h2&gt;&lt;p&gt;There has recently been talk of &lt;a href=&quot;https://www.google.com/search?q=Cloud+Repatriation&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;cloud repatriation&lt;/a&gt; where companies are moving from the cloud to on premises, with some particularly &lt;a href=&quot;https://basecamp.com/cloud-exit&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;noisy examples&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Fastmail has a long history of using our &lt;a href=&quot;https://www.fastmail.com/blog/standalone-mail-servers/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;own&lt;/a&gt; &lt;a href=&quot;https://www.fastmail.com/blog/getting-the-most-out-of-hardware/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;hardware&lt;/a&gt;. We have over two decades of experience running and optimising our systems to use our own &lt;a href=&quot;https://en.wikipedia.org/wiki/Bare-metal_server&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;bare metal&lt;/a&gt; servers efficiently.&lt;/p&gt;&lt;p&gt;We get way better cost optimisation compared to moving everything to the cloud because:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;We understand our short, medium and long term usage patterns, requirements and growth very well. This means we can plan our hardware purchases ahead of time and don’t need the fast dynamic scaling that cloud provides.&lt;/li&gt; &lt;li&gt;We have in house operations experience installing, configuring and running our own hardware and networking. These are skills we’ve had to maintain and grow in house since we’ve been doing this for 25 years.&lt;/li&gt; &lt;li&gt;We are able to use our hardware for long periods. We find our hardware can provide useful life for anywhere from 5-10 years depending on what it is and when in the global technology cycle it was bought, meaning we can amortise and depreciate the cost of any hardware over many years.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;Yes, that means we have to do more ourselves, including planning, choosing, buying, installing, etc, but the tradeoff for us has and we believe continues to be significantly worth it.&lt;/p&gt;&lt;h2 id=&quot;hardware-over-the-years&quot; tabindex=&quot;-1&quot;&gt;Hardware over the years&lt;/h2&gt;&lt;p&gt;Of course over the 25 years we’ve been running Fastmail we’ve been through a number of hardware changes. For many years, our IMAP server storage platform was a combination of &lt;a href=&quot;https://www.urbandictionary.com/define.php?term=Spinning%20Rust&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;spinning rust&lt;/a&gt; drives and &lt;a href=&quot;https://www.areca.com.tw/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ARECA RAID controllers&lt;/a&gt;. We tended to use faster 15k RPM SAS drives in &lt;a href=&quot;https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;RAID1&lt;/a&gt; for our hot meta data, and 7.2k RPM SATA drives in &lt;a href=&quot;https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;RAID6&lt;/a&gt; for our main email blob data.&lt;/p&gt;&lt;p&gt;In fact it was slightly more complex than this. Email blobs were written to the fast RAID1 SAS volumes on delivery, but then a separate archiving process would move them to the SATA volumes at low server activity times. Support for all of this had been added into &lt;a href=&quot;https://www.cyrusimap.org&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;cyrus&lt;/a&gt; and our tooling over the years in the form of separate “meta”, “data” and &lt;a href=&quot;https://www.cyrusimap.org/3.8/imap/reference/admin/locations/archive-partitions.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;“archive”&lt;/a&gt; partitions.&lt;/p&gt;&lt;h2 id=&quot;moving-to-nv-me-ssds&quot; tabindex=&quot;-1&quot;&gt;Moving to NVMe SSDs&lt;/h2&gt;&lt;p&gt;A few years ago however we made our biggest hardware upgrade ever. We moved all our email servers to a new &lt;a href=&quot;https://www.supermicro.com/en/aplus/system/2u/2113/as-2113s-wn24rt.cfm&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;2U AMD platform&lt;/a&gt; with pure &lt;a href=&quot;https://www.solidigm.com/products/data-center.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;NVMe SSDs&lt;/a&gt;. The density increase (24 x 2.5&amp;quot; NVMe drives vs 12 x 3.5&amp;quot; SATA drives per 2U) and performance increase was enormous. We found that these new servers performed even better than our initial expectations.&lt;/p&gt;&lt;p&gt;At the time we upgraded however NVMe RAID controllers weren’t widely available. So we had to decide on how to handle redundancy. We considered a RAID-less setup using raw SSDs drives on each machine with synchronous application level replication to other machines, but the software changes required were going to be more complex than expected.&lt;/p&gt;&lt;p&gt;We were looking at using classic Linux &lt;a href=&quot;https://en.wikipedia.org/wiki/Mdadm&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;mdadm RAID&lt;/a&gt;, but the &lt;a href=&quot;https://en.wikipedia.org/wiki/RAID#Atomicity&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;write hole&lt;/a&gt; was a concern and the &lt;a href=&quot;https://docs.kernel.org/driver-api/md/raid5-cache.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;write cache&lt;/a&gt; didn’t seem well tested at the time.&lt;/p&gt;&lt;p&gt;We decided to have a look at &lt;a href=&quot;https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ZFS&lt;/a&gt; and at least test it out.&lt;/p&gt;&lt;p&gt;Despite some of the cyrus on disk database structures being fairly hostile to &lt;a href=&quot;https://en.wikipedia.org/wiki/ZFS#Copy-on-write_transactional_model&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ZFS Copy-on-write&lt;/a&gt; semantics, they were still incredibly fast at all the IO we threw at them. And there were some other wins as well.&lt;/p&gt;&lt;h2 id=&quot;zfs-compression-and-tuning&quot; tabindex=&quot;-1&quot;&gt;ZFS compression and tuning&lt;/h2&gt;&lt;p&gt;When we rolled out ZFS for our email servers we also enabled &lt;a href=&quot;https://freebsdfoundation.org/wp-content/uploads/2021/05/Zstandard-Compression-in-OpenZFS.pdf&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;transparent Zstandard compression&lt;/a&gt;. This has worked very well for us, saving about 40% space on all our email data.&lt;/p&gt;&lt;p&gt;We’ve also recently done some additional calculations to see if we could tune some of the parameters better. We sampled 1 million emails at random and calculated how many blocks would be required to store those emails uncompressed, and then with &lt;a href=&quot;https://klarasystems.com/articles/tuning-recordsize-in-openzfs/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ZFS record sizes&lt;/a&gt; of 32k, 128k or 512k and zstd-3 or zstd-9 compression options. Although ZFS &lt;a href=&quot;https://en.wikipedia.org/wiki/ZFS#ZFS&#39;s_approach:_RAID-Z_and_mirroring&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;RAIDz2&lt;/a&gt; seems conceptually similar to classic RAID6, the way it &lt;a href=&quot;https://ibug.io/blog/2023/10/zfs-block-size/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;actually stores blocks of data&lt;/a&gt; is quite different and so you have to take into account volblocksize, how files are split into logical recordsize blocks, and number of drives when doing calculations.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;               Emails: 1,026,000
            Raw blocks: 34,140,142
  32k &amp;amp; zstd-3, blocks: 23,004,447 = 32.6% saving
  32k &amp;amp; zstd-9, blocks: 22,721,178 = 33.4% saving
 128k &amp;amp; zstd-3, blocks: 20,512,759 = 39.9% saving
 128k &amp;amp; zstd-9, blocks: 20,261,445 = 40.7% saving
 512k &amp;amp; zstd-3, blocks: 19,917,418 = 41.7% saving
 512k &amp;amp; zstd-9, blocks: 19,666,970 = 42.4% saving
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This showed that the defaults of 128k record size and zstd-3 were already pretty good. Moving to a record size of 512k improved compression over 128k by a bit over 4%. Given all meta data is cached separately, this seems a worthwhile improvement with no significant downside. Moving to zstd-9 improved compression over zstd-3 by about 2%. Given the CPU cost of compression at zstd-9 is about 4x zstd-3, even though emails are immutable and tend to be kept for a long time, we’ve decided not to implement this change.&lt;/p&gt;&lt;h2 id=&quot;zfs-encryption&quot; tabindex=&quot;-1&quot;&gt;ZFS encryption&lt;/h2&gt;&lt;p&gt;We always enable &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_at_rest#Encryption&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;encryption at rest&lt;/a&gt; on all of our drives. This was usually done with &lt;a href=&quot;https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;LUKS&lt;/a&gt;. But with ZFS this was &lt;a href=&quot;https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-to-openzfs-native-encryption/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;built in&lt;/a&gt;. Again, this reduces overall system complexity.&lt;/p&gt;&lt;h2 id=&quot;going-all-in-on-zfs&quot; tabindex=&quot;-1&quot;&gt;Going all in on ZFS&lt;/h2&gt;&lt;p&gt;So after the success of our initial testing, we decided to go all in on ZFS for all our large data storage needs. We’ve now been using ZFS for all our email servers for over 3 years and have been very happy with it. We’ve also moved over all our database, log and backup servers to using ZFS on NVMe SSDs as well with equally good results.&lt;/p&gt;&lt;h2 id=&quot;ssd-lifetimes&quot; tabindex=&quot;-1&quot;&gt;SSD lifetimes&lt;/h2&gt;&lt;p&gt;The flash memory in SSDs has a finite life and &lt;a href=&quot;https://en.wikipedia.org/wiki/Flash_memory#Write_endurance&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;finite number of times it can be written to&lt;/a&gt;. SSDs employ increasingly complex &lt;a href=&quot;https://en.wikipedia.org/wiki/Wear_leveling&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;wear levelling&lt;/a&gt; algorithms to spread out writes and increase drive lifetime. You’ll often see the quoted endurance of an enterprise SSD as either an absolute figure of “Lifetime Writes”/“Total bytes written” like 65 PBW (petabytes written) or a relative per-day figure of “Drive writes per day” like 0.3, which you can convert to lifetime figure by multiplying by the drive size and the drive expected lifetime which is often assumed to be 5 years.&lt;/p&gt;&lt;p&gt;Although we could calculate IO rates for existing &lt;a href=&quot;https://en.wikipedia.org/wiki/Hard_disk_drive&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;HDD&lt;/a&gt; systems, we were making a significant number of changes moving to the new systems. Switching to a COW filesystem like ZFS, removing the special casing meta/data/archive partitions, and the massive latency reduction and performance improvements mean that things that might have taken extra time previously and ended up batching IO together, are now so fast it actually causes additional separated IO actions.&lt;/p&gt;&lt;p&gt;So one big unknown question we had was how fast would the SSDs wear in our actual production environment? After several years, we now have some clear data. From one server at random but this is fairly consistent across the fleet of our oldest servers:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;# smartctl -a /dev/nvme14
 ...
 Percentage Used:                    4%
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;At this rate, we’ll replace these drives due to increased drive sizes, or entirely new physical drive formats (such &lt;a href=&quot;https://www.snia.org/forums/cmsi/knowledge/formfactors&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;E3.S&lt;/a&gt; which appears to finally be gaining traction) long before they get close to their rated write capacity.&lt;/p&gt;&lt;p&gt;We’ve also anecdotally found SSDs just to be much more reliable compared to HDDs for us. Although we’ve only ever used &lt;a href=&quot;https://www.micron.com/products/storage/ssd/data-center-ssd/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;datacenter&lt;/a&gt; &lt;a href=&quot;https://www.solidigm.com/products/data-center.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;class&lt;/a&gt; SSDs and &lt;a href=&quot;https://www.seagate.com/www-content/datasheets/pdfs/exos-7-e8-data-sheet-DS1957-1-1709US-en_US.pdf&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;HDDs&lt;/a&gt; failures and replacements every few weeks were a regular occurrence on the old fleet of servers. Over the last 3+ years, we’ve only seen a couple of SSD failures in total across the entire upgraded fleet of servers. This is easily less than one tenth the failure rate we used to have with HDDs.&lt;/p&gt;&lt;h2 id=&quot;storage-cost-calculation&quot; tabindex=&quot;-1&quot;&gt;Storage cost calculation&lt;/h2&gt;&lt;p&gt;After converting all our email storage to NVMe SSDs, we were recently looking at our data backup solution. At the time it consisted of a number of older 2U servers with 12 x 3.5&amp;quot; SATA drive bays and we decided to do some cost calculations on:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;Move to cloud storage.&lt;/li&gt; &lt;li&gt;Upgrade the HD drives in existing servers.&lt;/li&gt; &lt;li&gt;Upgrade to SSD NVMe machines.&lt;/li&gt; &lt;/ol&gt;&lt;h3 id=&quot;1-cloud-storage&quot; tabindex=&quot;-1&quot;&gt;1. Cloud storage:&lt;/h3&gt;&lt;p&gt;Looking at various providers, the per TB per month price, and then a yearly price for 1000Tb/1Pb (prices as at Dec 2024)&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://aws.amazon.com/s3/pricing/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Amazon S3&lt;/a&gt; - $21 -&amp;gt; $252,000/y&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://developers.cloudflare.com/r2/pricing/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Cloudflare R2&lt;/a&gt; - $15 -&amp;gt; $180,000/y&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://wasabi.com/pricing&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Wasabi&lt;/a&gt; - $6.99 -&amp;gt; $83,880/y&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://www.backblaze.com/cloud-storage/pricing&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Backblaze B2&lt;/a&gt; - $6 -&amp;gt; $72,000/y&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://aws.amazon.com/s3/pricing/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Amazon S3 Glacier Instant Retrieval&lt;/a&gt; - $4 -&amp;gt; $48,000/y&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://aws.amazon.com/s3/pricing/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Amazon S3 Glacier Deep Archive (12 hour retrieval time)&lt;/a&gt; - $0.99 -&amp;gt; $11,880/y&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;Some of these (e.g. Amazon) have potentially significant bandwidth fees as well.&lt;/p&gt;&lt;p&gt;It’s interesting seeing the spread of prices here. Some also have a bunch of weird edge cases as well. e.g. “The S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive storage classes require an additional 32 KB of data per object”. Given the large retrieval time and extra overhead per-object, you’d probably want to store small incremental backups in regular S3, then when you’ve gathered enough, build a biggish object to push down to Glacier. This adds implementation complexity.&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;em&gt;Pros&lt;/em&gt;: No limit to amount we store. Assuming we use S3 compatible API, can choose between multiple providers.&lt;/li&gt; &lt;li&gt;&lt;em&gt;Cons&lt;/em&gt;: Implementation cost of converting existing backup system that assumes local POSIX files to S3 style object API is uncertain and possibly significant. Lowest cost options require extra careful consideration around implementation details and special limitations. Ongoing monthly cost that will only increase as amount of data we store increases. Uncertain if prices will go down or not, or even go up. Possible significant bandwidth costs depending on provider.&lt;/li&gt; &lt;/ul&gt;&lt;h3 id=&quot;2-upgrade-hdds&quot; tabindex=&quot;-1&quot;&gt;2. Upgrade HDDs&lt;/h3&gt;&lt;p&gt;&lt;a href=&quot;https://www.seagate.com/au/en/products/enterprise-drives/exos-x/x24/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Seagate Exos 24 HDs&lt;/a&gt; are 3.5&amp;quot; 24T HDDs. This would allow us to triple the storage on existing servers. Each HDD is about $500, so upgrading one 2U machine would be about $6,000 and have storage of 220T or so.&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;em&gt;Pros&lt;/em&gt;: Reuses existing hardware we already have. Upgrades can be done a machine at a time. Fairly low price&lt;/li&gt; &lt;li&gt;&lt;em&gt;Cons&lt;/em&gt;: Will existing units handle 24T drives? What’s the rebuild time on drive failure look like? It’s almost a day for 8T drives already, so possibly nearly a week for a failed 24T drive? Is there enough IO performance to handle daily backups at capacity?&lt;/li&gt; &lt;/ul&gt;&lt;h3 id=&quot;3-upgrade-to-new-hardware&quot; tabindex=&quot;-1&quot;&gt;3. Upgrade to new hardware&lt;/h3&gt;&lt;p&gt;As we know, SSDs are denser (2.5&amp;quot; -&amp;gt; 24 per 2U vs 3.5&amp;quot; -&amp;gt; 12 per 2U), more reliable, and now higher capacity - &lt;a href=&quot;https://www.solidigm.com/products/data-center/d5/p5336.html#form=U.2%2015mm&amp;amp;cap=61.44TB&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;up to 61T per 2.5&amp;quot; drive&lt;/a&gt;. A single 2U server with 24 x 61T drives with 2 x 12 RAIDz2 = 1220T. Each drive is &lt;a href=&quot;https://www.newegg.com/solidigm-61-44tb-d5-p5336/p/N82E16820318031&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;about $7k&lt;/a&gt; right now, prices fluctuate. So all up 24 x $7k = $168k + ~$20k server =~ $190k for &amp;gt; 1000T storage one-time cost.&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;em&gt;Pros&lt;/em&gt;: &lt;strong&gt;Much&lt;/strong&gt; higher sequential and random IO than HDDs will ever have. Price &amp;lt; 1 year of standard S3 storage. Internal to our WAN, no bandwidth costs and very low latency. No new development required, existing backup system will just work. Consolidate on single 2U platform for all storage (cyrus, db, backups) and SSD for all storage. Significant space and power savings over existing HDD based servers&lt;/li&gt; &lt;li&gt;&lt;em&gt;Cons&lt;/em&gt;: Greater up front cost. Still need to predict and buy more servers as backups grow.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;One thing you don’t see in this calculation is datacenter space, power, cooling, etc. The reason is that compared to the amortised yearly cost of a storage server like this, these are actually reasonably minimal these days, on the order of $3000/2U/year. Calculating person time is harder. We have a lot of home built automation systems that mean installing and running one more server has minimal marginal cost.&lt;/p&gt;&lt;h3 id=&quot;result&quot; tabindex=&quot;-1&quot;&gt;Result&lt;/h3&gt;&lt;p&gt;We ended up going with the the new 2U servers option:&lt;/p&gt;&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;/assets/images/nvme-imap-servers-AqR6yL3DlW-375.webp 375w, /assets/images/nvme-imap-servers-AqR6yL3DlW-750.webp 750w, /assets/images/nvme-imap-servers-AqR6yL3DlW-1500.webp 1500w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;img alt=&quot;NVME IMAP Servers&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;/assets/images/nvme-imap-servers-AqR6yL3DlW-375.png&quot; width=&quot;1500&quot; height=&quot;559&quot; srcset=&quot;/assets/images/nvme-imap-servers-AqR6yL3DlW-375.png 375w, /assets/images/nvme-imap-servers-AqR6yL3DlW-750.png 750w, /assets/images/nvme-imap-servers-AqR6yL3DlW-1500.png 1500w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;/picture&gt;&lt;/p&gt;&lt;ul&gt; &lt;li&gt;The 2U AMD NVMe platform with ZFS is a platform we have experience with already&lt;/li&gt; &lt;li&gt;SSDs are much more reliable and much higher IO compared to HDDs&lt;/li&gt; &lt;li&gt;No uncertainty around super large HDDs, RAID controllers, rebuild times, shuffling data around, etc.&lt;/li&gt; &lt;li&gt;Significant space and power saving over existing HDD based servers&lt;/li&gt; &lt;li&gt;No new development required, can use existing backup system and code&lt;/li&gt; &lt;li&gt;Long expected hardware lifetime, controlled upfront cost, can depreciate hardware cost&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;So far this has worked out very well. The machines have bonded 25Gbps networks and when filling them from scratch we were able to saturate the network links streaming around 5Gbytes/second of data from our IMAP servers, compressing and writing it all down to a RAIDz2 zstd-3 compressed ZFS dataset.&lt;/p&gt;&lt;h2 id=&quot;conclusion&quot; tabindex=&quot;-1&quot;&gt;Conclusion&lt;/h2&gt;&lt;p&gt;Running your own hardware might not be for everyone and has distinct tradeoffs. But when you have the experience and the knowledge of how you expect to scale, the cost improvements can be significant.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 21: Fastmail in a box</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/fastmail-in-a-box/' />
 			<id>https://www.fastmail.com/blog/fastmail-in-a-box/</id>
 			<updated>2024-12-21T00:00:01Z</updated><author>
 				<name>Andrew Davis</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the twenty-first post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/how-fastmail-uses-fastmail/&quot;&gt;Dec 20: How Fastmail uses Fastmail!&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/why-we-use-our-own-hardware/&quot;&gt;Dec 22: Why we use our own hardware at Fastmail&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;They say everybody has a testing environment. Some people are just lucky enough enough to have a separate environment for production. At Fastmail, every staff member can get their own isolated testing and development sandbox. We call this Fastmail-In-A-Box, or more commonly just “fminabox”.&lt;/p&gt;&lt;p&gt;Like many technologists, I learn most by fiddling with things, often breaking them along the way and putting them back together again. With fminabox, we give everyone their own world to break apart and put back together, risk free. This makes it an invaluable place for new hires to cut their teeth, and for existing staff to come up to speed in an area of Fastmail’s stack they haven’t worked on before.&lt;/p&gt;&lt;p&gt;Fminabox is a complete Fastmail deployment on a single host. This includes Cyrus for IMAP storage, Postfix for incoming and outgoing mail, MySQL for non-mail data, our JMAP web API, and all the frontend assets. It also runs the ancillary services we use to monitor Fastmail such as Prometheus, all managed by the same configuration and service management system we use in production. This allows for fast, iterative development with very little waiting time between making a change and seeing the effect, while eliminating most “it worked on my machine” bugs.&lt;/p&gt;&lt;p&gt;We use &lt;a href=&quot;https://www.packer.io/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Hashicorp Packer&lt;/a&gt; to create fminabox, following the provisioning scripts we use in production as closely as possible. Who hasn’t made a change to a system where it works going from state N → N+1, but then discovered weeks later that it’s broken when bootstrapping from nothing? Each night we build a new image from scratch. This allows us to catch those types of failures, and to do so while the changes are still front-of-mind in the developers that made them.&lt;/p&gt;&lt;p&gt;Any staff member can tell our chatbot &lt;a href=&quot;https://github.com/fastmail/Synergy&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Synergy&lt;/a&gt; to &lt;code&gt;box create&lt;/code&gt;, and Synergy will handle provisioning a VM in the cloud, set up DNS, and provide VPN configuration upon request. Fastmail continues to eschew the public cloud in favour of our own hardware to run our product, but it turns out the public cloud is really useful for creating test environments.&lt;/p&gt;&lt;p&gt;Fminabox is also a key part of our testing workflow. Fastmail has thousands of tests, from simple sanity compile checks to complex integration tests between systems. We use fminabox with our CI/CD pipeline so every change is automatically tested before it is merged. This was the ultimate progression from developers just running a handful of tests manually, to overnight runs, to fully integrated continuous testing.&lt;/p&gt;&lt;p&gt;As new needs arise, we continue to evolve the infrastructure. A few years ago I was making an improvement to our tooling that balances users between machines in our &lt;a href=&quot;https://www.fastmail.com/blog/building-a-backup-system-for-cyrus/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Cyrus backup system&lt;/a&gt;. At the time, fminabox only had a single target that all users were backed up to, so my first step was to add support for multiple backup targets. Only then did I feel comfortable that I could properly test any changes to the tooling.&lt;/p&gt;&lt;p&gt;I’m not the only user, so I asked some other Fastmail staff members “what’s your favourite feature of fminabox?”, and here’s what they had to say.&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;Fastmail is able to send via externally authenticated submission via OAuth, but Fastmail is also an OAuth provider and provides authenticated SMTP submission via OAuth. We were able to update our test suite to do full end-to-end OAuth authentication with ourself, send an email back to ourselves, and see that this entire path works.&lt;/p&gt; &lt;/blockquote&gt;&lt;p&gt;—Rob Mueller, CTO&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;The best thing about fminabox is that it’s cheap and disposable. If I mess it up, I throw it away and make a new one and act like nothing happened. (The previous solution took hours to create a new box.)&lt;/p&gt; &lt;/blockquote&gt;&lt;p&gt;—Ricardo Signes, Head of Special Projects&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;Although it only takes 5 minutes to setup, inaboxes provide a fully contained sandbox that includes all of our code ready to test. Minutes to build, seconds to tear down.&lt;/p&gt; &lt;/blockquote&gt;&lt;p&gt;—Marcus Love, System Engineer&lt;/p&gt;&lt;p&gt;And that’s the story of fminabox. It isn’t perfect but it’s pretty damn good and it helps enable my colleagues to get their work done.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 20: How Fastmail uses Fastmail!</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/how-fastmail-uses-fastmail/' />
 			<id>https://www.fastmail.com/blog/how-fastmail-uses-fastmail/</id>
 			<updated>2024-12-20T00:00:01Z</updated><author>
 				<name>Anju Manohar</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the twentieth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/offline-mail-storage/&quot;&gt;Dec 19: Building offline: mail storage&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/fastmail-in-a-box/&quot;&gt;Dec 21: Fastmail in a box&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;At Fastmail, our features are designed to make email management seamless and efficient. But how do our own staff use these tools in their daily lives? While all the features are incredibly helpful, there will always be some personal favorites for one.&lt;/p&gt;&lt;p&gt;We asked team members to share their favorite Fastmail features and how they’ve customized them to fit their workflows. As support staff, we’re familiar with every feature, but each of us uses them in unique ways—and you might just discover a hidden gem in this article by seeing how we put them to work. And here’s what they had to say.&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Vysakh: Mastering Inbox Zero and Organization&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;&lt;em&gt;“I enjoy maintaining an Inbox Zero approach, so I organize my emails into folders for various services.”&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Vysakh takes organization to the next level with &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/1500000280301-Setting-up-and-using-folders&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Folders&lt;/strong&gt;&lt;/a&gt; and &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/1500000278122-Mail-rules&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Rules&lt;/strong&gt;&lt;/a&gt;. He creates folders for specific categories like “Bank” (with subfolders for each bank) and “Purchases” (with subfolders for each website he purchases from like Amazon and Flipkart).&lt;/p&gt;&lt;p&gt;He also raves about &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360060591213-Searching-your-mail#:~:text=Saved%20searches,for%20the%20same%20thing%20later.&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Saved Searches&lt;/strong&gt;&lt;/a&gt;, calling them an &lt;em&gt;“underrated feature”&lt;/em&gt;:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;A saved search for &lt;strong&gt;Unread Emails&lt;/strong&gt; allows him to see all unread messages across folders.&lt;/li&gt; &lt;li&gt;Another for &lt;strong&gt;Emails Delivered Today&lt;/strong&gt; is perfect for quickly finding new emails, even if they’re in Spam or Trash.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;For self-organization, Vysakh uses &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360060591053-Plus-addressing-and-subdomain-addressing&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Plus Addressing&lt;/strong&gt;&lt;/a&gt; to save important documents by emailing them to himself with a folder-specific alias like &lt;code&gt;username+Docs@fastmail.tld&lt;/code&gt;. He adds, &lt;em&gt;“Fastmail supports searching inside attachments, so finding them later is super easy.”&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Vysakh also highlights the efficiency of &lt;strong&gt;Keyboard Shortcuts&lt;/strong&gt; for email navigation and &lt;strong&gt;Customizable Notification Actions&lt;/strong&gt; in the Android app, which let him manage emails without opening the app.&lt;/p&gt;&lt;p&gt;I must say, he’s truly a pro-user of all our power features!&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Merlin: Timing and Efficiency Made Easy&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Merlin’s favorite is &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/4686576659727-Scheduled-Send&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Schedule Send&lt;/strong&gt;&lt;/a&gt;, which allows drafting emails at their convenience and sending them at the perfect time. I couldn’t agree more—it’s like having a personal assistant keeping your inbox in check!&lt;/p&gt;&lt;p&gt;She goes on to say, &lt;em&gt;“My second favorite is Snooze—it lets me come back to emails when I have time to act on them.”&lt;/em&gt; Smart, right? Fastmail features truly help you work smarter, not harder.&lt;/p&gt;&lt;p&gt;Merlin also loves &lt;strong&gt;Mail rules&lt;/strong&gt; to keep her emails organized effortlessly and appreciates their simplicity, saying, &lt;em&gt;&amp;quot;It is a cinch to use even for beginners.”&lt;/em&gt;&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Maya: A Domain for Every Interest&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Maya’s love for email personalization shines through her extensive use of &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360058753394-Custom-domains-with-Fastmail&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Domains&lt;/strong&gt;&lt;/a&gt; and &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360060591073-How-to-set-up-aliases&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Aliases&lt;/strong&gt;&lt;/a&gt;:&lt;/p&gt;&lt;p&gt;&lt;em&gt;“I have a whole bunch of domains just for fun—some professional, some for hobbies like photo essays and writing.”&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Her go-to feature is &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/4406536368911-Masked-Email&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Masked Email&lt;/strong&gt;&lt;/a&gt; combined with a custom domain. Whenever she signs up for a service, she creates a unique masked address, assigning each one to a folder. This setup lets her instantly identify breaches, unsubscribe from unwanted notifications, and organize her inbox for easy prioritization.&lt;/p&gt;&lt;p&gt;Maya also uses:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;strong&gt;Pins&lt;/strong&gt; to flag important emails in folders.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Snooze&lt;/strong&gt; for emails she wants to revisit later without cluttering her inbox.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Tamper-Proof Retention&lt;/strong&gt;, ensuring emails she’s deleted can still be recovered if needed.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;She admits with a laugh, &lt;em&gt;“It’s probably not something a non-business user like me should need, but it’s handy!”&lt;/em&gt;&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Thu: Seamless Sharing and Catchall Convenience&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Thu finds &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/1500000277942-Catch-all-wildcard-aliases&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Catchall aliases&lt;/strong&gt;&lt;/a&gt; and &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360060590733-Sharing-mail&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Mail sharing&lt;/strong&gt;&lt;/a&gt; to be game-changers. &lt;em&gt;“The catchall is super handy for creating new email addresses anytime I want”.&lt;/em&gt; She’s even set up a Fastmail account for her partner with shared aliases, mail, and calendars. Excitedly, she adds, &lt;em&gt;“The sharing function works very well with integration to Apple devices and Gmail account,”&lt;/em&gt; making collaboration a breeze. You can almost feel her joy through those words—don’t you? I bet you do!&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Leslie: Staying Organized with Pins&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Leslie swears by the &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/1500000280341-Pin-important-messages&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Pin&lt;/strong&gt;&lt;/a&gt; and &lt;strong&gt;Keep pinned on top&lt;/strong&gt; features to stay on top of essential emails. &lt;em&gt;“I use this on my staff account quite a lot because we get various emails with a bit of information in them. When I go through my emails every morning, I will pin the emails that I want to go back and read and also emails about support trends that are occurring. This allows me to quickly refer back to the emails since I’m seeing them at the top of the list in my inbox.”&lt;/em&gt; Leslie clearly has a knack for using this feature to keep important information right at her fingertips—guess I know who to turn to for a quick reference!&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Yassar: Power User of Labels and Searches&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;For Yassar, &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360058753554-Setting-up-and-using-labels&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Labels&lt;/strong&gt;&lt;/a&gt; are indispensable. &lt;em&gt;“I love organizing everything in my mailbox, and I can’t imagine working without labels!”&lt;/em&gt; If you’re a fan of staying organized, you’re probably nodding in agreement right now.&lt;/p&gt;&lt;p&gt;But Yassar doesn’t stop there, his other go-to features include:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;strong&gt;Masked Emails&lt;/strong&gt; for secure online shopping.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Aliases&lt;/strong&gt; to classify emails and route them to specific folders such as &lt;code&gt;documents@mydomain.com&lt;/code&gt; or &lt;code&gt;gmail@mydomain.com&lt;/code&gt;.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Saved Search&lt;/strong&gt; for quick filtering of emails without creating a label for everything.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;Yassar’s approach combines security, organization, and speed, showing how powerful these tools can be for anyone looking to optimize their email experience!&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Jed: Streamlining Inbox Management with Keyboard Shortcuts&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Jed relies heavily on &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360058753534-Keyboard-shortcuts&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Keyboard Shortcuts&lt;/strong&gt;&lt;/a&gt; as one of his favorite Fastmail features:&lt;/p&gt;&lt;p&gt;&lt;em&gt;“As someone who lets a lot of email build up before I get around to sorting them, I like how quickly I can label, archive, and delete large numbers of messages using these shortcuts.”&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Wow! That’s brilliant—such a simple feature, yet so powerful! He’s turned what could be a daunting task into a seamless process. I’m definitely going to start using these more, and you should too!&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Jess: Memos, Snoozing, and Smart Searching&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Jess praises &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/1500001969861-Conversations#:~:text=conversations%20help%20page.-,Memos,-Use%20memos%20to&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Memos&lt;/strong&gt;&lt;/a&gt; for saving key details from lengthy emails. With a grin, she shares, &lt;em&gt;“I just used it to note a coupon hidden in a long marketing email.”&lt;/em&gt; Who else can relate? I know I can—if you’re like me, always hunting for those elusive discount codes, Memos are here to save the day!&lt;/p&gt;&lt;p&gt;She’s also a big fan of the &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360058753634-Snoozing-mail&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Snooze&lt;/strong&gt;&lt;/a&gt; feature, which she frequently uses for bills or shipping notifications that need attention later.&lt;/p&gt;&lt;p&gt;Lastly, Jess makes extensive use of &lt;strong&gt;Search&lt;/strong&gt; to streamline her inbox. The &lt;code&gt;is:unread&lt;/code&gt; search helps her capture all unread emails across labels, enabling her to quickly sort through them and get closer to Inbox Zero. That’s an amazing tip—you’ll get a unified inbox with all unread emails in one place!&lt;/p&gt;&lt;hr&gt;&lt;p&gt;&lt;strong&gt;Aric: Sending at the Perfect Time&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Scheduled send&lt;/strong&gt; is one of the features Aric relies on the most, and it’s no surprise why, as he shares: “&lt;em&gt;Working with teams across the globe and working on a non-traditional schedule, I often find myself sending mail outside of people’s general working hours. Using scheduled send allows me to send mail so that my message is one of the first things the recipient sees when they check their inbox.&amp;quot;&lt;/em&gt; Impressive—that explains why his emails always seem to land at just the right time!&lt;/p&gt;&lt;hr&gt;&lt;p&gt;It’s been such a joy hearing from our team members about their favorite features. Inspired by them, how can I not share mine? I’m certainly not stepping back—here are my favorites.&lt;/p&gt;&lt;p&gt;One feature I find myself using a lot is the &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360058752514-Logged-in-sessions#:~:text=wish%20to%20end.-,View%20all%20logins%20in%20the%20last%204%20weeks,-At%20the%20bottom&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Login Log&lt;/strong&gt;&lt;/a&gt;, which is crucial for maintaining the privacy and security of my account. It lets me instantly review login activities whenever I spot something suspicious. The interface is simple and intuitive, showing logins from the Fastmail web UI, Fastmail app, and third-party apps, along with any failed login attempts. This feature gives me the confidence to monitor and act swiftly when needed—no more panic attacks when something looks off!&lt;/p&gt;&lt;p&gt;Next up is the &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360060591213-Searching-your-mail#:~:text=searches%20with%20syntax.-,Advanced%20search,-If%20you%27re%20having&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Advanced Search&lt;/strong&gt;&lt;/a&gt;, a powerful yet user-friendly tool. For someone new to search tools and struggling to remember search syntax, this feature would be a real lifesaver. I use it all the time to refine my searches by selecting criteria from the available fields—whether it’s finding emails with specific attachment types, emails within a date range, or so much more. It’s fast, easy, and incredibly efficient! If you haven’t tried it yet, give it a go today—you’ll be amazed at how simple it is!&lt;/p&gt;&lt;p&gt;Then comes the &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/10351615144335-Passkeys&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;strong&gt;Passkey&lt;/strong&gt;&lt;/a&gt; feature making my life easier and logging into my account across different devices both simpler and more secure.&lt;/p&gt;&lt;p&gt;Honestly, I love all the features! They’ve taken me—once a self-proclaimed email management avoider (guilty as charged!)—and turned me into a full-fledged Inbox enthusiast.&lt;/p&gt;&lt;hr&gt;&lt;p&gt;A big thank you to my team for sharing their insights. It’s been great learning how everyone makes the most of these tools!&lt;/p&gt;&lt;p&gt;That wraps up an exciting dive into these amazing features and creative ways to use them—I hope you’ve found some inspiration to explore and make them your own!&lt;/p&gt;&lt;p&gt;Still not on Fastmail? Now’s the perfect time! Get your whole family on board with the Family plan and enjoy an &lt;a href=&quot;https://app.fastmail.com/signup/?discount=MjAsMTIsMTczNTY0NzQ0MCxhbWFub2hhciwsZjMyN2RlNGU4ZDVmZTM4MmVmNDM3YzI2YzRhNDFhY2EyYmZmMDA4MA&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;exclusive 20% off your first year&lt;/a&gt;—don’t miss out!&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 19: Building offline: mail storage</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/offline-mail-storage/' />
 			<id>https://www.fastmail.com/blog/offline-mail-storage/</id>
 			<updated>2024-12-19T00:00:01Z</updated><author>
 				<name>Neil Jenkins</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the nineteenth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/offline-sync/&quot;&gt;Dec 18: Building offline: syncing changes back to the server&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/how-fastmail-uses-fastmail/&quot;&gt;Dec 20: How Fastmail uses Fastmail!&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Yesterday, we looked at &lt;a href=&quot;/blog/offline-sync/&quot;&gt;how we store changes you make offline&lt;/a&gt; so we can accurately and efficiently sync them back to the server when you come online. Today, we’ll discuss why email is special, and what else we do to make this super fast, with support for full-text search offline.&lt;/p&gt;&lt;h2 id=&quot;why-offline-email-is-hard&quot; tabindex=&quot;-1&quot;&gt;Why offline email is hard&lt;/h2&gt;&lt;p&gt;As discussed earlier, because we use JMAP for all of our APIs, once we can implement generic offline support and have it work for everything (currently 56 data types and counting in our app!). However, mail is special. And the reason it’s special is purely the volume of data.&lt;/p&gt;&lt;p&gt;Most web apps severely underestimate how small their data is. In almost all cases, you will be more efficient and way faster to just suck it all into memory and do a linear filter pass whenever you need to query it. This is the difference between response as-you-type autocomplete and frustrating loading spinners on each key stroke. Even for users with 10,000 contacts this is only a few megabytes of data — perfectly cacheable.&lt;/p&gt;&lt;p&gt;Email is different though. We have users with millions of messages. Even with attachments handled separately in JMAP, each message could have hundreds of kilobytes of HTML as the body. But we expect opening a mailbox to load a listing pretty much instantly, and searches to be fast too. To make this work, we have to add a number of tricks to our standard offline approach.&lt;/p&gt;&lt;h2 id=&quot;splitting-the-data&quot; tabindex=&quot;-1&quot;&gt;Splitting the data&lt;/h2&gt;&lt;p&gt;The first trick is to split the data into two separate &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/IDBObjectStore&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;object stores&lt;/a&gt;:&lt;/p&gt;&lt;ol&gt; &lt;li&gt; &lt;p&gt;&lt;strong&gt;EmailMetadata&lt;/strong&gt;: this stores just the data that’s not parsed from the email content, like the id, thread id, keywords it has, and mailboxes it’s in. This keeps it small, but crucially also contains all the mutable data. This is treated like our standard JMAP object store for a data type.&lt;/p&gt; &lt;/li&gt; &lt;li&gt; &lt;p&gt;&lt;strong&gt;EmailContent&lt;/strong&gt;: this stores the email content; who it was sent from/to, the subject, body, list of attachments (but not the attachment data itself) etc.&lt;/p&gt; &lt;/li&gt; &lt;/ol&gt;&lt;p&gt;Due to the volume of data, we can’t load everything at once. We page in the data in stages instead:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;We fetch a list of just the ids and create placeholder entries in the EmailMetadata object store.&lt;/li&gt; &lt;li&gt;We page in the metadata and basic headers (like to/from/subject) for all messages in batches. This gives us everything we need to show the listing for any folder or label.&lt;/li&gt; &lt;li&gt;We page in the body for pinned and recent messages, or everything if the user has selected this option in settings, again in batches.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;This split is useful, because for most queries we can get away with just loading the metadata into memory, not the content. This is a big saving in time and memory when deserialising the objects from the underlying datastore.&lt;/p&gt;&lt;h2 id=&quot;efficient-mailbox-querying&quot; tabindex=&quot;-1&quot;&gt;Efficient mailbox querying&lt;/h2&gt;&lt;p&gt;A linear pass through all the metadata is surprisingly tractable, even for large mailboxes, however it’s slower than we want for common queries (like opening your inbox). This is where we introduce a couple of extra custom indexes — separate object stores we are careful to update in lock step with any changes to our data.&lt;/p&gt;&lt;p&gt;The first of these is &lt;strong&gt;EmailMailboxes&lt;/strong&gt;. This stores an entry for each addition or removal of a message from a folder/label, allowing us to both very efficiently compute the list of messages/conversations in a particular mailbox, and also calculate a delta update to the query when making changes.&lt;/p&gt;&lt;p&gt;The key for this object store is:&lt;/p&gt;&lt;pre class=&quot;language-javascript&quot;&gt;&lt;code class=&quot;language-javascript&quot;&gt;&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;MAILBOX_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;constant token&quot;&gt;REMOVED_MODSEQ&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;constant token&quot;&gt;ADDED_MODSEQ&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The values look like:&lt;/p&gt;&lt;pre class=&quot;language-javascript&quot;&gt;&lt;code class=&quot;language-javascript&quot;&gt;&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;EMAIL_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;constant token&quot;&gt;THREAD_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;constant token&quot;&gt;DATE&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;constant token&quot;&gt;IS_UNREAD&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Whenever a message is added to a mailbox, a new entry is created. &lt;code&gt;ADDED_MODSEQ&lt;/code&gt; is the current “updated” moseq of the message, and &lt;code&gt;REMOVED_MODSEQ&lt;/code&gt; is 0.&lt;/p&gt;&lt;p&gt;If the message is removed from the mailbox, the old entry is deleted, and a new one added with the same &lt;code&gt;ADDED_MODSEQ&lt;/code&gt;, but &lt;code&gt;REMOVED_MODSEQ&lt;/code&gt; set to the new “updated” modseq of the message.&lt;/p&gt;&lt;p&gt;From this, we can quickly get the list of current messages in a particular mailbox by doing a range query for entries with keys that start: &lt;code&gt;[MAILBOX_ID, 0]&lt;/code&gt;. The values include the date and thread id, allowing us to do the most common sort, and remove duplicates for the same thread id, without having to even fetch the metadata objects for the emails.&lt;/p&gt;&lt;h2 id=&quot;delta-query-updates&quot; tabindex=&quot;-1&quot;&gt;Delta query updates&lt;/h2&gt;&lt;p&gt;JMAP has a way for a client to &lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc8620.html#section-5.6&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ask for what’s changed in a query&lt;/a&gt;. This allows it to more efficiently update its local store and uses less bandwidth. With the EmailMailboxes index, we can also implement this. First we fetch the entries for the current messages as before, but then we also fetch the entries for messages that have been removed since our last state (this is a range query between &lt;code&gt;[MAILBOX_ID, sinceModSeq + 1]&lt;/code&gt; and &lt;code&gt;[MAILBOX_ID, max_int]&lt;/code&gt;). We sort these entries together according to the sort order the user has requested, normally date descending:&lt;/p&gt;&lt;pre class=&quot;language-javascript&quot;&gt;&lt;code class=&quot;language-javascript&quot;&gt;mailboxRecords&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;sort&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;a&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; b&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt;
         b&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;DATE&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;-&lt;/span&gt; a&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;DATE&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;||&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;a&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;EMAIL_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;&amp;lt;&lt;/span&gt; b&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;EMAIL_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;?&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; a&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;EMAIL_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;&gt;&lt;/span&gt; b&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;EMAIL_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;?&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;number token&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;||&lt;/span&gt;
         a&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;ADDED_MODSEQ&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;-&lt;/span&gt; b&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;ADDED_MODSEQ&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Then we can iterate through to calculate what has been added or removed from the query, like so. (“Exemplar” is our term for the email that’s representing a thread when &lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc8621.html#section-4.4.3&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;the “collapseThreads” argument&lt;/a&gt; is true.)&lt;/p&gt;&lt;pre class=&quot;language-javascript&quot;&gt;&lt;code class=&quot;language-javascript&quot;&gt;&lt;span class=&quot;keyword token&quot;&gt;let&lt;/span&gt; index &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;number token&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; seenExemplar &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; collapseThreads &lt;span class=&quot;operator token&quot;&gt;?&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;new&lt;/span&gt; &lt;span class=&quot;class-name token&quot;&gt;Set&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;null&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; seenOldExemplar &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; collapseThreads &lt;span class=&quot;operator token&quot;&gt;?&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;new&lt;/span&gt; &lt;span class=&quot;class-name token&quot;&gt;Set&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;null&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;let&lt;/span&gt; uptoHasBeenFound &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;let&lt;/span&gt; total &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; added &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; removed &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; record &lt;span class=&quot;keyword token&quot;&gt;of&lt;/span&gt; mailboxRecords&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; isDeleted &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;record&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;REMOVED_MODSEQ&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// Created and deleted after our previous state? Ignore.&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; isNew &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; record&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;ADDED_MODSEQ&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;&gt;&lt;/span&gt; sinceModSeq&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;isNew &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; isDeleted&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;continue&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
     &lt;span class=&quot;comment token&quot;&gt;// Is this message the current exemplar?&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;let&lt;/span&gt; isNewExemplar &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;let&lt;/span&gt; isOldExemplar &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; emailId &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; record&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;EMAIL_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; threadId &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; record&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;THREAD_ID&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;isDeleted &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;collapseThreads &lt;span class=&quot;operator token&quot;&gt;||&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;seenExemplar&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;has&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;threadId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         isNewExemplar &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         index &lt;span class=&quot;operator token&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         total &lt;span class=&quot;operator token&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;collapseThreads&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             seenExemplar&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;add&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;threadId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// Was this message an old exemplar?&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// 1. Must not have been added to mailbox after the client&#39;s state&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// 2. Must have been removed from mailbox before the client&#39;s state&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// 3. Must not have already found the old exemplar.&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;isNew &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;collapseThreads &lt;span class=&quot;operator token&quot;&gt;||&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;seenOldExemplar&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;has&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;threadId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         isOldExemplar &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;collapseThreads&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             seenOldExemplar&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;add&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;threadId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
     &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;isOldExemplar &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;isNewExemplar&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         removed&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;push&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;emailId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;else&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;isOldExemplar &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; isNewExemplar&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         &lt;span class=&quot;comment token&quot;&gt;// If the message has been moved out and back in again&lt;/span&gt;
         &lt;span class=&quot;comment token&quot;&gt;// we&#39;ll have separate mailbox records for added/removed&lt;/span&gt;
         &lt;span class=&quot;comment token&quot;&gt;// so not detect it&#39;s both the old and new exemplar;&lt;/span&gt;
         &lt;span class=&quot;comment token&quot;&gt;// check for that here.&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; removedIndex &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; isMutableSort &lt;span class=&quot;operator token&quot;&gt;?&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;number token&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; removed&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;indexOf&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;emailId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;removedIndex &lt;span class=&quot;operator token&quot;&gt;&gt;&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;number token&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             removed&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;splice&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;removedIndex&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;else&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             added&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;push&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
                 index&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
                 &lt;span class=&quot;literal-property property token&quot;&gt;id&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; emailId&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
             &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
     &lt;span class=&quot;comment token&quot;&gt;// Special case for mutable sorts (based on isFlagged/isUnread)&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;isMutableSort &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; isOldExemplar &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; isNewExemplar&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         &lt;span class=&quot;comment token&quot;&gt;// Has the isUnread/isFlagged status of the message/thread&lt;/span&gt;
         &lt;span class=&quot;comment token&quot;&gt;// (as appropriate) possibly changed since the client&#39;s state?&lt;/span&gt;
         &lt;span class=&quot;comment token&quot;&gt;// If so, we need to remove the exemplar from the client view&lt;/span&gt;
         &lt;span class=&quot;comment token&quot;&gt;// and add it back in at the correct position.&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; mayHaveMoved &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; collapseThreads
             &lt;span class=&quot;operator token&quot;&gt;?&lt;/span&gt; threadChanged&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;has&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;threadId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
             &lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; emailChanged&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;has&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;emailId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;mayHaveMoved&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             removed&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;push&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;emailId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
             added&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;push&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
                 index&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
                 &lt;span class=&quot;literal-property property token&quot;&gt;id&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; emailId&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
             &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// If this is the last message the client cares about, we can stop&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// here and just return what we&#39;ve calculated so far. We already&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// know the total count for this message list as we keep it pre&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// calculated and cached in the Mailbox object.&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// However, if the sort is mutable we can&#39;t break early, as&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// messages may have moved from the region we care about to lower&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// down the list.&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;isMutableSort &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;isNew &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; emailId &lt;span class=&quot;operator token&quot;&gt;===&lt;/span&gt; upToId&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         uptoHasBeenFound &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;break&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;mail-search&quot; tabindex=&quot;-1&quot;&gt;Mail search&lt;/h2&gt;&lt;p&gt;Fastmail supports an &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/360060591213-Searching-your-mail&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;extremely powerful set of search operators&lt;/a&gt;, allowing for &lt;a href=&quot;https://www.fastmail.com/features/search/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;fast, precise searching&lt;/a&gt;. We support almost all of it offline, with a few caveats discussed below.&lt;/p&gt;&lt;p&gt;To make full-text search work and be performant, we need to build another index. If you have hundreds of thousands of messages, it would be unusably slow to scan through all of them looking for a word, phrase or email address.&lt;/p&gt;&lt;p&gt;Our index is stored in another &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;IndexedDB&lt;/a&gt; object store called &lt;strong&gt;EmailSearch&lt;/strong&gt;. The key for each entry is &lt;code&gt;[token, emailId]&lt;/code&gt;. The token is usually a word or other sequence of letters and numbers extracted from the email. We also have special token variations to represent a list-id or email addresses found in the headers. We create an entry in EmailSearch for each such token we find in the email. The value encodes where the token was found (e.g. in the &lt;code&gt;To&lt;/code&gt; header, or the message body), and the index(es) of the token so we can do &lt;a href=&quot;https://en.wikipedia.org/wiki/Phrase_search&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;phrase searches&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;We decided to index the content on the device, rather than download the indexes from the server. This ensured our search index would be completely in sync with the cached messages you have on your device, and we could index and make searchable messages and memos you wrote while you were offline.&lt;/p&gt;&lt;p&gt;However, this does mean the offline search works a little differently to our server-based search, so may return slightly different results (although we think both will do a great job in most cases). In particular:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Our offline search doesn’t index any text inside attachments. When online you can search for content in attached PDFs, spreadsheets, and other documents.&lt;/li&gt; &lt;li&gt;Our offline search doesn’t do &lt;a href=&quot;https://en.wikipedia.org/wiki/Stemming&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;stemming&lt;/a&gt;. Stemming tries to reduce a word to its common root, so if you search in English for &lt;code&gt;bus&lt;/code&gt; you would also match emails containing &lt;code&gt;buses&lt;/code&gt;, but not &lt;code&gt;business&lt;/code&gt;. Stemming requires language analysis of the email content and custom stemming algorithms for each language, and we decided the extra complexity and code download size was not currently worth it for our offline search. Instead, our offline search does prefix matching by default, so &lt;code&gt;bus&lt;/code&gt; will still match &lt;code&gt;buses&lt;/code&gt; but also &lt;code&gt;business&lt;/code&gt;. Of course, if you wrap the term in quotes (like &lt;code&gt;&amp;quot;bus&amp;quot;&lt;/code&gt;) it will only look for exact matches, just like with server-based search.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;And of course, the search index will only contain messages you have downloaded for offline, which might not be everything in your account. We therefore try to do a search on the server first and only fallback to the local search if you are offline.&lt;/p&gt;&lt;h2 id=&quot;search-tokenisation&quot; tabindex=&quot;-1&quot;&gt;Search tokenisation&lt;/h2&gt;&lt;p&gt;To create our index we have to be able to extract the tokens from a sequence of text. We have users around the world, so we knew we had to handle multilingual text and scripts. In the end, we settled on a simple but effective tokenisation algorithm:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;We normalise the string into Unicode &lt;a href=&quot;https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;NFKD normal form&lt;/a&gt;. This will decompose diacritics to make it easy to strip them, and replace various variations of letters and numbers (such as typographic ligatures, or subscript numbers) with the baseline equivalent.&lt;/li&gt; &lt;li&gt;We divide the string into segments according to the &lt;a href=&quot;https://www.unicode.org/reports/tr29/#Word_Boundaries&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Unicode text segmentation word boundary algorithm&lt;/a&gt;.&lt;/li&gt; &lt;li&gt;For each segment, we apply the full &lt;a href=&quot;https://www.unicode.org/Public/16.0.0/ucd/CaseFolding.txt&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Unicode case folding substitutions&lt;/a&gt; (for example, this will replace uppercase letters with lowercase for Latin text), then we strip every code point that’s not categorised by Unicode as a number, letter, joining punctuation, or emoji.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;If we have anything left, that’s our token. So to give an example, supposing we had the text:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;The café is über cheap — only $3.60 a ☕️!!
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;We would end up with the following tokens:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;the
 cafe
 is
 uber
 cheap
 only
 360
 a
 ☕️
 &lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;wrapping-it-up&quot; tabindex=&quot;-1&quot;&gt;Wrapping it up&lt;/h2&gt;&lt;p&gt;We now have the indexes we need for fast, precise search. There’s still a lot of work involved in putting it all together though! When you search for something complex like &lt;code&gt;in:inbox from:@example.com (is:pinned OR &amp;quot;very important&amp;quot;)&lt;/code&gt;, we analyse the query to work out which indexes to use and efficiently combine them to compute the results. The speed will depend on how much mail you have—and how fast your device is!—but we believe it lives up to the Fastmail promise of great search everywhere.&lt;/p&gt;&lt;p&gt;There’s so much interesting tech behind our offline support, but for now I need to stop writing. If you’ve read all of this mini series on how we are making our app work offline: thank you, and I hope you found it interesting! Please give the beta a go, and let us know any feedback you might have. We’re excited to finish polishing this highly requested feature and we hope to ship it to everyone early in the new year.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 18: Building offline: syncing changes back to the server</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/offline-sync/' />
 			<id>https://www.fastmail.com/blog/offline-sync/</id>
 			<updated>2024-12-18T00:00:01Z</updated><author>
 				<name>Neil Jenkins</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the eighteenth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/offline-architecture/&quot;&gt;Dec 17: Building offline: general architecture&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/offline-mail-storage/&quot;&gt;Dec 19: Building offline: mail storage&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Yesterday, we looked at &lt;a href=&quot;/blog/offline-architecture/&quot;&gt;how our offline caching layer fits into our app&lt;/a&gt;, and the way it stores data to efficiently respond to JMAP requests. Today, we’ll dive into how it keeps track of changes the user makes while offline, so it can reconcile this with the server.&lt;/p&gt;&lt;h2 id=&quot;keeping-track-of-changes&quot; tabindex=&quot;-1&quot;&gt;Keeping track of changes&lt;/h2&gt;&lt;p&gt;When a client makes a change offline, we update our local cache and have to keep track of it so we can sync that change back to the server when we come online. There are two main approaches you could take:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;You keep a time-ordered log of every change, then replay the log against the server. One record may appear multiple times in the log if it has multiple modifications applied.&lt;/li&gt; &lt;li&gt;You keep a set of created/updated/destroyed records, along with the current server value. Each record can only appear once, in at most one of these categories. You calculate the difference between the server state and the current state to update the server.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;The benefit of the first approach is it ensures we maintain any ordering dependencies. The benefit of the second approach is it’s more efficient in terms of both storage and synchronisation speed when there are multiple changes made to the same record.&lt;/p&gt;&lt;p&gt;The Fastmail offline cache uses a hybrid of these approaches to try to get the best of both worlds:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;A log stores (in order) the &lt;code&gt;[data type, account id, id]&lt;/code&gt; of any changes, along with what type of change this is (create/update/destroy).&lt;/li&gt; &lt;li&gt;The record itself stores the last known server state if it’s been updated, stored efficiently as a patch to get back to the server state from the updated state.&lt;/li&gt; &lt;li&gt;If the record is updated a second time, it: &lt;ul&gt; &lt;li&gt;stays in its current position in the log if not yet present on the server (this is a create); or&lt;/li&gt; &lt;li&gt;moves to the end of the log (remove the old entry and add a new one) if it already exists on the server (this is an update/destroy); or&lt;/li&gt; &lt;li&gt;is removed entirely from the log if the change reverted it back to the last-known server state.&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;/ul&gt;&lt;p&gt;If we’re updating a record that’s not yet been created on the server, we may have to do an update as well as a create, due to an ordering problem. For example, suppose you do the following:&lt;/p&gt;&lt;ol type=&quot;a&quot;&gt; &lt;li&gt;Create Mailbox X&lt;/li&gt; &lt;li&gt;Create Emails A &amp;amp; B in Mailbox X&lt;/li&gt; &lt;li&gt;Create Mailbox Y&lt;/li&gt; &lt;li&gt;Move Mailbox X to be a child of Y&lt;/li&gt; &lt;li&gt;Move Email A to be in Mailbox Y&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;You can’t move (a) later because (b) depends on it. You can’t move (d) earlier because it depends on (c). So if we update a record that’s not yet been created on the server, and we set a property that includes a local id (i.e., it references another object that’s been created locally but not yet synced to the server), we add it as a patch and apply it as an update later.&lt;/p&gt;&lt;p&gt;When loading data from the server, we do not need to look for an entry in the log of changes still to sync. We can just update the server state in the record. If the change is now inert, we’ll delete it from the log when we go to sync it.&lt;/p&gt;&lt;p&gt;For example, suppose we have a mailbox, id &lt;code&gt;1&lt;/code&gt;, with two messages in it, ids &lt;code&gt;A&lt;/code&gt; &amp;amp; &lt;code&gt;B&lt;/code&gt;, and the user does the following (contrived) actions:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Creates a new mailbox: &lt;code&gt;2&lt;/code&gt;&lt;/li&gt; &lt;li&gt;Creates a new child mailbox of that: &lt;code&gt;3&lt;/code&gt;&lt;/li&gt; &lt;li&gt;Moves A and B into mailbox &lt;code&gt;3&lt;/code&gt;&lt;/li&gt; &lt;li&gt;Marks B as read&lt;/li&gt; &lt;li&gt;Moves A back to its original mailbox.&lt;/li&gt; &lt;li&gt;Renames mailbox &lt;code&gt;2&lt;/code&gt;.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;Our log will end up looking like this:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;    [Mailbox, &amp;quot;#2&amp;quot;, CREATE]
     [Mailbox, &amp;quot;#3&amp;quot;, CREATE]
     [Email, &amp;quot;B&amp;quot;, UPDATE]
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Because &lt;code&gt;A&lt;/code&gt; is back to its original state, we’ve eliminated it entirely from the log, and do not need to send anything to the server. Because &lt;code&gt;#2&lt;/code&gt; was a create in the log, we do not move it when we renamed it at the end, which is good because otherwise the other changes in the log would both fail as they depend on it. Despite making two changes to &lt;code&gt;B&lt;/code&gt;, we only have to send a single update to the server for it.&lt;/p&gt;&lt;h2 id=&quot;conflicts&quot; tabindex=&quot;-1&quot;&gt;Conflicts&lt;/h2&gt;&lt;p&gt;Suppose you have a shared contact, let’s call him Joe Bloggs. While offline you edit to add his phone number. Meanwhile, a colleague updates his email address. This means when your client comes back online and synchronises the changes, the object it is updating has already changed. This is called a conflict.&lt;/p&gt;&lt;p&gt;For the data types we have to handle, we believe automatic resolution (rather than presenting the conflict to the user and asking them to choose what should happen) is the right way to go. We follow these simple rules:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Last write wins.&lt;/li&gt; &lt;li&gt;All updates are patches.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;This means if the same object is updated by two different people, whichever client writes second will overwrite the data of the one that wrote first. (The first client will then sync this change back so you get a consistent state.) However, since all updates are patches, it will merge the changes unless they apply to the same property on the object. So in the case above, although there were two writes to the same contact, they were updating different properties. One user was updating the “&lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc9553.html#name-emails&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;emails&lt;/a&gt;”, the other the “&lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc9553.html#name-phones&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;phones&lt;/a&gt;”. So in this case, both changes would be preserved.&lt;/p&gt;&lt;h2 id=&quot;next-up-mail-storage&quot; tabindex=&quot;-1&quot;&gt;Next up, mail storage&lt;/h2&gt;&lt;p&gt;In this post we looked at how we store changes you make offline so we can accurately and efficiently sync them back to the server when you come online. Like our discussion of data storage yesterday, everything here applies generically to all data types.&lt;/p&gt;&lt;p&gt;Tomorrow, we’ll discuss &lt;a href=&quot;/blog/offline-mail-storage/&quot;&gt;why email is special&lt;/a&gt;, and what else we do to make this super fast in our offline store.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 17: Building offline: general architecture</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/offline-architecture/' />
 			<id>https://www.fastmail.com/blog/offline-architecture/</id>
 			<updated>2024-12-17T00:00:01Z</updated><author>
 				<name>Neil Jenkins</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the seventeenth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/offline-in-beta/&quot;&gt;Dec 16: Offline support now in public beta&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/offline-sync/&quot;&gt;Dec 18: Building offline: syncing changes back to the server&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Yesterday &lt;a href=&quot;/blog/offline-in-beta/&quot;&gt;we announced full offline support for Fastmail&lt;/a&gt; is now available in public beta, both in our app and on the web. Today, and over the next few days, I’ll dive into some of the technical aspects about how we’re making this work.&lt;/p&gt;&lt;p&gt;Making Fastmail work offline has been our most popular feature request for some time (ever since &lt;a href=&quot;/blog/more-swipe-options-on-mobile-let-you-work-faster/&quot;&gt;we added support for custom swipe actions&lt;/a&gt;, our previous top request!), and we wanted to make sure our support was done &lt;em&gt;right&lt;/em&gt;. It should just work, seamlessly, and as far as possible you should be able to do everything you can do online. Open your calendar and update an event, perhaps inviting someone using autocomplete from your contacts. Search your mail and triage it. Write a new note. Add a memo. We want it all to &lt;em&gt;just work&lt;/em&gt;.&lt;/p&gt;&lt;p&gt;When you come online it should seamlessly sync these changes back to the server, and fetch any new mail and other changes.&lt;/p&gt;&lt;h2 id=&quot;general-architecture&quot; tabindex=&quot;-1&quot;&gt;General architecture&lt;/h2&gt;&lt;p&gt;The Fastmail app is very cleanly separated from our server, which was a huge benefit when adding offline support. All data in the app is loaded via a &lt;a href=&quot;https://jmap.io/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;JMAP&lt;/a&gt; API, with all UI rendering and routing happening in the app. This means the data flow looks a bit like this:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;[App] ← JMAP → [Server]
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This gave us a really well defined boundary on which to build the offline support. If we built something that could handle and respond to the JMAP requests directly on your device then the app would work offline, and we wouldn’t really have to change anything else in it. So the architecture we came up with looks like this:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;[App] ← JMAP → [Caching layer] ← JMAP → [Server]
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Looking at our new diagram, we can see that we are building two things:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;A JMAP server that can understand the requests the client makes, fetch and write the data from/to a local store, and return a JMAP response.&lt;/li&gt; &lt;li&gt;A JMAP client that can fetch data it doesn’t have from the server, and write back changes made while we were offline.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;This is actually quite a lot harder than just building a JMAP server! We will often only have partial information, and we have to transparently pass through requests for data we don’t have to the server, and gracefully handle a fallback if we are offline.&lt;/p&gt;&lt;p&gt;The core function our caching layer needs to perform is handling a JMAP request from the client. Each JMAP request is a sequence of method calls. Once we have locally cached data, we may be able to handle the method call entirely locally, but we may always run into one or more that we can’t.&lt;/p&gt;&lt;p&gt;For performance, we want to batch our method calls into a single database transaction where possible. But we can’t hold open a transaction over a network request, so we divide up the execution into phases.&lt;/p&gt;&lt;p&gt;First, we attempt to execute the method calls locally. If all complete, we’re done! If any require us to fallback to the server, we stop and send it everything from that point on, as we want to avoid making multiple HTTP requests, which would be slow.&lt;/p&gt;&lt;p&gt;If the request completed successfully, we process the responses to save any new data into our local datastore. If the request failed, we call the offline fallback methods, which may be able to still return a response to the UI.&lt;/p&gt;&lt;h2 id=&quot;a-separate-thread-for-the-caching-layer&quot; tabindex=&quot;-1&quot;&gt;A separate thread for the caching layer&lt;/h2&gt;&lt;p&gt;The caching layer runs on your device, just like the rest of the app. Because JMAP requests are already asynchronous network calls from the UI, we can easily run the caching layer in a separate OS thread so it never blocks the UI thread, which could cause &lt;a href=&quot;https://en.wiktionary.org/wiki/jank&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;jank&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Our app is built using web technology, which allows us, as a small company, to build an app that runs everywhere our users are, with a single code base and feature parity across all platforms. Separate threads are represented as &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;workers&lt;/a&gt; in the web API. There are three types of worker:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/Worker&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Dedicated worker&lt;/a&gt; — this is a worker that is tied to a particular window or tab in your browser. If you have Fastmail open in multiple tabs, each would have to create its own worker.&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Shared worker&lt;/a&gt; — this is a worker that’s shared between tabs or windows, so no matter how many you have there’s only a single instance of this worker.&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Service worker&lt;/a&gt; — this is a special type of shared worker that can intercept network requests and change their response.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;At first glance, a service worker seems the place to handle all of this, and this is what we tried first. However, we soon switched over to using a shared worker instead:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;The service worker is designed to be short lived and only spun up when needed, but we wanted to hold open a persistent &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;EventSource push connection&lt;/a&gt; while the app is open, for instant updates. The shared worker is a better conceptual fit for this.&lt;/li&gt; &lt;li&gt;We don’t need to intercept network requests, as we can just pass the JMAP request object directly to the worker using the &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/Worker/postMessage&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;postMessage&lt;/a&gt; API, avoiding some serialisation overhead.&lt;/li&gt; &lt;li&gt;We ran into a bug in iOS where network requests would sometimes not be intercepted even though the service worker was registered when the app was running in the background. This meant we &lt;em&gt;had&lt;/em&gt; to pass the request directly to the worker anyway instead of intercepting it at the network level to ensure we didn’t hit this bug.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;We wanted to use a shared worker rather than a dedicated worker for more efficiency when there were multiple tabs open — we can avoid some contention and locking issues, and ensure we have a single push connection open to the server.&lt;/p&gt;&lt;h2 id=&quot;storing-data-indexed-db&quot; tabindex=&quot;-1&quot;&gt;Storing data: IndexedDB&lt;/h2&gt;&lt;p&gt;The web API for storing large volumes of structured data is &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;IndexedDB&lt;/a&gt;. This lets you create multiple &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/IDBObjectStore&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;object stores&lt;/a&gt; (the equivalent of SQL tables), which offer simple key-value storage with ordered keys. Indexes can be automatically built based on properties in the object being stored. Transactions ensure data consistency.&lt;/p&gt;&lt;p&gt;The IndexedDB API was unfortunately designed just before promises became ubiquitous in the web world. This means just fetching a record from a store requires code a bit like this:&lt;/p&gt;&lt;pre class=&quot;language-javascript&quot;&gt;&lt;code class=&quot;language-javascript&quot;&gt;&lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; request &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; store&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;get&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;id&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function function-variable token&quot;&gt;onerror&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;callErrorHandler&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function function-variable token&quot;&gt;onsuccess&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; record &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;result&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// Do something&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This is clunky and becomes hard to read and follow. However, we wrote one tiny little wrapper function that converts it into a promise-based API:&lt;/p&gt;&lt;pre class=&quot;language-javascript&quot;&gt;&lt;code class=&quot;language-javascript&quot;&gt;&lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;function function-variable token&quot;&gt;_&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;request&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;new&lt;/span&gt; &lt;span class=&quot;class-name token&quot;&gt;Promise&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;resolve&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; reject&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function function-variable token&quot;&gt;onsuccess&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;resolve&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;result&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function function-variable token&quot;&gt;onerror&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;reject&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;error&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Using this function, we can rewrite the above fetch like this:&lt;/p&gt;&lt;pre class=&quot;language-javascript&quot;&gt;&lt;code class=&quot;language-javascript&quot;&gt;&lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; record &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;await&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;_&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;store&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;get&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;id&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// Do something&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;(Note, if the fetch has an error this will result in an exception being thrown, which is generally handled at a higher layer, so avoids that cluttering our code here at all!)&lt;/p&gt;&lt;p&gt;With this simple addition, I found the IndexedDB API consistent and easy to work with.&lt;/p&gt;&lt;h2 id=&quot;the-standard-object-store-structure&quot; tabindex=&quot;-1&quot;&gt;The standard object store structure&lt;/h2&gt;&lt;p&gt;The consistency of JMAP means we can write one generic implementation and then use it to provide offline support for all our data types. For each data type (such as Calendar, Email, Contact, etc.) we create an object store to store the instances of that type.&lt;/p&gt;&lt;p&gt;When we create the object store we also store a single metadata object in it, using a special key (a zero-byte &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ArrayBuffer&lt;/a&gt;). This stores some important bookkeeping information, in particular the following properties:&lt;/p&gt;&lt;pre class=&quot;language-javascript&quot;&gt;&lt;code class=&quot;language-javascript&quot;&gt;&lt;span class=&quot;comment token&quot;&gt;// Do we have the full set of data from the server for this data&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// type?&lt;/span&gt;
 &lt;span class=&quot;literal-property property token&quot;&gt;hasAllRecords&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// State string representing the current server state we have&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// synced with. The store may also contain newer information, but&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// that&#39;s ok as it will still get to the correct state when we&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// update from the old state.&lt;/span&gt;
 &lt;span class=&quot;literal-property property token&quot;&gt;serverState&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// This is the highest modseq of a record in the store.&lt;/span&gt;
 &lt;span class=&quot;literal-property property token&quot;&gt;lastModSeq&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// This is the highest modseq of a record that was destroyed that&#39;s&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// now been removed entirely from the store; we can only calculate&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// changes accurately from this point on.&lt;/span&gt;
 &lt;span class=&quot;literal-property property token&quot;&gt;highestPurgedModSeq&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// This is the number of records currently marked destroyed in the&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// store. We keep them there so we can calculate changes. Once we&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// cross a threshold, we&#39;ll clean up old ones.&lt;/span&gt;
 &lt;span class=&quot;literal-property property token&quot;&gt;numDestroyed&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;A key concept here is &lt;em&gt;modseq&lt;/em&gt;, which stands for “modification sequence”. It’s a counter we keep per account, per data type. Every time we make a change to a record in our local store we bump the sequence number and assign that as the new “updated” modseq for that record. We also store a “created” modseq on each record, which is the same as the “updated” modseq when the record is first created. These simple bookkeeping properties allow us to efficiently calculate changes, as needed for &lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc8620.html#section-5.2&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;the JMAP “/changes” method&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Aside from the metadata object, every other entry in the object store is a record — an instance of that data type.&lt;/p&gt;&lt;p&gt;The key for each record is &lt;code&gt;[account id, id]&lt;/code&gt;, because some data types exist in multiple accounts (e.g. shared contacts and your personal contacts) and ids are only unique within an account. As far as I could see from inspecting the source, string keys are stored as &lt;a href=&quot;https://en.wikipedia.org/wiki/UTF-16&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;UTF-16&lt;/a&gt; in all major IndexedDB implementations. This is a fairly inefficient encoding, especially as we know JMAP ids can only use the &lt;a href=&quot;https://datatracker.ietf.org/doc/html/rfc4648#section-5&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;base64url characters&lt;/a&gt;, so for efficiency we encode this data into an &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ArrayBuffer&lt;/a&gt;, making use of this fact.&lt;/p&gt;&lt;p&gt;The value associated with the key is the record itself — an object representing an instance of that data type, as fetched from the server. In addition, we add a few bookkeeping properties, as discussed above:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;The created modseq&lt;/li&gt; &lt;li&gt;The updated modseq&lt;/li&gt; &lt;li&gt;Is the record destroyed?&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;Each object store has an index built automatically based on the updated modseq of the records.&lt;/p&gt;&lt;p&gt;The modseq is used as the “state” string over JMAP. When asked for what’s changed since a particular state, we know:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Only records with a higher modseq have changed (which we can efficiently get from the index).&lt;/li&gt; &lt;li&gt;If the record’s &lt;code&gt;created&lt;/code&gt; modseq is higher than &lt;code&gt;lastModSeq&lt;/code&gt;, it’s new. Otherwise it’s been updated or destroyed (depending on whether the record is now destroyed). If it’s new and also destroyed, we can ignore it entirely.&lt;/li&gt; &lt;/ul&gt;&lt;h2 id=&quot;next-up-local-changes&quot; tabindex=&quot;-1&quot;&gt;Next up, local changes&lt;/h2&gt;&lt;p&gt;In this post we looked at the basic overview of how our offline caching layer fits into our app, and the way it stores data to efficiently respond to JMAP requests. Tomorrow, we’ll dive into &lt;a href=&quot;/blog/offline-sync/&quot;&gt;how it keeps track of changes the user makes while offline&lt;/a&gt;, so it can reconcile this with the server.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 16: Offline support now in public beta</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/offline-in-beta/' />
 			<id>https://www.fastmail.com/blog/offline-in-beta/</id>
 			<updated>2024-12-16T00:00:01Z</updated><author>
 				<name>Neil Jenkins</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the sixteenth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/platform-team-working-agreement/&quot;&gt;Dec 15: Platform Team working agreement&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/offline-architecture/&quot;&gt;Dec 17: Building offline: general architecture&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;On an aeroplane, or down a tunnel. Roaming in a foreign land, or just out for a walk in the country. There are still times when we aren’t connected to the internet. As a strong supporter of open standards, you’ve always been able to use Fastmail with any email app you like, many of which work offline. However, many of our customers prefer the Fastmail app (we certainly do!), and offline support there has been our most popular request for some time. As Bron &lt;a href=&quot;/blog/2023-advent-post-bron-gondwana/&quot;&gt;foreshadowed last year&lt;/a&gt;, we’ve been working hard on this big project for some time, and and we’re very pleased to announce it’s now ready for public beta testing.&lt;/p&gt;&lt;h2 id=&quot;what-s-a-beta&quot; tabindex=&quot;-1&quot;&gt;What’s a beta?&lt;/h2&gt;&lt;p&gt;Fastmail runs a beta version of our app to allow our users that like to live on the cutting edge to get early access to new features before they’re finished, and send us feedback while we’re still working on them. Things may be broken occasionally, but generally it’s pretty stable.&lt;/p&gt;&lt;h2 id=&quot;how-do-i-get-the-beta&quot; tabindex=&quot;-1&quot;&gt;How do I get the beta?&lt;/h2&gt;&lt;p&gt;If you are using our iOS or Android apps:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;Check for updates in the app store — make sure you have the latest version.&lt;/li&gt; &lt;li&gt;Go to Settings → Device settings → Show Advanced Settings, and change the “server backend” to “Beta”.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;If you are using Fastmail on the web, just log in at &lt;a href=&quot;https://betaapp.fastmail.com&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;betaapp.fastmail.com&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;In either case, once on beta you will find a new settings screen — &lt;a href=&quot;https://betaapp.fastmail.com/settings/offline&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Settings → Offline&lt;/a&gt; — where you can toggle on offline support. You must have ticked “Keep me logged in” when you logged in to enable offline support.&lt;/p&gt;&lt;h2 id=&quot;what-can-i-do-offline&quot; tabindex=&quot;-1&quot;&gt;What can I do offline?&lt;/h2&gt;&lt;p&gt;Almost everything! You can read mail, reply, view and edit your contacts/calendar, change settings, etc.…&lt;/p&gt;&lt;p&gt;Probably easier is to describe what &lt;em&gt;won’t&lt;/em&gt; work offline:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Mail search will not look inside attachments, and will give slightly different results to when online. If you don’t choose to make every message available offline, it won’t be able to match against content it hasn’t downloaded!&lt;/li&gt; &lt;li&gt;Snoozed messages will not move back to the inbox while offline.&lt;/li&gt; &lt;li&gt;Calendar reminders will not show a notification.&lt;/li&gt; &lt;li&gt;You can’t delete attachments.&lt;/li&gt; &lt;li&gt;You can’t attach files to a message you are composing.&lt;/li&gt; &lt;li&gt;You can’t add or change users or domains, change your plan or update your billing details, or change your security settings.&lt;/li&gt; &lt;/ul&gt;&lt;h2 id=&quot;how-do-i-send-feedback&quot; tabindex=&quot;-1&quot;&gt;How do I send feedback?&lt;/h2&gt;&lt;p&gt;Found a bug? Got an idea? Something you like, or don’t like? &lt;a href=&quot;/support/&quot;&gt;Let us know&lt;/a&gt;! Our support team will pass all feedback on to the product team. We promise we read and carefully consider it all.&lt;/p&gt;&lt;h2 id=&quot;what-s-the-tech-behind-your-offline-support&quot; tabindex=&quot;-1&quot;&gt;What’s the tech behind your offline support?&lt;/h2&gt;&lt;p&gt;Interested in the technical details? Over the next few days I’m going to dive into how we built offline support into our app, starting with &lt;a href=&quot;/blog/offline-architecture/&quot;&gt;the general architecture&lt;/a&gt;.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 15: Platform Team working agreement</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/platform-team-working-agreement/' />
 			<id>https://www.fastmail.com/blog/platform-team-working-agreement/</id>
 			<updated>2024-12-15T00:00:01Z</updated><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the fifteenth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/on-call-systems/&quot;&gt;Dec 14: On-call systems&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/offline-in-beta/&quot;&gt;Dec 16: Offline support now in public beta&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Another Sunday, another document from our internal collection!&lt;/p&gt;&lt;p&gt;Over the past couple of years, we’ve been taking the bits that we like from scrum, agile, and all the other buzzword methodologies that are out there. One of the best things that all the good frameworks to is give a space for setting behaviours for ourselves and each other within a team — so you know what to expect from your colleagues, and also what they expect of you!&lt;/p&gt;&lt;p&gt;I posted about our Mission Statement and Guiding Principles already, now we narrow down to look at a single team. Our platform team maintains the infrastructure that the Fastmail product runs on top of.&lt;/p&gt;&lt;p&gt;We started from here:&lt;/p&gt;&lt;h2 id=&quot;principles-and-duties&quot; tabindex=&quot;-1&quot;&gt;Principles and duties&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;We are lifeguards:&lt;/strong&gt;&lt;/p&gt;&lt;ul&gt; &lt;li&gt;We watch out for upcoming risks and for systems in distress.&lt;/li&gt; &lt;li&gt;Part of our day is keeping a watchful eye and scanning for strangeness and hints of something going down.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;&lt;strong&gt;We are first-aiders:&lt;/strong&gt;&lt;/p&gt;&lt;ul&gt; &lt;li&gt;When something goes wrong, our first principle is to keep the patient alive&lt;/li&gt; &lt;li&gt;For Fastmail: the patient is the data stored on our systems; the data we store on the behalf of others. This is a higher priority than uptime.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;&lt;strong&gt;We are paramedics:&lt;/strong&gt;&lt;/p&gt;&lt;ul&gt; &lt;li&gt;We do more than just keep the patient alive! We perform surgery and repairs to the level of our ability.&lt;/li&gt; &lt;li&gt;For more complex things, we stabilise and bring things to the specialists in that system to make more permanent repairs.&lt;/li&gt; &lt;li&gt;We get the service back up for as many as possible, as quickly as possible, without compromising data.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;&lt;strong&gt;We are specialists:&lt;/strong&gt;&lt;/p&gt;&lt;ul&gt; &lt;li&gt;For the basics of first aid and early response, we all need to become experts - and we all need to be vigilant.&lt;/li&gt; &lt;li&gt;But; we all have our areas of specialty, and it’s OK to lean into that!&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;&lt;strong&gt;The duties of a platformer:&lt;/strong&gt;&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Monitoring - looking at the alert emails, the metrics, etc and making sure things are nominal.&lt;/li&gt; &lt;li&gt;First-aid first - if the platform is in crisis, drop everything and fix it. This is the oncall duty, we all do this.&lt;/li&gt; &lt;li&gt;Janitorial - mop the floors, oil the joints, replace worn parts. We all do this, on rotation and as we see things. We keep the ship ship-shape.&lt;/li&gt; &lt;li&gt;Initiatives - look into possible improvements, experiment with possibilities, and build the future. Everyone should have an initiative they are leading or collaborating on. These make progress in the time we’re not doing first-aid or janitorial work.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;A platform team shouldn’t be particularly busy. The first-aid and janitorial work should not be a large chunk of your day most of the time.&lt;/p&gt;&lt;p&gt;Everyone on the platform team is an experienced and senior Platform Engineer, though most of the current team are quite new to Fastmail, which is why I’m also embedded with them as the crusty old expert who knows where many of the bodies are buried!&lt;/p&gt;&lt;p&gt;Since the team is split between USA and Australia; we only spend an hour of dedicated synchronous time per week. So we brought everyone together in Melbourne in November, and came up with the following schedule and working agreement:&lt;/p&gt;&lt;p&gt;Over each 4 week period, this includes two set of task list gardening, two technical deep-dive sessions, a retrospective where we can change how we operate as a team! One member of the team runs all the ceremonies for a 4 week period, and hands off after the retrospective.&lt;/p&gt;&lt;h2 id=&quot;working-agreement&quot; tabindex=&quot;-1&quot;&gt;Working Agreement&lt;/h2&gt;&lt;p&gt;We value action, forward progress and take ownership of tasks&lt;/p&gt;&lt;p&gt;We learn from each others successes and mistakes, and use good processes to protect ourselves from human error, always asking how we can be better.&lt;/p&gt;&lt;p&gt;We give and seek feedback, and encourage asking questions (no dumb questions)&lt;/p&gt;&lt;p&gt;We generate discussion and directional buy in, and make time per site for mobbing/pairing and getting clarity on solutions.&lt;/p&gt;&lt;p&gt;We celebrate our successes and show them off to the rest of the company.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;To resolve conflict we:&lt;/strong&gt;&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Check with the customer (the Fastmail product team) - what do they need?&lt;/li&gt; &lt;li&gt;Check with an area expert&lt;/li&gt; &lt;li&gt;Use experiments&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;&lt;strong&gt;To agree things we:&lt;/strong&gt;&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Make solo calls if comfortable; or&lt;/li&gt; &lt;li&gt;Bring it ‘To Discuss’ - the fortnightly deep dive meeting&lt;/li&gt; &lt;/ul&gt;</content>
         </entry><entry>
             <title>Dec 14: On-call systems</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/on-call-systems/' />
 			<id>https://www.fastmail.com/blog/on-call-systems/</id>
 			<updated>2024-12-14T00:00:02Z</updated><author>
 				<name>Luke Erlacher</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the fourteenth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/moving-fastmail-dns-to-knot/&quot;&gt;Dec 13: It’s knot DNS. There’s no way it’s DNS. It is DNS!&lt;/a&gt; The next post is &lt;a href=&quot;/blog/platform-team-working-agreement/&quot;&gt;Dec 15: Platform Team working agreement&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;After the scary interlude for Friday the 13th, here’s a follow-up to our blog about support and on-call.&lt;/p&gt;&lt;p&gt;At Fastmail, we own and operate most of our stack, from networking gear and servers in the hosted datacenters we use, server OS setup, internal services, backups and redundancy, through to user-facing services and mail flow.&lt;/p&gt;&lt;p&gt;In the last post, we introduced our organizational model of how our support and on-call team manage on-call and incidents. In this blog post, we get a little more technical into the systems we use to help us do this.&lt;/p&gt;&lt;h2 id=&quot;how-incidents-are-raised&quot; tabindex=&quot;-1&quot;&gt;How Incidents are raised&lt;/h2&gt;&lt;p&gt;We source alerts from our service stack through a number of mechanisms:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Prometheus alerts for infrastructure and service metrics such as disk usage and replication delay&lt;/li&gt; &lt;li&gt;Logwatchers that alert on things like Cyrus errors&lt;/li&gt; &lt;li&gt;Cron scripts that run regular tests and alert on failures&lt;/li&gt; &lt;li&gt;Alerts from external monitoring&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;We’re currently experimenting with integrating all of these sources into a single observability stack (I’m a big fan of the unified observability model) - maybe you will read about that in next year’s advent blog post series!&lt;/p&gt;&lt;p&gt;But right now, all of these alerts go straight to Pagerduty to create an incident there.&lt;/p&gt;&lt;p&gt;Our support team and any staff member is empowered to raise an incident at any time when they observe issues that indicate an infrastructure or service failure. This is done via slack pings during working hours, and otherwise by paging through our slackops bot.&lt;/p&gt;&lt;h2 id=&quot;pagerduty-alerts&quot; tabindex=&quot;-1&quot;&gt;Pagerduty alerts&lt;/h2&gt;&lt;p&gt;Our Pagerduty on-call rotation and escalation process ensure that incidents are always promptly responded to.&lt;/p&gt;&lt;p&gt;The primary on-call engineer is also responsible for Business As Usual (BAU) tasks such as reviewing non-critical errors and escalations. When incidents have to be escalated past the primary on-call engineer, they go first to the team lead. This allows the on-call engineers that are off rotation to focus on project work during the day, and decompress outside of work hours without worrying about on-call.&lt;/p&gt;&lt;p&gt;In order to minimize the attention load of on-call alerts on the platform engineers, engineers can override the on-call schedule to take on-call when they do large deploys that result in alerts.&lt;/p&gt;&lt;p&gt;We have recently created a dedicated incident discussion channel. Previously, incident communication would take place either in the alerts feed channel, the “ops” channel where we post updates for visibility into changes and deploys, or in a team channel for the team principally responding to the incident.&lt;/p&gt;&lt;p&gt;A dedicated incident discussion channel removes incident discussion noise from other channels. We decided against going with the “create a dedicated channel for every incident” as we think this would create too much churn and reduce visibility.&lt;/p&gt;&lt;h2 id=&quot;outage-notifications&quot; tabindex=&quot;-1&quot;&gt;Outage notifications&lt;/h2&gt;&lt;p&gt;When our service is experiencing issues or outages, we want our users to know as soon as possible, both for their benefit and our support team. As we have support staff on shift 24/7, they handle outage notifications via our status page on https://fastmailstatus.com/, Zendesk banners, and posting on our social channels. During incidents, they act as the communication conduit to our users - ascertaining the extent of outages, ensuring timely updates, and checking that when services are restored this is reflected in user experience.&lt;/p&gt;&lt;h2 id=&quot;pagerduty-nitty-gritty&quot; tabindex=&quot;-1&quot;&gt;Pagerduty nitty-gritty&lt;/h2&gt;&lt;p&gt;Some of the things we do to automate and simplify our life as on-call engineers butt up against the limits of Pagerduty.&lt;/p&gt;&lt;p&gt;For example, it would be nice to have an empty escalation layer as the first layer that during normal times falls through to the next layer instantly. Then people can add themselves in this layer to override the on-call escalation chain for doing deploys. However Pagerduty does not allow this so we have to override the schedules for primary on-call instead.&lt;/p&gt;&lt;p&gt;We also have regular unavailabilities during the week for on-call engineers for scheduled events such as gym training or dance classes. For this, we want to add exclusions to the schedule so that the alerts go to the fallback on-call immediately.&lt;/p&gt;&lt;p&gt;These unavailabilities are different for every individual engineer. Pagerduty doesn’t allow to model this - on-call schedules can have almost arbitrary scheduling through weekdays, but this is per schedule and not per on-call user. So we have to split the schedule and make a separate schedule for every engineer. However, we then can no longer make a rotating schedule - inside one schedule, a user can’t be on for a week and then off for a week - unless that off week is taken by another user.&lt;/p&gt;&lt;p&gt;To work around this we have created (and paid for) a “Blank” dummy user that has no contact methods and is only there to fill the off week for a user.&lt;/p&gt;&lt;p&gt;&lt;a href=&quot;/assets/blog/2024-12-14-oncall-systems/pagerduty_screenshot_3x.png&quot; target=&quot;_blank&quot;&gt;&lt;picture&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;/assets/images/pagerduty_screenshot_3x-4tkoE_Vw31-375.webp 375w, /assets/images/pagerduty_screenshot_3x-4tkoE_Vw31-750.webp 750w, /assets/images/pagerduty_screenshot_3x-4tkoE_Vw31-1500.webp 1500w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;img alt=&quot;Pagerduty schedule&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;/assets/images/pagerduty_screenshot_3x-4tkoE_Vw31-375.png&quot; width=&quot;1500&quot; height=&quot;514&quot; srcset=&quot;/assets/images/pagerduty_screenshot_3x-4tkoE_Vw31-375.png 375w, /assets/images/pagerduty_screenshot_3x-4tkoE_Vw31-750.png 750w, /assets/images/pagerduty_screenshot_3x-4tkoE_Vw31-1500.png 1500w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;/picture&gt;&lt;/a&gt;&lt;/p&gt;&lt;p&gt;As you can see in the picture, there are 3 layers in the schedule: The first layer is a base layer that is a weekly rotation between the fallback engineer and the Blank user.&lt;/p&gt;&lt;p&gt;The second layer has the primary oncall engineer’s shift for the first half of the day, and the third layer has the second half of the day. This is also in a weekly rotation between the oncall engineer and the Blank user.&lt;/p&gt;&lt;p&gt;To understand what’s going on here, imagine that the primary oncall engineer needs to be offline every monday from 4PM to 6PM. For that, we put an 8AM - 4PM shift time for every monday in the second layer, and 6PM to 10PM in the third layer. During the 4PM to 6PM time, there is no on-call user in the second or third layer, so it falls back to the base layer and so the fallback engineer will be active.&lt;/p&gt;&lt;p&gt;Finally, we make a separate schedule for every on-call engineer following the same pattern, but offset by one week.&lt;/p&gt;&lt;h2 id=&quot;what-works-well&quot; tabindex=&quot;-1&quot;&gt;What works well&lt;/h2&gt;&lt;p&gt;I have done overnight on-call at previous companies and that is a lot more stressful. I definitely think being able to split on-call is a lot better for engineers’ quality of life!&lt;/p&gt;&lt;p&gt;We have ops tools to do things like log searching, show replication stats, grafana dashboards, and prometheus alerts that give us quick insights into the state of our infrastructure and services to pin down the root cause of an incident quickly. These could be more comprehensive and better organized but they work well.&lt;/p&gt;&lt;p&gt;We have runbooks for some, but not all, alerts. Where they exist they are quite good and comprehensive and we frequently review and update them after incidents.&lt;/p&gt;&lt;h2 id=&quot;what-could-work-better&quot; tabindex=&quot;-1&quot;&gt;What could work better&lt;/h2&gt;&lt;p&gt;Most of our incidents auto-resolve when the service / monitor returns to nominal service. This is good! We are lucky to have very few flappy alerts that bounce up and down.&lt;/p&gt;&lt;p&gt;But a handful of alerts do not auto-resolve because the alerting mechanism is not stateful. This is confusing for engineers, and it is an extra annoyance to clean up alerts after an incident.&lt;/p&gt;&lt;p&gt;We don’t currently have good categorization and prioritization of alerts. The only way to know whether an alert is important or not is to know from experience, or asking someone with experience. We need to spend more time and be more ruthless in weeding out low-quality alerts!&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 13: It’s knot DNS. There’s no way it’s DNS. It is DNS!</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/moving-fastmail-dns-to-knot/' />
 			<id>https://www.fastmail.com/blog/moving-fastmail-dns-to-knot/</id>
 			<updated>2024-12-13T00:00:01Z</updated><author>
 				<name>Rob Mueller</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the thirteenth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/following-the-sun/&quot;&gt;Dec 12: Following the Sun&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/on-call-systems/&quot;&gt;Dec 14: On-call systems&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Ten years ago on December 13, 2014, I talked about how Fastmail had moved its DNS system &lt;a href=&quot;/blog/fastmail-dns-hosting/&quot;&gt;from TinyDNS to PowerDNS&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;In 2023, we made another big move, switching all our DNS serving from PowerDNS to &lt;a href=&quot;https://www.knot-dns.cz/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Knot DNS&lt;/a&gt;. This turned out to be a fairly large change and we ended up going down a couple of different paths before landing on a solid implementation.&lt;/p&gt;&lt;h2 id=&quot;where-were-we-up-to-again&quot; tabindex=&quot;-1&quot;&gt;Where were we up to again?&lt;/h2&gt;&lt;p&gt;As a reminder where we &lt;a href=&quot;/blog/fastmail-dns-hosting/&quot;&gt;left off in 2014&lt;/a&gt;, we’d moved DNS serving for our &lt;code&gt;ns[12].messagingengine.com&lt;/code&gt; DNS servers from static TinyDNS files to using PowerDNS with its &lt;a href=&quot;https://doc.powerdns.com/authoritative/backends/pipe.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;pipe backend&lt;/a&gt; to generate content dynamically on each DNS request via internal logic in code. Modulo some caching, this removed the latency from when you made an update to your domain’s DNS in our UI to when those changes became visible at our DNS servers.&lt;/p&gt;&lt;p&gt;If you own your own domain and host it at Fastmail, it’s easy to customise the DNS for it. Just go to &lt;a href=&quot;https://app.fastmail.com/settings/domains&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Settings -&amp;gt; Domains&lt;/a&gt; and click &lt;strong&gt;Edit&lt;/strong&gt; next to the domain.&lt;/p&gt;&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;/assets/images/customise-dns-1-F6B9iDf5KA-375.webp 375w, /assets/images/customise-dns-1-F6B9iDf5KA-750.webp 750w, /assets/images/customise-dns-1-F6B9iDf5KA-1024.webp 1024w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;img alt=&quot;Screenshot of domain settings&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;/assets/images/customise-dns-1-F6B9iDf5KA-375.png&quot; width=&quot;1024&quot; height=&quot;100&quot; srcset=&quot;/assets/images/customise-dns-1-F6B9iDf5KA-375.png 375w, /assets/images/customise-dns-1-F6B9iDf5KA-750.png 750w, /assets/images/customise-dns-1-F6B9iDf5KA-1024.png 1024w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;/picture&gt;&lt;/p&gt;&lt;p&gt;Then &lt;strong&gt;Customise DNS&lt;/strong&gt;.&lt;/p&gt;&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;/assets/images/customise-dns-2-q21fuWTaz8-375.webp 375w, /assets/images/customise-dns-2-q21fuWTaz8-750.webp 750w, /assets/images/customise-dns-2-q21fuWTaz8-1024.webp 1024w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;img alt=&quot;Screenshot of Customise DNS link&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;/assets/images/customise-dns-2-q21fuWTaz8-375.png&quot; width=&quot;1024&quot; height=&quot;80&quot; srcset=&quot;/assets/images/customise-dns-2-q21fuWTaz8-375.png 375w, /assets/images/customise-dns-2-q21fuWTaz8-750.png 750w, /assets/images/customise-dns-2-q21fuWTaz8-1024.png 1024w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;/picture&gt;&lt;/p&gt;&lt;p&gt;By default we generate a number of records to make using your domain with email easy. We recommend leaving these as is, but you have full control and it’s easy to add or change additional records.&lt;/p&gt;&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;/assets/images/customise-dns-3-_fxZZelxrZ-375.webp 375w, /assets/images/customise-dns-3-_fxZZelxrZ-750.webp 750w, /assets/images/customise-dns-3-_fxZZelxrZ-1024.webp 1024w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;img alt=&quot;Screenshot of custom DNS page&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;/assets/images/customise-dns-3-_fxZZelxrZ-375.png&quot; width=&quot;1024&quot; height=&quot;270&quot; srcset=&quot;/assets/images/customise-dns-3-_fxZZelxrZ-375.png 375w, /assets/images/customise-dns-3-_fxZZelxrZ-750.png 750w, /assets/images/customise-dns-3-_fxZZelxrZ-1024.png 1024w&quot; sizes=&quot;(max-width: 425px) 375px, 750px&quot;&gt;&lt;/picture&gt;&lt;/p&gt;&lt;pre&gt;&lt;code&gt;# dig +short a.b.c.uberengineer.com TXT
 &amp;quot;txt for a.b.c&amp;quot;
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;While this solved the original latency problem we had, it introduced a few others.&lt;/p&gt;&lt;p&gt;The pipe backend was computationally considerably more expensive. Under normal DNS load this wasn’t a problem and the system was scaled appropriately to handle it just fine. However if we got hit with excessive load mostly due to some form of &lt;a href=&quot;https://en.wikipedia.org/wiki/Denial-of-service_attack#Distributed_DoS_attack&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;DDoS&lt;/a&gt; attack, the system could easily come under strain and start to fail. When this happens people can’t access our website reliably, or our IMAP/POP/SMTP servers, and worst of all other sites might have problems working out which servers to deliver email for Fastmail customers to.&lt;/p&gt;&lt;p&gt;To protect our servers from DDoS attacks, we had put them behind &lt;a href=&quot;https://developers.cloudflare.com/dns/dns-firewall/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Cloudflare’s DNS firewall&lt;/a&gt; product, which is a global distributed DNS cache.&lt;/p&gt;&lt;p&gt;The Cloudflare DNS firewall product works great at edge caching if there’s a flood of DNS queries to a particular domain (or small set of) domain names and solved most of the DDoS flooding issues we saw. However we experienced cases where we were flooded with DNS queries of the form &lt;code&gt;$randomdomain.fastmail.com&lt;/code&gt; (called a &lt;a href=&quot;https://developers.cloudflare.com/dns/dns-firewall/random-prefix-attacks/about/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;pseudo random prefix attack&lt;/a&gt; or random subdomain attack). If every DNS query is to a random different sub-domain, then Cloudflare has to pass all those queries straight through to us as it has nothing cached. In theory Cloudflare say they can mitigate this. Unfortunately we felt their mitigation didn’t work particularly well and still resulted in a significant overload of incoming DNS queries.&lt;/p&gt;&lt;p&gt;Again, this flood of queries could cause an overload of the PowerDNS pipe backend processes which caused visible DNS downtime. We couldn’t find any sensible tuning that would allow PowerDNS and our pipe backend to correctly operate under one of these sustained floods.&lt;/p&gt;&lt;h2 id=&quot;mitigating-random-prefix-attacks&quot; tabindex=&quot;-1&quot;&gt;Mitigating random prefix attacks&lt;/h2&gt;&lt;p&gt;At the time we needed a quick solution to this problem. So we ended up splitting our DNS in two. We noted that basically all the random prefix attacks were against our system domains like fastmail.com and not user domains. Since DNS for our system domains doesn’t change much at all we basically backtracked and put our system domains onto separate nameservers that ran TinyDNS using a mostly static database of DNS records. Although this felt hacky, it worked. TinyDNS was able to absorb the higher query load in these attack situations quite well.&lt;/p&gt;&lt;p&gt;This however complicated our DNS setup even more. We also knew if a user domain experienced one of these attacks, it wasn’t using the TinyDNS system. Obviously it was possible an attack could take out not just the one user domain, but &lt;em&gt;all&lt;/em&gt; user domains using our DNS if it overloaded the PowerDNS server. We wanted a simpler, better, and more permanent solution.&lt;/p&gt;&lt;h2 id=&quot;replacing-all-dns-with-knot-dns-server&quot; tabindex=&quot;-1&quot;&gt;Replacing all DNS with Knot DNS server&lt;/h2&gt;&lt;p&gt;So the main observation about DNS is that in general it doesn’t actually change that often. So what we wanted was a solution where each of the 100,000’s of domains in our system represents a DNS zone that can be individually built into a static database, and each zone can be added/updated/deleted separately without having to reload/rebuild all zones.&lt;/p&gt;&lt;p&gt;We looked around at a few servers and went with &lt;a href=&quot;https://www.knot-dns.cz/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Knot DNS&lt;/a&gt;.&lt;/p&gt;&lt;ul&gt; &lt;li&gt;It looks like a “standard” DNS server with zone files, replication, etc, so it’s easier for new staff to understand&lt;/li&gt; &lt;li&gt;A single zone can be easily added/updated/deleted at runtime into its live database&lt;/li&gt; &lt;li&gt;It’s a known &lt;a href=&quot;https://www.knot-dns.cz/benchmark/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;high performance server out of the box&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;In theory then what we want isn’t actually that hard:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Whenever a domain is added/updated/deleted, we create/replace/remove a zone file for that domain&lt;/li&gt; &lt;li&gt;We tell the DNS server to add/update/drop the corresponding zone&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;The devil turned out to be in many small details and mis-adventures.&lt;/p&gt;&lt;h2 id=&quot;lets-not-unknot-the-knot&quot; tabindex=&quot;-1&quot;&gt;Lets not unknot the knot&lt;/h2&gt;&lt;p&gt;First things first. Although Knot as a DNS server works great, we’ve found its name to be a bit annoying. It’s a play on the fact that the most common DNS server on the internet is &lt;a href=&quot;https://www.isc.org/bind/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;bind&lt;/a&gt; so… knot. Unfortunately, you’ll find yourself at some point saying something &lt;a href=&quot;https://www.youtube.com/watch?v=8H1u-zh9dmU#t=0m53&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Bernard Woolley-esque&lt;/a&gt; like “it was not obvious that it did not work because knot was not bound to the knot ips”. Which is easier to understand when you read it compared to when you say it. We keep trying to keep discussions sane by explicitly saying “ka-not” whenever we refer to the server/software.&lt;/p&gt;&lt;h2 id=&quot;detecting-zone-changes&quot; tabindex=&quot;-1&quot;&gt;Detecting zone changes&lt;/h2&gt;&lt;p&gt;There were two main ways we could do this:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;Catch everywhere in application code we insert/update/delete records for the &lt;code&gt;Domains&lt;/code&gt;, &lt;code&gt;CustomDNS&lt;/code&gt; or &lt;code&gt;DKIMRecords&lt;/code&gt; tables&lt;/li&gt; &lt;li&gt;Use DB triggers to do the same thing&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;We ended up going with (2) because it felt like the right choice for rock solid data reliability.&lt;/p&gt;&lt;p&gt;We update the DB for domains related changes in a number of places, from code, to scripts, to cron jobs. Each of those might need DNS zones to be updated and if we miss something unexpected now or in the future, it might create subtle bugs such as “domain DNS didn’t get updated” or worse, “domain SOA serial number didn’t get bumped and so the Knot replica server didn’t get the updated domain data, so we’re serving different DNS from two different nameservers”.&lt;/p&gt;&lt;p&gt;These are the sorts of problems that could cause really hard to debug customer issues, that then randomly disappear when the domain gets touched in some other way and everything gets updated correctly again, making them extremely hard to track down and debug.&lt;/p&gt;&lt;p&gt;DNS is already hard enough for many customers to understand. We wanted to be 100% sure that our DNS servers are rock solid and the data they are generating is completely consistent with what users see in our UI.&lt;/p&gt;&lt;p&gt;Now actually getting the triggers working turned out to have a number of issues and unexpected edge cases, but also ended up with some nice results.&lt;/p&gt;&lt;ol&gt; &lt;li&gt;We moved the logic that bumps an SOA serial number into the trigger. Originally this was done on the “active primary” server, and we had to make sure this happened before the non-active failover primary rebuilt the zone. By doing this in the trigger, we ensure that the serial can never be out of date after a change.&lt;/li&gt; &lt;li&gt;We ended up with a nice design to track changed domains.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;We have a &lt;code&gt;KnotDomainsChanged&lt;/code&gt; table that looks like:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;+--------------+--------------+------+-----+---------+-------+
 | Field        | Type         | Null | Key | Default | Extra |
 +--------------+--------------+------+-----+---------+-------+
 | Server       | varchar(255) | NO   | PRI | NULL    |       |
 | DomainId     | int          | NO   | PRI | NULL    |       |
 | Domain       | varchar(255) | YES  |     | NULL    |       |
 | NeedsRebuild | tinyint      | YES  |     | 1       |       |
 | ErrorCount   | int          | YES  |     | 0       |       |
 +--------------+--------------+------+-----+---------+-------+
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;There is another table &lt;code&gt;KnotServers&lt;/code&gt; that has a list of all currently running Knot servers. Whenever a domain is added/updated/deleted, a trigger executes this query:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;    INSERT INTO KnotDomainsChanged (Server, DomainId, Domain)
     SELECT Server, BumpDomainId, BumpDomain
     FROM KnotServers
     ON DUPLICATE KEY UPDATE
       NeedsRebuild = 1,
       ErrorCount = 0;
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;With this, the &lt;code&gt;KnotDomainsChanged&lt;/code&gt; table effectively maintains a set of changed domains for each primary Knot server to pick up and build. By using &lt;code&gt;(Server, DomainId)&lt;/code&gt; as the primary key, we ensure this table can’t grow without bound even if a primary Knot server is down for a while.&lt;/p&gt;&lt;p&gt;The &lt;code&gt;NeedsRebuild&lt;/code&gt; flag allows us to correctly rebuild a zone without a race condition. The process for keeping zones up-to-date is to effectively run the following in an infinite loop.&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Fetch and iterate over all &lt;code&gt;KnotDomainsChanged&lt;/code&gt; records for this Server &lt;ul&gt; &lt;li&gt;Set &lt;code&gt;NeedsRebuild = 0&lt;/code&gt; for this &lt;code&gt;(Server, DomainId)&lt;/code&gt; record&lt;/li&gt; &lt;li&gt;If the domain exists in the &lt;code&gt;Domains&lt;/code&gt; table, add/rebuild the zone into Knot&lt;/li&gt; &lt;li&gt;If the domain does not exist in the &lt;code&gt;Domains&lt;/code&gt; table, purge the zone from Knot&lt;/li&gt; &lt;li&gt;Delete the &lt;code&gt;(Server, DomainId)&lt;/code&gt; record from &lt;code&gt;KnotDomainsChanged&lt;/code&gt; iff &lt;code&gt;NeedsRebuild = 0&lt;/code&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;/ul&gt;&lt;p&gt;So if the domain changes while a rebuild is in progress, the trigger will set &lt;code&gt;NeedsRebuild = 1&lt;/code&gt;, which means it won’t be deleted from the table after the zone build finishes, which means it’ll be picked up again the next time the sync runs again in a few seconds. This avoids the race if the domain is changed while it is in the process of being rebuilt.&lt;/p&gt;&lt;h2 id=&quot;using-standard-dns-axfr-ixfr-for-replication&quot; tabindex=&quot;-1&quot;&gt;Using standard DNS AXFR/IXFR for replication&lt;/h2&gt;&lt;p&gt;Our initial plan was to use a more “standard” DNS setup. We would have an internal hidden primary server and a number of secondary servers that pull from the primary via &lt;a href=&quot;https://datatracker.ietf.org/doc/html/rfc5936&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;AXFR&lt;/a&gt;/IXFR. Only the secondary servers would handle DNS queries from the world.&lt;/p&gt;&lt;p&gt;This turned out to have a number of annoying edge cases:&lt;/p&gt;&lt;ol&gt; &lt;li&gt;It required two completely separate Knot server setups with completely separate and quite different configurations. Although this is a more “standard” DNS management approach, it reduced some of benefit of moving to a single DNS server.&lt;/li&gt; &lt;li&gt;Using AXFR for updates caused unexpected problems. AXFR connections aren’t reused, so every domain transferred required a separate TCP connection. When a lot of domains needed to be updated at once, we &lt;a href=&quot;https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ran out of TCP socket tuples because of sockets in TIMEWAIT state&lt;/a&gt;. This caused AXFRs to the secondaries to start failing. We fixed this by enabling the kernel &lt;code&gt;tcp_tw_reuse&lt;/code&gt; tunable, but it felt… hacky.&lt;/li&gt; &lt;li&gt;At Fastmail, whenever we have a singleton service (e.g. Knot primary), we want to make sure that we can take the machine it’s running on down safely. To allow that we have the service run on at least two separate servers and use a failover IP to bind to the current up/active server.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;Unfortunately this combined with the way Knot does catalog zones completely broke AXFR/IXFR replication. What is a &lt;a href=&quot;https://datatracker.ietf.org/doc/rfc9432/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;catalog zone&lt;/a&gt;? It’s a standard way to allow primary and secondary DNS servers to keep the complete list of zones actually managed in sync.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;   The content of a DNS zone is synchronized among its primary and
    secondary nameservers using AXFR and IXFR.  However, the list of
    zones served by the primary (called a catalog in [RFC1035]) is not
    automatically synchronized with the secondaries.  To add or remove a
    zone, the administrator of a DNS nameserver farm not only has to add
    or remove the zone from the primary, they must also add/remove
    configuration for the zone from all secondaries.  This can be both
    inconvenient and error-prone; in addition, the steps required are
    dependent on the nameserver implementation.
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;It’s a classic example of taking a system that already has a way of storing data (DNS records) and replicating that data (AXFR/IXFR), and reusing those mechanisms to sync something else. In this case, a specially configured catalog zone that itself contains a list of all other member zones managed by the server in a standard defined format.&lt;/p&gt;&lt;p&gt;The basic format of the catalog zone is that each member zone managed by the server exists in the catalog zone as the RDATA of a PTR record. Then because each PTR record needs to be a unique domain name, a unique identifier is used as a sub-domain of the catalog zone itself. e.g. &lt;code&gt;unique-N.catalog.zone. PTR member-domain.org.&lt;/code&gt;&lt;/p&gt;&lt;p&gt;The problem here is that the unique identifiers are not generated in a consistent way if you have multiple different primary servers!&lt;/p&gt;&lt;pre&gt;&lt;code&gt;k1: 8815a670d8fa9032.zones.catalog.dns.internal. 0        PTR     uberengineer.com.
 k2: 09b6b01fb75e81e6.zones.catalog.dns.internal. 0        PTR     uberengineer.com.
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;During testing k1 was the entry on our first server (e.g. active primary), the k2 on the second server (e.g. backup primary).&lt;/p&gt;&lt;p&gt;So when we did a failover from the current active primary server to promote the backup primary to the active primary, the catalog zone on the new active primary is completely out of sync with the catalog zone on all the downstream secondary servers. This caused a massive amount of resyncing and general Knot confusion. There didn’t seem to be an easy solution to this problem without ultimately having some singleton source of truth server, which is what we wanted to avoid.&lt;/p&gt;&lt;h2 id=&quot;switching-to-a-single-server-type&quot; tabindex=&quot;-1&quot;&gt;Switching to a single server type&lt;/h2&gt;&lt;p&gt;After this, we decided to dump the whole Knot primary/secondary system and AXFR/IXFR replication, and instead have just a single type of Knot server. This ended up having a number of advantages.&lt;/p&gt;&lt;ol&gt; &lt;li&gt;Only one type of Knot server and one type of Knot server configuration. Less configurations to understand.&lt;/li&gt; &lt;li&gt;No catalog zone. Less concepts to learn about.&lt;/li&gt; &lt;li&gt;No AXFR/IXFR. AXFR/IXFR is harder to reason about and less visible to operators.&lt;/li&gt; &lt;li&gt;No worry about zone serial numbers going backwards if you add -&amp;gt; update -&amp;gt; delete -&amp;gt; re-add a domain which can cause AXFR/IXFR replication weirdness.&lt;/li&gt; &lt;/ol&gt;&lt;p&gt;Doing this definitely felt like a better solution. Additionally it already all “just worked” because we had already built the trigger system that could keep an arbitrary number of primary servers (originally for failover) up-to-date. Now it was just keeping all our Knot instances up to date.&lt;/p&gt;&lt;h2 id=&quot;testing-the-new-system&quot; tabindex=&quot;-1&quot;&gt;Testing the new system&lt;/h2&gt;&lt;p&gt;Since DNS is so critical and we have 100,000’s of domains, we wanted to test as carefully as possible that the new Knot system would generate the same results as the existing PowerDNS system.&lt;/p&gt;&lt;p&gt;The &lt;code&gt;Net::Pcap&lt;/code&gt;, &lt;code&gt;Net::Frame&lt;/code&gt; and &lt;code&gt;Net::DNS&lt;/code&gt; modules in perl made this straight forward. We were able to write a script that captured packets with libpcap, unpacked them into DNS queries and responses, and then replayed them against the new Knot servers to compare the results. This allowed us to see with real world query data that we would get back the same responses.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;my $resolver = Net::DNS::Resolver-&amp;gt;new(nameservers =&amp;gt; [ &#39;...existing DNS ip...&#39; ]);
 ... libpcap setup ...
 my $link_class;
 if ($linktype == Net::Pcap::DLT_EN10MB) {
   $link_class = &#39;Net::Frame::Layer::ETH&#39;;
 } elsif ($linktype == Net::Pcap::DLT_LINUX_SLL) {
   $link_class = &#39;Net::Frame::Layer::SLL&#39;;
 } else {
   die &amp;quot;unknown link layer: $linktype&#92;n&amp;quot;;
 }
 ... run libpcap loop ...
 sub process_packet ($user_data, $header, $packet) {
   my $p_link = $link_class-&amp;gt;new(raw =&amp;gt; $packet);
   $p_link-&amp;gt;unpack;
   my $p_ip4 = Net::Frame::Layer::IPv4-&amp;gt;new(raw =&amp;gt; $p_link-&amp;gt;payload);
   $p_ip4-&amp;gt;unpack;
   my $p_udp = Net::Frame::Layer::UDP-&amp;gt;new(raw =&amp;gt; $p_ip4-&amp;gt;payload);
   $p_udp-&amp;gt;unpack;
   my $p_dns = Net::DNS::Packet-&amp;gt;decode( &#92;$p_udp-&amp;gt;payload );
 
   if (my @a = $p_dns-&amp;gt;answer) {
     my @q = $p_dns-&amp;gt;question;
     my $q = $q[0];
 
     my $dns_q = Net::DNS::Packet-&amp;gt;new();
     $dns_q-&amp;gt;push(question =&amp;gt; $q);
     my $kres = $resolver-&amp;gt;send($dns_q);
     my @ka = $kres-&amp;gt;answer;
 
 ... compare @a (pdns answer) vs @ka (knot answer) modulo some known differences ...
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Mostly it showed that everything was working as expected, though there were a few interesting edge cases that ended up needing to be dealt with.&lt;/p&gt;&lt;h2 id=&quot;non-terminal-nodes-and-wildcards&quot; tabindex=&quot;-1&quot;&gt;Non-terminal nodes and wildcards&lt;/h2&gt;&lt;p&gt;The biggest subtle difference we discovered was around non-terminal nodes and wildcards.&lt;/p&gt;&lt;p&gt;This is subtly documented in the tinydns &lt;a href=&quot;http://cr.yp.to/djbdns/axfr-get.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;axfr-get&lt;/a&gt; program:&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;axfr-get does not precisely simulate BIND’s handling of &lt;code&gt;*.dom&lt;/code&gt;. Under BIND, records for &lt;code&gt;*.dom&lt;/code&gt; do not apply to &lt;code&gt;y.dom&lt;/code&gt; or &lt;code&gt;anything.y.dom&lt;/code&gt; if there is a normal record for &lt;code&gt;x.y.dom&lt;/code&gt;. With axfr-get and tinydns, the records apply to &lt;code&gt;y.dom&lt;/code&gt; and &lt;code&gt;anything.y.dom&lt;/code&gt; except &lt;code&gt;x.y.dom&lt;/code&gt;.&lt;/p&gt; &lt;/blockquote&gt;&lt;p&gt;Knot DNS follows the traditional BIND and &lt;a href=&quot;https://datatracker.ietf.org/doc/html/rfc1034#section-4.3.3&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;RFC 1034&lt;/a&gt; intepretation of wildcard records. Our PowerDNS backend was built to follow the TinyDNS model because that’s what we were migrating from at the time. This can cause subtle differences in DNS results if you have any wildcard domains.&lt;/p&gt;&lt;p&gt;An example. If you have a domain &lt;code&gt;example.com&lt;/code&gt; setup at Fastmail with our standard DNS configuration, then we add default A records for &lt;code&gt;*.example.com&lt;/code&gt;. If you add a DNS record for the subdomain &lt;code&gt;*.foo&lt;/code&gt; that creates a record for &lt;code&gt;*.foo.example.com&lt;/code&gt;. However that implied existence of &lt;code&gt;foo.example.com&lt;/code&gt; means that the &lt;code&gt;*.example.com&lt;/code&gt; record no longer exists for &lt;code&gt;foo.example.com&lt;/code&gt;. At least, that’s true in the Knot/Bind DNS implementation, but not true in our PowerDNS backend implementation, so this will get you subtly different results.&lt;/p&gt;&lt;p&gt;In quite a few cases this won’t be a problem. We see queries like &lt;code&gt;_domainkey.example.com A&lt;/code&gt;, which under PowerDNS return an IP, but won’t under knot, but that’s fine. Nothing should really be using that IP anyway, it’s probably just some gateway device somewhere doing DNS querying of passively seen domains in email headers or the like. But in some cases it might be.&lt;/p&gt;&lt;p&gt;We initially thought we could fix this automatically. For everyone that has a &lt;code&gt;foo.bar.example.com&lt;/code&gt; subdomain without a &lt;code&gt;bar.example.com&lt;/code&gt; subdomain, if there’s any wildcard &lt;code&gt;*.example.com&lt;/code&gt; records, we make copies of them at &lt;code&gt;bar.example.com&lt;/code&gt;. This should make everything “just work”.&lt;/p&gt;&lt;p&gt;The problem is, there’s actually a deeper problem that affects basically every domain and every single subdomain in a subtle way. As noted, by default we publish &lt;code&gt;*.example.com&lt;/code&gt; records. However these effectively work for all sub-sub domains as well. For example I have a domain &lt;code&gt;uberengineer.com&lt;/code&gt; and the standard &lt;code&gt;*.uberengineer.com&lt;/code&gt; wildcard means that all sub-domains, sub-sub-domains, etc resolve:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;# dig +short that.uberengineer.com
 103.168.172.37
 103.168.172.52
 # dig +short this.that.uberengineer.com
 103.168.172.37
 103.168.172.52
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Now I have a TXT record at &lt;code&gt;test.uberengineer.com&lt;/code&gt;, so that hides the A records&lt;/p&gt;&lt;pre&gt;&lt;code&gt;# dig +short test.uberengineer.com
 #
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;But in our PowerDNS implementation, that doesn’t hide sub-domains of &lt;code&gt;test.uberengineer.com&lt;/code&gt; from the original wildcard.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;# dig +short foo.test.uberengineer.com
 103.168.172.37
 103.168.172.52
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;But with Knot, it does:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;# dig +short foo.test.uberengineer.com @knottest.internal
 #
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;So basically to just fix users automatically, for every single subdomain X they have configured, if there’s a wildcard at a lower level, we’d have to explicitly create a copy of all the lower level wildcard records at &lt;code&gt;*.X&lt;/code&gt; as well. This was going to be just way too much magic, especially for a rare edge case that it’s possible no one was even relying on anyway!&lt;/p&gt;&lt;p&gt;In the end, we analysed the DNS queries we were seeing and also all the email deliveries we saw. We can looked at the email logs on our MX servers for any RCPT TO address, and then checked if the domain that was delivered to is hosted by us, and compared if it would resolve under Knot as well. Combining the data convinced us that no one was going to be actively affected by this change.&lt;/p&gt;&lt;h2 id=&quot;conclusion&quot; tabindex=&quot;-1&quot;&gt;Conclusion&lt;/h2&gt;&lt;p&gt;This has now been running for over a year in production and has been working extremely well. We were able to remove two existing systems and replace them with a single consistent Knot DNS based system for all our Fastmail domains and 100,000’s of user domains. The triggers that track zones that need rebuilding work reliably and consistently. The new system performs enormously better than the existing PowerDNS pipe backend based system. We can easily scale it to add additional servers if needed.&lt;/p&gt;&lt;p&gt;This all fits with a mantra we’ve been working with recently, “fewer better ways”. We’ve been running an email service for over 25 years and it’s easy to accumulate a plethora of different services and systems with varying levels of polish and performance. Revisiting what you’re doing and running to try and consolidate to a smaller number of systems working in better ways can reduce long term debt. This makes systems more reliable, easier to manage, and also allows new staff to understand them more quickly as well.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 12: Following the Sun</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/following-the-sun/' />
 			<id>https://www.fastmail.com/blog/following-the-sun/</id>
 			<updated>2024-12-12T00:00:01Z</updated><author>
 				<name>Andria DeFulio</name>
 			</author><author>
 				<name>Luke Erlacher</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the twelfth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/meet-the-team-marc/&quot;&gt;Dec 11: Meet the team—Marc&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/moving-fastmail-dns-to-knot/&quot;&gt;Dec 13: It’s knot DNS. There’s no way it’s DNS. It is DNS!&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;As Fastmail provides a global service for end-users, we operate 24/7 and respond to issues around the clock. Given that humans do not operate around the clock, this requires some thoughtful process to effectively maintain our service.&lt;/p&gt;&lt;p&gt;To do this, our support team is split across three timezones, and our on-call engineers are split across two timezones.&lt;/p&gt;&lt;p&gt;As a term of art, this is often referred to as “Follow the Sun support”: Wherever the sun is shining right now, agents and support engineers are working and ready there, resulting in 24/7 coverage.&lt;/p&gt;&lt;h2 id=&quot;support&quot; tabindex=&quot;-1&quot;&gt;Support&lt;/h2&gt;&lt;p&gt;Our support staff is located in both of our offices in Melbourne and Philadelphia, and remotely in India. Having staff located around the world works really well for us and for our customers, who are also spread across the globe!&lt;/p&gt;&lt;p&gt;Having staff spread across multiple time zones allows us to provide support to you, our customer, when you need it. We aim to respond to routine questions within a few hours. We usually do much better than that, and respond within an hour!&lt;/p&gt;&lt;p&gt;If we were all located in a single office, not only would our customers potentially be left waiting for a full day to get a response to a ticket, but our support team would start each morning with a daunting backlog of tickets. It’d be much trickier to surface customers’ most urgent tickets, and the natural response to a backlog might be to rush through tickets. Instead, we have opted out of that unnecessary stress and can give each user the time it takes to fully research their issue and then send them a thorough, thoughtful response.&lt;/p&gt;&lt;p&gt;Having a team across multiple locations does present challenges, too! Particularly with collaboration. To mitigate that, the support team has multiple brief huddles, with each location having some hours of overlap within their schedule. At the end of our day, we hand the baton off to those who are just starting their day, letting them know if there is anything impacting multiple customers, like &lt;a href=&quot;/blog/moving-house-new-datacentre/&quot;&gt;moving to a new data center&lt;/a&gt; or &lt;a href=&quot;/blog/sunsetting-pobox/&quot;&gt;migrating Pobox users&lt;/a&gt;. We also use these huddles to help each other with particularly challenging tickets, to make sure the team is aware of work on the horizon, and to share newly acquired knowledge across the team.&lt;/p&gt;&lt;p&gt;Outside of our daily huddles, we communicate with everyone on the Fastmail team synchronously in Slack and asynchronously via our sister product, &lt;a href=&quot;https://www.topicbox.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Topicbox&lt;/a&gt;.&lt;/p&gt;&lt;h2 id=&quot;on-call-engineers&quot; tabindex=&quot;-1&quot;&gt;On-Call Engineers&lt;/h2&gt;&lt;p&gt;At Fastmail, we own and operate most of our stack, from networking gear and servers in the datacenter space we use, server OS setup, internal services, backups, and redundancy, through to user-facing services and mail flow.&lt;/p&gt;&lt;p&gt;To maintain the availability of all these 24/7, our platform team operates an on-call process split between our US and Australian teams.&lt;/p&gt;&lt;p&gt;With the Australian team working to Australia East timezone, and the US team working to Eastern Standard timezone, we have set this up for the Australian team to be on call from 8AM to 10PM their time, and the US team from 6AM to 4PM their time.&lt;/p&gt;&lt;p&gt;So the times are a little less friendly for the US team, but in return, they have less time to cover overall.&lt;/p&gt;&lt;p&gt;As well as being the primary on-call engineers, the platform team also owns our observability stack and is empowered to make changes to all parts of our stack to improve monitoring and alerting. Nothing is worse than being responsible for something you can’t fix, so we make sure to avoid that!&lt;/p&gt;&lt;p&gt;Another practice we follow to reduce on-call load is to help the support team troubleshoot and fix issues without needing to escalate to engineers. Our support team can search logs and run admin and fixup tools to resolve problems themselves.&lt;/p&gt;&lt;p&gt;One of Fastmail’s operational maxims is “the spice must flow”. So, the first priority during incidents is to restore service—the mail has to flow, and users need to be able to access their mailboxes and use all the other features we provide. Our on-call engineers are experts in most, but not all, systems and processes at Fastmail, so sometimes this involves paging experts or technical owners to help troubleshoot.&lt;/p&gt;&lt;p&gt;Once service is restored, everything else has less urgency, and incident reports and follow-ups are coordinated via - what else - emails on our internal mailing lists. We make sure that we understand what happened, fix bugs, and make the system more robust for the future.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 11: Meet the team—Marc</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/meet-the-team-marc/' />
 			<id>https://www.fastmail.com/blog/meet-the-team-marc/</id>
 			<updated>2024-12-11T00:00:01Z</updated><author>
 				<name>The Fastmail Team</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the eleventh post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/sunsetting-pobox/&quot;&gt;Dec 10: Sunsetting Pobox&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/following-the-sun/&quot;&gt;Dec 12: Following the Sun&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Meet Marc, our Head of the new Trust, Abuse, and Deliverability team — and also source of bad puns. (Oy, good puns!)&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Name:&lt;/strong&gt; Marc Bradshaw&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Role:&lt;/strong&gt; Head of Trust, Abuse, and Deliverability&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What do you work on?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;The Trust, Abuse, and Deliverability (TAD) team is responsible for mail flow, and anti-abuse. It is our task to make sure the good mail gets delivered and the bad mail does not. This is a fairly new team here at Fastmail, with the responsibility having previously been split between the backend and platform teams.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;How long working at Fastmail, how did you get involved?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Too bloody long! 10 years now. I started in 2014. Rob Norris called me and asked if I was still looking for a job. I said “no, but keep talking”. I was another refugee from Monash University - we had worked there together. I had started somewhere else, but that was not a good fit.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What’s a project you have worked on that you’re particularly proud of?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;The answer to this is quite often “the next one”, we are rewriting our inbound mail delivery software, and have lots of great ideas which should allow some great new features in the future.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What are your favourite Fastmail features?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;I love the memos feature. Being able to put a note on an email so you can remember something about it. Also, labels! Labels make organising email easy.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Other than Fastmail, what’s your favourite or most used piece of technology?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;I have a supernote nomad, which is an e-ink tablet which I use to take and organise notes, and as an e-reader. Being e-ink allows me to write notes in a more natural way, but still be able to organise them digitally, all without the distractions I would get from a more fully featured tablet.&lt;/p&gt;&lt;p&gt;&lt;a href=&quot;https://nothingbutstatic.dev/tech/2024-07-13-the-digital-analog-2/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;I blogged about it&lt;/a&gt; back in April.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What are you listening to these days?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;My musical taste is quite varied, my recently played list on the music app ranges from 80s synth pop, to Madonna, Linkin Park, SOPHIE, Veruca Salt, and Chappell Roan&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What are you watching these days?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;I have just finished watching Fallout, I can’t believe we need to wait until 2026 for the next season.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What do you like to do outside of work?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;I like to ride my bike and try to keep fit, with varying levels of success.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What’s your favourite animal?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Hmm. Just got a new puppy. He’s pretty cute, when he’s not being naughty and getting stuck under the house!&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Any Fastmail staff you want to brag on?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;I mean, they’re all pretty good. Andrew has moved from the Platform team into Development and is quickly getting up to speed on all things backend and mail flow.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What do you like best about working at Fastmail?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;I love that we are a small company making a big difference in our field.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 10: Sunsetting Pobox</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/sunsetting-pobox/' />
 			<id>https://www.fastmail.com/blog/sunsetting-pobox/</id>
 			<updated>2024-12-10T00:00:01Z</updated><author>
 				<name>Ricardo Signes</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the tenth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/building-a-blog/&quot;&gt;Dec 9: Building a blog&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/meet-the-team-marc/&quot;&gt;Dec 11: Meet the team—Marc&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;On November 12th of this year, our service Pobox was finally merged into Fastmail. This had been inevitable since &lt;a href=&quot;/blog/exciting-news-about-pobox-and-fastmail/&quot;&gt;Fastmail’s acquisition of Pobox&lt;/a&gt; in 2015, but we’d put it off over and over. In the end, it took a lot more time than we expected, but it paid off. The cutover went well… but it was still just a little bittersweet, ending a long era. Pobox was around for thirty years. In Internet years, that’s an eternity. Seeing something so long-lasting cease to exist as itself is a bit, well, &lt;em&gt;confronting&lt;/em&gt;.&lt;/p&gt;&lt;p&gt;It hit home for me, personally: I’d been working on Pobox for nearly 20 years, and the shutdown of Pobox meant turning off hundreds of thousands of lines of code I wrote or maintained. A professional programmer does well to avoid identifying too much with their work product, but on some level it’s hard to avoid. Still, it was better to do it myself. I had a good knowledge of both systems and felt like I was best placed to make things as seamless as possible.&lt;/p&gt;&lt;p&gt;So, how’d it all go? I’ll walk you through it.&lt;/p&gt;&lt;h2 id=&quot;what-even-was-pobox&quot; tabindex=&quot;-1&quot;&gt;What even was Pobox?&lt;/h2&gt;&lt;p&gt;Pobox was, at its heart, an email forwarding service. It was started by Meng Wong and Helen Horstmann-Allen in 1995, while they were both still in college. College students in the 90s had email addresses, but they knew they wouldn’t last. Eventually you’d leave school and you’d need to tell everybody your new address. The same thing went for people who had email at work. Pobox was the answer to this problem: you’d get an address with Pobox and set it up to forward your mail to whatever your current “real” address was. Unlike Fastmail, Pobox didn’t include mail storage by default. You could get it, but it was an add-on.&lt;/p&gt;&lt;p&gt;Broadly speaking, Pobox and Fastmail were very similar. Users would have one or more email addresses, and those addresses would deliver to an inbox or to another address. Spam got filtered out, and users could write their own mail filters. One “group” could have a bunch of users on it. Users could send mail through relay servers.&lt;/p&gt;&lt;h2 id=&quot;so-what-was-the-problem&quot; tabindex=&quot;-1&quot;&gt;So what was the problem?&lt;/h2&gt;&lt;p&gt;Yes, broadly speaking, Pobox and Fastmail &lt;em&gt;were&lt;/em&gt; very similar. In the fine details, though, everything was just kind of different in tedious little ways. Email aliases were per-group in Fastmail, per-user in Pobox. Spam filtering exceptions worked differently. The rules on sharing domains were different. In short, no feature &lt;em&gt;really&lt;/em&gt; worked the same in both services, and every exception had an exception.&lt;/p&gt;&lt;p&gt;Obviously, maintaining many of the same features (but differently) in two places took a lot more time and effort than just maintaining one version. The lion’s share of development always went to Fastmail while the number of staff with expertise in Pobox dwindled. We kept training new support staff on Pobox, but new programmers almost never touched it. Pobox had emergency support staff, but no full time developers, and it showed, with nearly no changes of any kind after 2015. That didn’t just mean they weren’t getting cool new features. It meant they weren’t getting as many deliverability improvements or updates to service monitoring.&lt;/p&gt;&lt;p&gt;In order to keep Pobox users happy, online, and connected, we were going to have to turn them into Fastmail users. We wanted to make this as seamless as possible, which was going to be real work. Remember: everything was different enough to make “just import them to Fastmail” not work. Still, the goal was that cutover would happen without significant downtime, and without users needing to reconfigure their mail clients, change passwords, or (worst of all) fiddle with their DNS.&lt;/p&gt;&lt;p&gt;This was going to be a complicated piece of work, and risky, and was going to require significant knowledge of both Pobox and Fastmail internals. Also, it just wasn’t going to be exciting work day to day. The final payoff would be great, but otherwise it would involve writing a lot of fiddly and risky code that was just going to get deleted later. So we put it off a long time, even though we’d known since 2015 that we’d have to do it eventually. In 2023, we finally started in earnest.&lt;/p&gt;&lt;h2 id=&quot;the-upgrade-process&quot; tabindex=&quot;-1&quot;&gt;The Upgrade Process&lt;/h2&gt;&lt;p&gt;Back when Fastmail acquired Pobox, the first thing we changed was the Pobox mail storage. Both Fastmail and Pobox used &lt;a href=&quot;https://github.com/cyrusimap/cyrus-imapd&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Cyrus IMAP&lt;/a&gt;, but Fastmail ran it at larger scale and with better tooling. Very early on, we created a secret Fastmail user for each Pobox Mailstore account and moved mail out of Pobox and into Fastmail. This also let us replace Pobox’s webmail with Fastmail’s best-in-class UI.&lt;/p&gt;&lt;p&gt;It didn’t replace almost anything else, though: IMAP and SMTP connections went through Pobox. Pobox handled the mail forwarding. Settings and billing were all in Pobox. That meant that those “secret Fastmail users” had all kinds of settings that nothing used. The Pobox Upgrade Project (known internally as PUP) used these as the starting point: they would be configured to act almost exactly like their Pobox owners, and then they’d take over.&lt;/p&gt;&lt;p&gt;First, we created these Fastmail users for all the non-Mailstore Pobox accounts. This was pretty painless. Next, we merged these users into customers to match the billing setup in Pobox. That’s where the pain began. Fastmail supported merging multiple users into one customer (to get just one bill)… but not when both of those users are free and have no expiration date. We had to get into the billing system and add ways to override the safety mechanisms keeping us from breaking the rules that we so desperately needed to break. I won’t provide a litany of every such complication, but: there were plenty. Each ported feature involved one or two big mismatches, one or two edge cases, and one or two places where everything would work just fine, if you just bypassed all the business logic briefly.&lt;/p&gt;&lt;p&gt;For months, the process moved forward feature by feature. At Pobox, the “migration planner” kept learning how to describe more and more of each Pobox account’s configuration in terms of Fastmail. The planner wrote out new plans constantly, and those were shipped from Pobox to Fastmail. On the Fastmail side, the “migration executor” would take those plans and reconfigure things on the matching Fastmail users. Every week, the Fastmail users were configured more and more like their counterpart Pobox users. The big question was: when could we cut over?&lt;/p&gt;&lt;h2 id=&quot;cutover&quot; tabindex=&quot;-1&quot;&gt;Cutover&lt;/h2&gt;&lt;p&gt;When making a big change, we generally like a staged rollout. First, we apply it to some of our internal test customers. Then we apply it to ourselves. Then one percent of users, then five… you get the idea. We’re looking to reduce the damage that a bug can cause. The Pobox cutover was exactly the kind of big (huge!) change where we’d like roll things out in stages.&lt;/p&gt;&lt;p&gt;Unfortunately, it wasn’t going to be that simple. Email is routed by domain. You can’t really send one user’s mail through one system and another user’s mail through another mail system. Or, you can, but it requires building a &lt;em&gt;third&lt;/em&gt; system that sits in front of both, which is just another thing that can go wrong!&lt;/p&gt;&lt;p&gt;We decided that all users would get cut over at once. To mitigate that big change, we further decided that we’d cut over different components over time, as much as possibly invisibly to the user, like this:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;First, anybody connecting to IMAP or POP was connecting directly to Fastmail, not Pobox.&lt;/li&gt; &lt;li&gt;Later, DNS for our domains (and user domains) was moved to Fastmail.&lt;/li&gt; &lt;li&gt;Later still, anybody sending mail through Pobox was actually connecting directly to Fastmail, not Pobox.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;Each of these cutovers was a big change that we’d been able to test well in advance, and each one turned up one or two weird edge cases that took a little time to fix. Doing them one at a time let us focus and get things right. As far as I know, nobody reported noticing that any service had moved to Fastmail.&lt;/p&gt;&lt;p&gt;Finally, the big cutover day was set for November 12th. That’s when we’d have all mail destined for Pobox customers start hitting Fastmail servers, and when we’d make &lt;a href=&quot;https://pobox.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;pobox.com&lt;/a&gt; start taking people to Fastmail. We had a big checklist, and the plan was that we’d start our day at 8:00, run a series of programs, watch some logs, and be done around lunchtime. That was the prediction if everything went &lt;em&gt;perfectly&lt;/em&gt;. In reality, we weren’t done until about three in the afternoon, and we did spend a decent amount of time responding to unexpected cases. Mostly, though, it went smoothly and we were done by the end of the day.&lt;/p&gt;&lt;p&gt;The goal had been to avoid a painful reconfiguration or transition process for users, and I think we did pretty well on that front! I’m proud of that. Pobox is how I got involved in working on email, and it’s been one of the major things I’ve worked on for twenty years. Working on Pobox made me a better programmer and helped me understand what it means to be a good internet citizen and a good steward of customers’ data. Astoundingly, most Pobox customers have been with Pobox &lt;em&gt;at least&lt;/em&gt; as long as I have. I’m pleased we could bring them smoothly into Fastmail, where we can keep trying to deliver the best email service around.&lt;/p&gt;&lt;p&gt;Everybody at Fastmail contributed to the success of the project, but I think I should especially shout out two teams. Team PUP (meaning Mark Jason Dominus and Matthew Horsfall) worked on the project full time, dealing with weird legacy systems, obnoxious edge cases, and plenty of complexity arising from wiring together two incompatible systems. It was a bit of a slog, and they did great. Also, our support team (Team SUP!) were invaluable in this project. They wrote the &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/9822848635919-Guide-to-Fastmail-for-Pobox-users&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Guide to Fastmail for Pobox users&lt;/a&gt;, they provided feedback on what decisions would help make things easiest on users (and on them!), and they took care of our users whenever things weren’t &lt;em&gt;quite&lt;/em&gt; as seamless as we’d hoped. One of the most common concerns I heard from Pobox customers was “the support won’t be as good.” The great news is that it’s just one support team that’s been supporting both products, so the same great support is going to be there for you.&lt;/p&gt;&lt;p&gt;Thanks for a few good decades. I’m looking forward to what comes next!&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 9: Building a blog</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/building-a-blog/' />
 			<id>https://www.fastmail.com/blog/building-a-blog/</id>
 			<updated>2024-12-09T02:00:01Z</updated><author>
 				<name>Callum Skeet</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the ninth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/principles/&quot;&gt;Dec 8: Guiding principles&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/sunsetting-pobox/&quot;&gt;Dec 10: Sunsetting Pobox&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Today we’ll look at how we went about creating our new blog and marketing site, why we chose the tools we did, and how we resolved issues we bumped into along the way.&lt;/p&gt;&lt;h2 id=&quot;moving-away-from-word-press&quot; tabindex=&quot;-1&quot;&gt;Moving away from WordPress&lt;/h2&gt;&lt;blockquote&gt; &lt;p&gt;N.B. Eleventy now &lt;a href=&quot;https://github.com/11ty/eleventy-import&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;provides a tool&lt;/a&gt; to assist with migrating content from WordPress&lt;/p&gt; &lt;/blockquote&gt;&lt;p&gt;Our last site was powered by WordPress and, despite its age, it also powers a &lt;a href=&quot;https://w3techs.com/technologies/details/cm-wordpress&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;large proportion&lt;/a&gt; of the websites out there. Just because something’s popular doesn’t mean it’s the right tool for us. The added costs of running a WordPress instance aren’t insignificant, and making large, sweeping changes to any aspect of the site required a level of WordPress knowledge that few people wanted to learn (or admit they knew). So, what we wanted was something simple, that anyone can contribute to, and considerably lowered our running costs.&lt;/p&gt;&lt;h2 id=&quot;meet-eleventy&quot; tabindex=&quot;-1&quot;&gt;Meet Eleventy&lt;/h2&gt;&lt;p&gt;&lt;a href=&quot;https://www.11ty.dev/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Eleventy (11ty)&lt;/a&gt; is a Static Site Generator (SSG). It’s also a tool that fit our needs perfectly. It was written with Node (1 fewer language to know), allowed us to write content in markdown, and spat out HTML which can be hosted anywhere. It’s also really fast. Adding extra functionality to it is also dead simple.&lt;/p&gt;&lt;p&gt;For example, we can easily add a new font family like so:&lt;/p&gt;&lt;pre class=&quot;language-jsx&quot;&gt;&lt;code class=&quot;language-jsx&quot;&gt;&lt;span class=&quot;comment token&quot;&gt;// -----------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// source/_data/webfonts.js&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// -----------------&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; config &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// ...&lt;/span&gt;
     &lt;span class=&quot;literal-property property token&quot;&gt;roca&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;weight&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;constant token&quot;&gt;THIN&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;th&#39;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;weight&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;constant token&quot;&gt;LIGHT&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;lt&#39;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;weight&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;constant token&quot;&gt;REGULAR&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;rg&#39;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;function&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;getRocaVariant&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;&lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt; weight&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; name &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         &lt;span class=&quot;property string-property token&quot;&gt;&#39;@font-face&#39;&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             &lt;span class=&quot;property string-property token&quot;&gt;&#39;font-family&#39;&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;roca&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
             &lt;span class=&quot;property string-property token&quot;&gt;&#39;src&#39;&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;template-string token&quot;&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;url(/assets/fonts/roca/rocaone-&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;name&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;-webfont.woff2) format(&#39;woff2&#39;)&lt;/span&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
             &lt;span class=&quot;property string-property token&quot;&gt;&#39;font-weight&#39;&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; weight&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
             &lt;span class=&quot;property string-property token&quot;&gt;&#39;font-style&#39;&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;normal&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
             &lt;span class=&quot;property string-property token&quot;&gt;&#39;font-display&#39;&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;swap&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;export&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;default&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;roca&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; config&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;roca&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;map&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;getRocaVariant&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// -----------------------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// source/_includes/layouts/base.liquid&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// -----------------------------&lt;/span&gt;
 
 &lt;span class=&quot;tag token&quot;&gt;&lt;span class=&quot;tag token&quot;&gt;&lt;span class=&quot;punctuation token&quot;&gt;&amp;lt;&lt;/span&gt;style&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;plain-text token&quot;&gt;
 {%- toCSS from: webfonts.roca -%}
 &lt;/span&gt;&lt;span class=&quot;tag token&quot;&gt;&lt;span class=&quot;tag token&quot;&gt;&lt;span class=&quot;punctuation token&quot;&gt;&amp;lt;/&lt;/span&gt;style&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;&gt;&lt;/span&gt;&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// ------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// .eleventy.js&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// ------------&lt;/span&gt;
 
 eleventyConfig&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;addPassthroughCopy&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;source/assets/fonts&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 eleventyConfig&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;addLiquidTag&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;toCSS&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;function&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;comment token&quot;&gt;/* liquidEngine */&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         &lt;span class=&quot;function token&quot;&gt;parse&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;tagToken&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;this&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;args &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;new&lt;/span&gt; &lt;span class=&quot;class-name token&quot;&gt;Hash&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;tagToken&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;args&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
         &lt;span class=&quot;operator token&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;render&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;context&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; emitter&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt; from &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;yield&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;this&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;args&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;render&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;context&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
             &lt;span class=&quot;comment token&quot;&gt;// Always a good idea to cache processed CSS/JS where you can.&lt;/span&gt;
             &lt;span class=&quot;comment token&quot;&gt;// This can speed your builds up significantly.&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;cssCache&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;has&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;from&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
                 emitter&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;write&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;cssCache&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;get&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;from&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
                 &lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
             &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;let&lt;/span&gt; css &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; ruleset &lt;span class=&quot;keyword token&quot;&gt;of&lt;/span&gt; from&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
                 &lt;span class=&quot;keyword token&quot;&gt;let&lt;/span&gt; stringifiedDecl &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
                 &lt;span class=&quot;keyword token&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;selector&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; declarations&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;of&lt;/span&gt; Object&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;entries&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;
                     ruleset&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
                 &lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
                     &lt;span class=&quot;keyword token&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;prop&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; value&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;of&lt;/span&gt; Object&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;entries&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;
                         declarations&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
                     &lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
                         stringifiedDecl &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;template-string token&quot;&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;stringifiedDecl&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;prop&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;value&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;;&lt;/span&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
                     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
                     css &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;template-string token&quot;&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;css&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;selector&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;stringifiedDecl&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
                 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
             &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
             &lt;span class=&quot;comment token&quot;&gt;// Process your CSS however you like&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;css&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; processedCss &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;yield&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;processCss&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;css&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
             cssCache&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;set&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;from&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; processedCss&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
             emitter&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;write&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;processedCss&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;There, not much code and now adding a font to the site is super easy!&lt;/p&gt;&lt;p&gt;Eleventy doesn’t make you do everything though, they provide plugins for &lt;a href=&quot;https://www.11ty.dev/docs/plugins/image/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;optimising images&lt;/a&gt;, &lt;a href=&quot;https://www.11ty.dev/docs/plugins/bundle/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;per-page CSS/JS/HTML bundling&lt;/a&gt;, &lt;a href=&quot;https://www.11ty.dev/docs/plugins/syntaxhighlight/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;syntax highlighting&lt;/a&gt; and &lt;a href=&quot;https://www.11ty.dev/docs/plugins/official/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;many, many more&lt;/a&gt;. There’s also a very &lt;a href=&quot;https://11tybundle.dev/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;active community&lt;/a&gt; out there full of tips, tricks and guides that’ll help you get the most out of the software.&lt;/p&gt;&lt;h2 id=&quot;leveraging-our-design-system&quot; tabindex=&quot;-1&quot;&gt;Leveraging our design system&lt;/h2&gt;&lt;p&gt;The design system we use in the Fastmail client is called Elemental. It’s well suited to displaying content in an information-dense environment without overwhelming users. While in the exploratory phase of rebuilding the site, we quickly discovered our existing spacing and typography didn’t fit our vision for the new site. Everything was too compact! We wanted a new set of &lt;a href=&quot;https://thedesignsystem.guide/design-tokens&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;design tokens&lt;/a&gt; that adapted well to any screen size and reduced the burden on a designer to create different layouts for different viewports.&lt;/p&gt;&lt;p&gt;&lt;a href=&quot;https://utopia.fyi/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Utopia&lt;/a&gt; met these needs precisely for us and provided a great framework for thinking about new layouts. In hindsight, I think we could have gotten away with a much simpler set of tokens, but I won’t deny the confidence it gave us in moving forward on this project.&lt;/p&gt;&lt;h2 id=&quot;partial-site-building&quot; tabindex=&quot;-1&quot;&gt;Partial site building&lt;/h2&gt;&lt;p&gt;We’ve got quite a few images on our site, mainly due to the number of blog posts we have—&lt;a href=&quot;https://www.fastmail.com/blog/imported-older-news-10/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;it goes all the way back to 2001&lt;/a&gt;! An unfortunate side-effect of this is that it significantly increases how long a build takes simply due to the sheer number of images we need to process and generate.&lt;/p&gt;&lt;p&gt;For draft posts, we don’t need to build every post on the blog. In fact, we only need the post (or posts) the author is working on. To achieve this, we can use permalink objects to tell Eleventy that it shouldn’t render the content of a particular page.&lt;/p&gt;&lt;pre class=&quot;language-jsx&quot;&gt;&lt;code class=&quot;language-jsx&quot;&gt;&lt;span class=&quot;comment token&quot;&gt;// -------------------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// source/_data/NO_RENDER.js&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// -------------------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// If we set the value of a page&#39;s permalink to this object, then the content&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// won&#39;t be rendered during build. Under the hood, Eleventy is checking if the&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// object has a `build` property to decide if it should render the page during&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// static generation.&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;export&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;default&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;norender&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;&#39;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// --------------------------------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// source/content/posts/+data.11tydata.js&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// --------------------------------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// Now, we can do something like this in a directory data file:&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;function&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;shouldRenderPost&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;comment token&quot;&gt;// Posts to render&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt; data&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;env&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;postsToRender
         &lt;span class=&quot;operator token&quot;&gt;?&lt;/span&gt; data&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;env&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;postsToRender&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;some&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;
               &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;id&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt;
                   id &lt;span class=&quot;operator token&quot;&gt;===&lt;/span&gt; data&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;id &lt;span class=&quot;operator token&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;&#39;&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;||&lt;/span&gt;
                   id &lt;span class=&quot;operator token&quot;&gt;===&lt;/span&gt; data&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;permalink &lt;span class=&quot;operator token&quot;&gt;||&lt;/span&gt;
                   &lt;span class=&quot;template-string token&quot;&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;/blog/&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;id&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;/&lt;/span&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;===&lt;/span&gt; data&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;permalink &lt;span class=&quot;operator token&quot;&gt;||&lt;/span&gt;
                   path&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;normalize&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;id&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;===&lt;/span&gt; path&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;normalize&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;data&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;page&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;inputPath&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
           &lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
         &lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;export&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;default&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;literal-property property token&quot;&gt;layout&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;layouts/post&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
     &lt;span class=&quot;literal-property property token&quot;&gt;tags&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;posts&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
     &lt;span class=&quot;literal-property property token&quot;&gt;eleventyComputed&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         &lt;span class=&quot;function function-variable token&quot;&gt;permalink&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;function&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;shouldRenderPost&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;data&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
                 &lt;span class=&quot;operator token&quot;&gt;?&lt;/span&gt; data&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;permalink
                 &lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; data&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;NO_RENDER&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;You can get the list of posts that have changed with a few git commands. Some of this will depend on your build environment, the following code is applicable to Cloudflare Pages:&lt;/p&gt;&lt;pre class=&quot;language-bash&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;&lt;span class=&quot;important shebang token&quot;&gt;#!/usr/bin/env bash&lt;/span&gt;
 
 &lt;span class=&quot;builtin class-name token&quot;&gt;set&lt;/span&gt; &lt;span class=&quot;parameter token variable&quot;&gt;-euxo&lt;/span&gt; pipefail
 
 &lt;span class=&quot;function function-name token&quot;&gt;filter&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;builtin class-name token&quot;&gt;read&lt;/span&gt; line&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;do&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;for-or-select token variable&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$line&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;do&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$1&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&quot;&lt;span class=&quot;token variable&quot;&gt;$x&lt;/span&gt;&quot;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;then&lt;/span&gt;
                 &lt;span class=&quot;builtin class-name token&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&quot;&lt;span class=&quot;token variable&quot;&gt;$x&lt;/span&gt;&quot;&lt;/span&gt;
             &lt;span class=&quot;keyword token&quot;&gt;fi&lt;/span&gt;
         &lt;span class=&quot;keyword token&quot;&gt;done&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;done&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
 &lt;span class=&quot;function function-name token&quot;&gt;isBlogContent&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;builtin class-name token&quot;&gt;local&lt;/span&gt; &lt;span class=&quot;assign-left token variable&quot;&gt;EXT&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token variable&quot;&gt;${1&lt;span class=&quot;operator token&quot;&gt;##&lt;/span&gt;*.}&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$1&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;==&lt;/span&gt; source/content/posts/* &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$EXT&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;==&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;md&#39;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt; &lt;span class=&quot;parameter token variable&quot;&gt;-n&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&quot;&lt;span class=&quot;token variable&quot;&gt;${CF_PAGES_BRANCH+x}&lt;/span&gt;&quot;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt; &lt;span class=&quot;token variable&quot;&gt;$CF_PAGES_BRANCH&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;==&lt;/span&gt; draft* &lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;then&lt;/span&gt;
     &lt;span class=&quot;function token&quot;&gt;git&lt;/span&gt; remote set-branches origin &lt;span class=&quot;string token&quot;&gt;&#39;*&#39;&lt;/span&gt;
     &lt;span class=&quot;function token&quot;&gt;git&lt;/span&gt; fetch &lt;span class=&quot;parameter token variable&quot;&gt;--depth&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;number token&quot;&gt;100&lt;/span&gt; origin production &lt;span class=&quot;token variable&quot;&gt;$CF_PAGES_BRANCH&lt;/span&gt;
     &lt;span class=&quot;assign-left token variable&quot;&gt;CHANGES&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token variable&quot;&gt;&lt;span class=&quot;token variable&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;git&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;diff&lt;/span&gt; --name-only origin/production&lt;span class=&quot;punctuation token&quot;&gt;..&lt;/span&gt;.HEAD &lt;span class=&quot;operator token&quot;&gt;|&lt;/span&gt; filter isBlogContent &lt;span class=&quot;operator token&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;tr&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;&#92;n&#39;&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39; &#39;&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;function token&quot;&gt;xargs&lt;/span&gt;&lt;span class=&quot;token variable&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
     &lt;span class=&quot;assign-left token variable&quot;&gt;POST_IDS&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;token variable&quot;&gt;$CHANGES&lt;/span&gt; npx eleventy
 &lt;span class=&quot;keyword token&quot;&gt;else&lt;/span&gt;
     npx eleventy
     &lt;span class=&quot;function token&quot;&gt;npm&lt;/span&gt; run pagefind:index
 &lt;span class=&quot;keyword token&quot;&gt;fi&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This significantly sped up build times on draft branches, going from ~5 minutes to ~1:30 per build.&lt;/p&gt;&lt;h2 id=&quot;creating-a-workflow-for-developers&quot; tabindex=&quot;-1&quot;&gt;Creating a workflow for developers&lt;/h2&gt;&lt;p&gt;We want people to be able to publish blog posts by simply merging a PR. But if anyone can contribute, how will they know what should go in each field of the front matter? What tags can they use? How do they find their author ID? Sure, you can write some documentation, but wouldn’t it be better to write a few simple scripts that help us enforce content rules?&lt;/p&gt;&lt;p&gt;These scripts go a long way to empowering a developer to contribute without getting bogged down in the idiosyncrasies of our particular system. Tools like &lt;a href=&quot;https://github.com/enquirer/enquirer&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;enquirer&lt;/a&gt; help you bash out a quick script that keeps your content consistent while providing an easy-to-use CLI.&lt;/p&gt;&lt;p&gt;Eleventy also provides hooks to &lt;a href=&quot;https://www.11ty.dev/docs/data-validate/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;validate your data&lt;/a&gt; during build. And don’t forget, if you want to use existing site data in these scripts, use the same tools as Eleventy such as LiquidJS and gray-matter.&lt;/p&gt;&lt;h2 id=&quot;enabling-non-technical-contributors&quot; tabindex=&quot;-1&quot;&gt;Enabling non-technical contributors&lt;/h2&gt;&lt;p&gt;So far this is all sounding great if you know how to use git. Just write some markdown and go. But if you’re unfamiliar with git, then writing a simple blog post can seem daunting. We needed a CMS that integrated with our repository, didn’t keep our data behind a proprietary API, and allowed technical and non-technical contributors to use the workflow that felt most comfortable to them.&lt;/p&gt;&lt;p&gt;&lt;a href=&quot;https://cloudcannon.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;CloudCannon&lt;/a&gt; meets each of these requirements and fully enables anyone to contribute to the website. Rather than storing your data, CloudCannon integrates with your site by reading your existing content and providing an interface to create, edit, and delete it. You can further enhance your integration by using their component workflow, &lt;a href=&quot;https://github.com/CloudCannon/bookshop&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Bookshop&lt;/a&gt;. This will also allow you to &lt;a href=&quot;https://cloudcannon.com/documentation/guides/bookshop-eleventy-guide/page-building/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;build pages&lt;/a&gt; using a visual editor by composing Bookshop components in their UI.&lt;/p&gt;&lt;p&gt;I recommend checking out CloudCannon’s &lt;a href=&quot;https://github.com/CloudCannon&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;GitHub profile&lt;/a&gt;, they have a lot of interesting projects geared toward enhancing static sites.&lt;/p&gt;&lt;h2 id=&quot;edge-worker-enhancements&quot; tabindex=&quot;-1&quot;&gt;Edge worker enhancements&lt;/h2&gt;&lt;p&gt;We can’t quite get all the way with static assets alone. For our use case, we wanted to display localised currencies on our pricing page based on the user’s country. We chose &lt;a href=&quot;https://pages.cloudflare.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Cloudflare Pages&lt;/a&gt; as our hosting platform, so here we’ll use a &lt;a href=&quot;https://developers.cloudflare.com/pages/functions/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Pages Function&lt;/a&gt; to achieve the desired outcome.&lt;/p&gt;&lt;p&gt;Cloudflare provide a great interface for transforming HTML on the fly. Simply build an &lt;code&gt;HTMLRewriter&lt;/code&gt; instance and use the &lt;code&gt;transform&lt;/code&gt; method to modify the body of your response. For us, it looks something like this:&lt;/p&gt;&lt;pre class=&quot;language-jsx&quot;&gt;&lt;code class=&quot;language-jsx&quot;&gt;&lt;span class=&quot;comment token&quot;&gt;// ------------------------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// functions/pricing/[country].ts&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// ------------------------------&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// Imported from our 11ty project&lt;/span&gt;
 &lt;span class=&quot;keyword token&quot;&gt;import&lt;/span&gt; billingCountries &lt;span class=&quot;keyword token&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;../../source/_data/billingCountries.json&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;export&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;onRequest&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; PagesFunction&lt;span class=&quot;operator token&quot;&gt;&amp;lt;&lt;/span&gt;Env&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;country&#39;&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;&gt;&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;async&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;context&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	&lt;span class=&quot;comment token&quot;&gt;// ...&lt;/span&gt;
 	&lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;new&lt;/span&gt; &lt;span class=&quot;class-name token&quot;&gt;HTMLRewriter&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
 	    &lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;on&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;[data-plan-id] [data-swap]&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	        &lt;span class=&quot;function token&quot;&gt;element&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;el&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	            el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;replace&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;swapElements&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;getAttribute&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;data-swap&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	                &lt;span class=&quot;literal-property property token&quot;&gt;html&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;boolean token&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 	            &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 	        &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 	    &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
 	    &lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;on&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;[name=&quot;select-currency&quot;] option&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	        &lt;span class=&quot;function token&quot;&gt;element&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;el&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	            &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;hasAttribute&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;selected&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	                el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;removeAttribute&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;selected&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 	            &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 	            &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;getAttribute&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;value&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;toLowerCase&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;===&lt;/span&gt; country&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	                el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;setAttribute&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;selected&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 	            &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 	        &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 	    &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
 	    &lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;on&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;[data-tax-notice]&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	        &lt;span class=&quot;function token&quot;&gt;element&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;el&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	            el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;setInnerContent&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;swapElements&lt;span class=&quot;punctuation token&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;taxNotice&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 	        &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 	    &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
 	    &lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;on&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;[name=&quot;plan-type&quot;]&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	        &lt;span class=&quot;function token&quot;&gt;element&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;el&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	            &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;url&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;searchParams&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;has&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;plan-type&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;business&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	                &lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 	            &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 	            &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; isBusiness &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;getAttribute&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;value&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;===&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;business&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 	            &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;isBusiness&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	                el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;setAttribute&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;checked&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;checked&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 	            &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;else&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
 	                el&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;removeAttribute&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;&#39;checked&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 	            &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 	        &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt;
 	    &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;
         &lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;keyword token&quot;&gt;await&lt;/span&gt; context&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;env&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;constant token&quot;&gt;ASSETS&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;fetch&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;pageURL&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
 
 &lt;span class=&quot;comment token&quot;&gt;// --------------------------&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// functions/pricing/index.ts&lt;/span&gt;
 &lt;span class=&quot;comment token&quot;&gt;// --------------------------&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;import&lt;/span&gt; billingCountries &lt;span class=&quot;keyword token&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;../../source/_data/billingCountries.json&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; supportedCountries &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; billingCountries&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;reduce&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;parameter token&quot;&gt;countries&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; country&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;operator token&quot;&gt;=&gt;&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     countries&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;add&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;country&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;code&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt; countries&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;new&lt;/span&gt; &lt;span class=&quot;class-name token&quot;&gt;Set&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 
 &lt;span class=&quot;keyword token&quot;&gt;export&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;literal-property property token&quot;&gt;onRequest&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;:&lt;/span&gt; PagesFunction&lt;span class=&quot;tag token&quot;&gt;&lt;span class=&quot;tag token&quot;&gt;&lt;span class=&quot;punctuation token&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;class-name token&quot;&gt;Env&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;plain-text token&quot;&gt; = (context) =&gt; &lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;let&lt;/span&gt; country &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; context&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;cf&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;country&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;operator token&quot;&gt;!&lt;/span&gt;supportedCountries&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;has&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;country&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;punctuation token&quot;&gt;{&lt;/span&gt;
         country &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;string token&quot;&gt;&#39;US&#39;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;const&lt;/span&gt; url &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;keyword token&quot;&gt;new&lt;/span&gt; &lt;span class=&quot;class-name token&quot;&gt;URL&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;context&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;request&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;url&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     url&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;pathname &lt;span class=&quot;operator token&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;template-string token&quot;&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;/pricing/&lt;/span&gt;&lt;span class=&quot;interpolation token&quot;&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;${&lt;/span&gt;country&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;toLowerCase&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;interpolation-punctuation punctuation token&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;string token&quot;&gt;/&lt;/span&gt;&lt;span class=&quot;string template-punctuation token&quot;&gt;`&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
     &lt;span class=&quot;keyword token&quot;&gt;return&lt;/span&gt; Response&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;function token&quot;&gt;redirect&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;(&lt;/span&gt;url&lt;span class=&quot;punctuation token&quot;&gt;.&lt;/span&gt;href&lt;span class=&quot;punctuation token&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;number token&quot;&gt;302&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;punctuation token&quot;&gt;;&lt;/span&gt;
 &lt;span class=&quot;punctuation token&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;plain-text token&quot;&gt;;
 &lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Now, when a user visits &lt;a href=&quot;http://www.fastmail.com/pricing/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;fastmail.com/pricing/&lt;/a&gt;, they’ll be redirected to a page with prices in their currency (if we support it).&lt;/p&gt;&lt;p&gt;Unfortunately, this is the part of our site that isn’t portable. Hopefully, once &lt;a href=&quot;https://wintercg.org/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;WinterCG&lt;/a&gt; picks up steam, we’ll start seeing much better compatibility between these different runtimes. But for now, using an Edge worker does mean accepting a certain level of lock-in with your provider.&lt;/p&gt;&lt;h2 id=&quot;wrapping-up&quot; tabindex=&quot;-1&quot;&gt;Wrapping up&lt;/h2&gt;&lt;p&gt;Personally, I’ve found Eleventy to be a great piece of software. It puts you in full control of the site you want to build and gives you the tools to extend its functionality with ease.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 8: Guiding principles</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/principles/' />
 			<id>https://www.fastmail.com/blog/principles/</id>
 			<updated>2024-12-08T00:00:01Z</updated><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the eighth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/revision-of-core-email-specifications/&quot;&gt;Dec 7: Revision of the core email specifications&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/building-a-blog/&quot;&gt;Dec 9: Building a blog&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;When it’s a week before December and you suddenly remember “oh yeah, we were going to do an advent blog series” — it does help to have some pre-written material! So I’m cheating a bit and pulling from our internal Notion, getting the words we’ve already spent a bunch of time honing.&lt;/p&gt;&lt;p&gt;I wrote last Sunday about our &lt;a href=&quot;/blog/mission-statement/&quot;&gt;Mission Statement&lt;/a&gt;. A mission statement is all well and good, but it’s very high level and aspirational. It needs to be distilled down to something that’s actionable and useful on a daily basis.&lt;/p&gt;&lt;p&gt;So we wrote the following:&lt;/p&gt;&lt;h2 id=&quot;our-guiding-principles&quot; tabindex=&quot;-1&quot;&gt;Our guiding principles&lt;/h2&gt;&lt;p&gt;How we go about achieving that vision is also very important to who Fastmail is as a company. We feel these principles really capture how Fastmail has operated as a company in the past, and should do in the future.&lt;/p&gt;&lt;blockquote&gt; &lt;h3 id=&quot;we-are-a-good-internet-citizen&quot; tabindex=&quot;-1&quot;&gt;We are a good internet citizen&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;We believe in open protocols, standards and interoperability.&lt;/li&gt; &lt;li&gt;We build, share, and support technology to make email better for everyone, not another walled garden.&lt;/li&gt; &lt;li&gt;We foster positive relationships with our customers, partners, suppliers, and staff.&lt;/li&gt; &lt;/ul&gt; &lt;/blockquote&gt;&lt;p&gt;This is also one of our &lt;a href=&quot;/company/values/&quot;&gt;public values&lt;/a&gt;. We encourage our employees to contribute to open source projects, to be involved in the technical communities in their local area, and to be involved in the standards development process.&lt;/p&gt;&lt;blockquote&gt; &lt;h3 id=&quot;we-build-the-future&quot; tabindex=&quot;-1&quot;&gt;We build the future&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;We are not content to just accept the status quo.&lt;/li&gt; &lt;li&gt;If the right tool doesn’t exist, we make it.&lt;/li&gt; &lt;li&gt;If the open standards aren’t good enough, we improve them.&lt;/li&gt; &lt;li&gt;We are leaders in our industry.&lt;/li&gt; &lt;/ul&gt; &lt;/blockquote&gt;&lt;p&gt;Yes, we’re a bit “not invented here”. We maintain the &lt;a href=&quot;https://cyrusimap.org/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Cyrus IMAP&lt;/a&gt; server. We built our own &lt;a href=&quot;https://github.com/fastmail/overture&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Javascript framework&lt;/a&gt; and &lt;a href=&quot;https://github.com/fastmail/Squire&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;email editor&lt;/a&gt;. We buy our own hardware made to spec for what we need, and manage everything from the operating system up.&lt;/p&gt;&lt;p&gt;We like to make things ourselves, because then we understand how they work; which leads into the next point!&lt;/p&gt;&lt;blockquote&gt; &lt;h3 id=&quot;we-seek-understanding&quot; tabindex=&quot;-1&quot;&gt;We seek understanding&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;We are curious about how things work, and how they came to be.&lt;/li&gt; &lt;li&gt;We seek deep and actionable understanding of our systems.&lt;/li&gt; &lt;li&gt;We recognise that we can’t understand everything, but strive to know where the boundaries of our knowledge lie.&lt;/li&gt; &lt;/ul&gt; &lt;/blockquote&gt;&lt;p&gt;We really don’t like unexplained behaviours in our system. If we don’t know why it happened, that’s a problem! It’s this attitude which led us (see the past point) to &lt;a href=&quot;/blog/twoskip-and-more/&quot;&gt;debug and then replace the skiplist database format in Cyrus&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;We’re not writing our own filesystem or operating system though! At least not this week.&lt;/p&gt;&lt;blockquote&gt; &lt;h3 id=&quot;we-value-discussion&quot; tabindex=&quot;-1&quot;&gt;We value discussion&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;We reach agreement through constructive, iterative collaboration.&lt;/li&gt; &lt;li&gt;Everyone is encouraged to express their theories, and explain the basis on which they are formed.&lt;/li&gt; &lt;li&gt;We test our assumptions against the real world.&lt;/li&gt; &lt;li&gt;What works is more important than who thought of it.&lt;/li&gt; &lt;/ul&gt; &lt;/blockquote&gt;&lt;p&gt;We really believe in confirming theories against reality. Nothing beats a good hypothesis, and a well constructed experiment to determine whether it adequately predicted what happens.&lt;/p&gt;&lt;hr&gt;&lt;p&gt;These principles really do reflect how we think and talk about ourselves at Fastmail. We live by these every day.&lt;/p&gt;&lt;p&gt;There have been times when we have considered that we need an additional principle “We get shit done”, which is in a degree of opposition to both “seek understanding” and “value discussion”. That’s the thing with principles like this, they are a tradeoff between different valuable properties. These are the ones we choose to prioritize! So we discuss, we understand, then we do.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 7: Revision of the core email specifications</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/revision-of-core-email-specifications/' />
 			<id>https://www.fastmail.com/blog/revision-of-core-email-specifications/</id>
 			<updated>2024-12-07T00:00:01Z</updated><author>
 				<name>Ken Murchison</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the seventh post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/twoskip-and-more/&quot;&gt;Dec 6: Twoskip and more&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/principles/&quot;&gt;Dec 8: Guiding principles&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;There are many specifications that form the current internet messaging ecosystem, but the most integral of these are &lt;a href=&quot;https://datatracker.ietf.org/doc/rfc5321/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;RFC 5321&lt;/a&gt; and &lt;a href=&quot;https://datatracker.ietf.org/doc/rfc5322/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;RFC 5322&lt;/a&gt;. RFC 5321 specifies the protocol for the transport of electronic mail messages on the internet, known as the Simple Mail Transfer Protocol (SMTP). RFC 5322 specifies the base syntax of these messages, known as the Internet Message Format (IMF). These documents were both published in 2008, and each are the third version of the “core” specifications (see RFCs &lt;a href=&quot;https://datatracker.ietf.org/doc/rfc821/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;821&lt;/a&gt; / &lt;a href=&quot;https://datatracker.ietf.org/doc/rfc2821/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;2821&lt;/a&gt; and RFCs &lt;a href=&quot;https://datatracker.ietf.org/doc/rfc822/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;822&lt;/a&gt; / &lt;a href=&quot;https://datatracker.ietf.org/doc/rfc2822/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;2822&lt;/a&gt; respectively).&lt;/p&gt;&lt;p&gt;Over the sixteen years since publication, several errata have been filed against both documents. As a result, an effort is currently underway within the Internet Engineering Task Force (IETF) &lt;a href=&quot;https://datatracker.ietf.org/wg/emailcore/about/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;EMAILCORE&lt;/a&gt; working group to revise these specifications yet again. Per its charter, the working group is tasked with updating the documents with “corrections and clarifications only, with a strong emphasis on keeping these minimal and avoiding broader changes to terminology or document organization”.&lt;/p&gt;&lt;p&gt;Work on the revision to the transport protocol specification (&lt;a href=&quot;https://datatracker.ietf.org/doc/draft-ietf-emailcore-rfc5321bis/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;RFC 5321bis&lt;/a&gt;) continues on a few remaining issues, but is nearing completion. Work on the revision to the message format specification (&lt;a href=&quot;https://datatracker.ietf.org/doc/draft-ietf-emailcore-rfc5322bis/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;RFC 5322bis&lt;/a&gt;) has already been completed, provided that no further changes are prompted by changes to RFC 5321bis. Remarkably, both documents have the same editors as the previous two revisions going back to 2001 — John Klensin and Pete Resnick respectively. For those that are interested in what has been corrected and/or clarified in these revisions, each of the documents have appendices that contain a list of changes, with the discussions of these changes having taken place on the working group’s &lt;a href=&quot;https://mailarchive.ietf.org/arch/browse/emailcore/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;mailing list&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;In addition to revising the core specifications, the group is also working on an &lt;a href=&quot;https://datatracker.ietf.org/doc/draft-ietf-emailcore-as/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;applicability statement&lt;/a&gt; to document other relevant specifications that implementors should be aware of, such as the use of Multipurpose Internet Mail Extensions (MIME) and Transport Layer Security (TLS). It also documents current email best practices, such as which provisions of the transport protocol and message format have proven to cause interoperability issues, and how to properly reuse an existing email message as a template for a new one.&lt;/p&gt;&lt;p&gt;All three documents are expected to be submitted to the RFC Editor for publication in early 2025.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 6: Twoskip and more</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/twoskip-and-more/' />
 			<id>https://www.fastmail.com/blog/twoskip-and-more/</id>
 			<updated>2024-12-06T00:00:01Z</updated><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the sixth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post is &lt;a href=&quot;/blog/mysql-innodb-trx-id/&quot;&gt;Dec 5: MySQL InnoDB innodb_trx is cached&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/revision-of-core-email-specifications/&quot;&gt;Dec 7: Revision of the core email specifications&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Rob wrote yesterday about some of the issues he ran into using MySQL triggers to implement JMAP access to our database tables, so I figured I’d follow up with talking about the other database that a lot of our data is stored in.&lt;/p&gt;&lt;p&gt;I love talking about twoskip. It’s used within the &lt;a href=&quot;https://cyrusimap.org/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Cyrus IMAP server&lt;/a&gt; for all our big databases. And I wrote something about &lt;a href=&quot;/blog/cyrus-databases-twoskip-and-beyond/&quot;&gt;possible futures for the format&lt;/a&gt; back in the 2016 advent. Well, we over-engineered our attempt at zeroskip, a more copy-on-write friendly alternative to twoskip, so we’ve been running twoskip basically unmodified since then despite moving to ZFS for all our email storage.&lt;/p&gt;&lt;p&gt;It turns out that when you have really fast NVMe, even suboptimal filesystem usage is still pretty fast.&lt;/p&gt;&lt;h2 id=&quot;before-twoskip&quot; tabindex=&quot;-1&quot;&gt;Before twoskip&lt;/h2&gt;&lt;p&gt;When I first started at Fastmail, we were using the Cyrus with the skiplist database format. In the early days, it could get corrupted pretty easily on a busy server. I wrote a tool in perl back in 2006:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;commit 20e5db17fd5be878a84c9b122fc22f49fe0f7ea2
 Author: Bron Gondwana &amp;lt;brong@fastmailteam.com&amp;gt;
 Date:   Wed Sep 27 02:59:54 2006 +0000
 
     tool to dump even corrupted skiplists in something resembling a usable form
 
 diff --git a/utils/oneoff/skiplist_dump.pl b/utils/oneoff/skiplist_dump.pl
 new file mode 100644
 index 0000000000..e8bf1b27d1
 --- /dev/null
 +++ b/utils/oneoff/skiplist_dump.pl
 @@ -0,0 +1,115 @@
 +#!/usr/bin/perl
 +
 +use strict;
 +use warnings;
 +
 +use IO::File;
 +use Data::Dumper;
 +
 +use constant INORDER =&amp;gt; 1;
 +use constant ADD =&amp;gt; 2;
 +use constant DELETE =&amp;gt; 4;
 +use constant COMMIT =&amp;gt; 255;
 +use constant DUMMY =&amp;gt; 257;
 +
 +my $fh = IO::File-&amp;gt;new(shift);
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Because we were getting so much corruption, I wound up figuring out what was going on and writing a patch to Cyrus. Here was the description:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;conf/patches/cyrus_quilt/cyrus-skiplist-bugfixes-2.3.10.diff
 
 SKIPLIST bugfixes
 
 &amp;lt;b&amp;gt;ACCEPTED UPSTREAM 2007-11-16&amp;lt;/b&amp;gt;
 
 In the past we have had issues with bugs in skiplist on seen
 files, and we truncated files at the offset with the issue
 since they were only seen data.
 
 Lately, we have had more tools updating mailboxes.db more
 often, and have lost multiple mailboxes.db files.
 
 There are two detectable issues:
 
 1) incorrect header &amp;quot;logstart&amp;quot; values causing recovery to
    fail with either unexpected ADD/DELETE records or
    unexpected INORDER records depending which side of the
    correct location the logstart value is wrong.
 2) a bunch of zero bytes between transactions in the log
    section.
 
 The attached patch fixes the following issues:
 
 a) recovery failed to update db-&amp;gt;map_base if it truncated
    a partial transaction.  This reliably recreated the
    zero bytes issues above by having the next store command
    lseek to a location past the new end of the file, and
    hence fill the remainder with blanks.
 
 b) the logic in the &amp;quot;delete&amp;quot; handler for detecting &amp;quot;no
    record exists&amp;quot; (ptr == db-&amp;gt;map_base) was backwards,
    meaning that a delete on a record which didn&#39;t exist
    caused reads of PTR(db-&amp;gt;map_base, i), which is bogus
    and nasty.  This is the suspect for logstart breakage
    though I haven&#39;t proven this yet.
 
 c) unsure if this is a real risk, but added a ftruncate
    to checkpoint to ensure new file really is empty,
    since we don&#39;t open it with O_EXCL.
 
 d.1) when abort is called it needs to update_lock() to
      ensure that the records it&#39;s about to rollback are
      actually locked.  This fix stopped segfaults in
      my testing.
 
 d.2) delete didn&#39;t check retry_write for success, and
      also suffered from the same problem as:
 
 d.3) if retry_writev in store failed, then it called
      abort, but without the records actually written,
      abort had no way of knowing which offsets needed
      to be switched back, meaning bogus pointers could
      be left in the file until the next recovery().
      Changed it to write the records first, then only
      once that succeeded update the pointers.  This way
      abort will do the right thing regardless.
 
 *PHEW* - that&#39;s three days I want back, but it has survived
 10 concurrent processes doing nasty things to it for 10000
 operations each and it&#39;s still going strong:
 skiplist: recovered /tmp/hammer.db (143177 records, 5965096 bytes) in 0 seconds
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;So that was that. But skiplist was still expensive to recover after a crash, so by 2011 I had replaced it with twoskip. Twoskip had many design goals (this is all in comments at the top of the source file, you can &lt;a href=&quot;https://github.com/cyrusimap/cyrus-imapd/tree/master/lib/cyrusdb_twoskip.c&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;read it on Github&lt;/a&gt;).&lt;/p&gt;&lt;h3 id=&quot;design-goals&quot; tabindex=&quot;-1&quot;&gt;Design Goals:&lt;/h3&gt;&lt;ul&gt; &lt;li&gt;64 bit throughout&lt;/li&gt; &lt;li&gt;Checksums on all content&lt;/li&gt; &lt;li&gt;Single on-disk file&lt;/li&gt; &lt;li&gt;Fully embedded, no daemons or external helper&lt;/li&gt; &lt;li&gt;Readable without causing writes (if clean)&lt;/li&gt; &lt;li&gt;May require that an aligned 512 bytes either fully writes or doesn’t write anything.&lt;/li&gt; &lt;li&gt;May require fsync to work&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;We achieved all those, which is why twoskip has been such a workhorse. With that said,&lt;/p&gt;&lt;h2 id=&quot;a-corrupt-twoskip-file&quot; tabindex=&quot;-1&quot;&gt;A corrupt twoskip file&lt;/h2&gt;&lt;p&gt;Last year, we had our only incident of corrupt twoskip files in this entire time!&lt;/p&gt;&lt;p&gt;I’m basically just going to dump the entire Topicbox thread here, because it’s a peek behind the scenes at how we debugged the issue, and also how having redundant servers with failover meant that we could do all this debugging and recovery without impacting access to email for users.&lt;/p&gt;&lt;h3 id=&quot;first-email-by-me&quot; tabindex=&quot;-1&quot;&gt;First email, by me:&lt;/h3&gt;&lt;p&gt;Sunday, February 05, 2023 12:28&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;I got paged about:&lt;/p&gt; &lt;pre&gt;&lt;code&gt;2023-02-04T15:54:56.525746-05:00 imap41 sloti41n57/syncserver[2135571]: DBERROR: twoskip checksum head error: filename=&amp;lt;/mnt/i41n57/sloti41n57/store56/conf/user/uuid/c/9/c993a9a5-5fed-4515-807e-0b1326c7200d/conversations.db&amp;gt; offset=&amp;lt;00370B30&amp;gt; syserror=&amp;lt;No such file or directory&amp;gt; func=&amp;lt;read_onerecord&amp;gt;
 &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;It turns out there’s a block of corruption from 00360000 to 00380000 in that file.  And it’s not all zeros.  I’m still investigating.&lt;/p&gt; &lt;p&gt;Anyway, failed everything off the machine, rebooted - corruption is still there.  Initiated a new zfs scrub.  That slot is left down, all others are up.&lt;/p&gt; &lt;p&gt;Marc and I are both heading out for a bit to do personal things today, but I’ll look in again on the scrub later and also keep looking at the corruption.  It looks like this:&lt;/p&gt; &lt;pre&gt;&lt;code&gt;00360000  2b 01 00 34 00 00 00 19  00 00 00 00 00 76 e3 80  |+..4.........v..|
 00360010  00 00 00 00 00 36 00 70  28 b1 a3 a4 fa ef 97 5e  |.....6.p(......^|
 00360020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 003600e0  2b 01 00 30 00 00 00 19  00 00 00 00 00 82 4a c8  |+..0..........J.|
 003600f0  00 00 00 00 00 36 01 50  11 20 22 7c 62 dd 0d d3  |.....6.P. &amp;quot;|b...|
 00360100  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00360150  2b 02 00 31 00 00 00 19  00 00 00 00 00 00 00 00  |+..1............|
 00360160  00 00 00 00 00 36 01 c8  00 00 00 00 00 36 03 f8  |.....6.......6..|
 00360170  62 6b f2 75 12 c0 98 2e  00 00 00 00 00 00 00 00  |bk.u............|
 00360180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00360230  00 00 00 00 00 00 00 00  2b 01 00 31 00 00 00 19  |........+..1....|
 00360240  00 00 00 00 00 75 4d d8  00 00 00 00 00 36 02 a8  |.....uM......6..|
 00360250  f8 55 a3 e8 24 1c 9e 87  00 00 00 00 00 00 00 00  |.U..$...........|
 00360260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00360310  00 00 00 00 00 00 00 00  2b 01 00 30 00 00 00 19  |........+..0....|
 00360320  00 00 00 00 00 76 df 08  00 00 00 00 00 36 03 88  |.....v.......6..|
 00360330  0d 41 4e 13 67 8d aa b4  00 00 00 00 00 00 00 00  |.AN.g...........|
 00360340  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 
 &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;Which is so weird because it looks like keys but not values being written or something.  But the checksums also aren’t consistent so it’s not pure on-write corruption.  I don’t know yet.&lt;/p&gt; &lt;p&gt;I’m not thrilled with the number of filesystem level blue moons we’ve had in the past few weeks.  I want to do more digging later, but for now we’ll let the scrub run.&lt;/p&gt; &lt;/blockquote&gt;&lt;h3 id=&quot;i-came-back-later-with-more-detail&quot; tabindex=&quot;-1&quot;&gt;I came back later with more detail&lt;/h3&gt;&lt;p&gt;Sunday, February 05, 2023 22:04&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;OK - this IS consistent with a zeroed out area being written over later.  So the file was written, then this block got zeroed, then later writes came back and updated just the headers of these records, which is how it works.&lt;/p&gt; &lt;p&gt;So it SMELLS like maybe this is a pwrite vs mmap issue.  Linux (maybe ZFS) failed to page in the content, and then wrote bits of it, and wrote back the block with zeros for all the bits it wasn’t writing.&lt;/p&gt; &lt;p&gt;So it’s for sure a page cache bug.  OUCH.  We can maybe work around this by using mmap for writes as well as reads, certainly if we see this again we’re in trouble.&lt;/p&gt; &lt;p&gt;I looked to see if there were ANY other files of the same type on the same machine, and no - everything else was consistent.  This took hours:&lt;/p&gt; &lt;pre&gt;&lt;code&gt;[fastmail root(brong)@imap41 ~]# find /mnt/*/*/*/conf/user/uuid -name conversations.db | xargs -I{} sh -c &#39;echo -n {} &amp;quot;: &amp;quot;; sudo -u cyrus /mnt/i41n57/sloti41n57/store56/conf/pkg/bin/cyr_dbtool -C /etc/cyrus/imapd-sloti41n57.conf {} twoskip consistent&#39; | grep No
 /mnt/i41n57/sloti41n57/store56/conf/user/uuid/c/9/c993a9a5-5fed-4515-807e-0b1326c7200d/conversations.db : No, not consistent
 [fastmail root(brong)@imap41 ~]#
 
 &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;So I moved it aside and reconstructed the user:&lt;/p&gt; &lt;pre&gt;&lt;code&gt;[fastmail root(brong)@imap41 ~]# mv /mnt/i41n57/sloti41n57/store56/conf/user/uuid/c/9/c993a9a5-5fed-4515-807e-0b1326c7200d/conversations.db /mnt/i41n57/sloti41n57/store56/conf/user/uuid/c/9/c993a9a5-5fed-4515-807e-0b1326c7200d/conversations.db.old
 [fastmail root(brong)@imap41 ~]# cyr sloti41n57 ctl_conversationsdb -R [REDACTED USER]
 [REDACTED DETAIL]
 [fastmail root(brong)@imap41 ~]#
 
 &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;And finally started up the slot.&lt;/p&gt; &lt;/blockquote&gt;&lt;h3 id=&quot;rob-m-our-cto-chimed-in&quot; tabindex=&quot;-1&quot;&gt;Rob M, our CTO, chimed in&lt;/h3&gt;&lt;p&gt;Monday, February 06, 2023 10:26&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;I note that this is 128kb, which is the ZFS recordsize we use for the cyrus spools. That seems awfully coincidental.&lt;/p&gt; &lt;p&gt;It’s well known that the way ZFS uses memory (ARC) and it’s interaction with the Linux page cache is messy and caused problems in the past, but that it had been basically dealt with.&lt;/p&gt; &lt;p&gt;This problem though suggests there might still be a very rare hole somewhere. Given this is the first corruption we’ve seen in over a year of heavy usage across many machines, it seems it’s going to be a very very rare edge case, and likely to be extremely hard to ever track down :(&lt;/p&gt; &lt;/blockquote&gt;&lt;h3 id=&quot;rob-n-senior-sysadmin&quot; tabindex=&quot;-1&quot;&gt;Rob N, Senior Sysadmin&lt;/h3&gt;&lt;p&gt;Saturday, February 11, 2023 12:21&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;This is likely to be https://github.com/openzfs/zfs/issues/13608, or very adjacent to it. Its describing an mmap read returning wrong/corrupted data.&lt;/p&gt; &lt;p&gt;No one yet really has a handle on it. It is being investigated a little elsewhere and was discussed at the OpenZFS dev call in early January. It seems clear that its possible for the page cache to be dirty when ZFS thinks it should be clean, but no one has been able to track it down reliably. It might be related to concurrent mmap reads and modifies over the same record, and then being invalidated inconsistently (which is why clones are in play in the above issue).&lt;/p&gt; &lt;p&gt;Assuming it is the same issue, we might in a position to assist because we’re coming at it from a different angle.&lt;/p&gt; &lt;p&gt;What I want to do next is try to determine the rough sequence events (mapped read -&amp;gt;  write -&amp;gt; read -&amp;gt; whatever) including any other programs that might have been operating on the same data at the same time. I suspect we can put together a rough sequence just by reading code. Bron, if you have some time this week, I would like to go through this with you (mostly: show me the code and I’ll nod along). If we can pin down a clear sequence, maybe we can write something that can reproduce it if we drive it hard enough. And if not, maybe we can at least write this all down for the ZFS folk to consider.&lt;/p&gt; &lt;/blockquote&gt;&lt;h3 id=&quot;and-finally-rob-n-wrote-up-a-great-summary&quot; tabindex=&quot;-1&quot;&gt;And finally Rob N wrote up a great summary&lt;/h3&gt;&lt;p&gt;Sunday, February 19, 2023 22:08&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;Bron and I spent a couple of hours last Tuesday going over exactly how twoskip does its work and trying to get a sense of what happened. This is something of a writeup. Thin on technical detail, because that would take even longer for me to write.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;Tiny tiny twoskip overview&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;It helps to know what a twoskip file is before we talk about how it writes to disk.&lt;/p&gt; &lt;p&gt;From outside, its a KV store, with variable-length keys and values. Internally, its effectively a linked list of records, sorted by key (its a skiplist, so there’s actually multiple linked lists, but that doesn’t matter for our purposes).&lt;/p&gt; &lt;p&gt;Each record is either a “RECORD”, that is, a real active item in the store, or a “DELETE”, which is a tombstone in place of a thing that used to exist. (There’s a couple of others, that don’t matter). Making a modification means writing a new record to the end of the file, and then going back through the file to fix the linked list pointers to hook that record up in the right place (if you ever wrote a sorted linked list and remember the fiddling to insert a thing in the middle of the list, its just a slightly fancier version of that).&lt;/p&gt; &lt;p&gt;Cyrus does fairly regular repack operations, which means creating a new file and writing all the “live” records to it, removing any RECORDs that have a DELETE, and of course writing all the records in order in the file, rather than having pointers back and forth through the file. Same end result, but a better optimised file.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;Reading and writing&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;Accessing a twoskip file is done via two different mechanisms: &lt;code&gt;mmap()&lt;/code&gt; for reads, &lt;code&gt;write()&lt;/code&gt; (actually &lt;code&gt;pwritev()&lt;/code&gt;) for writes.&lt;/p&gt; &lt;p&gt;For the uninitiated, &lt;code&gt;mmap()&lt;/code&gt; is (in this case) a way to pretend a file is really a region of memory. Rather than having to read the contents of a file into memory piece by piece, you can just tell the kernel to give you a big chunk of memory. When you try to read from it, the kernel will go and get the data from the file and make it appear in that memory space, so you can kind of pretend that its already been read into memory. The main reason to do this is performance - the kernel is usually given you a view over its own filesystem cache memory, where its already likely got a copy of a busy file available, and it doesn’t have to copy that data into a user-space memory buffer.&lt;/p&gt; &lt;p&gt;For writing, we doing a more conventional &lt;code&gt;write()&lt;/code&gt; at specific offsets. You can set up a &lt;code&gt;mmap()&lt;/code&gt; for writes too, but for small random writes throughout a file (as twoskip writes are) it doesn’t really gain much on performance (the kernel has to copy from userspace) and can be tougher for the user program to manage (in a variety of subtle handwavey ways that tbh I am not an expert in). The calls to &lt;code&gt;write()&lt;/code&gt; will be reflected in the existing read map anyway (the kernel will copy it there before it sends it down to disk anyway), so it all works out quite nicely in the end.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;The twoskip write cycle&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;When we open a twoskip file, we maps the entire file into memory with &lt;code&gt;mmap()&lt;/code&gt;. The map region is the length of the file, rounded up to the nearest 16K boundary. At this point we could happily read the existing contents out of memory.&lt;/p&gt; &lt;p&gt;When we write a new record, we seek to the end of the file, and then write four distinct parts (in a single &lt;code&gt;write()&lt;/code&gt; call):&lt;/p&gt; &lt;ul&gt; &lt;li&gt;a 32-byte header, which includes the lengths of the key and value, linked list pointers, and checksums for the header and the data&lt;/li&gt; &lt;li&gt;variable-length key&lt;/li&gt; &lt;li&gt;variable-length value&lt;/li&gt; &lt;li&gt;0-7 padding bytes to bring the entire record to an 8-byte boundary&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Then, we go back through the file (before the record), finding the nodes that should point to this new record, and overwrite their 32-byte header updating the pointers to splice it into the list. Each header is a separate call to &lt;code&gt;write()&lt;/code&gt;.&lt;/p&gt; &lt;p&gt;Now, a &lt;code&gt;mmap()&lt;/code&gt; region has a fixed length, and we started with it as the length of the file, plus padding. After writing a record, the file may now actually be larger than the map. There’s nothing wrong with this; &lt;code&gt;mmap()&lt;/code&gt; is done by offset and length; it doesn’t have to cover the whole file. This does of course mean, if we have just written a record past the end of the map, we can’t actually read it back (it would cause a segfault). To keep everything nice, after each &lt;code&gt;write()&lt;/code&gt;, we check to see if we’ve gone past the end of the map and if so, we extend it out past the end (to the nearest 16K).&lt;/p&gt; &lt;p&gt;An important thing to note: if you &lt;code&gt;mmap()&lt;/code&gt; past the end of a file (and we do as a matter of course, with the 16K alignment) and then you read from that area, you get zeroes back.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;ZFS is not like other filesystems&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;There’s a few things to know about ZFS that is different to a more conventional overwriting filesystem in this situation.&lt;/p&gt; &lt;p&gt;ZFS’ unit of IO is the “record size”. Its configurable, but we’re using the default, which is 128K. Which is to say, any write to an existing file of smaller than 128K requires loading a full 128K record from disk, modifying it in memory, then writing the full 128K back down again (to a new record; ZFS never overwrites existing records, but that’s not really relevant here).&lt;/p&gt; &lt;p&gt;ZFS does not use the kernel-provided filesystem cache (the “page cache”) because it has its own cache (the ARC) which has quite different characteristics to support all sorts of ZFS features. However, &lt;code&gt;mmap()&lt;/code&gt; by definition exposes memory in the page cache. To make this work, ZFS copies data between the page cache and the ARC as necessary when a file is mapped.&lt;/p&gt; &lt;p&gt;For asynchronous writes, a conventional filesystem will end up having written data staged in the page cache, which is periodically flushed to disk. ZFS has the same concept, but it uses the ARC as the cache, and writes are attached to a “transaction group”, which is flushed periodically. This is mostly interesting here because I’m going to say “transaction group” soon. The details past that don’t matter, mostly its just a collection of writes waiting to go to the storage pool.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;What even happened?&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;So, what we saw is a single 128K section of the file that had record headers, but zeroes through the data sections. Recall that new records are written to the end of the file, making it larger, then we extend the map over it and past the real end, and then we go back through the file and update the headers.&lt;/p&gt; &lt;p&gt;This record has headers, but no data. It seems that the only way this could have happened is if the initial record writes were lost somehow, but the header writes survived. The initial record writes are the only thing that can write “outside” of the currently mapped region; the header fixups always happen on existing records that the map will always be covering.&lt;/p&gt; &lt;p&gt;This and the fact that the data is all-zeroes (and not scrambled random memory contents) suggests that &lt;code&gt;write()&lt;/code&gt; successfully staged stuff to be written (that is, an ARC buffer for the record was allocated and was attached to a transaction group), but when &lt;code&gt;mmap()&lt;/code&gt; was called to extend the map, ZFS decided that the page cache contents (all zeroes) was dirty, and overwrote the pending ARC buffer with its contents, and so those zeroes were written down. Whatever the bug is, this is the heart of it.&lt;/p&gt; &lt;p&gt;Another interesting factor is that last twoskip record in the previous 128K ZFS record ends exactly on at the end of the record. This means that the map wouldn’t have been extended, as it was already right on the end of the block, covering the entire file. So the very first write to the new record would have been the one that allocated it, and the map extension immediately after would have been the first that covered that record, so the record would have only existed in any form in the ARC at that moment, not yet on disk.&lt;/p&gt; &lt;p&gt;(This is interesting, because complex interactions at object boundaries are a traditionally a rich source of bugs).&lt;/p&gt; &lt;p&gt;But the thing we must remember is that we have been running this code for nearly two years. We have written billions, maybe even trillions, of records in that time. If this was a simple case of a small handful of conditions occurring at the same time, we’d almost certainly have hit this already, and we know we haven’t. That almost certainly points to a locking bug; ZFS is working to keep the ARC and the page cache in sync, which means knowing their states in a given moment and choosing whether or not to copy data back and forth. It very much feels like there is a tiny, probably microsecond-long gap, where something is unlocked that shouldn’t be, and if you’re very very unlucky, a decision is made right in the gap based on faulty knowledge.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;Making a test case&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;So Bron and I got to the point where we at least felt like we knew the moving parts that were likely involved, and we could start thinking about a test case. To reproduce a bug that only happens in a very specific set of conditions that you can control and one that you can’t, you just have to write a program that hits those conditions as often as possible, and then run it over and over in a short space of time, trying to hit the elusive condition that explodes the whole thing.&lt;/p&gt; &lt;p&gt;So, Bron wrote &lt;a href=&quot;https://github.com/cyrusimap/cyrus-imapd/tree/skipwork/contrib/skipwhack&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;skipwhack&lt;/a&gt;, which attempts to write to a file in the shape described above over and over, and then check back to see that it actually wrote stuff out properly.&lt;/p&gt; &lt;p&gt;Now, at time of writing, I haven’t had chance to exercise this - all of my test systems have some deficiency (disks too slow, CPU too slow, not enough memory) that make it unable to generate enough load to trigger the bug. Of course, it could be that its also not an accurate enough reflection of the situation (maybe we missed a condition) to trigger it, but I think its far too soon to call that. I’ll be trying to find a suitable machine this week.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;Maybe none of this matters&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;Around the time Bron emailed me a PR appeared that I think will probably fix this anyway: https://github.com/openzfs/zfs/pull/14498. It’s claiming to fix the previous mmap bug I mentioned upthread. I don’t really follow it well, but it seems like ZFS was expecting a page fault to either succeed or fail, but there’s a rare “in between” state that it wasn’t handling properly. This fix seems to make it handle that case.&lt;/p&gt; &lt;p&gt;I suspect this is the whole story, but I’m not totally sure, I’m still going to be trying to get skipwhack to run and blow up, because then we can try this patch and see if it stops doing that. If it does, great, if not, I can write this up in more detail as there may be further things. We’ll see!&lt;/p&gt; &lt;/blockquote&gt;&lt;h3 id=&quot;postscript&quot; tabindex=&quot;-1&quot;&gt;Postscript&lt;/h3&gt;&lt;p&gt;We never saw another one of these corruptions, and we’re now on a newer ZFS which includes that fix, so we believe we’re all good for this. But in all these years, the only twoskip corruption we have seen has been an operating system level filesystem corruption!&lt;/p&gt;&lt;p&gt;Yes, we did blow away and recreate the filesystem which had given us the error as well. Our hardware really is super reliable these days, so we really don’t see this kind of issue very often.&lt;/p&gt;&lt;p&gt;Rob N is now working for &lt;a href=&quot;https://klarasystems.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Klara Systems&lt;/a&gt;, on the OpenZFS filesystem. He writes more great material about it over on &lt;a href=&quot;https://despairlabs.com/blog/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;his blog&lt;/a&gt;. We’re all very happy to be using a filesystem that he’s working on improving!&lt;/p&gt;&lt;h2 id=&quot;replacing-twoskip&quot; tabindex=&quot;-1&quot;&gt;Replacing twoskip&lt;/h2&gt;&lt;p&gt;Having said all this, the issues that we identified even back in 2016 still exist. Twoskip is a little high on random IO for writes. That’s OK for us. But we have single file databases with over a gigabyte; and the “checkpoint” command, where a file gets rewritten from scratch, can take many minutes. That’s painful, because it’s a stop the world lock.&lt;/p&gt;&lt;p&gt;So I’m working on a &lt;a href=&quot;https://github.com/cyrusimap/cyrus-imapd/pull/5157&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;new format&lt;/a&gt;, &lt;code&gt;twom&lt;/code&gt; - pronounced “tomb”. It’s basically what I described back in 2016:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Ancestor pointers so you can perform an &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiversion_concurrency_control&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;MVCC&lt;/a&gt; read without blocking writes&lt;/li&gt; &lt;li&gt;a checkpoint operation which uses MVCC plus log replay, without ever needing long locks&lt;/li&gt; &lt;li&gt;a faster hash algorithm (&lt;a href=&quot;https://xxhash.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;xxHash&lt;/a&gt; rather than CRC32)&lt;/li&gt; &lt;li&gt;direct mmap reads and writes rather than mixing mmap with file IO&lt;/li&gt; &lt;li&gt;blank slop space on the end of files to avoid needing so many syscalls to extend the file&lt;/li&gt; &lt;li&gt;more direct file handle and mmap manipulation rather than using our library wrappers, so we can be smarter about duplicating key records.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;That last one this is a bigger deal than you might imagine. We spend a lot of CPU in twoskip just duplicating data that could be only be lost in rare edge cases. By being more lazy about that, we can save CPU in the common case.&lt;/p&gt;&lt;p&gt;All this needs to be done without losing any of the existing high reliability that twoskip gives us. I’m very much hoping to have twom stable by early next year and start testing it in production!&lt;/p&gt;&lt;h2 id=&quot;starvation&quot; tabindex=&quot;-1&quot;&gt;Starvation&lt;/h2&gt;&lt;p&gt;The thing that triggered me coming back to this problem was an archive user having a couple of million emails deleted. One week later, we run the &lt;code&gt;cyr_expire&lt;/code&gt; tool to clean up the delete email, and this tool currently takes a lock and then does all the deletes. A million updates to a conversations database takes a long time, even just for deletes! There’s indexes to re-calculate and update, files to unlink.&lt;/p&gt;&lt;p&gt;It took a couple of hours, and while it did so, it blocked deliveries for that one user. Archiveusers for big customers get a lot of mail, this started using up all the processes and slowing email delivery for about 8% of our customers. Ouch. We paused delivery to that one user and got everyone else back working, but that’s annoying having to manually intervene, particularly outside working hours.&lt;/p&gt;&lt;p&gt;So we’re also working out way through the Cyrus code looking for things like this, and batching them up. In future, &lt;code&gt;cyr_expire&lt;/code&gt; is going to do a bunch of messages, release all its locks, then come back and start again with another batch. That way, other tasks like delivering a new email get a chance to interleve.&lt;/p&gt;&lt;h2 id=&quot;fastmail-really-cares-about-the-detail&quot; tabindex=&quot;-1&quot;&gt;Fastmail really cares about the detail&lt;/h2&gt;&lt;p&gt;Hopefully you can see from this, we’re right down in the weeds looking at exactly what’s going on, and building components with the kind of reliability that allows us to keep the service up, and keep email flowing, even as we run into nasty edge cases with our systems.&lt;/p&gt;&lt;p&gt;See you again in the next one.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 5: MySQL InnoDB innodb_trx is cached</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/mysql-innodb-trx-id/' />
 			<id>https://www.fastmail.com/blog/mysql-innodb-trx-id/</id>
 			<updated>2024-12-05T00:00:01Z</updated><author>
 				<name>Rob Mueller</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the fifth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/meet-the-team-bek/&quot;&gt;Dec 4: Meet the team — Bek&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/twoskip-and-more/&quot;&gt;Dec 6: Twoskip and more&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;This is a technical post about an aspect of MySQL InnoDB and transaction ids.&lt;/p&gt;&lt;p&gt;One of the features of &lt;a href=&quot;https://datatracker.ietf.org/doc/html/rfc8620&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;JMAP&lt;/a&gt; is that it allows a client to &lt;a href=&quot;https://datatracker.ietf.org/doc/html/rfc8620#section-5.2&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;fetch changes&lt;/a&gt; that have occurred on the server since the client last synced with the server. This is done with a &lt;code&gt;sinceState&lt;/code&gt; string. Although no specific implementation is required, we’ve found that using a system based on “modification sequences” (aka MODSEQs) as originally described in the &lt;a href=&quot;https://datatracker.ietf.org/doc/html/rfc4551&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;IMAP CONDSTORE&lt;/a&gt; extension works well.&lt;/p&gt;&lt;p&gt;I have been doing some work internally to allow us to calculate MODSEQs for all JMAP objects stored in our MySQL database. The basic idea was to setup triggers on any INSERT/UPDATE/DELETE actions to update the appropriate MODSEQ on the corresponding table row. The exact structures required for implementing the JMAP /changes method on all database tables is something for another blog post, but this post is more about an unexpected oddity I found while trying to do this.&lt;/p&gt;&lt;p&gt;Conceptually what I need is reasonably straightforward. There is a table to store all the current MODSEQs for each User owned table tuple.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;CREATE TABLE UserModSeqs (
   UserId INT NOT NULL,
   TableName VARCHAR(255) NOT NULL,
   CurrentModSeq BIGINT NOT NULL DEFAULT 0,
   HighestPurgedModseq BIGINT NOT NULL DEFAULT 0,
   PRIMARY KEY (UserId, TableName),
   CONSTRAINT UserModSeqsFK FOREIGN KEY (UserId)
     REFERENCES Users (UserId) ON DELETE CASCADE
 );
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;There is a generic “bump” function to increment the current MODSEQ for a particular User/table&lt;/p&gt;&lt;pre&gt;&lt;code&gt;CREATE FUNCTION BumpUserModSeq (ModSeqOwnerValue INT, DataTableName VARCHAR(255))
 RETURNS BIGINT
 BEGIN
   DECLARE NewModSeq BIGINT;
 
   INSERT INTO UserModSeqs
     (UserId, TableName, CurrentModSeq, HighestPurgedModseq)
   VALUES
     (ModSeqOwnerValue, DataTableName, 1, 0)
   ON DUPLICATE KEY UPDATE
     CurrentModSeq = CurrentModSeq + 1;
 
   SELECT CurrentModSeq
   FROM UserModSeqs
   WHERE UserId = ModSeqOwnerValue
     AND TableName = DataTableName
   INTO NewModSeq;
 
   RETURN NewModSeq;
 END
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And then the actual trigger which looks something like:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;CREATE TRIGGER ${table}UpdateModSeq
 BEFORE UPDATE ON $table
 FOR EACH ROW
 BEGIN
   SET NEW.UpdatedModSeq = BumpUserModSeq(&amp;quot;UserId&amp;quot;, &amp;quot;$table&amp;quot;);
 END
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;That you’d create for each &lt;code&gt;$table&lt;/code&gt; that’s “owned” by a User (i.e. has foreign key UserId to the Users table).&lt;/p&gt;&lt;p&gt;One of the issues with this is that every single row updated on a table generates a new MODSEQ for each updated row.&lt;/p&gt;&lt;p&gt;I had an idea to make it so we only bump the modseq number once for each transaction rather than each row. Searching around you can find that the &lt;code&gt;information_schema.innodb_trx&lt;/code&gt; table has a &lt;code&gt;trx_id&lt;/code&gt; field, so a query like:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;SELECT trx_id
 FROM information_schema.innodb_trx
 WHERE trx_mysql_thread_id = connection_id()
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Will get the transaction id of your current session, great. So we can create a function like:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;CREATE FUNCTION GetCurrentTrxId ()
 RETURNS BIGINT UNSIGNED
 BEGIN
   DECLARE CurrentTrxId BIGINT UNSIGNED;
 
   SELECT trx_id
   FROM information_schema.innodb_trx
   WHERE trx_mysql_thread_id = connection_id()
   INTO CurrentTrxId;
 
   RETURN CurrentTrxId;
 END
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And then change the “bump” function to:&lt;/p&gt;&lt;pre&gt;&lt;code&gt;CREATE FUNCTION BumpUserModSeq (ModSeqOwnerValue INT, DataTableName VARCHAR(255))
 RETURNS BIGINT
 BEGIN
   DECLARE NewModSeq BIGINT;
   DECLARE CurrentTrxId BIGINT UNSIGNED DEFAULT GetCurrentTrxId();
 
   IF CurrentTrxId IS NULL OR
      @LastBumpUserModSeqTrxId IS NULL OR
      @LastBumpModSeqOwnerValue IS NULL OR
      @LastBumpDataTableName IS NULL OR
      @LastBumpUserModSeqTrxId &amp;lt;&amp;gt; CurrentTrxId OR
      @LastBumpModSeqOwnerValue &amp;lt;&amp;gt; ModSeqOwnerValue OR
      @LastBumpDataTableName &amp;lt;&amp;gt; DataTableName
     THEN
 
     INSERT INTO UserModSeqs
       (UserId, TableName, CurrentModSeq, HighestPurgedModseq)
     VALUES
       (ModSeqOwnerValue, DataTableName, 1, 0)
     ON DUPLICATE KEY UPDATE
       CurrentModSeq = CurrentModSeq + 1;
 
     # InnoDB may only create trx_id on first write (which we just did)
     SET @LastBumpUserModSeqTrxId = IFNULL(CurrentTrxId, GetCurrentTrxId());
     SET @LastBumpModSeqOwnerValue = ModSeqOwnerValue;
     SET @LastBumpDataTableName = DataTableName;
   END IF;
 
   SELECT CurrentModSeq
   FROM UserModSeqs
   WHERE UserId = ModSeqOwnerValue
     AND TableName = DataTableName
   INTO NewModSeq;
 
   RETURN NewModSeq;
 END
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;That way we bump the MODSEQ for the given object and user if it’s a new transaction, but use the existing value if we’re in a transaction where we already bumped the MODSEQ.&lt;/p&gt;&lt;p&gt;Now if you try this by hand a bit, it all seems to work great.&lt;/p&gt;&lt;p&gt;But, if you start trying to write tests for this, you start noticing it doesn’t always work as expected. Multiple updates in quick succession don’t bump the modseq correctly.&lt;/p&gt;&lt;p&gt;And so down the rabbit hole you go.&lt;/p&gt;&lt;p&gt;The first thing you learn is that InnoDB has an optimisation where it won’t generate a transaction id until you actually perform either a write statement or a &lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt; (that’s the “InnoDB may only create trx_id on first write (which we just did)” comment in the above code).&lt;/p&gt;&lt;pre&gt;&lt;code&gt;mysql (root@127.0.0.1) [fastmail]&amp;gt; begin; select GetCurrentTrxId(); commit;
 Query OK, 0 rows affected (0.00 sec)
 
 +-------------------+
 | GetCurrentTrxId() |
 +-------------------+
 |              NULL |
 +-------------------+
 1 row in set (0.00 sec)
 &lt;/code&gt;&lt;/pre&gt;&lt;pre&gt;&lt;code&gt;mysql (root@127.0.0.1) [fastmail]&amp;gt; begin; select count(*) from Users where UserId=7 for update into @foo; select GetCurrentTrxId(); commit;
 Query OK, 0 rows affected (0.00 sec)
 
 Query OK, 1 row affected (0.00 sec)
 
 +-------------------+
 | GetCurrentTrxId() |
 +-------------------+
 |           4769402 |
 +-------------------+
 1 row in set (0.00 sec)
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Fine, but then you notice strange behavior like.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;mysql (root@127.0.0.1) [fastmail]&amp;gt; begin; select GetCurrentTrxId(); select count(*) from Users where UserId=7 for update into @foo; select GetCurrentTrxId(); commit;
 Query OK, 0 rows affected (0.00 sec)
 
 +-------------------+
 | GetCurrentTrxId() |
 +-------------------+
 |              NULL |
 +-------------------+
 1 row in set (0.00 sec)
 
 Query OK, 1 row affected (0.00 sec)
 
 +-------------------+
 | GetCurrentTrxId() |
 +-------------------+
 |              NULL |
 +-------------------+
 1 row in set (0.01 sec)
 
 Query OK, 0 rows affected (0.00 sec)
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;So even though a &lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt; should create a transaction id, the subsequent call to &lt;code&gt;GetCurrentTrxId()&lt;/code&gt; still didn’t return one.&lt;/p&gt;&lt;p&gt;But then if you try.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;mysql (root@127.0.0.1) [fastmail]&amp;gt; begin; select GetCurrentTrxId(); select sleep(1) into @foo; select count(*) from Users where UserId=7 for update into @foo; select GetCurrentTrxId(); commit;
 Query OK, 0 rows affected (0.00 sec)
 
 +-------------------+
 | GetCurrentTrxId() |
 +-------------------+
 |              NULL |
 +-------------------+
 1 row in set (0.00 sec)
 
 Query OK, 1 row affected (1.00 sec)
 
 Query OK, 1 row affected (0.00 sec)
 
 +-------------------+
 | GetCurrentTrxId() |
 +-------------------+
 |           4770035 |
 +-------------------+
 1 row in set (0.00 sec)
 
 Query OK, 0 rows affected (0.00 sec)
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;So putting a sleep between them, suddenly you do get a new transaction id.&lt;/p&gt;&lt;p&gt;Digging into MySQL code you end up finding&lt;/p&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/mysql/mysql-server/blob/61a3a1d8ef15512396b4c2af46e922a19bf2b174/storage/innobase/handler/i_s.cc#L751&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;i_s.cc#L751&lt;/a&gt;&lt;/p&gt;&lt;pre&gt;&lt;code&gt;/** Common function to fill any of the dynamic tables:
  INFORMATION_SCHEMA.innodb_trx
  @return 0 on success */
 static int trx_i_s_common_fill_table(
 ...
   /* update the cache */
   trx_i_s_cache_start_write(cache);
   trx_i_s_possibly_fetch_data_into_cache(cache);
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/mysql/mysql-server/blob/61a3a1d8ef15512396b4c2af46e922a19bf2b174/storage/innobase/trx/trx0i_s.cc#L817&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;trx0i_s.cc#L817&lt;/a&gt;&lt;/p&gt;&lt;pre&gt;&lt;code&gt;int trx_i_s_possibly_fetch_data_into_cache(
     trx_i_s_cache_t *cache) /*!&amp;lt; in/out: cache */
 {
   if (!can_cache_be_updated(cache)) {
     return (1);
   }
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/mysql/mysql-server/blob/61a3a1d8ef15512396b4c2af46e922a19bf2b174/storage/innobase/trx/trx0i_s.cc#L681&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;trx0i_s.cc#L681&lt;/a&gt;&lt;/p&gt;&lt;pre&gt;&lt;code&gt;static bool can_cache_be_updated(trx_i_s_cache_t *cache) /*!&amp;lt; in: cache */
 {
 ...
   /** The minimum time that a cache must not be updated after it has been
   read for the last time. We use this technique to ensure that SELECTs which
   join several INFORMATION SCHEMA tables read the same version of the cache. */
   constexpr std::chrono::milliseconds cache_min_idle_time{100};
 
   return std::chrono::steady_clock::now() - cache-&amp;gt;last_read.load() &amp;gt;
          cache_min_idle_time;
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;So the &lt;code&gt;information_schema.innodb_trx&lt;/code&gt; table is cached internally for 100ms, so trying to use it to fetch the transaction for the current connection is not guaranteed to be up to date. Ouch! This does not appear to be documented anywhere obvious that I could find.&lt;/p&gt;&lt;p&gt;Unfortunately I can’t see any way to &lt;em&gt;reliably&lt;/em&gt; find out if you’re in a transaction or way to identify the current transaction id.&lt;/p&gt;&lt;p&gt;In the end, I had to resort to something at the application level. Internally our DB library encourages you to use a guard object to start and end transactions.&lt;/p&gt;&lt;pre&gt;&lt;code&gt;my $committer = $dbh-&amp;gt;begin_work_auto();
 $dbh-&amp;gt;do(...);
 $committer-&amp;gt;commit();
 &lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The &lt;code&gt;-&amp;gt;begin_work_auto()&lt;/code&gt; call starts a transaction and returns a guard object. If the &lt;code&gt;$committer&lt;/code&gt; guard object is destroyed before the &lt;code&gt;-&amp;gt;commit()&lt;/code&gt; call, then it will automatically rollback the transaction in progress.&lt;/p&gt;&lt;p&gt;I added a small bit of code in &lt;code&gt;begin_work_auto()&lt;/code&gt; to do &lt;code&gt;$self-&amp;gt;do(&#39;SET @CurrentTrxId=?&#39;, {}, ++$count);&lt;/code&gt;, and then set it back to NULL when we commit/rollback. That simulates a guaranteed changing transaction id within each MySQL session as long as you use the &lt;code&gt;begin_work_auto()&lt;/code&gt; guard method which most code does when using transactions. Anything that doesn’t use &lt;code&gt;begin_work_auto()&lt;/code&gt; leaves &lt;code&gt;@CurrentTrxId&lt;/code&gt; as NULL.&lt;/p&gt;&lt;p&gt;The &lt;code&gt;BumpUserModSeq&lt;/code&gt; function used in the trigger checks the &lt;code&gt;@CurrentTrxId&lt;/code&gt; session variable to see if it’s NULL or has been incremented from it’s last value to decide whether to bump the MODSEQ or not.&lt;/p&gt;&lt;p&gt;So transactions that use &lt;code&gt;begin_work_auto()&lt;/code&gt; get optimised MODSEQ bumping (a single update to the modseq table for all rows added/updated/deleted for a user within the transaction), and everything else falls back to still working correctly but with a bit more overhead (an update to the modseq table for every single row added/updated/deleted).&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 4: Meet the team — Bek</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/meet-the-team-bek/' />
 			<id>https://www.fastmail.com/blog/meet-the-team-bek/</id>
 			<updated>2024-12-04T00:00:01Z</updated><author>
 				<name>The Fastmail Team</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the fourth post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/moving-house-new-datacentre/&quot;&gt;Dec 3: On moving house — bringing a new data centre online&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/mysql-innodb-trx-id/&quot;&gt;Dec 5: MySQL InnoDB innodb_trx is cached&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Meet Bek, our Head of People and Culture based in Melbourne.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Name:&lt;/strong&gt; Bek Fraser&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Role:&lt;/strong&gt; Head of People and Culture&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What do you work on?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Anything people related, including all our policies, processes, compliance. All the really boring stuff. Also, how to keep our team members happy.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;How long working at Fastmail, how did you get involved?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;10 months. I was looking for a role in an IT company. I heard of Fastmail many years ago when I first started my IT degree, and Fastmail was leading the email industry.&lt;/p&gt;&lt;p&gt;I’m Fastmail’s first people and culture hire. The exciting part was starting from a blank slate — it’s been a good opportunity to look at how to align our teams, which are global with many remote team members.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What’s a project you have worked on that you’re particularly proud of?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Implementation of a skill and career framework which has created equality for all our team members globally, and is allowing us to look at what skills we need for the future.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Who is somebody who inspires you?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Early on in life it was Mother Theresa, because she just wanted to help everybody — but she didn’t hesitate to break the mould. Something I learned being a female in IT, you never can hesitate to break the mould.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What are your favourite Fastmail features?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Memos! It means I can keep information about something without having to go out to a separate tool. It took me a while to get used to conversation mode, but I’ve found it useful to help my husband manage his time better — I’ve taught him how to use conversations and colour coding to keep track of things.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Other than Fastmail, what’s your favourite or most used piece of technology?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;My robot that mops and vacuums at the same time! It notices when it’s on carpet and lifts the mop. It saves me tons of time and I can vacuum the floor every day.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What are you listening to these days?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;John Farhnam — I did the backing for Age of Reason! You can see tiny little me in the music video. I also enjoy Guy Sebastian, Adele, that type of music. I also love musicals. I’m old school, I love music that wasn’t created by computer. My grandfather was a concert pianist.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What do you like to do outside of work?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;I like to challenge myself. I built outside stairs last week, now I need to repaint the whole deck to match them. I love working in the garden. I make resin chopping boards and sell them at the market.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What’s your favourite animal?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;My puppy dog. I love dogs, especially my puppy Baxter.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Any Fastmail staff you want to brag on?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;All of them. We’ve achieved a lot in the last 12 months, and everyone really lives the Fastmail culture.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;What do you like best about working at Fastmail?&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;The people and their commitment. It’s been great helping everyone find the type of growth that works best for them, and creating a culture at Fastmail that truly aligns with what we believe in.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 3: On moving house — bringing a new data centre online</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/moving-house-new-datacentre/' />
 			<id>https://www.fastmail.com/blog/moving-house-new-datacentre/</id>
 			<updated>2024-12-03T00:00:01Z</updated><author>
 				<name>Graeme Lee</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the third post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/throwback-security-confidentiality-integrity-availability/&quot;&gt;Dec 2: Thowback: security — confidentiality, integrity, and availability&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/meet-the-team-bek/&quot;&gt;Dec 4: Meet the team — Bek&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Who hasn’t moved house at some point in their lives? Whether we choose to rent or buy, board, or backpack, each relocation in premises has its own challenges which have to be considered.&lt;/p&gt;&lt;p&gt;You might be striking out, leaving the nest and finding your own dwelling. Or you may have been comfortably in the one location for many years, and now it’s time to move on. Maybe you live in a motor home? Then relocating is probably something you &lt;em&gt;just do&lt;/em&gt;. It’s different for everyone, and the decisions and circumstances are unique to each individual.&lt;/p&gt;&lt;p&gt;Well, for reasons, Fastmail decided it was time to relocate our DC’s. And it felt a lot like moving house. How? Well, here are some cross-overs.&lt;/p&gt;&lt;ul&gt; &lt;li&gt;How much space do we need? Usually the first question asked, I suppose. Do we have enough already? Are we wasting space? What if we need more? How do we expand? If the family has moved out, we can downsize. Or there might be another project on the way, and we need another room.&lt;/li&gt; &lt;li&gt;What’s the neighbourhood like? When house-hunting, we will consider how we might fit in with the community we are integrating into. We will look at services, schools, shopping facilities. Whatever we may prioritise, it will be unique to each one’s circumstances. When we look at data centres, we might consider their physical location. What about connectivity? Do we have specific requirements for any services? What are the facilities like when connecting, upgrading, or disconnecting?&lt;/li&gt; &lt;li&gt;What’s the access like? Is it miles from anywhere, or is it in the CBD? Are we prepared to commute, or do we want quick and easy access? Does it have a garage or is it street parking only? The same questions can be asked when choosing a new DC. We might appreciate the opportunity to get out and visit. Or do we want something close that has convenient parking.&lt;/li&gt; &lt;li&gt;Is the furniture we have going to work in our new abode? Maybe it’s all we need, and we’ll “make it fit”. Do we have the budget to refurbish? What can we throw away? What do we keep? We could go on-and-on here. But it’s important to accept that what may work for one person may not necessarily be the right fit for the next. The same goes for moving to a new data centre. You might need new hardware, so it’s time to find a new space and start fresh. Whatever. What you choose to bring and what you choose to leave behind is always your choice alone.&lt;/li&gt; &lt;/ul&gt;&lt;h2 id=&quot;timing-is-everything&quot; tabindex=&quot;-1&quot;&gt;Timing is everything&lt;/h2&gt;&lt;p&gt;How long will the move take? Maybe you’re moving across the continent, so it’s going to take a few days travel by road. Or it’s just around the corner, so logistically, the move looks like it will be fairly quick. But now, you’re moving data centres. The lights need to stay on! We now have a conundrum. Your servers might be fine with being turned off. But not your service!&lt;/p&gt;&lt;p&gt;How do you go about moving a live production system from one data centre to another with zero downtime as the goal? This is where your existing data integrity, resilience and backup strategies come in to play.&lt;/p&gt;&lt;p&gt;At Fastmail, data integrity is of utmost importance. It was time to take our integrity policy to task! We were about to stretch our network’s capabilities to the limit, break some old rules and make new ones, and find some new corner cases that we had not considered.&lt;/p&gt;&lt;p&gt;And you can’t just do “1 thing”, and move on to the next. A lot of things need to be thought out and prepared for before you can turn up at the doorstep and ask for the keys!&lt;/p&gt;&lt;p&gt;We did the tours. We shopped around. And we found a data centre that we felt good about. It was accessible, and had the right demographic of network carriers in it for our needs. Our floorplan was drawn up, cables were run, and the plumbing was good to go.&lt;/p&gt;&lt;h2 id=&quot;let-s-do-this&quot; tabindex=&quot;-1&quot;&gt;Let’s do this!&lt;/h2&gt;&lt;p&gt;New switching and routing hardware was delivered and installed. We needed some way to bridge the two data centres together, so we used &lt;a href=&quot;https://www.megaport.com/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Megaport&lt;/a&gt; to provide a private circuit between our sites so that we could transparently continue to provide services from our existing network connections without disrupting normal operations.&lt;/p&gt;&lt;p&gt;We decided to divide and conquer our data. We already have data replicated between our primary and backup DC’s. It made sense to employ a similar technique. We transferred the operational load onto the servers that were staying up, offlined our standby systems, put them on a truck, and shipped. Things were ok. But as time passed, we discovered that the load now on our remaing hosts was taxing them to their limits! Fortunately, the time-in-transit was only a few hours, and we were able to keep things running.&lt;/p&gt;&lt;p&gt;Once our standby hosts were in place, we began to re-sync them with our master systems. This was fairly straightforward. But we quickly discovered that we had not provisioned enough bandwidth between our DC’s to cater for the high intra-network load. Extra bandwidth on our DC cross-connect was provisioned, and things synced up nicely.&lt;/p&gt;&lt;p&gt;Because we experienced a higher than expected load on our existing servers, we had to re-think our redundancy strategy. We had already moved 2/3 of our compute, with 1/3 remaining (excluding our backup DC), and we felt that it wasn’t desirable to rely on 1/3 in the event of an emergency, so a rethink was in order. Our server pool was reconfigured to provide more compute and more resilience to our primary DC.&lt;/p&gt;&lt;p&gt;Finally, we brought the remainder of our servers across. This was possibly the most straight-forward part of the move. Our new DC was promoted to master, and the servers were offlined, shipped, and installed.&lt;/p&gt;&lt;p&gt;Happy days!&lt;/p&gt;&lt;h2 id=&quot;but-wait-there-s-more&quot; tabindex=&quot;-1&quot;&gt;But wait, there’s more!&lt;/h2&gt;&lt;p&gt;A lot more! TL;DR more (At this point, does ‘TL’ mean ‘Too Late!’?) It wasn’t plain sailing. We had some hurdles that had to be swiftly cleared so that we could continue with our migration. Cables didn’t arive. We ran out of certain types of SFP modules. We upgraded every NIC on every blade. And we managed to migrate our backup DC right on the heels of our primary!&lt;/p&gt;&lt;p&gt;I’m sure we could write another post or two about these things. But we are still happy with the outcome. Some of our tooling got a fresh look at under stress, and we were able to improve even further our backup and migration tools for both present and future use.&lt;/p&gt;&lt;p&gt;The whole global team stepped up for this, and as a result, we were successful. Our network has been revitalised for the future, and we are in a great position to grow and improve our backend to provide the excellent service our customers expect and deserve.&lt;/p&gt;&lt;p&gt;No emails were harmed or lost in the migration of our data centres!&lt;/p&gt;&lt;hr&gt;&lt;blockquote&gt; &lt;p&gt;Postscript by Bron: the whole team did amazing work here. While we didn’t lose any email, we did have some customer visible downtime unfortunately. This all happened in just a few weeks - shipping equipment from Seattle to Philly over a week first, then the crazy day where we were running fully split as we shipped half of New Jersey to Philly, then finally I was off at a conference while the remaining crew handled the move of the last parts from New Jersey to our new secondary location.&lt;/p&gt; &lt;p&gt;We identfied three major learning points from this move:&lt;/p&gt; &lt;ol&gt; &lt;li&gt;We hadn’t done enough testing of the states we were going to be in during the move. We believed things would work that turned out to not be able to handle the complete real-world load during the US daytime hourly spikes.&lt;/li&gt; &lt;li&gt;We didn’t actually have the capacity! We also had a Cyrus replication repair inefficiency which, along with a very inopportune IMAP server crash due to the extra load of running overloaded - caused days of scrambling to get things back into the right state. We didn’t lose any data, but we were running with lower redundancy than I wanted for longer than I wanted! We have ordered a bunch of new servers which were shipped just last week and will be up and running by the end of the year.&lt;/li&gt; &lt;li&gt;There was no single coordinator driving the entire process. We still need to be better at designating a single responsible person. With such a senior and self-directing team it wasn’t a disaster, but we could have been even more efficient with a bit more up-front planning.&lt;/li&gt; &lt;/ol&gt; &lt;p&gt;There were also some network hiccups that happened in the weeks immediately following; but we’re now in a really stable configuration, with more options for routing traffic than we’ve ever had before, so we can work around faults faster in future.&lt;/p&gt; &lt;/blockquote&gt;</content>
         </entry><entry>
             <title>Dec 2: Throwback: security — confidentiality, integrity, and availability</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/throwback-security-confidentiality-integrity-availability/' />
 			<id>https://www.fastmail.com/blog/throwback-security-confidentiality-integrity-availability/</id>
 			<updated>2024-12-02T00:00:01Z</updated><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the second post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The previous post was &lt;a href=&quot;/blog/mission-statement/&quot;&gt;Dec 1: Mission statement&lt;/a&gt;. The next post is &lt;a href=&quot;/blog/moving-house-new-datacentre/&quot;&gt;Dec 3: On moving house — bringing a new data centre online&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Throwback time! &lt;a href=&quot;/blog/security-confidentiality-integrity-and-availability/&quot;&gt;This was the post&lt;/a&gt; which inspired our first ever Advent series. All I’ve changed is putting an Oxford comma in the title.&lt;/p&gt;&lt;p&gt;Honestly, very little has changed since then. Even the Wikipedia link is still valid. The James Mickens paper has disappeared, Microsoft’s legendary commitment to backwards compatibility clearly doesn’t extend that far, so if you’re interested you’ll have to &lt;a href=&quot;https://scholar.harvard.edu/files/mickens/files/thisworldofours.pdf&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;get the file from Harvard&lt;/a&gt; (pdf). I strongly recommend reading anything he has written.&lt;/p&gt;&lt;h2 id=&quot;integrity&quot; tabindex=&quot;-1&quot;&gt;Integrity&lt;/h2&gt;&lt;p&gt;We did an &lt;a href=&quot;/blog/security-integrity/&quot;&gt;explicit post on integrity in 2014&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;I’m going to write a whole separate post in this series about how the data integrity checks we have added over the years have really held up. We haven’t had a major data loss in those 10 years, through many cases of equipment failure and the occasional human error. Designing with data resilience as a key goal has paid off.&lt;/p&gt;&lt;h2 id=&quot;availability&quot; tabindex=&quot;-1&quot;&gt;Availability&lt;/h2&gt;&lt;p&gt;We also wrote an &lt;a href=&quot;/blog/security-availability/&quot;&gt;explicit post on availability in 2014&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Sadly we haven’t been with NYI for a while. Miss those guys. We moved to New Jersey, then they sold that data centre to a new provider, and we were there for a while. Our current data centres are in Philadelphia and St Louis.&lt;/p&gt;&lt;p&gt;We did a major data centre move earlier this year. We will write in this series about the experience, some lessons learned about our preparedness, and how the move has improved our resilience against some of the risks out there. Unfortunately, it did lead to some higher levels of downtime during and immediately after the move as things settled.&lt;/p&gt;&lt;h2 id=&quot;confidentiality&quot; tabindex=&quot;-1&quot;&gt;Confidentiality&lt;/h2&gt;&lt;p&gt;Finally the first one everybody thinks about! &lt;a href=&quot;blog/security-confidentiality&quot;&gt;We wrote about confidentiality in 2014&lt;/a&gt; as well.&lt;/p&gt;&lt;p&gt;The reasoning here is still the same, though a few things have changed. Our data centre structure is a bit different, we handle the networking in-house now and use a single switching infrastructure with VLANs rather than airgapped networks. The legal framework has changed too, Australia now has a Cloud Act agreement with the USA, so we receive requests directly from the USA for investigations of serious crimes by non-Australians.&lt;/p&gt;&lt;p&gt;By far the biggest risk to confidentiality that didn’t exist 10 years ago to the same extent is AI training! It seems that the temptation is for services that are “freemium” to &lt;a href=&quot;https://www.linkedin.com/help/linkedin/answer/a5538339&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;use your&lt;/a&gt; &lt;a href=&quot;https://slack.com/intl/en-au/trust/data-management/privacy-principles&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;data for&lt;/a&gt; &lt;a href=&quot;https://www.cnet.com/tech/services-and-software/how-to-opt-out-of-instagram-and-facebook-using-your-posts-for-ai/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;training their&lt;/a&gt; &lt;a href=&quot;https://support.google.com/mail/answer/10079371&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;AI models&lt;/a&gt;, either without choice or by forcing you to find a &lt;a href=&quot;https://www.goodreads.com/quotes/40705-but-the-plans-were-on-display-on-display-i-eventually&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;well hidden&lt;/a&gt; way to opt out.&lt;/p&gt;&lt;p&gt;We aren’t doing any AI, and if we did it would be very carefully and with the only goal being to help our users get better insight into their email. We would use per-user training models, in the same way that we currently create per-user search indexes, ensuring that data is segregated and can’t leak across.&lt;/p&gt;&lt;p&gt;The great thing about having a paid product is that we don’t have split loyalties. Fastmail has been profitable every one of the past 10 years, with no outside investment. We expect to remain so into the future without having to “diversify” into shady shit. We sleep well knowing we run an ethical business, and we’re very grateful to the people who trust us with their email and pay us to keep providing them with the service.&lt;/p&gt;&lt;hr&gt;&lt;p&gt;In conclusion, Fastmail has exactly the same attitude to security that we had 10 years ago. It’s important. It’s not a marketing dot-point, it’s table stakes — you have to be secure if you want to be trusted with other people’s precious emails.&lt;/p&gt;&lt;p&gt;Other than the areas where we’re the ones creating new and better standards, we are cautious about adopting new technologies and the latest fads. This has served us very well over the years. Fastmail cares about real security, actionable changes that make things better. We don’t do security theatre. When we do something in the name of security, it’s because it makes a meaningful difference to your risk profile.&lt;/p&gt;&lt;p&gt;See you again tomorrow.&lt;/p&gt;</content>
         </entry><entry>
             <title>Dec 1: Mission statement</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/mission-statement/' />
 			<id>https://www.fastmail.com/blog/mission-statement/</id>
 			<updated>2024-12-01T00:00:01Z</updated><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;This is the first post in the &lt;a href=&quot;/blog/fastmail-advent-2024/&quot;&gt;Fastmail Advent 2024&lt;/a&gt; series. The next post is &lt;a href=&quot;/blog/throwback-security-confidentiality-integrity-availability/&quot;&gt;Dec 2: Thowback: security — confidentiality, integrity, and availability&lt;/a&gt;.&lt;/p&gt;&lt;p&gt;Mission statements have a bad reputation. For good reason. They’re generally somewhere between aspirational goal and corporate bullshit: don’t be evil, think different, enhance shareholder value…&lt;/p&gt;&lt;p&gt;So anyway. We decided to do one:&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;Make email better&lt;/p&gt; &lt;/blockquote&gt;&lt;p&gt;The very first page in our Notion workspace starts with that phrase, and then explains it:&lt;/p&gt;&lt;blockquote&gt; &lt;p&gt;Fastmail is a small company making a big difference.&lt;/p&gt; &lt;p&gt;We make email better for our customers by providing the email service that people are proud to pay for. And we make email better for the world by leading standards, open source, and advocacy work.&lt;/p&gt; &lt;/blockquote&gt;&lt;p&gt;People who use our service know that the first item is true. Our customers are very loyal, for good reason. Sure, you can get email for free, if free is your only consideration, but our interface is smooth and fast, we have real humans staffing our support desk, and they sit RIGHT next to us. I am less than 2 metres away from the closest support agent right now.&lt;/p&gt;&lt;p&gt;Fastmail has its mission statement, and I have my own personal mission statement as well: “keep email open”. We had fully ⅓ of our engineering staff at the &lt;a href=&quot;https://www.ietf.org/meeting/121/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;IETF meeting in Dublin&lt;/a&gt; a few weeks ago. We’re heavily involved in &lt;a href=&quot;https://www.ietf.org/about/introduction/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;developing the standards&lt;/a&gt; that will keep email as the number one social network in the world, and we’re widely respected in the email industry.&lt;/p&gt;&lt;p&gt;I firmly believe that email is the &lt;a href=&quot;https://www.fastmail.com/blog/email-is-your-electronic-memory/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;electronic memory&lt;/a&gt; that we all need in a world where online content is changed frequently. I routinely find emails from 10 or even 20 years ago to remind me what happened then, and what I thought about it.&lt;/p&gt;&lt;p&gt;So there it is — Fastmail’s mission statement. We strive to live up to it every day. I judge myself as CEO, and prioritise our work, around meeting it. Creating a great product, and leading the world in making email better for everyone. I hope it comes through in every interaction everyone has with us, and you — the reader — agree with me that it’s how you see Fastmail as well.&lt;/p&gt;&lt;p&gt;See you tomorrow for the next post!&lt;/p&gt;</content>
         </entry><entry>
             <title>Fastmail Advent 2024</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/fastmail-advent-2024/' />
 			<id>https://www.fastmail.com/blog/fastmail-advent-2024/</id>
 			<updated>2024-12-01T00:00:00Z</updated><author>
 				<name>Bron Gondwana</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;Hello everyone, and welcome to Fastmail’s Advent blog post series for 2024!&lt;/p&gt;&lt;p&gt;After an experiment with &lt;a href=&quot;/blog/fastmail-advent-2023-25-days-of-better-email/&quot;&gt;Mastodon posts last year&lt;/a&gt;, we’re back to our own platform, where we can do longer posts with multiple links again.&lt;/p&gt;&lt;p&gt;This is a special year for us: 10 years since our &lt;a href=&quot;/blog/fastmail-advent-2014/&quot;&gt;very first Advent blog post series&lt;/a&gt;, and 25 years since the founding of Fastmail. More importantly, it’s my own 20 year anniversary; I started with FastMail (as we spelled it back then) as a sysadmin/programmer in October 2004!&lt;/p&gt;&lt;p&gt;In this series, you’ll see some throwbacks to posts we made 10 years ago, discussing what’s changed and what’s still the same. You’ll meet some of our newer staff members. You’ll see some of the documents that describe what we do and, more importantly, why we do it. What makes Fastmail special and why after all this time we’re still doing email and still passionate about making email better for everyone.&lt;/p&gt;&lt;p&gt;So pop our RSS feed into your reader, follow us on the socials, or just come back and visit our blog every day between your coffee and your wordle — or whatever guilty pleasure you prefer. Let’s get started:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;a href=&quot;/blog/mission-statement/&quot;&gt;Dec 1: Mission statement&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/throwback-security-confidentiality-integrity-availability/&quot;&gt;Dec 2: Throwback: security — confidentiality, integrity, and availability&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/moving-house-new-datacentre/&quot;&gt;Dec 3: On moving house — bringing a new data centre online&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/meet-the-team-bek/&quot;&gt;Dec 4: Meet the team — Bek&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/mysql-innodb-trx-id/&quot;&gt;Dec 5: MySQL InnoDB innodb_trx is cached&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/twoskip-and-more/&quot;&gt;Dec 6: Twoskip and more&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/revision-of-core-email-specifications/&quot;&gt;Dec 7: Revision of the core email specifications&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/principles/&quot;&gt;Dec 8: Guiding principles&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/building-a-blog/&quot;&gt;Dec 9: Building a blog&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/sunsetting-pobox/&quot;&gt;Dec 10: Sunsetting Pobox&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/meet-the-team-marc/&quot;&gt;Dec 11: Meet the team—Marc&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/following-the-sun/&quot;&gt;Dec 12: Following the Sun&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/moving-fastmail-dns-to-knot/&quot;&gt;Dec 13: It’s knot DNS. There’s no way it’s DNS. It is DNS!&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/on-call-systems/&quot;&gt;Dec 14: On-call systems&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/platform-team-working-agreement/&quot;&gt;Dec 15: Platform Team working agreement&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/offline-in-beta/&quot;&gt;Dec 16: Offline support now in public beta&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/offline-architecture/&quot;&gt;Dec 17: Building offline: general architecture&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/offline-sync/&quot;&gt;Dec 18: Building offline: syncing changes back to the server&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/offline-mail-storage/&quot;&gt;Dec 19: Building offline: mail storage&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/how-fastmail-uses-fastmail/&quot;&gt;Dec 20: How Fastmail uses Fastmail!&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/fastmail-in-a-box/&quot;&gt;Dec 21: Fastmail in a box&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/why-we-use-our-own-hardware/&quot;&gt;Dec 22: Why we use our own hardware at Fastmail&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/ten-years-of-jmap/&quot;&gt;Dec 23: Ten years of JMAP&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;/blog/twenty-five-years-of-fastmail/&quot;&gt;Dec 24: Twenty five years of Fastmail&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt;</content>
         </entry><entry>
             <title>Introducing memos: stick private notes on your email</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/introducing-memos/' />
 			<id>https://www.fastmail.com/blog/introducing-memos/</id>
 			<updated>2024-08-29T00:00:00Z</updated><author>
 				<name>Neil Jenkins</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;Today we’re launching a new feature at Fastmail: memos.&lt;/p&gt;&lt;p&gt;Add a memo to an email to jot down private notes — remind yourself of tasks, record when you paid a bill, capture notes from a side conversation, or anything else!&lt;/p&gt;&lt;p&gt;Your memo will stay stuck to the top of the email, so you won’t forget. Want to change it? Just click and type, it saves automatically. It’s super simple, but surprisingly powerful.&lt;/p&gt;&lt;p&gt;Memos are private, only visible to you. They are not sent to anyone else in the conversation.&lt;/p&gt;&lt;p&gt;When you add a memo it will show up in the message list as well, so you can see at a glance the memos you’ve added to conversations in a label or folder. Here’s a quick look at it in action:&lt;/p&gt;&lt;div class=&quot;mt-xl&quot;&gt;&lt;video class=&quot;aspect-video relative rounded-lg shadow-card z-50&quot; playsinline muted controls width=&quot;2048&quot; height=&quot;1152&quot;&gt; &lt;source src=&quot;/assets/video/tour/Memos.webm&quot; type=&quot;video/webm; codecs=vp9,vorbis&quot;&gt; &lt;source src=&quot;/assets/video/tour/Memos.mp4&quot; type=&quot;video/mp4&quot;&gt; &lt;/video&gt;&lt;/div&gt;&lt;p&gt;We’ve integrated memos deeply into our search as well. Text in your memos will be matched against words you search for, just like with your messages. You can search for text specifically inside memos with the operator &lt;code&gt;memo:&amp;lt;text&amp;gt;&lt;/code&gt;. Or, want to just find all the messages with a memo? Search for &lt;code&gt;has:memo&lt;/code&gt;.&lt;/p&gt;&lt;p&gt;If you also use another email app to access your Fastmail account, such as Apple Mail or Thunderbird, you’ll still have access to your memos. You’ll find them in the Memos folder, as a reply to the message your memo is attached to.&lt;/p&gt;&lt;p&gt;Memos are available in Fastmail now. Please note, if you have turned conversation grouping off, you’ll need to &lt;a href=&quot;https://www.fastmail.help/hc/en-us/articles/1500001969861-Conversations#turnoff&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;turn it back on&lt;/a&gt; to use memos. In multi-user accounts, memos cannot be added to messages that are shared with you by another user; you can only add them to your own messages.&lt;/p&gt;&lt;p&gt;Not a Fastmail user yet? &lt;a href=&quot;/features/&quot;&gt;Discover all our powerful features&lt;/a&gt; to make you more productive, or &lt;a href=&quot;https://app.fastmail.com/signup/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;start your 30-day free trial today&lt;/a&gt;.&lt;/p&gt;</content>
         </entry><entry>
             <title>Introducing passkey support to Fastmail</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/introducing-passkeys/' />
 			<id>https://www.fastmail.com/blog/introducing-passkeys/</id>
 			<updated>2024-08-14T01:30:00Z</updated><author>
 				<name>Neil Jenkins</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;From today, we’re pleased to announce you can create passkeys for your Fastmail account, giving you a faster, more secure way to log in. Open the &lt;a href=&quot;https://app.fastmail.com/settings/security&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Privacy &amp;amp; Security&lt;/a&gt; settings in your account to get started, or read on to learn more about passkeys.&lt;/p&gt;&lt;h2 id=&quot;the-problem-with-passwords&quot; tabindex=&quot;-1&quot;&gt;The problem with passwords&lt;/h2&gt;&lt;p&gt;To understand why passkeys are useful, we should start by looking at the problems with passwords that we want to solve.&lt;/p&gt;&lt;p&gt;A password is a shared secret between you and the website you’re logging into. You tell the website who you are (your username/email address) and the secret only you know, the website checks it matches the secret you gave them before, and if it’s a match then you’re logged in.&lt;/p&gt;&lt;p&gt;So far, so good. The trouble is, people are terrible at using passwords.&lt;/p&gt;&lt;ul&gt; &lt;li&gt; &lt;p&gt;&lt;strong&gt;We’re not good at coming up with hard to guess passwords&lt;/strong&gt;. In security, we have the concept of entropy, which is a mathematical way of calculating how predictable something is. The higher the entropy, the less predictable, and the stronger the security. This is important to stop people from being able to guess your password. Unfortunately, when most people try to come up with a password, they choose something with low entropy, which makes it susceptible to “brute force” attacks, where a computer can try billions of passwords very quickly until it finds which one is yours.&lt;/p&gt; &lt;/li&gt; &lt;li&gt; &lt;p&gt;&lt;strong&gt;We’re not good at remembering lots of different passwords&lt;/strong&gt;. As we saw above, a truly secure password is something that’s unpredictable. And if it’s unpredictable, it’s probably hard to memorise. Memorising one strong password is probably doable, but using the same password at every site is a terrible idea from a security point of view: all it takes is one website to store its passwords insecurely and a hacker could now access all your accounts, anywhere on the internet!&lt;/p&gt; &lt;/li&gt; &lt;li&gt; &lt;p&gt;&lt;strong&gt;We’re not good at only giving our password to the right website&lt;/strong&gt;. You may have a phenomenal memory, and maybe &lt;a href=&quot;https://diceware.dmuth.org/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;roll dice to create a unique password with high entropy&lt;/a&gt; for every account you create. Unfortunately, this doesn’t help if you click a link to a phishing website and hand it straight over to an attacker. This is the biggest cause of stolen Fastmail accounts by a long way.&lt;/p&gt; &lt;/li&gt; &lt;/ul&gt;&lt;p&gt;Now, you may be thinking there’s already an answer to the above: password managers. And that’s absolutely right. If there’s one thing you can do to improve your security on the web, it’s use a password manager. A password manager:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Creates secure, high-entropy passwords for you.&lt;/li&gt; &lt;li&gt;Remembers them, so you don’t have to.&lt;/li&gt; &lt;li&gt;Will only automatically fill it in on the website you created the password. (But as autofill isn’t 100% reliable, users can be tricked into manually copying the password out of the manager and into a phishing site).&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;That mostly solves our problems! But if we’re using a password manager anyway, we can make it even more secure by storing passkeys in it instead of passwords. All modern password managers also support passkeys (there’s a built-in one on every device these days, or we recommend &lt;a href=&quot;https://1password.com&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;1Password&lt;/a&gt; for a good cross-platform password manager).&lt;/p&gt;&lt;h2 id=&quot;what-are-passkeys-and-why-are-they-better-than-passwords&quot; tabindex=&quot;-1&quot;&gt;What are passkeys, and why are they better than passwords?&lt;/h2&gt;&lt;p&gt;Instead of a password - which is a shared secret with the website - a passkey a is a super-secure cryptographic key. It uses public key cryptography, which means your password manager stores a key that can prove it’s you, while the website gets a different key that can only &lt;em&gt;verify&lt;/em&gt; this assertion. This has a number of advantages over passwords:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;&lt;strong&gt;It’s replay resistant&lt;/strong&gt;. A password is the same every time, so if an attacker can observe it being sent to the website, they can use it themselves. With passkeys, you sign a different random “challenge” from the website every time, so even if an attacker can intercept your network traffic, they can’t steal your passkey and log in as you.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;It’s database-leak resistant&lt;/strong&gt;. If a website’s password database gets hacked, there’s a risk the attacker could get your password. (Good websites will have &lt;a href=&quot;https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;hashed the passwords&lt;/a&gt; to slow the attackers down, but it’s still possible.) With passkeys, you have both a private key (that can create a signature to prove it’s you) and a public key (that can only verify the signature is real, but not create it). The website only ever has the public key, so even if this were stolen it couldn’t be used to access your account.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;It’s phishing proof&lt;/strong&gt;. Passkeys are strongly tied to the website they were created for. If you click a phishing link and end up on a malicious website, your passkey simply won’t appear, so you can’t get phished.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;It’s quicker and easier&lt;/strong&gt;. Browsers can provide secure, privacy-preserving APIs for integrating passkeys on a website to make logging in as easy as a single click.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;Your password manager handles all of this for you. When you want to log in to a website or app, you just use your fingerprint, face, or PIN to verify it’s you (this happens entirely locally - your biometrics never leave your device). The website then receives a digital signature proving you have the passkey, and you’ll be logged in.&lt;/p&gt;&lt;h2 id=&quot;how-do-i-log-in-on-a-new-device&quot; tabindex=&quot;-1&quot;&gt;How do I log in on a new device?&lt;/h2&gt;&lt;p&gt;Your passkey will be stored in your password manager. This will often automatically sync between all your devices. For example, if you use multiple Apple devices, the iCloud keychain will securely sync your passkeys to all of them.&lt;/p&gt;&lt;p&gt;If you don’t have the same password manager available on all your devices, you can also:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Use the passkey on your phone to log in on another device.&lt;/li&gt; &lt;li&gt;Create another passkey on the other device to make it easier to log in there going forwards.&lt;/li&gt; &lt;/ul&gt;&lt;p&gt;You can create as many passkeys for your account as you need; any of them can be used to log in. You can review and revoke the passkeys created for your account at any time from the &lt;a href=&quot;https://app.fastmail.com/settings/security&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;Privacy &amp;amp; Security&lt;/a&gt; settings.&lt;/p&gt;&lt;h2 id=&quot;how-does-this-interact-with-my-existing-password-what-if-i-have-two-step-verification&quot; tabindex=&quot;-1&quot;&gt;How does this interact with my existing password? What if I have two-step verification?&lt;/h2&gt;&lt;p&gt;Passkeys are an additional way to log in, not currently a replacement for passwords. If you create a passkey, you will still be able to log in with your password, just as you did before. If you have two-step verification, this will still be required when you use your password. Two-step verification is not required when you use your passkey, as this already has two factors:&lt;/p&gt;&lt;ul&gt; &lt;li&gt;Something you have (the passkey on your device).&lt;/li&gt; &lt;li&gt;Something you are or something you know (the touch/fingerprint or PIN your device requires to use it).&lt;/li&gt; &lt;/ul&gt;&lt;h2 id=&quot;the-state-of-passkeys-on-the-web-today&quot; tabindex=&quot;-1&quot;&gt;The state of passkeys on the web today&lt;/h2&gt;&lt;p&gt;We’re a big believer in passkeys, and hope they are the future for authentication. Phishing is a blight on the internet, and this is our best hope of eliminating it forever. We’ve actually had passkeys implemented for over a year at Fastmail, but we’ve been waiting for the ecosystem to mature before releasing it. While it’s come along way, there are still some rough edges.&lt;/p&gt;&lt;p&gt;The biggest issue is the integration of third-party password managers in browsers is still not as polished as it should be. Browser vendors are working on an API for passkey integration but it’s not widely supported yet, so instead the password managers are injecting JavaScript into the page to overwrite the native &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/Navigator/credentials&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;&lt;code&gt;navigator.credentials&lt;/code&gt;&lt;/a&gt; object. The result is more fragile - for example, we ran into a bug with 1Password in Firefox, where the &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/AuthenticatorAttestationResponse/getAuthenticatorData&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;getAuthenticatorData&lt;/a&gt; method returned an object that throws an error whenever you try to access it, due to extension security controls. It also can result in confusing, competing UIs for the user - the password manager intercepts the calls and presents its own UI, but if you cancel this you might get a browser dialog, which might itself hand over to a system dialog, all with their own style. Integration with the browser’s built-in webauthn support will provide a more seamless experience and allow the user to have passkeys from multiple password managers all available in one place.&lt;/p&gt;&lt;p&gt;Even the native browser APIs can have their quirks, though. For example, Safari requires the &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;webauthn API&lt;/a&gt; call to be made in an event directly triggered by the user, like a click. However, the webauthn API requires a challenge, which requires an asynchronous HTTP call to the server to fetch it. So if you trigger this request on the user click, then call the webauthn API with the response, it will fail. Annoying.&lt;/p&gt;&lt;p&gt;The &lt;a href=&quot;https://fidoalliance.org/specifications-credential-exchange-specifications/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;standard for people to export and import passkeys&lt;/a&gt; is not yet complete, although it’s getting close. This will allow you to move between password managers without having to recreate your passkeys on every site. Once this is complete and widely adopted it will remove the risk of lock-in, which we believe is currently hampering adoption.&lt;/p&gt;&lt;p&gt;The good news is these problems are all solvable, and the ongoing work shows there is strong industry desire to do so. Despite the current minor issues, we believe the time is now right to start adopting passkeys, and we hope to see their continued success.&lt;/p&gt;</content>
         </entry><entry>
             <title>Get the best in email: how Domenic uses Fastmail</title>
 			<link rel='alternate' type='text/html' href='https://www.fastmail.com/blog/get-the-best-in-email-how-domenic-uses-fastmail/' />
 			<id>https://www.fastmail.com/blog/get-the-best-in-email-how-domenic-uses-fastmail/</id>
 			<updated>2024-06-04T16:00:00Z</updated><author>
 				<name>The Fastmail Team</name>
 			</author><content xml:lang='en' type='html'>&lt;p&gt;Fastmail makes it easy to organize and prioritize the messages that are most important to you. Domenic, a software developer, uses Fastmail personal domains, aliases, folders, labels, and more. Read on to learn how Domenic gets the most out of his Fastmail inbox and why he switched from Gmail.&lt;/p&gt;&lt;h3 id=&quot;what-are-your-favorite-fastmail-features-and-what-purpose-do-you-use-them-for&quot; tabindex=&quot;-1&quot;&gt;&lt;strong&gt;What are your favorite Fastmail features, and what purpose do you use them for?&lt;/strong&gt;&lt;/h3&gt;&lt;p&gt;My favorite Fastmail feature is definitely personal domains. The ability to use my own domain names and create as many aliases for as many domains as I want makes my workflow so efficient.&lt;/p&gt;&lt;p&gt;I chose the Dark theme, and using the customizable colors and buttons is incredible. It’s great that Fastmail allows me to choose how I want my email client to look (and it does look beautiful 😊).&lt;/p&gt;&lt;p&gt;Email and calendaring are done well and feel very easy to use. It was a breeze to import all of my emails from my Gmail and set up a connection to send and receive emails via the Fastmail interface.&lt;/p&gt;&lt;h3 id=&quot;how-do-you-maintain-your-workflow&quot; tabindex=&quot;-1&quot;&gt;&lt;strong&gt;How do you maintain your workflow?&lt;/strong&gt;&lt;/h3&gt;&lt;p&gt;I use Fastmail Rules to help categorize my incoming emails into a folder structure. The rules engine is powerful enough to perform all the options I need: move emails, delete emails, set follow-up flags, etc.&lt;/p&gt;&lt;p&gt;I pin all the important emails that I need to take further action on. Those emails appear at the top of my Inbox until I have dealt with them. This allows me to immediately see what items I need to follow up on, or messages that are awaiting a response.&lt;/p&gt;&lt;hr&gt;&lt;p&gt;Fastmail is email for people created by people. With endless possibilities for personalizing your workflow and an expert support team available at any time, it’s easy to get better email today. &lt;a href=&quot;/try-it/&quot;&gt;Try Fastmail free for 30 days&lt;/a&gt;!&lt;/p&gt;</content>
         </entry>
 </feed>