Back
November 19, 2020

Report Regarding the November 16, 2020 Email Incident

This report was written by Coil's founder and CEO, Stefan Thomas, with input from the Coil team.

A few days ago, we sent out a routine account update email which unintentionally disclosed email addresses from other users in the To field. We immediately took steps to diagnose the issue and determine what we could do to mitigate the impact. However, once an email is sent, there isn't much that can be done to take it back. Once we identified and confirmed the root cause, we sent a follow-up email apologizing to our users. We then set out to further analyze what happened and develop a plan to prevent anything like this from ever happening again.

I'm writing this post to document the exact mistakes we made leading up to the incident. Obviously, this doesn't change the outcome, but we believe we owe the community an explanation, and perhaps seeing our analysis and response to an incident like this will help those of you who run an online service avoid making the same mistakes.

Overview

On November 16, 2020 at 6:02 pm PT using an internal admin tool we sent out an email notifying all Coil Users with verified emails about the updates which were made to our Terms of Service & Privacy Policy. This email exposed other users' email addresses in the To field, meaning the email recipients were able to see each other's email addresses within the same batch.

Root Cause

We made a seemingly innocuous change to our custom mailing list service used for account emails. This change triggered a different code path in our email provider's API which unexpectedly changed the handling of the To field. Because we only expected content changes—not a change in the email metadata—our testing process did not require a review of the To field.

Detailed Analysis

In order to notify users via email about important account updates such as changes to our Terms of Service, we use a dedicated service which interfaces with the API provided by our email provider (Mailgun). We had used this service before without issue.

These previous Terms of Service & Privacy Policy email updates included a personalized greeting. This personalization was accomplished by using Recipient Variables, which is a Mailgun feature, to enable dynamic content in the body of the email such as: Hi %recipient.firstName%, where %recipient.firstName% would be replaced with the user's given name.

However, we recently changed our signup flow to no longer ask users for their name. This change was done both to simplify the signup process and to reduce the amount of personal information we collect. Because we no longer ask for a first name, we removed the personalized greeting and replaced it with a simpler generic greeting: Hi,

The following code changes were done to accomplish this:

1. Update to the greeting


2. Removal of the recipient-variables

The change was then put through our code review process. In addition to the original author, two other Coil engineers and one other team member reviewed the change before it was approved. It was then tested against a local database containing fictional test accounts. During the testing of the emails we were focused on testing the formatting and content of the emails and did not notice that there were multiple email addresses in the To field.

Since the formatting was correct and links were working, we moved forward with sending the emails to users notifying them of the Terms of Service and Privacy Policy changes. When this notification went out the extremely unfortunate impact was that users were able to see the email addresses of other users in the same batch.

The logic in the above code calls a function emailAllUsers and passes in the emails object. That function then calls the /messages endpoint from Mailgun's API. This endpoint is used for both normal emails and batch emails. We did not expect the absence of the recipient-variables parameter to affect anything other than disabling the templating feature. However, unbeknownst to us, Mailgun's API documentation states:

"Warning: It is important when using Batch Sending to also use Recipient Variables. This tells Mailgun to send each recipient an individual email with only their email in the to field. If they are not used, all recipients’ email addresses will show up in the to field for each recipient."

When we removed the Recipient Variables we implicitly and unexpectedly changed the semantics of the To field which caused all of the email addresses to be populated within each batch. We want to make it perfectly clear that we do not blame Mailgun for our incorrect usage of their API.

Our Response

  • Fixed the immediate bug by reintroducing a dummy recipient-variables object.
  • Sent an apology to all impacted users on November 16, 2020 at 8:31 pm PT.
  • Deprecated the custom mailing list service. Future account updates such as this will be sent via a specialized mailing list provider.
  • Changed our code review checklist to include an API docs review for any changes to external API calls.
  • Improved our testing plans for mass mailings to include a review of email metadata.
  • Increased monitoring for failed account logins and similar indicators of malicious activity.

These are the immediate steps taken but we will continue to analyze and learn from this incident and implement further improvements to our processes.

Questions

Why did we use the To field instead of the Bcc field?

  • Our mail provider's API does not allow sending emails without a To field.
  • Our original implementation using the To field was in fact following best practices for our mail provider's API as per their documentation. However, when removing the recipient-variables parameter, we failed to realize the implications of that change.

Was this not tested?

  • The emails were tested but with a limited group in which no one spotted the issue.
  • The functionality had been tested rigorously upon initial deployment and was used for previous Terms of Service and Privacy Policy update emails without issue. The code change was incorrectly classified as a content change which did not trigger a review of the email sending mechanism itself.

What is Coil doing to prevent this from happening in the future?

  • We have removed the ability to send account emails to all users via the internal admin tool.
  • We will no longer use our mail provider's API to send general account updates to all users.
  • For future account update emails and other mass mailings we will use a specialized mailing list provider rather than a custom service.

Did you notify all the users whose emails had been leaked?

  • Yes, this was done via an apology email on November 16, 2020 at 8:31 pm PT.

What went right?

Aside from the mistakes we made, there are some things that went right.

  • The employee responsible immediately noticed what happened, and reported the issue. There was no delay, finger-pointing, or attempts to conceal culpability. It's extremely important to have a culture where employees are accountable for their mistakes, but also confident to report issues right away and not hold back any information.
  • We had emergency communication processes in place which allowed us to get all of the relevant engineers and leadership members onto a conference call within minutes of the incident.
  • As a standard practice, Coil minimizes data collection, access, and use. As an example, we stopped collecting users' real names, which might otherwise have been part of the To field and could have resulted in additional information being unintentionally disclosed.
  • This incident underscores that human error is always a risk and reducing data collection is an important tool for reducing the risk for our users and for Coil as an organization.
  • Other examples of this policy include:
  • Not having access to users' full credit/debit card information which is stored by our payment processor (Stripe).

Closing thoughts

I'd like to once again offer my deepest apologies to our users. I let you down. Our entire team and I are committed to making sure we never let anything like this happen again.

Next, I'd like to thank you for your understanding. Many of you have reached out to us to express your empathy and support and that means a lot to us. Thank you!

Finally, I'd like to say that I hope that our mistake does not reflect poorly on the Web Monetization community overall. Web Monetization is a proposed open standard and there are many websites that support it, but that does not automatically mean those sites are endorsing Coil. This incident only underscores the need for more Web Monetization providers.

How to contact us

If you require any assistance or help, please contact us at support@coil.com.

Go to top