On Saturday, March 17th at 19:05 CET/14:05 EST, an issue occurred which caused Hotjar to stop collecting data from a number of web visitors on sites with Hotjar installed. The issue also inadvertently triggered browser warnings for a much smaller portion of web visitors who had anti-virus suites installed with HTTPS Scanning enabled. This issue was fully resolved by Monday, March 19th 13:00 CET/08:00 EST and occurred due to misconfiguration caused by a human error on our end.

What effect did this issue have on my website?

If you have a website with Hotjar installed, the issue affected your website in two ways:

1. Depending on your visitors’ browser and their internet security settings, Hotjar may have stopped collecting data from your web visitors. 
   
   
2. A small percentage of web visitors who had an antivirus suite installed (ESET NOD32, Avast or AVG) with HTTPS Scanning enabled may have seen a message in their browser warning them about a revoked website certificate. This message would have appeared as a small unobtrusive message in their browser with minimal impact to their experience on your website. An example of the message is shown below.
   
   Note that any web visitors without an antivirus suite or without HTTPS Scanning enabled would NOT have been impacted in any way.
   
   

An example of a warning shown by ESET Smart Security

How many of my web visitors were affected?

The message shown above only appeared for web visitors with ESET NOD32 Antivirus, Avast Antivirus or AVG Antivirus installed with the HTTPS Scanning feature turned on. 

All other users were not impacted in any way.

How did this issue occur?

The issue was caused by a revoked security certificate for the Hotjar.com domain. During a planned security improvement, our team inadvertently triggered a series of events which led to our domain security certificate being unintentionally revoked by our security provider.

For a full technical explanation, please see our In-Depth Technical Explanation below.

Why was the issue not immediately detected?

While our engineering team did have certificate monitoring in place, this was only set up to check exclusively for certificate expirations, not revocations. As a result, our team was not immediately alerted when the issue first occurred.

What are we doing to avoid the incident happening again?

The team carried out a full investigation and post-mortem to establish what processes and monitoring we need to set up immediately to avoid a repeat of the incident.

Here’s what we’ve already done:

• We’ve set up SSL revocation monitoring for all our services.

Here’s what we’re planning on doing:

• We’re improving our incident handling procedures to ensure we can tackle future issues more efficiently.

In-depth technical explanation

Below is a timeline explaining the series of events which led to the issue happening.

• On March 15th 19:05 CET/14:05 EST, our engineering team issued a new SSL certificate for *.hotjar.com to provide certificate transparency in a way suggested by our SSL provider. After testing the certificate, the team realised it was not signed with a X.509v3 extension and therefore wouldn’t provide certificate transparency. As a result, it was decided not to install the new certificate.
• On March 17th 19:05 CET/14:05 EST, our active certificate was added to our providers certificate revocation list. This happened because of a miscommunication between us and the provider, and we were not aware that the active certificate would be revoked.
• At this point in time new certificate revocation lists started to be propagated to browsers, operating systems and internet security software. It was not possible to establish exactly how many people got the updated revocation list since the change was not immediate.
• On March 19th 10:37 CET/05:37 EST, we discovered the issue and it was soon confirmed by our engineering team.
• As a first step, the team immediately tried to get a new certificate to work. When that failed, it was decided to switch to a new certificate provider, Let's Encrypt. At 13:00 CET/08:00 EST, we had new certificates in place everywhere and the issue was confirmed to be solved.
• On March 21st 17:30 CET/12:30 EST, we discovered a set of Hotjar domains which hadn't been updated to using the new certificate. The affected domains were hotjar.com (without "www") and assets.hotjar.com. This was fixed within minutes of discovery.