Why Was marblehead.org Down for Hours Yesterday?

Looking to pay your tax bill? Find links for School Committee meetings? Email any town employee? For part of yesterday, accessing these or any other matters on the town website was impossible. www.marblehead.org's webpage was down for several hours with no explanation provided to town residents via a robocall, social media post, or other communication. Emails to town employees came back undeliverable during these hours.

Last evening Thatcher Kezer, Marblehead’s Town Administrator, confirmed that Marblehead Beacon’s research suggesting that the Town had failed to renew its domain name, www.marblehead.org, was correct. The reason, Kezer said, was that the bills and notices regarding the renewal of the town page had been sent to an email address of an employee no longer employed by the Town of Marblehead. Asked who this employee was and why that email address had not been forwarded or shut down, Kezer stated, “I don’t have that information.”

As noted in our prior reporting regarding an independently performed town audit that had highlighted multiple problems with processes in place for off-boarding of employees and their email addresses, this sort of oversight was foreseeable. Asked about how the matter of thousands of vendor bills, emails regarding town business, and other potentially critical matters would be handled when employees depart, Kezer said that he is instituting a new process. “We will have general email boxes monitored by multiple people in each department,” he said, which will prevent, for example, invoices from going to employees who are no longer with the Town. 

Kezer–who has only been on the job since this past summer–appears to have inherited systemic problems that are sometimes exposed in a less-than-optimal way, as when an entire town’s website and all its associated emails suddenly become unavailable. When a webpage goes down, a variety of things may be happening, including a glitch on the server side, a problem with the host of the website, malware hijinks, or a failure by the owner of the domain to pay to renew the domain or web hosting subscription. Nearly all websites in the world–large and small–have had circumstances outside their own control cause site disruptions. But the reason for yesterday’s hours-long outage was not beyond the Town’s control; it was a mistake that arguably has exposed a deeper problem first noted in the 2021 audit about which Marblehead Beacon reported back in June. 

Our original outreach to Kezer yesterday morning shortly before 9:00 AM via email resulted in a bounced-back message landing in our spam folder with the notation, “unroutable address.” This appears to have been the case with all town email addresses during the hours-long outage, suggesting that all town business involving anything to do with www.marblehead.org was unsuccessful, and some outside vendors or others may be unaware that messages did not arrive at their intended destinations. 

Initially, Marblehead Beacon informed Kezer that our independent research had suggested a reason for the website’s failure, but that we awaited an opportunity to speak with him to receive confirmation. Kezer inquired as to what our research had uncovered, and was advised that publicly-available online evidence showed that the Town’s domain had expired on October 25, 2022, sometime in the afternoon, and had been renewed in the middle of the night on October 26th, likely upon town officials noticing that the website had gone down. Additionally, the status of www.marblehead.org was in a “Client Hold,” defined by the non-profit global internet protocol (IP) address repository–Internet Corporation for Assigned Names and Numbers (ICANN)–as follows: “This status code tells your domain’s registry to not activate your domain in the DNS and as a consequence, it will not resolve. It is an uncommon status that is usually enacted during legal disputes, non-payment, or when your domain is subject to deletion.” 

Additionally, we advised Kezer that we’d communicated with Civic Plus, the host of the town’s website (and those of hundreds of other municipalities across the country), to see if perhaps it was experiencing problems on its end that would shed more light on the failure of marblehead.org. This possibility appeared unlikely, though, as our call to the company revealed, according to a member of its technical support team, that there had been no problem on its end, as “there was no server outage,” and that meant, according to the individual, that the issue was most likely “with Marblehead’s domain.” Kezer confirmed our findings regarding the domain's expiration and non-payment.

Resolving future instances in which email communications sent to former employees are relegated to some unknown location in cyberspace rather than landing in any current employee’s mailbox, Kezer admitted, will take time. As part of its migration to newer software, he says, “we will be doing house cleaning of old email addresses and things will self correct over time, one system at a time.” 

Asked about the Town’s failure to communicate the website shutdown yesterday via social media or robocall, Kezer said, “we will not use robocalls for that purpose; it’s for emergencies only.” He additionally stated that the page had “only been down for two hours,” but when asked if he meant “two business hours,” given that our research had reflected the domain’s expiration late on October 25th, he said, “I don’t have that level of detail” and noted that he’d learned about the issue at approximately 8:00 AM yesterday. 

In the future, in terms of communicating issues such as the Town website being down, Kezer said social media would be used “depending on the expected duration and judgment of severity.” In this case, he stated, “it was temporary and had a known fix,” and he did not believe that needed to be communicated to the Town. 

Kezer said the domain www.marblehead.org had been renewed for a one-year period yesterday.