As promised, this is the email describing the details of the SmarterMail outage that occured on Thursday, August 10th 2017. While we do not have a line-by-line breakdown of the exact root cause, we do have details about what caused the spool to stall as well as both a temporary and coming permanent fix.
The issue begins when SmarterMail receives multiple badly formed emails in the spool at nearly the same time. The badly formed emails cause the spam-checking thread to stall until a timeout has been reached. Normally, this is not an issue, SmarterMail simply ignores the issue and continues forward, however, when multiple messages stall together, this creates an unhandled exception and does not free the threads for each message. SmarterMail will attempt to continue to re-scan the messages using additional threads and the cycle continues until SmarterMail has exhausted all available spool-processing threads.
This ultimately causes all emails that have been sent or received to sit in the spool directory waiting for the spool to complete its work and hand them off to the next step, either delivery to the mailbox or processed by SMTP for remote delivery. We do actively monitor the spool size as well as numerous other aspects of SmarterMail as reported by the SmarterMail API. However, since the spool system was no longer responding, it was not properly reporting the correct number of emails in spool.
SmarterTools has a fix coming that will make sure this never occurs again, as well as has provided a temporary fix, which we have applied. We have also added two additional probes to our PRTG monitoring system to provide the file count for the SmarterMail spool directory using the Win32 API as well as the SmarterTools API and one to monitor the number of threads used by the SmarterMail service process according to the Win32 API.
We appreciate your patience and understanding while we work to address this. If you have any questions, please open a ticket with our technical support department and we will be glad to answer any questions!