ASP1 Smartermail POP3 Bug and Outage (resolved)

We are aware of an ongoing issue with stability in smartermail that started around December 23rd.  This was due to a bug related to a critical update they released and pushed out to the server automatically.  We reported the issue immediately and they made one fix over Christmas which fixed many of the issues yet some do remain.  Below are known issues we are aware of and have reported.  This issue is affecting thousands of users on the server farm itself so we are sure it is getting their attention.  Its scope is beyond just our customers.  While some customers are not seeing issues at all, many are.:

- POP3 mailbox users are getting errors when checking mailboxes with Outlook (server updated / resolved)
- Users getting server emails that state the message was removed by the server administrator (resolved)
- Webmail users have difficulty loading some messages (resolved)
- Webmail instability (resolved)

Potential Intermediate Resolutions:
1) for the time being use webmail to check email, we expect an updated release from Smartermail soon that should address this issue.
2) some customers report that logging into webmail and deleting messages from the Inbox has corrected the issue for them.  Also consider to toggle the "delete messages off the server" option in your outlook client to "off".

Updates:
12/27 3pm:
We are aware that many services on the server are up and down at this time.  Smartermail is working on the server which may be causing instability.  Our latest information from SmarterMail and the datacenter is that they are working on a patch for the current issues.  We had expected it to be released yesterday.  That is all the information we have at this time unfortunately as we allow them to do their work.

12/27 5:45pm:
Latest update from our datacenter techs: "The issue is being investigated by SmarterTools developers. We are hoping to have this fixed by this evening."

12/28 2am:
As a final update this evening it seems that the webmail platform has been re-established and is stable from our end.  SMTP services are active and mail should be able to be sent and received via webmail during this time.  From what we can tell the Denver datacenter/Smartermail Engineers still have POP and IMAP services offline we suspect working on mailbox repairs.

12/28 8am:
This morning the SNMP systems indicate that all email services are back online.  So far we have not had an official update regarding the outcome of the update to the server but it is encouraging to see the server responding to requests on all protocols including POP3.  Will continue monitoring the situation.

12/28 11:30am:
We recieved the following message from the Datacenter:  "
The issues with POP, IMAP and the others have been resolved. There may still be a few minor hiccups here and there, but the major issues have been ironed out. Let us know if you are experiencing any other issues.

12/28 7pm:
Update from datacenter:  
Currently POP3 is running as it should be, note that the issues with SM haven't been 100% ironed out, we are at about 80% right now, so any hiccups could be caused by that last little bit as we transition over to the newer codebase and storage systems.
We expect within the next business week, SM should be back to 99% with only minor issues.

12/28 9:15pm:
We have confirmed that clients running outdated verisons of Outlook such as v2003-2007 may not work as they should.  We believe this issue will be addressed in an update this week.  Till then users should consider using Webmail or testing if an upgraded version helps.

12/30 11pm:
The datacenter alerted us that SmarterMail Support staff have made a recent update to the server(s) that should have the POP3 errors fixed.  We will check with some of you to determine if any issues remain.

12/31:  Generally speaking all feedback indicates that all issues are now resolved.