PidgeMe - Welcome!

2 new in last 7 days

10/03/2010
Brief Outage [Resolved]
20:36 - 20:42
Brief outage caused by 8 servers all appearing to loose power at the same time despite being on battery/generator backups in case of failure. Currently investigating with datacenter the exact cause.

20:55
One of the storage servers isn't responding, some of you will be unable to login at the current time as a result. Should have it online in a few minutes.

20:56
Storage remounted, service fully online. Still determining original cause of the outage.

22:57
Datacenter has confirmed that a faulty power breaker tripped and had to be replaced with an on-site spare. Power was down for under 5 minutes but it took a few further minutes for the servers to boot back up. As you know we have been investigating A/B power feeds to prevent any single power fault from taking down the service.

04/03/2010
Datacenter Power Failure [Resolved]
19:37 - Power circuit failed in the datacenter, knocking out 3 servers. Service was slow as two of the servers were webservers, leaving web access restricted in speed. Client access and inbound email are both unaffected as no mail servers were on the failed circuit.

20:07 - All affected servers back online, web access stabilising across the webservers. Service may be a little slow for the next 10-15 minutes as users online are redistributed over available webservers.

03/03/2010
Temporary access issues
There was a temporary problem with access this morning between 9.45am and 10.15am due to a database server issue.

This has now been resolved.

10:25am - Updated, issue has been found and fixed. If you have any problems, try resending your email.

03/03/2010
Thursday Emails Redelivery +Service Maintenance Notice [COMPLETED]
To those members concerned about the Thursday redelivery please be advised that this is now scheduled to happen tonight from 10pm. Other members please be aware that we are scheduling this off-peak because during this time there will be some delays on inbound mail as we process about 30,000 emails through the system. We will update this post when this has completed. 22:15 - this work is starting now. Update in approx 45 minutes. 11:05 - Mail redelivery is under way. This should hopefully complete within 1-2 hours. We will give a progress update at midnight. 12:05 - Redelivery is progressing well and should not take too much longer. Next update within 30 minutes. 00:30 - Restoration complete and full email flow restored. The held mail backlog will be cleared shortly. 01:00 - Mail backlog cleared and all operating as normal.

01/03/2010
Attachment Issues (Resolved)
There was an issue with not being able to send attachments over the weekend. This was caused by new attachments being unable to be written while sending. The problem was fixed at around 7pm Sunday evening.

28/02/2010
Mail between Saturday 20th - Thursday 25th
A small portion of members will have noticed that emails are not showing up between the 20th and 25th. These have not been deleted but need to be resynced with your account following last nights server fault. This has been happening throughout the day but may take some time. Please bear with us. We will let you know when all accounts have finished syncing.

[Update]

The bulk of the resync is complete. Some users may find messages not present from Thursday, these will be resyncing Monday morning.

26/02/2010
Temporary Access Problems [Resolved]
17:15 - Dear Members, there is a temporary mail access issue we are looking into now. Please bear with us. 17:30 - Identified the problem - working to rectify. Next update within 15 minutes. 17:45 - Subsystem is back online and we are connecting it to the front end now to restore mail access. Next update within 15 minutes. 18:00 - Service restored for about 75% of accounts. Working on the others now and will update again within 15 minutes. 18:15 - servers back online for all users. 18:20 - sorry guys - spoke too soon. One more server to go. 18:55 - sorry for delay updating this. We're having to resync the server so I'm afraid we're waiting on that to complete. Will post again within 20 minutes. 19:40 - still waiting on the sync I'm afraid. Another update in 20 mins or so. 20:20 - last step should be a final reboot. Updating again soon. 20:25 - Servers are up and we are doing final checks before pushing live. 20:30 - we have one server live for mail access but this may work very slowly. Bringing second into cluster shortly. 20:45 - second server coming up. First server coping fine and service is online. Unless there are other issues no further posts to come. Thanks for bearing with us on this.

23/02/2010
Outbound Email Issue [Resolved]
23:30 - Some of you will have found outbound emails to time out when sending through webmail. One of the web servers was affected causing this to happen. This was resolved this evening and an extra monitor activated to provide automatic recovery for this issue within 5 minutes of it being detected.

22/02/2010
Mail Filters and POP Downloads
21:15 - Dear Members, those of you who have set up mail filter rules are having these restored now. They were not done as part of the main mail data restoration. They should be back in place very shortly but let us know if they are still missing on Tuesday morning. We've also had to reset some mail cache data on the server to ensure that all old mail showed up in accounts. This may result in those members using POP to re-download messages. Unfortunately there was no easy way around this and it was a necessary step to ensure ALL your email was downloaded.

22/02/2010
Inbound Email Delays (Resolved), SMTP issues (Resolved) Secure SMTP Issues (Resolved) Helpdesk Issues (Resolved)
There were some issues with inbound emails being delayed, which has been fixed. Any delayed emails will be delivered to your account within the next couple of hours.Sending client (SMTP) issues are still ongoing and we are continuing to investigate. Unfortunately we are still unable to respond to help tickets as a result, but will reply as soon as we are able. In the meantime, we will continue to post frequent updates on the service news to ensure regular communication until this is resolved.
10.43am - A temporary email account has been created in order to respond promptly to helpdesk requests.
11.31am - Only secure (SSL/TLS) client SMTP remains offline. Standard SMTP is now fully functional from any client. As a temporary workaround, if you are using SSL/TLS you can turn it off to use SMTP sending.
11.47am - We are now able to respond via the helpdesk. Secure SMTP client sending has been fixed. Everything should return to normal.

Director's Report: DEBRIEF PART 2: Technical Details
and the Future

Welcome to PidgeMe

Director's Report: DEBRIEF PART 2: Technical Details and the Future

Welcome to PidgeMe

Director's Report: DEBRIEF PART 2: Technical Details
and the Future