The datacentre is performing a series of urgent unscheduled kernel upgrade on the nodes where the CPanel servers located. The nodes will need to be rebooted as part of the upgrade, which will cause a small amount of downtime while the reboots take place. 

Start Time: 25/06/2014 00:00 NZDT
End Time: 25/06/2014 02:00 NZDT

The following database servers will be affected

Our shared mail system users on the Auckland Network may be aware of an issue that occurred on the 18th June during a migration of data to a new mail platform. This resulted in mail delays for some of our clients. We apologise for the inconvenience caused, and have assurance from our upstream providers that future mail migrations will be broken down into a significantly smaller number of mailboxes, so the migrations complete within the planned maintenance window, and do not overrun into business hours. The full incident report is below.

Incident report:
A planned email storage migration to new hardware took longer than expected due to poor data synchronisation performance on the old mail storage hardware. Due to the ongoing mail synchronization issues we were unable to release new mail until the data synchronization had completed.This resulted in a mail delivery delay for many emails, of around 19 hours.

Root Cause Analysis:
The planned mail migration window of 3 hours overran into business hours, delaying delivery of new mail. Poor data synchronisation performance on the old mail hardware during the planned mail migration resulted in a mail delivery delays.

Improvements / Remedial Work:
Future mail migrations will be broken down into a significantly smaller number of mailboxes so the migrations complete within the planned maintenance window and do not overrun into business hours.

Event Sequence:
00:00: Synchronization start
03:00: Planned end of maintenance window reached, migration still ongoing
19:15: Synchronization completed, queued mail has been delivered to mailboxes
19:30: Issue resolved.

Update: 07.30pm
The mailserver migration is completed and all queued mail has been delivered to the mailboxes successfully. We sincerely apologise for the long delay this sheduled migration took . The migration commenced early this morning at 12am, and should have completed by 3am during off peak hours. However there were a number of unforseen issues that have led to the delay, including the vast amount of individual files on the server which has reduced file transfer speeds. We will put a thorough incident report on this RSS as soon as soon as we have received it from our upstream providers. 

-----

Update: 05.30 pm
This issue still continues. We sincerely apologise for the long delay. Our upstream providers are absolutely certain in their ability to restore service by tonight. There have been a number of unforseen issues that have led to the delay, including the vast amount of individual files on the server which has reduced file transfer speeds. We will put a thorough incident report on this RSS as soon as soon as we have received it from our upstream providers.

-----

Update: 4.00pm
Situation remains the same at this time. Please watch this page for further updates.

-----

Update: 2:50pm 

Unfortunately this issue is still ongoing. The sheer amount of emails and files stored on the servers to transfer is causing this delay. Our own email address is also on this network. So if you have emailed us, we may not have received your email yet. Please instead use emergencysupport@loungenetwork.co.nz

------

Update: 12:45pm 

The situation remains the same. We expect the migration to be complete and the mail issues resolved within a couple of hours. Please watch this page for further updates.

------

Update: 10:45am
Some customers may also be experiencing delayed delivery of mail while the storage migration is in progress. Further updates will be posted on this page

------

Update: 08.40am

Services Affected: Incoming email on POP and IMAP for some of our customers
ETA: 3 hours

Description: Further to our announcement on the 16th June, the migration of emails to the new platform overnight has taken longer than anticipated and mail was temporarily locked overnight while it was being done.
The mail migration is ongoing and as such, customers may receive some emails twice as this completes.

There is no risk to your data or service, but we appreciate that receiving any emails twice may cause inconvenience. The alternative was for us to lock email during business hours which we feel would cause more disruption.
If you have any questions, please contact us via our normal website.

---

We have scheduled a maintenance window to perform mail storage migration on Wednesday morning, starting at 00:00. This will address mail performance on our storage servers.

Start Time: 18/06/2014 00:00 NZST
End Time: 18/06/2014 03:00 NZST

During this maintenance window POP3 and IMAP will be unavailable.

We thank you for your understanding while this work is completed, and apologize for any inconvenience caused. If you have any questions, please do not hesitate to contact us.

There are reported power outages in Auckland. Please note that our hosting is not affected by this. Should there be a power outage at the datacentre, then they will run their servers off backup generators, which will keep all services up and running.

== re-scheduled ==

Dear Customers,

We have scheduled the installation of the latest Atmail software updates on Thursday, 29/05/2014, starting at midnight. The patches address mainly security and bug fixes and some general improvements.

Start Time: 29/05/2014 00:00 NZST
End Time: 29/05/2014 00:30 NZST

During this maintenance window our current Atmail installation will be sporadically unavailable for the duration of the update. Horde webmail, POP and IMAP services will continue to be available throughout the maintenance window, with no expected disruption.

We thank you for your understanding while this work is completed, and apologise for any inconvenience caused.

Dear Customers,

We are writing to advise that we have scheduled security patching that will require reboots for some of our mysql servers during the following period.

Start Time: 28/05/2014 00:00 NZDT
End Time: 28/05/2014 02:00 NZDT

The following database servers will be affected

mysql3

We thank you for your understanding while this working is completed and apologise for any inconvenience caused.

Dear Customers,

We have scheduled a maintenance window to perform mail storage migration on Thursday morning, starting at midnight. This will improve mail performance on the storage servers.

Start Time: 15/05/2014 00:00 NZST
End Time: 15/05/2014 03:00 NZST

During this maintenance window POP3 and IMAP will be unavailable.

Please bear with us while these changes are made, to improve overall email service.

We thank you for your understanding while this work is completed, and apologise for any inconvenience caused.