Post

2 followers Follow
0
Avatar

UNPLANNED OUTAGE - Server shopau8.info

11:04am QLD - UPDATE/ MONITORING

We're still monitoring for any further issues - however so far everything appears to be stable now with no further reports of any major issues. We're assisting with a couple syncing errors on a few sites - these will be resolved asap - but no wide-spread issues across the board have been reported.

9:23am QLD - UPDATE/ MONITORING

All images are now restored and all syncing should now be able to resume per normal.
We have one report of a server 500 error on one site's checkout - please call us if you have any issues with your checkout.

LOGIN/AUTHENTICATION - if any of your customers have issues logging into the website - please have them follow the usual "forgotten password"process.

9:02am QLD - UPDATE

As far as we can tell - all product images are now back up and running. If you still have any issues with your product images across the whole site - please give us a call and let us know so we can investigate it asap for you.

We're still working on the Sync database error and will put an update on here as soon as we have confirmation that's been resolved.

8:30am - 11 NOV QLD - INVESTIGATING

We're still investigating issues reported about product images missing on some sites even though all files have been copied over. We'll have another update about this in the next hour.

We're also investigating database errors that are preventing the sync from running on some sites. If you're affected by the syncing issue, please enter your orders manually for the time being until further notice from us that we've resolved the syncing.

5:26pm QLD

We're now seeing sites returning to normal operation - including the SSL Secure Server. However you'll still see images missing on some sites until later tonight just due to the volume of data being transferred.
We expect a full return to normal operation before the morning - however please do let us know if you have any syncing issues.

4:00pm QLD

PLEASE NOTE: as your site may become active before we contact you, you may find images are missing for a while longer as they take longer to copy across - but they shouldn't take long to show up.

SSL SECURE CHECKOUT: Some customers may try to checkout when your site becomes active again - they may or may not be successful as we're still in the process of re-instating the secure certificate on the server - so your customers may get warnings about an insecure connection on checkout. This is only temporary until we have the SSL Certificate back up which we expect to have happen very soon.

3:20pm QLD

We've been able to successfully restore the data and are in the process of switching the sites over live. We're moving as quickly as possible and will be in contact with individual sites as we're able to confirm each one. This process isn't instantaneous and will continue to roll out over the afternoon.

12:30pm QLD

Our Lead Development Engineer is still working on finalising the restoration of these sites and is still making progress - we don't have any further ETA just yet.

11:25am QLD

  • Further tests on the recovery procedure is proving to be very positive and we're hopeful to start seeing some sites return to a "LIVE" state in the next 1 - 2 hours - barring no unforeseen complications - however we still don't have a concrete time just yet. We'll advise a more specific time as soon as we know for sure.

10:35am QLD

  • The following email has been sent out to all affected websites: "We are experiencing a Hard Drive server issue and your website is currently offline.

The Server team are currently replacing the hardware as required and are testing a recovery procedure.
This procedure will take another 60 minutes to confirm our next course of action.

Your most recent site backup is from 1am on Wednesday 10th November and should the hard drive failure be terminal we will revert to this backup and restore your service.

Should we restore your backup please be aware the following data will not be recovered

  • Orders from 1am Wednesday to 11pm Wednesday will not be in your order panel. If these orders are not synched to your accounting software then you will need to re-key the orders via the email confirmation forms.
  • Any CMS Changes made from 1am Wednesday will need to be re-keyed
  • Any New Accounts Created online from 1am Wednesday to 11pm Wednesday will need to be recreated via the New Account confirmation email

We are making all attempts to ensure a timely resumption of service and will be in touch as soon as the recovery team has completed it’s testing."

Per the note above - we'll post another update here in the next hour.

10:15am QLD - TESTING

  • our Development Engineering team are currently testing data integrity and stability on new hardware to ensure the least impact to your data.

We'll be sending out an email with more details shortly

9:25am QLD

  • The issue has been identified and a plan of action is already being implemented - however it's going to take some time to update every site. As soon as we have a more definite timeline we'll post an ETA here.

8:45am QLD 10 Nov 2016 - UNSCHEDULED OUTAGE

on Shopau8.info server.

Our server is presently inaccessible and this is being treated with the absolute highest urgency possible (all of Webninja is solely focused on this right now) and we'll be posting updates here regularly.
We apologise greatly for any inconvenience incurred as a result of this issue.

Nolan

Please sign in to leave a comment.

3 comments

0
Avatar

After this is resolved I would like to understand the backup procedures better. I know from others in the IT industry that even with a full hardware server failure, the standard now is to be up and running within four hours for most cloud / online stores. What's your service standards for minor and critical failures which are not uncommon.

Abdullah Ramay 0 votes
Comment actions Permalink
0
Avatar

Hi Abdullah,
we'll be in contact tomorrow morning and can answer your questions then.
~Nolan

Nolan 0 votes
Comment actions Permalink