6 June 2014 at 5.38pm | 39 Comments
Updated 13 June with comment from Rob Greig
At 10am this morning booking opened to Supporting Friends of Covent Garden for our Autumn Season, but the experience was a poor one. Pages were slow to load, seat maps jammed and even if people were able to add tickets to their baskets they were often unable to complete the transaction. Many contacted us to communicate their anger and frustration, and our digital development team have been working throughout the day to resolve the problems. The website should now be working smoothly.
Once we know what has caused today's difficulties we will give you an update and – more importantly – ensure that it does not happen again. We value all of our customers, but the Supporting Friends of Covent Garden do seem to have had more than their fair share of difficult booking days and for that I apologize unreservedly. We value your support hugely, and are saddened when a booking day is a cause of frustration rather than excitement.
An update from Rob Greig, our Chief Technology Officer about the reason for the problems experienced on 6 June:
'An automatic overnight maintenance job developed a fault which caused it to still be working when booking started at 10am, this job caused our databases to slow down and therefore the website became very slow and created errors.
'Some audience members have asked about our testing regime. Ahead of every booking, we perform a load test that is many times greater than the traffic we expect. In this instance the faulty maintenance job was not load tested because it is not normally part of the load on a booking day. The incident we experienced was the equivalent of a fault developing in the electrical supply to the ROH or the server room getting too hot.
When we release new code we carry out unit tests, regression tests, functional tests, user acceptance tests and load tests. We also have a peer review mechanism where the developers review each other’s code. The external development agency we work with, POP, has a similar testing regime and they have a dedicated quality assurance and testing team which test before every release. Dozens of bugs get captured and fixed before code gets released into the wild.'