15 January 2014 at 1.45pm | 33 Comments
Tuesday was a difficult day for Royal Opera House customers who were trying to book for the Spring Season. After six months of error-free booking days, we suffered a serious incident during yesterday’s General Booking.
Firstly, we would like to apologize to anyone who had difficulty booking tickets on Tuesday. Caroline Bailey, ROH Director of Marketing, says, 'We all recognize that The Royal Opera and The Royal Ballet are very special to a lot of people. Our audience has an emotional and financial investment in what we do, as well as giving us their valuable time, which we very much appreciate. This is most definitely not the kind of service we wish to provide.'
Those of you who have felt the pain of booking days in the past may fear that this is more of the same. The nature of yesterday’s problems was very different, even if the frustration for ticket buyers was just as acute. We have cracked the problem of capacity on the website and have had some very good booking days, with no waiting rooms and no errors. Yesterday, however, the website displayed serious technical errors, which took us two hours to resolve. The problems were intermittent and caused errors at random for some customers. I would like to explain what happened and what we are doing about it.
We launched a new version of our website in December to make our website - among other things - compatible with any mobile device. As part of this work we took the opportunity to re-wire the engine behind the website with the intention of improving page load times and making future development more efficient. We moved from a development framework called Zend to a new framework called Symfony. Symfony works differently and allows us to re-use code and build websites in a more modular way. Sections of code then operate like building blocks rather than large, unwieldy chunks of programming. Why change a system that is working? We are continually developing and improving the website, and changing our development framework in this way will allow us to launch exciting new features in future.
We have now discovered that part of that code was causing a problem. One part of the code is meant to check that a user is logged into the website. It is meant to perform this check once on the relevant pages; in fact it was checking 10 to 15 times for each user. The ticketing part of the website was not expecting so many account check requests and started to return, well, nothing. As the website wasn’t returning an error that would give us guidance as to the nature of the problem, it took us a while to pinpoint the issue. Once we did, it was a quick fix. It won’t happen again.
This begs the question, how did this happen? Well, we have a thorough process of testing both code and load ahead of every booking day, and often reveal issues which we fix before they are seen by the customer. The tests, however, are mechanical and although they test the load to many times the size of a booking day, it’s still a mechanical test. So we now need to work on a more organic test, which will work more like a simulation of what actually happens on a booking day. As you can imagine, simulating the real activity of 100,000 people coming to our website could be tricky. There are lots of tools and methods to help achieve this and we are committed to improving the quality of experience for those who use our website and making sure this is a one-off incident.