Accessibility links

  • Home
  • News
  • Spring General Booking 2014: An apology

Spring General Booking 2014: An apology

Why the ROH website suffered serious technical problems and what we have done to fix them.

By Rob Greig (Chief Technology Officer)

15 January 2014 at 1.45pm | 33 Comments

Tuesday was a difficult day for Royal Opera House customers who were trying to book for the Spring Season. After six months of error-free booking days, we suffered a serious incident during yesterday’s General Booking.

Firstly, we would like to apologize to anyone who had difficulty booking tickets on Tuesday. Caroline Bailey, ROH Director of Marketing, says, ‘We all recognize that The Royal Opera and The Royal Ballet are very special to a lot of people. Our audience has an emotional and financial investment in what we do, as well as giving us their valuable time, which we very much appreciate. This is most definitely not the kind of service we wish to provide.’

Those of you who have felt the pain of booking days in the past may fear that this is more of the same. The nature of yesterday’s problems was very different, even if the frustration for ticket buyers was just as acute. We have cracked the problem of capacity on the website and have had some very good booking days, with no waiting rooms and no errors. Yesterday, however, the website displayed serious technical errors, which took us two hours to resolve. The problems were intermittent and caused errors at random for some customers. I would like to explain what happened and what we are doing about it.

We launched a new version of our website in December to make our website – among other things – compatible with any mobile device. As part of this work we took the opportunity to re-wire the engine behind the website with the intention of improving page load times and making future development more efficient. We moved from a development framework called Zend to a new framework called Symfony. Symfony works differently and allows us to re-use code and build websites in a more modular way. Sections of code then operate like building blocks rather than large, unwieldy chunks of programming. Why change a system that is working? We are continually developing and improving the website, and changing our development framework in this way will allow us to launch exciting new features in future.

We have now discovered that part of that code was causing a problem. One part of the code is meant to check that a user is logged into the website. It is meant to perform this check once on the relevant pages; in fact it was checking 10 to 15 times for each user. The ticketing part of the website was not expecting so many account check requests and started to return, well, nothing. As the website wasn’t returning an error that would give us guidance as to the nature of the problem, it took us a while to pinpoint the issue. Once we did, it was a quick fix. It won’t happen again.

This begs the question, how did this happen? Well, we have a thorough process of testing both code and load ahead of every booking day, and often reveal issues which we fix before they are seen by the customer. The tests, however, are mechanical and although they test the load to many times the size of a booking day, it’s still a mechanical test. So we now need to work on a more organic test, which will work more like a simulation of what actually happens on a booking day. As you can imagine, simulating the real activity of 100,000 people coming to our website could be tricky. There are lots of tools and methods to help achieve this and we are committed to improving the quality of experience for those who use our website and making sure this is a one-off incident.

By Rob Greig (Chief Technology Officer)

15 January 2014 at 1.45pm

This article has been categorised Off stage and tagged booking, Booking Day, booking days, digital, mobile, symfony, website, zend

This article has 33 comments

  1. Peter Lewis responded on 15 January 2014 at 2:15pm Reply

    Thank you for explaining what went wrong, apologising and confirming a plan for the future. As is always the case, confidence is damaged and nobody will be reassured until the next error free booking day. It was frustrating but ultimately successful for me yesterday. The end to end experience reflects on the Royal Opera 'brand'. I hope you are able to repair it for next time.

  2. Dr Solomon responded on 15 January 2014 at 2:17pm Reply

    Unfortunately whatever the cause it stopped many people getting tickets , mostly because they had given up long before the problem was fixed. Disappointing!

  3. Tony Boyd-Williams responded on 15 January 2014 at 2:58pm Reply

    A most thoughtful and generous explanation.

  4. JustWondering responded on 15 January 2014 at 3:18pm Reply

    Is there a reason why you don't open general bookings on different dates for opera and for ballet? Would that not (roughly) halve the load the website is taking on those days?

  5. Thanks for being upfront about the problem. I hope people understand your drive to constantly improve, it is this reason why the ROH has one of the best booking experiences available online. There is always a risk of development bringing instability, but personally I think it is worth it.

    A few days ago when the student booking opened I was noticing some odd behaviour. I had the page open on two machines, and they were reporting different availability of tickets. Sometimes I tried to book some which seemed available, but the system told me they weren't (despite them still being there when I refreshed the page). Was this a symptom of the same issue?

    I have also had issues with my Friend membership not being recognised on my Windows PC, performances showing 'Not On Sale' despite me being logged in. But this happened to me before the site redesign too.

    I'm sure there are lots of people (myself included) who would be happy to help by submitting bug reports or testing pages. Either way, well done for fixing the issue and I hope the next booking day runs smoothly.

  6. Stephen responded on 15 January 2014 at 4:05pm Reply

    An excellent message - apologetic, clear, open and empathetic. Thank you.

  7. Hariclea Darclee responded on 15 January 2014 at 4:35pm Reply

    I appreciate RG taking the time to explain the fault and the plan forward. It was indeed a frustrating instance when booking went back to old days time of taking more than 1h. Especially since the time tickets last in the basket is limited to 30 min...
    I'd like to point out some things which also went better than before, for example deleting a ticket from the basket updated the information about ticket limit immediately , ie one was able to go back to the performance and book more tickets, which previously took a while to update.

    On the new version of the website i find it personally a bit too red-heavy on the eyes and the white writing on the red background a bit too small on laptops/mobiles. Slightly bigger menu headers may help.

    Also, on the News section it's extremely cumbersome to go back to older news articles! If they are not the recent ones on the main page one starts clicking on opera, music, maybe other categories and it's not always obvious where things are. Was searching for an article KH wrote but it wasn't filed under opera?! Had to search by his name and scroll down in the list of results from all over the page. Can we please have some kind of chronological categorisation in News so we can view a full list of all articles published rather than the categories which are not always well allocated? (ie this month, past month, older etc) Thank you :-)
    And good luck for next public booking, with Manon L and Stuarda likely to be as busy as this one ;-)

    • Ellen West (Head of Online Content) responded on 15 January 2014 at 5:25pm

      Hi Hariclea

      Thanks for the feedback on the website - very useful - and also for the good wishes for Summer Season. We are expecting it to be a very busy one!

      Best wishes

      Ellen

  8. Chaz Dean Bennett responded on 15 January 2014 at 4:37pm Reply

    This explanation doesn't help that the fact I was unable to get a ticket for the event. I wanted despite being on-line and logged in way before 10am and tried for over an hour to purchase tickets. Seems the only way to ensure a ticket is to pay large sums of money to become a "Friend' of the R.O.H. Which we cannot all afford. This is now the 2nd time this has prevented me from getting tickets that I wanted, Very Disappointing.

    • Ellen West (Head of Online Content) responded on 15 January 2014 at 5:23pm

      Dear Chaz

      Sorry to hear that you had such a negative experience.

      Of course a smooth booking day is not dependent on being a Friend of Covent Garden. The code change that led to yesterday's problems fell between Friends' and General Booking in this instance and this is why the latter booking day was affected.

      Best wishes

      Ellen

  9. David responded on 15 January 2014 at 4:54pm Reply

    Thanks for this helpful and reassuring message. I for one have been delighted over the past 6 or so months with the smooth efficiency and general sophistication of the new site - frankly there isn't a website for a comparable House in the whole of Europe or indeed the States that begins to match it. It has been remarkably free of the inevitable early glitches and hopefully you will have nailed this one. I know you get complaints but speak as you find - I think it's a great site!

  10. Stuart Dixon responded on 15 January 2014 at 4:56pm Reply

    Being an ex programmer, system analyst, and designer I know one of the main problems is volume testing, trying to mimic 100k of people all hitting the system at the same time. Nevertheless I have made the point before that an easy fix for the volume issue is to separate the general booking dates for opera and ballet. I cannot figure out why they have to be the same day on the same hour.

    • Ellen West (Head of Online Content) responded on 15 January 2014 at 5:17pm

      Hi Stuart

      Load is no longer the issue - the website has coped with busier booking days than yesterday's. The problem in this case was an unexpected error.

      We are keen to keep the opera and ballet booking on the same day: if we split them we increase the number of booking days and see higher associated costs and additional risks. If we felt that splitting booking was a simple solution then we would do so, but we think that there are other areas we need to explore.

      Best wishes

      Ellen

  11. Karen Tkach responded on 15 January 2014 at 5:21pm Reply

    Thank you for the explanation, which at least ameliorates some of the frustration caused. I did manage to log on to the website at 10:00; although I was thrown out shortly later (and many times thereafter), I did have time to see that the Jonas Kaufmann recital on 6 April was Sold Out by 10:01! This seems very strange to me. Were no seats 'held back' for General Booking customers for this recital?

    • Ellen West (Head of Online Content) responded on 15 January 2014 at 5:32pm

      Hi Karen

      Seats were released for General Booking, but the website was not behaving as it would normally yesterday, so all sorts of unusual behaviour was apparent.

      Sorry that you missed out on tickets to see Jonas.

      Best wishes

      Ellen

  12. I had a nightmare getting on the site and then continually failed to complete a booking on Winterreise. However I did finally get a ticket at just after 11.00. So it sort of ended well. But certainly a disappointing experience after so many good ones recently

  13. Jeremy Fielding responded on 15 January 2014 at 6:48pm Reply

    We appreciate your information and honesty with the issues, but this has become such a regular event on big booking days your lead statement "what we have done to fix them" offers no confidence to those of us who have been the continual victims of such terrible experiences at the hands of your IT systems. We have heard so many apologies over the last couple of years for different issues each time (just look back in your archive of apologies you have had to issue after past booking dates); what will it be next time? And the time after that? And the time after that?
    One assumes you are investing huge amounts of taxpayers' money (from your subsidy) into companies providing these 'services' which are not meeting basic conditions. Surely somebody should be taking responsibility and accountability for these continued IT disasters?
    At the end of the day, whatever the cause was this time, you have masses of loyal patrons who are now empty-handed of tickets for performances we have been looking forward to all year - an apology is nice, but it doesn't undo the damage and disappointment your unreliable systems have caused.

    • William Griffin responded on 9 February 2014 at 7:51pm

      Your comments strike a chord, Jeremy. One hopes for a time when the testing will be adequate to avoid the continuing problems.

  14. Simon Fisher responded on 15 January 2014 at 7:39pm Reply

    Thank you for the explanation and apology - much appreciated. It certainly was frustrating, having re-arranged work in order to book, that the process was so erratic. As one of the other contributors says, confidence has been knocked! Let's hope all is well next time.

  15. Damian Riddle responded on 15 January 2014 at 9:55pm Reply

    Thanks for the explanation - it was frustrating to go back to the "bad old days" after the last couple of seasons had gone so smoothly.

    One piece of feedback - getting into the booking part of the website is not always intuitive. Yesterday - once I got onto the site - I followed several false leads through details of seasons, productions etc before finally finding a link to book for a production.

    For those of us who are a bit simple, maybe a big red button saying "BOOK TICKETS" would help in the future?

  16. Dana responded on 16 January 2014 at 8:42am Reply

    You could try a "mock" booking day - like those anti-terrorist drills, only less serious

  17. I am gad to see this explanation of Tuesday's problem. Having put Insight tickets into my basket I was relieved when I could finally pay for them with less than one minute to go before I lost them!
    The last few booking days have coped so well with the volume of traffic, I can't help feeling that the problems now may be exacerbated by the sheer number and spread of priority booking days... the next booking cycle starts on the 3rd February and goes on till 6th April. So it becomes more difficult to make/test changes during a quieter, less sensitive time. Now that most bookings are made online would it not be better to compress the priority period?

  18. chris mclaughlin responded on 16 January 2014 at 11:17am Reply

    I was interested to see that one of the people posting above was apparently able to get tickets for Winterreise at 11am since I saw a message on the site some 15 minutes before saying that the event was sold out!

    • Ellen West (Head of Online Content) responded on 16 January 2014 at 11:47am

      Hi Chris

      During booking it's not unusual for people to add tickets to their basket and then remove them if they spot better ones. This means that the tickets are then released back into the general pool.

      Also, and very unfortunately, some people did manage to get tickets into their baskets, which they then lost. These tickets were then picked up by others.

      Best wishes

      Ellen

  19. Wouldn't it be better to open ticket sales on different dates for different operas e.g. 3 months to the date beforehand?

    • Ellen West (Head of Online Content) responded on 16 January 2014 at 11:52am

      Hi Ed

      Now that volume is no longer the problem, splitting booking would actually increase the risk of something going wrong. More booking days means that there are more occasions for errors. This would also be inconvenient for people who come to a number of performances in one season, who would have to come back repeatedly for tickets.

      We are very sorry for the poor experience people had on Tuesday and are working hard to ensure it doesn't happen again.

      Best wishes

      Ellen

  20. Edward responded on 16 January 2014 at 11:52am Reply

    I think the explanation misses a fundamental point - repeated unexpected errors isn't bad luck but incompetence. It makes the whole booking process take hours and basically ends up a lottery.

  21. Dennis responded on 16 January 2014 at 2:42pm Reply

    Ellen's comment is not helpful for those of us who chose tickets, put them in the basket and then lost everything because the screen vanished for almost 2 hours, by which time all tickets (Faust) were gone. Advanced planning seems pointless as booking became a lottery.

    • Ellen West (Head of Online Content) responded on 16 January 2014 at 2:46pm

      Hi Dennis

      Agree that no amount of apologising will make up for having lost out on tickets. Sincerely hope that you will never have this experience with us again.

      Best wishes

      Ellen

  22. Don Aitkenhead responded on 16 January 2014 at 3:14pm Reply

    Rob, thanks for your explanation which is very clear. However, the fact that the management have fielded you to explain this from a "technical" point of view highlights the real problem. Many people have asked "why do all performances for a season go on sale to the public on the same day". This is clearly a policy and there has been no explanation. Several people have also drawn attention to the fact that by the time the public get to attempt to book, a large proportion of the tickets have already been taken by the various classes of "members" and corporate sponsors. For an organisation which receives the majority of its funding from the general public through taxation, this is invidious. While I don't expect the ROH to change its policy of giving priority to "members" I strongly urge it to look at the advance booking policy in other similar organisations such as the National Theatre. You will see a much fairer and inclusive policy there.

  23. Just a quick note, this organisation does not get the majority of it's funding from the taxpayer, in fact very little of it, i think it is less than 24% by now and decreasing.(ACE 24% of income in 2012, as per figures which anyone can read in the annual review, see menu at bottom of the page )
    The majority of funds come from ticket sales and private donors, like the friends, who contribute to the ROH funding with their membership and get early booking in return. I believe this is fair. At the time i joined i think there were over 20 thousand or so friends, i mention this to dispel the idea that a small number of people get tickets in advance, no, it is actually a very large number of people who support this organisation :-)
    At the same time the ROH allocates a contingent in all price categories to public booking as a rule, making tickets available to non-members.
    Many of said members/friends attend many performances, for example more than 10 each booking period. I'd have serious objections to being asked to buy these over 5, 6 or more booking dates! After all i have to work to pay for my tickets, not spend the working days in bookings :-)

    In terms of fairness, there are more discount tickets available for various categories of people than in most other opera houses in the world (i see more people with access tickets than in any other place, there are more performances accessible to children and families, students, etc)

    The fact remains, as has been mentioned before i believe, this house has 2,200 seats. It's is not only a UK institution, it is one of the top 3 opera houses in the world (same goes for ballet). With 100k+ people wanting to buy tickets to come here it will always be impossible to satisfy all buyers.

    Having said that, i believe what we can expect is fast, efficient sales - ie quick and easy access to the website at all times and quick booking time. Availability of tickets is something that can only be expected within the constraints mentioned above. They are 2 very different things - one where improvement needs to happen ( or rather more stability of improved service) and the other where, as long as good casting and high quality performance is a constant ( and hopefully will remain one :-)) improvement is almost impossible.

  24. Brigstocke HUGH responded on 17 January 2014 at 10:49am Reply

    I spent two hours from 10am using two computers. I got and then lost two upper slips for Kauffman,at around 11am (last in the house I think), failed to even access the Faust page but found seats for Die Frau. A very kind and helpful member of the box office responded to my help email sent 11am and at the end of the day I ended up more or less with what I wanted. Thank you.It must have been a dreadful day for you as well as us.
    But NB I still have a problem with the web site. My Apple computer which has always worked well via Safari now, even today, refuses to download the seating plan to 'choose your own seats', although my wife's non Apple computer is effective. Have the web site changes accidentally disenfranchised other Apple users?

  25. Ellen Moerman responded on 18 January 2014 at 9:34pm Reply

    Can't comment on the explanation, as I am not a IT buff. But what I don't understand is why you did you not close down - attend to repair and then open again for business. A plumber confronted with a leak switches off the water, an electrician would switch off the electricity. So what's so special about IT that you must keep it running while you tweak it? And would it not have been fairer on the customers?
    What I can say is that the previous 6 months weren't as flawless as is suggested. I had considerable problem during the last booking round, in October was it - I had to do two separate bookings after having lost my basket twice at the ckeck out stage. On Jan 13, I was on the webiste at 09.30, checking casts - and noticed that whereas casts for Winter's Tale were announced as having been confirmed, in reality, the message that showed up on the relevant page was exactly the same as it had been during the previous weeks. So that bit of "news" was either incorrect or hadn't reached the ROH's own pages yet. The second was still largely unknown. Hm. And we are supposed to buy tickets? The site went down at 09.58. I spent the next two hours and ten minutes looking at various error messages 1. this site can not be found 2. the home page froze. 3. the calendar froze right up to.... 27. seat availability froze to.... 39. most tickets now gone.
    Two hours and ten minutes (+ 1/2 hour preparation) is almost half a working day. That's an awful lot of money to pay for a theatre ticket. And so instead of buying tickets for 4 shows, I could only afford to buy tickets for 2 shows.
    I hope you will consider the proposal made by one of your correspondents: separate ballet and opera bookings. That should reduce the rush hour effect. And me? I'll return to the physical queue next time. It'll be so much more convivial than staring at a dumb computer screen...

Comment on this article

Connect with Facebook

Your email will not be published

Website URL is optional