Building my rebooted:bbc.co.uk homepage - #17: User Testing, QA and Launch Party
I've been putting together my own "entry" for the reboot:bbc.co.uk competition, and as I do so I've been going through how a project like this might develop within the BBC - and now with the competition closing today, I'm also at the final stage.
Once the new page was in the process of being built using the Scrum process I discussed yesterday, I'd expect to user test it again. Now that bits of it are working I would be looking to re-test the assumptions that were made earlier based on the previous testing.
You do have to be careful with this kind of late stage user-testing however, to make sure you put the correct amount of store by the findings. As I think I've mentioned before - when we did user testing on the BBC's search results interface back in 2002, we ended up finding out that something like only a third of people understood how to use the tabs. But we also found that only a third of people understood how to use radio buttons if we designed a page using them. And we knew that only a third of people understood how to use drop-down menus to 'scope' their searches. Which left us pretty much none-the-wiser as to how to improve the interface for everybody. The project ended up trapped in a cycle of producing design iterations that nobody was confident enough to use.
Likewise I've worked on a project where very near the end of the build user testing showed that production staff found the error messages "confusing". In the end I had to make a judgement call that it was a complicated application, and editorial staff were just going to have to get to grips with it - we didn't have time to build a whole new error handling module into the system.
After each scrum cycle has finished, I'd also expect to see QA work done on a project as big as redesigning the BBC homepage. I'd want to see end-to-end testing of the publication chain that made sure the News and Sports headlines were on the page, and I'd want to get some figures about the page footprint (i.e. how much each indivudal browser has to download to display the page) and expected bandwidth consumption to assist in optimising the design. I'd also want the page checked across a whole series of different browsers on PCs, Macs and LINUX machines, to make sure it worked well across all of them on the BBC's supported browser list. That also means checking that the page works for users who have cookie support or JavaScript switched off.
The BBC doesn't (well, didn't) have a dedicated testing team in-house. This work is usually carried out by producers, and then for very technical system testing, the BBC will hire in specialists on a contract basis. The BBC also uses a little volunteer group of testers, usually associated with the DNA powered sites like h2g2 or the message board system on bbc.co.uk - these can be potentially set loose on any new project to give an often very honest appraisal of it. Sometimes a little too honest perhaps for the liking of some BBC producers ;-)
Finally, it would be time for the project manager to concentrate on their most important task - organising a suitably riotous launch party. As a general rule of thumb it is best not to party on the actual night of launch. For one thing, people are liable to be very tired and possibly quite worked up having rushed to meet deadlines, and may not be in the best of moods to enjoy a good party. Secondly, you should always give a new system time to bed down before celebrating. Otherwise, there is a risk that something could go wrong just after launch, and there is no more thankless task than trying to drag software engineers away from a party in order to get broken things fixed at short notice...
Well, I've finished my theoretical entry to the competition - and it looks something like this.
It isn't a finished visual design, more of a coloured-in wireframe, but it pretty much lays out all the content and functionality I'd expect to see on the page.
Anyway I hope that my series of posts has been useful in giving a glimpse behind the scenes of how the BBC's New Media department builds products and services - well, at least how it did before I left the BBC six months ago anyway - and also has given an insight into how many requirements have to be balanced by the current design of BBC homepage.
I see there has been a flurry of new entries published in the gallery over the last couple of days. If you have entered the competition, which, as I say, closes today, then I wish you the best of luck when the judging starts.
- 25 May 2006 13:23
gallery
comments post a comment
Beer and nibbles all round!
Interesting that you built your design to fill a 1024 pixels wide screen, when the current one fits inside 800x600.
The proportion of people with 1024 or bigger screens is rising and rising, but there must still be a fair few with 800 wide monitors.
Mind you, I guess the proportion of people complaining about that 'huge white waste of space' are growing too :-)
You've also used up every pixel in your 1024 wide design - which means when I view the JPG within Firefox here you get a horizontal scrollbar as Firefox seems to add a margin.
My design is just 789 pixels wide, and I'd anticipate making it centred to making it feel less skewed on large monitors.
So, can we safely ignore people with 800 wide monitors yet? (of course, you could always do fluid designs, but they're tricky...)
>> Interesting that you built your design to fill a 1024 pixels wide screen, when the current one fits inside 800x600.
Well, I think "design" is a bit strong there - I wouldn't want to launch with anything that vaguely looked like that - it is more just a layout idea with some functionality exposed really.
>> You've also used up every pixel in your 1024 wide design - which means when I view the JPG within Firefox here you get a horizontal scrollbar as Firefox seems to add a margin.
Well spotted, I did throw it up rather quickly in the end - I've trimmed it a bit now to fit on screen without horizontal scrollbars (well, on my laptop anyway).