Most businesses can predict the amount of traffic that they’ll get to their online services and generally DWP is no exception to that. Under normal circumstances we’re good at predicting our levels of traffic and this helps us plan ahead to ensure we’re robust and ready. However, sometimes unexpected massive traffic spikes can happen.
COVID-19 had many effects on services and like most organisations, DWP wasn’t expecting a pandemic.
A few years ago we were asked to test how we’d cope if we were to experience a massive spike in traffic, for example due to a news story bringing attention to our services. We do proactive performance testing based on business predictions and we test 6 months ahead of ourselves, giving us an opportunity to resolve any issues. This kind of testing put us in a good position for the unexpected, and the events of 2020.
When COVID-19 hit we were faced with an unprecedented demand for Universal Credit.
About Universal Credit
Universal Credit is the biggest change to working-age benefits since the second world war. At full scale, it will serve 7 million people and pay out £67 billion each year. It’s a single monthly payment for people in, or out of work, replacing 6 working-age benefits and tax credits with a single combined monthly payment.
Back in 2013, when we started work on Universal Credit, DWP’s IT was outsourced, with in-house project management and requirements writing. We generally worked using waterfall methodology and cycles could take around 6 months. When we started the Universal Credit project, we knew we wanted to work differently. In collaboration with the Government Digital Service (GDS), we adopted modern agile ways of working. We design and build in-house with a user-centred approach. We work in multidisciplinary teams of a variety of roles, including developers, QAs, business analysts, user-centred design disciplines, and product and delivery managers. We also embed people from business operations and policy in those teams.
Using agile delivery, we take a minimal viable product approach to get our services out and in front of people as soon as possible. We test and learn, iterating quickly using feedback from users. Our platform is now hosted in a public cloud service that we operate ourselves 24-7.
We started with a proof of concept and by 2018 a full national roll-out had been completed. Over the years it’s been quite a challenge as it was a huge organisational change.
In 2020, COVID-19 changed everything.
Our COVID-19 story
In the early days of Universal Credit from late 2014, we rolled out really carefully to ensure we landed the service well. From 2018 to early 2020 we were on a standard roll-out, and were up to about 2 million people using the service. Our predictions were working well. We were experiencing around a 4% month-on-month growth and we would typically have around 100,000 successful claims to Universal Credit every fortnight.
COVID-19 changed all that. In early March we saw a small increase in claims, but when important announcements were made on 20 March, things changed. The first was the prime minister, Boris Johnson, announcing all non-essential businesses to close. The second was the chancellor, Rishi Sunak, announcing that the government would pay 80% of wages for employees unable to work. The next week, the chancellor made a further announcement that some self-employed people would be paid 80% of their expected profits to help them cope during the economic crisis. Each of these announcements resulted in a spike of traffic to our system.
The massive increase in traffic and claims that followed was highly unexpected and within a few months took us to 5 million active Universal Credit claims. We experienced a 40% increase in the initial weeks alone (up to 9 April), resulting in some spikes of a 10-fold increase. In one particular fortnight we had 950,000 applications.
How did we handle the unexpected?
We were in a fairly good state due to the 6-month ahead testing that we’d done regularly, so we knew that our service could withstand much more traffic than it was used to receiving.
Our initial response was to increase our capacity. We added more webservers, more application servers for a subset of microservices that were under pressure, and we increased the database capacity. We benefited from the fact that we use cloud hosting and so can scale-up our capacity on demand. We also benefited from a lot of the government announcements happening late in the day, when a lot of our usual daytime traffic had fallen away.
We then began to think about which parts of our systems might struggle next and how we could protect the system. Despite performance testing there are always surprises in store when you hit the real world. Fortunately, we never needed to control the amount of traffic coming in with web request queues, so our users could always access our service without waiting and continued to find it responsive.
In normal times we regularly deploy and release changes to production; in January and February we released 6 of these changes, in March and April that rose to 76. Our agile way of delivering Universal Credit allowed us to make these changes rapidly so we could adapt and evolve to the situation as it unfolded, allowing our customers to get the support they needed.
We also put on hold most of our plans for 2020 so we could concentrate on what was happening in the here and now.
Our colleagues also really rose to the challenge. Across DWP 10,000 employees were redeployed from their normal jobs to help with the demand, at the peak dealing with 2.2 million calls in a single day. It was a whole department effort to ensure that the people who needed assistance received it.
Universal Credit is a system that real people depend on. If people don’t get the support they need their families could go hungry, they could miss their rent or be unable to pay for heating. As we enter the biggest recession in the UK since records began, DWP’s services will continue to be critical for the foreseeable future. It’s been a real privilege to be able to contribute and make an impact through our work.
There’s a lot of exciting stuff happening in DWP and we’ve still got a lot more to do.
DWP Digital are recruiting digital specialists that want to help create solutions that can help change the lives of millions. To find out more, visit us at DWP Digital Careers site.
1 comment
Comment by Phil Davies posted on
Fantastic story - thanks. Shows the usefulness of cloud architectures for responding to demands quickly.