Analysis of New Deal and Programme Centre performance 2008-2009

This article builds on the analysis and explanations in Indus Delta's 2006-07 New Deal performance analysis. The figures used in this article cover every New Deal and Programme Centre contract in the country over the course of an entire year, end of March 2008 to end of March 2009. However, there are major added complexities on top of those faced in the previous analysis, that make a simple comparison table less useful. For an explanation of how we came by these figures, read this article

New Deal performance analysis

New Deal is not a single course. It's a range of separate modules that customers get sent on, and each time a customer starts or leaves a module is counted separately in these figures. The modules are:

• BET - Basic Employability Training
• ETF - Environment Task Force
• FTET - Full-time Education or Training
• FTET Basic Skills - focused on numeracy, literacy
• GtW - Gateway to Work. The two week full-time course that everyone takes at the beginning of New Deal
• IAP - Intensive Activity Period
• VS - Voluntary Sector option

The normal route through New Deal provision is to attend the GtW for 2 weeks in the New Deal 'Gateway', followed by one of the other options for up to 13 weeks during the 'Option' or IAP phase.

The 2008-09 job entry performance for each module, taken across the entire country, is as follows:

Course	Target JER %	Actual JER %	% of Target
BET	25%	21%	84%
ETF	45%	35%	78%
FTET	45%	34%	76%
FTET Basic Skills	25%	27%	108%
GtW	45%	22%	49%
IAP	40%	23%	58%
VS	45%	33%	73%

Most of the modules are substantially under target, but the bulk of the underperformance is concentrated in Gateway to Work, which averages 22% job entry rate rather than the 45% target figure. Gateway to Work is a two week course at the beginning of New Deal, and since everyone is meant to take it, this module alone has a huge impact on the overall figures.

What causes good or bad performance?

To figure out the impact of individual performance, I've categorised each performance figure into one big table in the attached spreadsheet. Categorisations include:

• Provider - the obvious one. Are different providers getting different results because they're innately better or worse?
• Provider type - by breaking providers down into third sector/private/public, their respective results can be compared
• Size of provider - this categorisation is rather less helpful, as the FND primes and the micro-providers are both largely absent from New Deal delivery
• Rurality - is the key in the contract itself? Do providers in rural areas or in London just have a tougher time of it? This (entirely subjective) split into rural/mixed/urban/London allows that to be tested

Assigning a relative level of importance to each of these requires some complex statistical analysis. So complex that I'm completely stuck on it, despite having a Maths degree! Anybody who knows how to apply multiple regression analysis to discrete non-numeric values please get in touch.

In the absence of a 'true' relative importance, it's still possible to get some rough-cut results for the impact of each possible cause, so I've done that in the following sections.

Are bad providers genuinely bad?

Let's take a simple comparison of performance in just one of the modules, say GtW. The performance across contracts has been added up to give a single figure for each provider:

As we can see, the job entry rate varies from a rather unbelievable 0% up to a just-above-target 47%. The margin of error is calculated using 0.98 / square root of no. of leavers, which gives a 95% certainty of being within the margin, supposing no complicating factors. Based on this, only a few providers have a high enough degree of certainty to provide a solid ranking. And even this ranking ignores the impact of local conditions, as discussed in the next section.

Are rural contracts harder to deliver?

A rough telltale of contracts being harder or easier to deliver would be to compare performance in different regions of the UK:

As we can see there are substantial, non-random variations in performance, with Scotland being exceptionally high-performing. Surprisingly, London appears to be easier to get results in than the North East, and not dissimilar in difficulty to a further four regions. This goes totally against the historical belief that London always does 10% worse than everywhere else. It's difficult to discern a pattern from this, other than perhaps a tendency for ex-mining areas to do slightly worse.

I've also categorised delivery districts by rurality (in a rather slipshod fashion). Let's see if that yields dividends:

This produces a rather different result. When comparing by type of area, London actually does worse than everywhere else. In each of the other categories, the above-average performance of some regions pulls up the below-average performance of others. Rural and mixed areas perform marginally better than London, but non-London urban areas perform best of all. Bear in mind that 'rurality' has been set a tad arbitrarily in this model.

The main thing these two sets of data tell us is that different areas have different characteristics, and comparisons of performance between areas have to take them into account. While 'rurality' can be used as a proxy for this, the completely different picture to emerge from the regional performance shows that it can't reflect the complexity of local needs. The Australian system has been using a complex model based on analysis of local conditions to assess provider performance for some time, although the details of it have been kept secret.

Are public providers better?

One of the more emotive questions in welfare-to-work is whether the public, private or third sectors should gain a monopoly over delivery. There is some debate as to whether this is even a valid question, given than current contracting practice is based on contestability, which posits that opening up who can deliver services (not who does, mind) actually improves the performance of everyone involved in delivery. Thus it might make sense to keep an underperforming sector on board in order to prevent monopolistic practices by other sectors. Anyway, here are the results of the GtW by provider status:

The '????' is Prospect Services, a company limited by guarantee but some uncertainty (to me, that is) about whether it counts as third sector or not. 'Mixed' is Working Links, which is jointly owned by organisations from all three sectors.

This is one of the few really unambiguous-looking results. The Third Sector languishes at the bottom of the rankings with just over 16% job entry rate, against almost 25% from private providers. Public sector providers (i.e. colleges and councils) come in a close second.

However, this is far from conclusive evidence of who delivers better. It's not clear that the third sector providers here are representative of the entire third sector. It's not clear that the third sector hasn't picked up more difficult-to-deliver contract areas. It's not clear that performance in GtW carries over to other programmes or FND. Third sector providers would also likely make the argument that they provide a more caring, holistic service that's genuinely in customer interests, instead of firing them into a job.

Why are job entries measured against leavers?

The New Deal and Programme Centre data include both number of starts and number of leavers. Normally, performance is taken as job entries against leavers, since job entries can only happen after leaving. However, where there's a large imbalance between starts and leavers, such as occurs with a sudden increase in the number of New Deal customers due to a recession, then the results can temporarily become artificially raised. This is because there's a lag between people who leave provision in order to start jobs, and their fellow job seekers who remain on provision until it finishes, potentially months later. Just to add complexity, we already know that Gateway is the lowest performing provision, and since that happens before everything else, it artificially lowers results! The total number of starts across all New Deal modules in the year was 215,536 and the total number of leavers was 185,612. This gives clear evidence that a large imbalance has occurred.

Is the performance data reliable?

There are a number of issues that raise questions about the reliability of the source data:

• Shaw Trust are down as delivering a London region contract. In that little-known district of London called Berkshire, Buckinghamshire and Oxfordshire. DMT Business Services have a long list of identical zero-rated contracts in Devon & Cornwall. Carnegie College are delivering a contract in '(blank)', presumably Dunfermline in Scotland
• Gateway to Work is a two week course, so starters and leavers really shouldn't be all that different when taken over an entire year. So how to explain that TWL have 1001 starts and 815 leavers in Lancashire, I2I 3094 starts and 2465 leavers in Greater Manchester East, Best 4982 starts and 4315 leavers in West Yorkshire, and Carnegie College 445 starts and 752 leavers?
• The contracts themselves have been recorded differently in different parts of the country. In some regions, every subcontractor is recorded as the deliverer of their own contract. In others, the prime is listed as delivering everything, even though this is demonstrably not the case
• I've been informed that, until recently, different regions measured leavers differently. Specifically, someone who started then left a course, then was sent on the same course again, was counted as multiple leavers in some regions and not in others. Having someone sent on the same course four times before completing it is apparently not uncommon, which would have an obviously bad effect on the figures as they would only ever be able to get one claimable job entry despite leaving four times

Conclusions

In order to reach an accurate comparison of provider performance, both local conditions and margins of error need to be taken into account. The New Deal contracting and performance measurement regimes are too localised and too sketchy to do this. The approach being taken by FND and Star Ratings at the moment is to measure providers against the target they set themselves in their initial bid. However, placing the onus of target-setting on the bidder assumes that bidders can put together a model of exactly how difficult it will be to get someone into work in a particular area for the next five to seven years, which is totally unachievable given the current recession wasn't being predicted even a year ago. Something similar to the Australian system, which takes local conditions into account when measuring performance, and compares providers in the same area against each other with rewards for the highest performer, would potentially be more useful.

Press Coverage

Just for reference, there are two articles in other press sources based on this story:

• Regen published a story in this week's edition
• The FT then picked up on the Regen article, but (ahem) failed to cite the original source. Grrr

Source data

The New Deal and Programme Centre performance figures are attached below. I've 'cleaned', collated and categorised the New Deal data. The Programme Centre info is far simpler to handle. Feel free to download them and do your own stuff, but please contribute back any findings!

Update 20/7 - I didn't post this at the time, but someone passed through an FoI response showing Gateway to Work performance in 2006, 2007, and 2008, broken down by delivery area. This is now attached. For reference, since New Deal and Programme Centre league tables have previously been released to me by the DWP in response to FoI requests, there are no grounds for arguing that any of the data in this article was exempt from publication.