Putting users at the heart of what we do

Ed-Penn.png

Ed Penn, who looks after the development of OSCAR, our online system that helps students write outstanding personal statements, reflects on a fascinating project that looked deeply at how users interact with technology, and how simple changes can make massive improvements.

OSCAR is one of the main ways we help students - it guides young people step-by-step through the often-difficult process of writing their personal statement. We know it’s effective: when we worked with the Sutton Trust to look at what makes a successful personal statement, we found that students who follow the approach outlined in OSCAR produce applications that are demonstrably better, and that a better application makes students more likely to get offers from top universities.

But we also know OSCAR’s personal statement tool isn’t perfect - writing the personal statement is a complicated process, and feedback from schools we work with sometimes suggested that students struggle to follow it. Rather than make changes unilaterally, we wanted the people who used it – young people – to be part of the design process.

Late last year we secured funding from Innovate UK, a public body that promotes human-centred design. We asked Innovate to help fund a design consultant to test young people’s views, learn more about what they needed, and test out ways to improve the design of OSCAR’s personal statement tool.

creative-design-drawing-1266012-crop.jpg

So in the three months from January to March, we worked with Rachel Smith to learn about student views and what to do to make improvements to OSCAR. Rachel is a user experience expert, and has worked with a variety of young people’s organisations to improve their digital engagement, and suggested we first test the platform as it stands to find out what really mattered to students. So we visited Cambridge Regional College, Herne Bay High School in Kent, and Brimsham Green School near Bristol, where we spoke to 19 students about their experiences of using OSCAR.

From our testing we could see straight away what needed to change. Students wanted lots more information about each stage of the writing tool – they were unsure what they were meant to be aiming for. They wanted the design of some pages to be simpler to make clear where they should enter information and discuss their work. And they wanted to be able to move around the tool more easily with more flexible navigation.

Screenshot.jpg

Taking that insight on board, Rachel, with our in-house team, built several prototypes that reflected what we’d heard. It wasn’t completely different – we knew that the current OSCAR structure does improve student writing. Instead, it aimed to clarify and simplify what young people were being asked to do.

Having sketched these prototypes out, we went back to see what young people made of the changes. We visited Long Road Sixth Form in Cambridge, Mildenhall College Academy in Suffolk, and Mascalls Academy in Kent, and asked students whether our changes matched their preferences.

Most of what we’d designed was better, some wasn’t – but crucially, we now know exactly what students want from OSCAR. Being able to get this level of insight is incredibly valuable, and means we’ll be able to make changes that will really benefit those we want to help most - young people.

We’ll make these changes to OSCAR in the next few months. They’ll be complete in time for the new academic year starting in September, and should help young people work their way through OSCAR with much less support.

As an organisation we’re always trying to do the thing that works, not the thing that’s easiest – which includes recognising when our own work can be better. We’re looking forward to making the personal statement tool more accessible for everyone.

The thought behind: Access Champions

Michael-Englard.png

Dr Michael Englard, our Director of Research and policy, and one of the co-founders of Causeway Education, explains the thought behind our Access Champions programme.

b87b7d7a77147d25b51dd2e389dd51ea.jpg

If you had a million pounds to improve the chances of bright students going to top universities, how would you start? As a keen empiricist, your first step might be to review the evidence. As a firm believer in making a difference, your next step might be to design a programme for disadvantaged students. Who could argue with your approach: analyse the evidence to find what works, then design a programme targeted at the students who need it most? 

But as many who work in Widening Participation might agree: what appears obvious rarely is. This is a curious story where dead-set certainties seem to melt away. 

This story began because, as teachers and academics, we knew that the transition from school to university was fraught with snares, misunderstandings, and unrealised aspirations. Some of the problems seen in Widening Participation are clearly structural and, to some degree, intractable without deep-seated changes in how we engage students, and their parents, from a young age and ensure they have an excellent education. 

But not all Widening Participation issues are impossible to solve.

In 2013, Professor Vikki Boliver at Durham University looked at ten years’ worth of UCAS data and showed that, even when state and privately educated students got exactly the same grades, state school applicants were up to a third less likely to get an offer from a leading university. Since then, strong evidence has pointed towards student attainment – which we might consider a structural problem – being the biggest key to increasing participation. But even with excellent grades state school students were not always getting the offers they deserved. 

Aside from grades, we felt that the most obvious place to start was improving the quality of applications made by disadvantaged students. There was some fine research in the area, not least from Dr Steven Jones and the Sutton Trust. Dr Jones had shown that there were big disparities in the quality of personal statements written by students in the private and state sectors. 

What, you might think, would be the point of working on personal statements? There are reams of information in the public domain and virtually every university offers sessions on writing. Looking more deeply, though, it seems that most information takes the form of guidelines that are easy to misinterpret and one-off events that don’t always have a bearing on the final product.

p034t6py.jpg

UCAS’ main guideline to students is “try to stand out” But how does an eighteen year old interpret this? In the final line of their personal statement, one finance applicant in Dr Jones’ study concluded with the memorable line “I want to study business and I mean business” – perfect for standing out in The Apprentice boardroom but utterly inappropriate for a personal statement. 

Working with the Sutton Trust, we developed our first programme, the Academic Apprenticeship, which focused on getting disadvantaged students to boost their personal statements by going into detail about an academic topic of interest that really appealed to them. 

Rather than saying how much they wanted to study Law, for instance, we encouraged students to research a particular case. We reasoned that getting disadvantaged applicants to show rather than state their enthusiasm for a subject could compensate for the inequalities in the quality of work experience and extra-curricular placements Dr Jones had identified. We paired students with online mentors and got them to follow subject-specific pathways.

We wanted the evaluation to be as rigorous as possible and set up participation and comparison groups. Students in both groups had attended a Sutton Trust Summer School, which meant they shared a similar academic profile and motivation. To ensure the scheme benefited applicants who needed it most, students in the participation group went to lower progression schools than those in the comparison group. In the end, UCAS data showed that 100% of students in the participation group got at least one offer from a Russell Group university compared to 73% in the comparison group.

Getting every student a Russell Group offer sounded like a huge success, but we had inadvertently discovered a different problem: when the students went back to school in September, they reported that many of their teachers had strongly advised them to remove the academic topic of interest which had been the key part of the programme. 

To see how widespread this was, we worked with Dr Jones and asked academics and teachers to ‘mark’ the same set of personal statements. The results were striking: teachers and academics agreed in only 23% of cases. Teachers found the valuable topic of interest “too long”, “essay-like” and “impersonal”. 

From this point on, our major focus has been to train senior teachers to lead change in their schools and colleges. 

AC-Causeway.png

Hence our new programme: Access Champions. It’s an intensive two-year programme that concentrates on improving the systems schools and colleges use to support their students. Reviewing the evidence and using an approach based on the excellent Gatsby Benchmarks, Access Champions charts progress in six key areas, from using destinations data to co-ordinating encounters with Higher Education. 

Currently in progress across England, the programme is showing good signs of success. 

So, if you won the lottery and decided to spend some of the money on improving access to higher education, would you necessarily start by working with students?
 

Better Statements

This blog was originally published by the Sutton Trust.

Dr Michael Englard, Sam Holmes and Helen Drummond discuss the implications of the evaluation of their Academic Apprenticeship programme.

At the end of 2012, the Sutton Trust published research from Dr Steve Jones at Manchester University which showed that differences in university offer rates between independent and state school pupils with the same A level grades could be linked, in part, to the quality of their personal statements. This followed Dr Vikki Bolliver’s stark finding that state school students with exactly the same grades as their privately-educated peers were a third less likely to receive an offer from a leading university.

In response to Dr Jones’ findings, The Sutton Trust funded the ‘Academic Apprenticeship’ programme for more than 100 of their summer school students prioritising those from the most deprived backgrounds. Run by our organisation – the HE Access Network (HEAN) –  the ‘Academic Apprenticeship’ programme had a simple premise. The key to writing an outstanding personal statement was to improve its academic content. This meant encouraging students to do some independent research on a topic which interested them and to write a short paragraph in which they analysed and reflected on their findings.

Much of the previous debate around the quality of personal statements had been focused on what academics call “social” or “cultural capital”. The state-educated students in Dr Jones’ report had fewer and less prestigious experiences than their privately educated counterparts. But did this really matter when applying for university? As teachers and academics, we had seen statements crammed full of EPQs, top work experience placements, and multiple university lectures and outreach programmes. On their own, these opportunities meant little because students were not actually analysing and reflecting properly on what they had seen, heard or researched.

As many teachers will tell you, setting a student a “wider reading” task without a clear structure to support analysis is more or less pointless. Working with academics and teachers, the HE Access Network designed specially structured subject-specific pathways which helped students to evaluate their reading or work experience in the manner of an undergraduate.

An evaluation of the ‘Academic Apprenticeship’ programme is included in Making a Statement, a follow-up research brief from Dr Jones, published today by the Sutton Trust. Encouragingly, it shows that, with the right support, the playing field can be levelled. Every one of the students in the study group – many applying for competitive courses like Medicine and Law – got at least one offer from a Russell Group university. The ‘Academic Apprentices’ significantly outperformed the comparison group which was made up of equal-attainment students who went to the same Sutton Trust summer school but attended colleges and schools with higher progression rates.

During the programme, we came across a major issue. Many of the ‘Academic Apprentices’ were being told by their teachers to remove the key paragraph of independent research and reflection which our subject pathways had encouraged. Repeatedly, teachers were saying that these sections were too “long”, “impersonal” or “essay-like”.

Working with Dr Jones and the Sutton Trust, we set up a second evaluation to compare how academics and teachers approached personal statements. Strikingly, it reveals that academics and teachers have very different views on what makes an excellent personal statement.

The academics’ comments indicate that the main function of the personal statement is for a student to demonstrate their suitability and motivation for a specific course. The research suggests that the best way for students to demonstrate their suitability is to analyse a topic which goes beyond their current syllabus. In subjects such as English, this might mean giving a developed claim about a specific feature of a text; for a more vocational course such as Medicine, a detailed reflection on work experience would serve a similar function.

But while academics consistently put a premium on these sections of reflective and detailed analysis, teachers often thought that these sections would actually lessen the chance of their students gaining an offer from a competitive university.

Dr Jones’ comparison shows that the “personal” element of the personal statement needs to be interpreted carefully. Unsurprisingly, admissions tutors seem to be primarily interested in a student’s academic personality. There is no evidence that students’ attempts to provide dubious “lightbulb” moments or to link multiple parts of the statement to autobiographical reasons improve their application.

To give just one example, here’s an extract from the personal statement of an Academic Apprentice:

“I especially enjoy reading War poetry as it offers subjective and emotive responses that are rarely found in historical accounts. To pursue this interest, I studied the pastoral idyll outside of the curriculum and explored the impact which the First World War had on ideas of rural land, particularly in Hardy’s poem ‘In time of the “Breaking of Nations”’ (1915) and Edward Thomas’s ‘As the Team’s Head Brass’ (1916). … I feel that the interlinking of the war and the land makes the war seem “nearer” in Thomas’ poem, whereas in Hardy’s work the presence of war is distanced through the use of everyday imagery such as the ‘maid and her wight’.”

Whilst the academic thought that this showed: “good evidence of wider reading and critical ability”, the teacher considered that the section was “more language analysis than overview”. While the academic likes the detail in the extract, the teacher wants a less focused, more generalised “overview”.

How can we explain such divergent views and ensure that advice is improved? Firstly, we need to support our teachers better. Staff working at low progression schools do a consistently heroic job educating their students and dealing with the turbulence thrown up by the move to linear A-Levels. As Dr Jones’ report recommends, we need universities to be much more transparent about what they want to see on personal statements, and, more importantly, how they evaluate them. There is no excuse for the continuing mystery that surrounds the applications process and the lack of clarity does a disservice to our teachers and our students. The first step to improving the quality of personal statements is for universities to publish explicit, subject-specific criteria which show what academics are really looking for.

Dr Michael Englard, Sam Holmes and Helen Drummond are directors of the HE Access Network.