This week I concluded a two-month long two-report impact evaluation project for an amazing charity. They provide independent advice services on areas such as benefits, immigration, debt and housing for their local community. Their community-led approach and ability to build trust with a community experiencing significant deprivation is hugely impactful (sorry, should have warned you – spoiler alert for the final report!). So, as usual in no particular order, here’s what I learnt:
1. GDPR is hard to navigate solo
While the tenants of it are (fairly) straightforward, the application in real life is trickier. Getting active, informed consent is harder remotely and there was often an additional language barrier to overcome. It was also a lesson in just how alienating a lot of the language in existing templates are, ironically as if they were almost designed so that people don’t understand what they are signing up to. I really struggle to see why more important documents like this don’t use simple, straightforward language – when it’s in the interests of both sides. The upshot is finding hope in the world, because I came across something phenomenal – Bright Harbour’s publicly available and open source research consent template , as part of their work to challenge existing habits and support practitioners to improve.
2. Recording can also get complicated
I was clear from the beginning that my typing speed was never going to match my talking speed, so the only way to properly capture the interviews was through recording them, and it wasn’t completely straightforward. In many ways it’s good, and absolutely right that technology is now clever enough to recognise you’re on a phonecall and by default block your standard voice recording app, but you do just need to get a bit creative across different devices. There’s also some trial and error about where to position devices so your desk set-up works and you’re picking up both sides clearly. Also, understanding where the data goes can be tricky using recording software and after hours of investigation, I decided I just wasn’t confident in guaranteeing data storage to use any of the specialist apps or websites out there.
3. Conversations win
It may sound so obvious its ridiculous but wherever possible, make it a conversation, and one where you genuinely care about what someone’s saying. It was easy in the prep stages to think of all the very specific questions I might want to ask with visions of being able to easily track trends across transcriptions. But it’s not a checklist. In reality, you need to be warm, open, inquisitive, relaxed, and flexible about where the conversation goes, as well as being adept at deciding quickly how far to let a particular train of thought go, and how firmly you steer back onto the original structure. And listen, really listen to what’s being said, what’s left unsaid, and explore those pregnant pauses – there’s often something really valuable there if you’re careful enough to read the silence.
4. Don’t underestimate the noisiness of life
This was work I did remotely which meant all interviews were done over the phone. The result was a huge range in clarity across the interviews, some being crystal clear and easy to work through and some with the TV going, kids chatting, traffic noise blaring, possibly even an email being answered. It gave an extra challenge to engaging and building a rapport, as well as the basics of simply hearing what was said. It turns out a bonus extra lesson is that if you listen to a clip 16,000 times, you might just make it out. You also get much more used to the regional dialect nuances that litter someone’s speech.
5. Transcribing is very labour intensive
Similarly to the question of recording, there are some interesting options out there now for transcribing recordings such as Google’s Speech to Text service or companies like Otter but they also were hard to unpick in terms of where the data goes geographically, who has access etc and it just didn’t feel right. I also looked at some of the pricing plans and snorted at the cost. Well, at least until I started doing my own transcribing and then I suddenly understood it a whole lot more. A key lesson was definitely about allowing at least three times the length of the interview for transcribing. There was some fast speech, some accidental speech clashes, some inaudible bits, some emotional bits and generally just covering a lot of ground.
6. Good enough
You will never catch all the punctuation gremlins. Ever. There’s a point I think, however much you love the work, that after you’ve looked at it for so many hours, you start not being able to read it properly. I definitely recognised that point towards the end with both reports, where I really struggled to tweak grammar, punctuation and layout because I just couldn’t see it. I had a real problem submitting the final reports in the end, knowing there may still be a stray set of quotation marks, or wonky formatting or where two people have edited text and we haven’t quite aligned with one voice and I’d find it staring me in the face when the final designed report is ready. As with the rest of life though, there is a point where you have to say, it’s good enough and make your peace with it. If I was doing it again though, I would definitely consider drafting in an extra pair of eyes.
7. Time to process
These interviews were with people who were struggling – struggling with everyday living costs, health conditions, poor housing and inadequate benefits or fleeing domestic violence, exacerbated by overcomplicated systems and structures that aren’t accessible, particularly in an area with low literacy and where many have English as an additional language. Some of the conversations were hard because their situation was so hard, despite the support of the amazing charity. Don’t underestimate the processing time needed in between interviews, where sometimes you just have to sit with what you’ve just heard.
8. Less is more
It’s a truth most people acknowledge, that hearing your own voice recorded is horrible. Well, I can confirm listening to yourself over hours of recording is as bad as you imagine, even when it’s just asking questions or steering the conversation. Even when you think you’re wise to some of the pitfalls you still occasionally catch yourself in danger of asking a leading question, sometimes I just plain talked too much and over-explained a question and ever wondered how often I say “erm?” 40,000 a sentence apparently. There was an accidental professional lesson in reviewing how you interact with people by being confronted with it in hard evidence, like one of those intense self-reflective sessions you can go on to learn about yourself. Informative, awkward at times and necessary.
9. Research truths
Accept certain inalienable truths – see Baz Luhrman for more details – but that in research some people will withdraw and some interviews just aren’t very useable. This is a really hard one to accept, particularly if your initial interviewee list isn’t huge to begin with. What I did manage though was rather than focus on the loss of an interviewee, instead to celebrate the clarity I must have worked through the information and consent with to have someone recognise that was an option with no repercussions. We were talking about deeply personal things and it was absolutely right that people were clear from the get go this was entirely voluntary, at every point in the process, not just at the beginning. So while sad to lose an interviewee, if it meant they stayed in their comfort zone and backed out, that’s far more important.
10. Qualitative is rich and beautiful
Qualitative data is rich and beautiful, but a lot more complicated than quantitative. It’s full of random thoughts, distractions, tangential wonders, misunderstanding the question or not quite answering the question…..and analysing it can feel daunting. Pages and pages of speech, not neat boxes of numbers and easy yes/no splits. I don’t know whether there’s an equation out there about how many hours of interviews equate to how many lines of useable quotes, but it’s an interesting question to ponder! Certainly, what I found was a sweet spot often after the halfway mark where people had settled into the conversation and I’d understood them and their situations in order to explore and nudge the conversation a little more into the right areas. And for all the tangents, personal stories or background information that you can’t use, there will be some absolute gold dust that brilliantly illustrates a core issue you’re exploring, and that’s where you do the happy dance.
And what about the financials?
This was a fixed price project so one where the parameters were set from the get-go and it was up to me to manage my time in order to make it feasible. As with other work, I tried to keep track of the hours I spent on it which gave me a rough day-rate equivalent at the end, and this one, on paper, came out well. It worked out almost exactly on my planned day-rate, although I’m not completely convinced I logged every block of time I spent on it (because ironically it was more time and admin to do that!). However, as a rough guide, this was a good piece of work to take on financially, it sat well with my own guidelines and was boundaried on both sides as a total sum piece.
Would I do it again?
Yes. But. One of my big reflections is that I worked with a great contact at the charity – she had worked with freelancers before and had done some consultancy herself so she was all the things you’d look for – quick to respond, clear in instruction, kind in feedback, upfront around payments and timings and generally a lovely person to work with. I recognise now lucky I was to have this, and how different the experience could have been if this relationship wasn’t as strong, and I may revisit my contract to reflect that and explore ways of setting firmer boundaries which may be seen as flexible with a different client. I’m awaiting formal feedback but we’ve both signed off happy with the final reports and I can’t wait to see whether they’re successful levering more funding for their vital services.