This week I concluded a two-month long two-report impact evaluation project for an amazing charity. They provide independent advice services on areas such as benefits, immigration, debt and housing for their local community. Their community-led approach and ability to build trust with a community experiencing significant deprivation is hugely impactful (sorry, should have warned you \u2013 spoiler alert for the final report!). So, as usual in no particular order, here\u2019s what I learnt:<\/p>\n\n\n\n
While the tenants of it are (fairly) straightforward, the application in real life is trickier. Getting active, informed consent is harder remotely and there was often an additional language barrier to overcome. It was also a lesson in just how alienating a lot of the language in existing templates are, ironically as if they were almost designed so that people don\u2019t understand what they are signing up to. I really struggle to see why more important documents like this don\u2019t use simple, straightforward language \u2013 when it\u2019s in the interests of both sides. The upshot is finding hope in the world, because I came across something phenomenal – Bright Harbour\u2019s publicly available and open source research consent template<\/a> , as part of their work to challenge existing habits and support practitioners to improve.<\/p>\n\n\n\n I was clear from the beginning that my typing speed was never going to match my talking speed, so the only way to properly capture the interviews was through recording them, and it wasn\u2019t completely straightforward. In many ways it\u2019s good, and absolutely right that technology is now clever enough to recognise you\u2019re on a phonecall and by default block your standard voice recording app, but you do just need to get a bit creative across different devices. There\u2019s also some trial and error about where to position devices so your desk set-up works and you\u2019re picking up both sides clearly. Also, understanding where the data goes can be tricky using recording software and after hours of investigation, I decided I just wasn\u2019t confident in guaranteeing data storage to use any of the specialist apps or websites out there.<\/p>\n\n\n\n It may sound so obvious its ridiculous but wherever possible, make it a conversation, and one where you genuinely care about what someone\u2019s saying. It was easy in the prep stages to think of all the very specific questions I might want to ask with visions of being able to easily track trends across transcriptions. But it\u2019s not a checklist. In reality, you need to be warm, open, inquisitive, relaxed, and flexible about where the conversation goes, as well as being adept at deciding quickly how far to let a particular train of thought go, and how firmly you steer back onto the original structure. And listen, really listen to what\u2019s being said, what\u2019s left unsaid, and explore those pregnant pauses \u2013 there\u2019s often something really valuable there if you\u2019re careful enough to read the silence.<\/p>\n\n\n\n This was work I did remotely which meant all interviews were done over the phone. The result was a huge range in clarity across the interviews, some being crystal clear and easy to work through and some with the TV going, kids chatting, traffic noise blaring, possibly even an email being answered. It gave an extra challenge to engaging and building a rapport, as well as the basics of simply hearing what was said. It turns out a bonus extra lesson is that if you listen to a clip 16,000 times, you might just make it out. You also get much more used to the regional dialect nuances that litter someone\u2019s speech.<\/p>\n\n\n\n Similarly to the question of recording, there are some interesting options out there now for transcribing recordings such as Google\u2019s Speech to Text service<\/a> or companies like Otter<\/a> but they also were hard to unpick in terms of where the data goes geographically, who has access etc and it just didn\u2019t feel right. I also looked at some of the pricing plans and snorted at the cost. Well, at least until I started doing my own transcribing and then I suddenly understood it a whole lot more. A key lesson was definitely about allowing at least three times the length of the interview for transcribing. There was some fast speech, some accidental speech clashes, some inaudible bits, some emotional bits and generally just covering a lot of ground. <\/p>\n\n\n\n You will never catch all the punctuation gremlins. Ever. There\u2019s a point I think, however much you love the work, that after you\u2019ve looked at it for so many hours, you start not being able to read it properly. I definitely recognised that point towards the end with both reports, where I really struggled to tweak grammar, punctuation and layout because I just couldn\u2019t see it. I had a real problem submitting the final reports in the end, knowing there may still be a stray set of quotation marks, or wonky formatting or where two people have edited text and we haven\u2019t quite aligned with one voice and I\u2019d find it staring me in the face when the final designed report is ready. As with the rest of life though, there is a point where you have to say, it\u2019s good enough and make your peace with it. If I was doing it again though, I would definitely consider drafting in an extra pair of eyes. <\/p>\n\n\n\n These interviews were with people who were struggling \u2013 struggling with everyday living costs, health conditions, poor housing and inadequate benefits or fleeing domestic violence, exacerbated by overcomplicated systems and structures that aren\u2019t accessible, particularly in an area with low literacy and where many have English as an additional language. Some of the conversations were hard because their situation was so hard, despite the support of the amazing charity. Don\u2019t underestimate the processing time needed in between interviews, where sometimes you just have to sit with what you\u2019ve just heard.<\/p>\n\n\n\n It\u2019s a truth most people acknowledge, that hearing your own voice recorded is horrible. Well, I can confirm listening to yourself over hours of recording is as bad as you imagine, even when it\u2019s just asking questions or steering the conversation. Even when you think you\u2019re wise to some of the pitfalls you still occasionally catch yourself in danger of asking a leading question, sometimes I just plain talked too much and over-explained a question and ever wondered how often I say \u201cerm?\u201d 40,000 a sentence apparently. There was an accidental professional lesson in reviewing how you interact with people by being confronted with it in hard evidence, like one of those intense self-reflective sessions you can go on to learn about yourself. Informative, awkward at times and necessary.<\/p>\n\n\n\n Accept certain inalienable truths – see Baz Luhrman for more details \u2013 but that in research some people will withdraw and some interviews just aren\u2019t very useable. This is a really hard one to accept, particularly if your initial interviewee list isn\u2019t huge to begin with. What I did manage though was rather than focus on the loss of an interviewee, instead to celebrate the clarity I must have worked through the information and consent with to have someone recognise that was an option with no repercussions. We were talking about deeply personal things and it was absolutely right that people were clear from the get go this was entirely voluntary, at every point in the process, not just at the beginning. So while sad to lose an interviewee, if it meant they stayed in their comfort zone and backed out, that\u2019s far more important.<\/p>\n\n\n\n Qualitative data is rich and beautiful, but a lot more complicated than quantitative. It\u2019s full of random thoughts, distractions, tangential wonders, misunderstanding the question or not quite answering the question\u2026..and analysing it can feel daunting. Pages and pages of speech, not neat boxes of numbers and easy yes\/no splits. I don\u2019t know whether there\u2019s an equation out there about how many hours of interviews equate to how many lines of useable quotes, but it\u2019s an interesting question to ponder! Certainly, what I found was a sweet spot often after the halfway mark where people had settled into the conversation and I\u2019d understood them and their situations in order to explore and nudge the conversation a little more into the right areas. And for all the tangents, personal stories or background information that you can\u2019t use, there will be some absolute gold dust that brilliantly illustrates a core issue you\u2019re exploring, and that\u2019s where you do the happy dance.<\/p>\n\n\n\n This was a fixed price project so one where the parameters were set from the get-go and it was up to me to manage my time in order to make it feasible. As with other work, I tried to keep track of the hours I spent on it which gave me a rough day-rate equivalent at the end, and this one, on paper, came out well. It worked out almost exactly on my planned day-rate, although I\u2019m not completely convinced I logged every block of time I spent on it (because ironically it was more time and admin to do that!). However, as a rough guide, this was a good piece of work to take on financially, it sat well with my own guidelines and was boundaried on both sides as a total sum piece.<\/p>\n\n\n\n Yes. But. One of my big reflections is that I worked with a great contact at the charity \u2013 she had worked with freelancers before and had done some consultancy herself so she was all the things you\u2019d look for \u2013 quick to respond, clear in instruction, kind in feedback, upfront around payments and timings and generally a lovely person to work with. I recognise now lucky I was to have this, and how different the experience could have been if this relationship wasn\u2019t as strong, and I may revisit my contract to reflect that and explore ways of setting firmer boundaries which may be seen as flexible with a different client. I\u2019m awaiting formal feedback but we\u2019ve both signed off happy with the final reports and I can\u2019t wait to see whether they\u2019re successful levering more funding for their vital services.<\/p>\n","protected":false},"excerpt":{"rendered":" This week I concluded a two-month long two-report impact evaluation project for an amazing charity. They provide independent advice services on areas such as benefits, immigration, debt and housing for their local community. Their community-led approach and ability to build trust with a community experiencing significant deprivation is hugely impactful (sorry, should have…<\/p>\n","protected":false},"author":1,"featured_media":174,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[34,26],"tags":[13,11,27,15,12,21,25,8],"class_list":["post-169","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-contracts","category-evaluation","tag-charity","tag-consultant","tag-evaluation","tag-founder","tag-freelance","tag-non-profit","tag-self-employment","tag-start-up"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t2. Recording can also get complicated<\/strong><\/h2>\n\n\n\n
3. Conversations win<\/strong><\/h2>\n\n\n\n
4. Don\u2019t underestimate the noisiness of life<\/strong><\/h2>\n\n\n\n
5. Transcribing is very labour intensive<\/strong><\/h2>\n\n\n\n
6. Good enough<\/strong><\/h2>\n\n\n\n
7. Time to process<\/strong><\/h2>\n\n\n\n
8. Less is more<\/strong><\/h2>\n\n\n\n
9. Research truths<\/strong><\/h2>\n\n\n\n
10. Qualitative is rich and beautiful<\/strong><\/h2>\n\n\n\n
And what about the financials?<\/strong><\/h2>\n\n\n\n
Would I do it again?<\/strong><\/h2>\n\n\n\n