Rebecca: Kia ora koutou. And welcome to today's Rethinking the Logic Model.
Rebecca: Sorry, that was just me hiccuping as the light went on. Ngā mihi nui ki a koutou katoa. I'm Rebecca Gray. I'm a social researcher and evaluator in the Allen + Clarke team. And I am your host for today's webinar with my learned colleagues here, Fiona and Marnie.
Marnie Kia ora koutou. I'm Marnie Carter and I'm a senior evaluator here at Allen + Clarke. So lately I've been getting very interested in systems thinking and how that can help us as we're doing evaluation.
Fiona: Kia ora koutou, I'm Fiona Scott-Melton and I do a lot of work in the outcomes space. I'm interested in what different services or organisations are making, the critical conditions for success and the assumptions they are based on. And as well as that, how we ensure that the people programmes are designed to serve are doing just that.
Rebecca: Awesome. So for those of you joining us for the first time, Allen + Clarke's a consultancy. It helps organisations make complex, high-stakes decisions using evidence. So those activities, as Fiona said, can work well for the people in the communities affected. Now, we are coming to you live from our occasional studio in the Wellington office today. Isn't that lovely? Yes, it's our studio. We also have a base in Australia and we work there and across the Pacific. So welcome to everyone joining us from further afield.
We specialise in strategy, change management, evaluation and policy. We're increasingly asked to help teams build their logic models and theories of change. So you've heard of these, right? Yep. I think, thank you, good. I think most of our audience have too. But people have some pretty varied experiences with the process of building, negotiating and actually applying logic models and theories of change once their program's live.
Marnie Yeah, absolutely. And it is quite a hot topic, actually. So Fiona and I both did presentations on logic models at the Aotearoa New Zealand Evaluation Association Conference. That was held in Ōtautahi Christchurch at the start of August. And we'll be giving you a bit of an overview of some of the key themes that we presented on and what we heard today.
Fiona: Yeah, sure. And I'm going to cover how personae, fictitional characters based on real information about different population groups, can be used to make logic models or theories of change more inclusive and used to make more robust assessments of how a program is performing.
Marnie Yeah, and similarly for me, I'm going to talk about how we can bring in some systems thinking concepts into our logic models. And the reason is to really bring in some thoughts about the wider context that a program or a policy or an initiative is delivered in and how that can affect whether the program actually achieves what it's meant to do.
Rebecca: Cool. And I'll be asking some questions as we go, including some that our attendees have sent through in advance, so thanks for that. Hey, if other questions come up, please feel free to add your own into that chat as we go, which is on your screen somewhere. And we may discuss these later on. Now, some of you have said that your big challenge with logic models is managing the assumptions that they tend to have built into them. Someone in the audience called these leaps of faith. So we're going to start by talking about what a logic model looks like and where these assumptions can be kind of hidden.
Fiona: Okay. So people often use terms such as logic model, theories of change, interchangeably.
Marnie There you go.
Fiona: A logic model is essentially a roadmap showing how you expect to get from where you are now to your desired or intended outcomes. As you can see on the screen, it maps out the logical connection between your inputs, so what you have, such as financial, human and physical resources, and activities, what you do, like delivering health services or housing. Which it is assumed will lead to outputs, what you produced or delivered, such as how many houses were built, and contribute to outcomes, the difference a program makes, like improved physical health.
Marnie Yeah, absolutely. And I have to say, I actually really like logic models. Bit of a fan. So the reason I like them is I think they're really useful for getting everyone on the same page. We can be really explicit about what are we actually expecting to do with this program and how it works. And then from an evaluation perspective, we can use logic models to provide a really good framework for understanding what we should be measuring. And why. So when we're designing our evaluation, we can use a logic model to identify what change to look for.
It helps us then measure and understand if the program is actually working as intended. And also if things aren't working and our evidence suggests there might be a challenge, we can go back to the logic model and help us to diagnose where that breakdown might be. be occurring.
Fiona: Yeah, I agree with all of that. But I think one of the challenges with logic models is that they tend to treat everyone the same, even though how people experience an initiative can vary hugely. This can lead to over-representation of what's important to certain population groups and under-representations of others. And as a result, they can unintentionally contribute to inequities in the system.
Marnie Yeah, true. I mean, can't argue with that. I would say the other problem that I think logic models, or limitations rather than problems, they really assume a linear pathway. So you've got A leads to B leads to C. But in the real world, when things are a bit more complex, it's just not that simple. And so when you've got a program or a policy, it often might have multiple decision makers. People might have competing priorities or there's contextual factors that you just can't capture within that A to B to C linear model.
Rebecca: Yeah, absolutely. And that's something that I know some of our audience are aware of from the pre-registration comments too. Like quite a few people did mention. Things aren't as linear in reality in a complex system and many influences. So there's quite a lot of awareness of that limitation, I think. And it seems kind of hard to capture in the type of logic model we just showed.
Marnie Yeah, 100% is. And then the other thing to think about is that when you've got a program that's the same design, but you're implementing it across a couple of different sites, that multiplies the complexity. At each site, you're probably going to have different resources. You might have different populations that are being served. The organisation that's delivering it might have a different culture. You're already going into an existing system. And so that just makes the complexity really difficult to capture in that linear model.
Rebecca: Yeah, yeah.
Marnie I guess if I think of an example, a programme that's based at a community centre, say, might work really well in an urban setting where there's good transport links to get there. But if you try and put that in a more rural area where there's no transport access, it makes the whole programme model potentially not work as well.
Rebecca: Yeah, yeah, for sure. So Fiona, you've been thinking about some of these challenges and you've got a suggestion or a fix for this. We've got a fix. So what have you got? Do tell. And also how do you pronounce it? Oh, that word.
Fiona: Oh, that word. So that word, I pronounce it persona. Cool. And I'm not sure that it's entirely a fix. Okay. No. No, because I think it's important to remember persona, like everything, have limitations. But I certainly think that they can be used to make logic models more inclusive.
Marnie I have a question. What is a persona?
Fiona: There you go. I was just about to say. A persona is a brief description of a fictional character that represents a specific population group that an initiative is designed to serve. It typically includes demographic information such as age and other identifying characteristics like living situation or cultural identity. In this context... They can set out the person's own aspirations or preferences, what they want for themselves, and sometimes their whānau and community.
Marnie Okay, yeah, I've heard it used, persona, from usability research in a sort of market segmentation exercise, you know, check how a service is likely to work for different types of users.
Fiona: Yeah, but it's also been used in evaluations to show different population groups. So in social service settings, the SONA might also include things like variation of functional support needs, such as hours of support required, level of oversight, or use of equipment or mobility aids. They could also include information such as behavioural risk factors, such as patterns of risk-taking, or non-compliance with recommended supports.
Marnie Okay, right, I do get that and it makes sense, but I'd quite like a concrete example where we could have something to hang this idea on.
Fiona: Yeah, for sure. If we're evaluating, for example, an employment program, we might develop a persona for a young person who's trying to enter the job market for the first time. And another for an older worker who has been made redundant.
Marnie Okay, got it. So for those two different people, their needs, they might be quite different, or the pathway they take through the program might be quite different, even though they're both participating in the same program.
Fiona: Yeah, absolutely. So, but so they help us to understand what good looks like for each of those two groups. Okay. Their needs and aspirations and what outcomes we should be measuring.
Marnie Okay, cool.
Rebecca: So how have you used them then, Fiona?
Fiona: So that's a good question. So when developing logic models, I've used them to identify the outputs and outcomes for different population groups, increasing visibility of what's important to them and the pathway to getting there. So most logic models, they indicate essentially a single pathway, as Marnie mentioned before, from activities through to outputs and outcomes. But in doing that, they don't reflect the different pathways and outcomes for different population groups.
Rebecca: Sure, sure. So let's discuss an example of a logic model that you've used in the past.
Fiona: Yeah, absolutely. So I've used this approach when developing a logic model for people with physical disabilities moving into a residential home. Personae were used to show different starting points and needs. So, for example, some residents came from their family home. Others were coming from a rest home. And others were moving from within the organisation itself. It meant some people moving in were well known to the organisation and already had things like individual plans in place. Individual plans set out individuals'goals and how they're going to be achieved.
A little bit like a professional development plan, only for everyday life.
Rebecca: Right, okay.
Fiona: I wasn't anticipated that settling into their new home would, it was anticipated for this group that settling into their new home would be fairly straightforward.
Rebecca: Right, because they've got that context around them already. And they've got their plan, so they kind of know what they're doing.
Fiona: They're well known, they've already got relationships, that makes it pretty easy.
Rebecca: Okay.
Fiona: Yeah.
Rebecca: But then others were different, right?
Fiona: Yes. So whereas others coming from rest homes or their family home were largely unknown to the organisations, of course, and it was anticipated that there would be a longer settling in period in which the relationships were built, individual goals were identified and individual plans developed. They may also initially need more support with daily living skills such as washing their clothes, cooking and cleaning because they may not have had those opportunities before.
Rebecca: Yeah, okay. So you've got the logic model for the one service, but not one pathway or experience because there's layers to it depending on what people are going to have experience. I mean, that's how things go in the real world. Cool.
Fiona: Right, so by developing four distinct personas, we were able to develop a more nuanced logic model showing different paths and trajectories. For example, one persona depicted someone arriving from their parents'home with a stable condition who was aspiring to become more independent over time. Their outcomes kind of focused on things like improved physical health and improved life satisfaction over time. Whereas someone with the same starting point but a degenerative diagnosis, which meant their physical ability was going to diminish over time, is aiming to maintain their independence for as long as possible.
So the outcomes were more about maintaining physical health and maintaining their life satisfaction.
Marnie Got it. So you've got two quite different groups represented by Persona and you make it really clear what those different pathways and outcomes are going to be. Okay, so that sounds good. But then, how do you actually use that in your evaluation practice using that persona approach?
Fiona: Yeah, good question. So in terms of identifying persona, my normal starting point is to work with an organisation to identify three or four key populations that they try to serve. So in the case of the adults with physical disabilities, we facilitated a workshop. With current residents to develop an understanding of what sorts of supports they need, what were their aspirations for the future. You could also consider which population groups are not represented or least likely to be seen and build a persona around them, preferably after engaging with that community.
Marnie Okay, so that's really the equity lens that you mentioned at the start and making sure those different groups really understand and have them reflected in the logic model.
Fiona: Exactly, and I think it's best trying to go directly and talking to them if possible because relying on organisational perspectives, which is the other option, means that the logics are built on the assumptions, and these could be wrong. They often actually don't really align very well.
Marnie Or at least incomplete, right?
Fiona: Or incomplete, yeah. And so once you've identified the persona, then you can map out the different causal pathways from activity to outputs and outcomes for each persona, and then map it out into an overarching logic model. A bit like our nested logics on the screen.
Marnie Okay, so that all sounds good and you've got your logics and they reflect different persona, but then how does that flow through into actually when you're doing your evaluation activities?
Fiona: Yeah, so there's a variety of ways. Persona can be used to help answer questions such as who does the program service? Are the outcomes equitable? They can be used to identify what success looks like when you're developing an evaluation framework. You could use them to develop interview guides. You could also use them to inform your sampling strategy to identify key population groups. We've also used them when developing performance measures and in social return on investment calculations to make them more nuanced and robust by weighting different outcomes.
Marnie Yeah, now as you can probably guess, Fiona could elaborate now on a bunch more of the
Rebecca: the technical content about what that social return on investment calculation involves. Totally. You could, but you're not allowed. So we're going to include a link to another webinar about this. So if that's your thing, check it out in the email afterwards. More SRI content is available. But we're going to move on to Marnie's alternative now, which was also what she presented at AES. Sorry, Anzia. So what does your thing look like, Marnie?
Marnie Yeah, so this is where I'm going to bring in my current pet interest, which is about systems thinking, which comes in really handy in this situation.
Rebecca: Yeah, and in this particular context, what do you mean by systems thinking?
Marnie Okay, so it's basically the idea that when we've got a program or an intervention, It doesn't operate in a vacuum. It's always delivered in a context. So that's within a web of existing people, relationships, there's pressures and there's conditions, and that all actually influences and shapes whether the program actually works. So while you might have your theory and your design, It's very much influenced by the context of the system in which it's delivered.
Fiona: Okay, that makes sense. But I'm curious, how do you incorporate that into a logic model?
Marnie Okay, well, I'm going to give you a metaphor. So I like to think of a logic model as something like a living being. So if you think back to that familiar linear structure, that's something like the skeleton. So we keep all of those inputs in the activities and the the outputs and the outcomes, that really is important and it does still matter. So that's central and that entails us what the intervention intends to do and how. But if we just think about that on its own, it's really the essential bare bones, but it's missing the complexity that makes it come to life.
Rebecca: Sure, yeah. She could have talked about ecosystems, but she's going creepy. So, okay, all right, move on.
Marnie Okay, ecosystems can work, but I'm going to stick with my slightly creepy skeleton. So if we think about that, the systems layer is the flesh or the blood or the nervous system that sits around the skeleton. And that's what makes it move and adapt. It can respond to the world around it rather than just sort of standing there and looking correct.
Rebecca: Okay, but so what goes into that lab? Because flesh and blood isn't exactly a methodology, right?
Marnie That's fair. So no, it's not enough. There are four system elements that I tend to focus on. So they're up there on the screen. And what I like to do is work through and think about each of those in turn. So the first is people in the system. Now, you would think that given most policies, programs, interventions are about people, they would be there in the linear logic model, but often they're not. So the first step is really identifying who is involved and what do they care about? What's their priority?
Rebecca: Sure, and that sounds a little bit like stakeholder mapping, which we do for other stuff.
Marnie It is a little bit like it, certainly related, but the difference is that what we're actually asking asking about is their priorities. And then importantly, how they create some friction or some momentum in our logic model, not just kind of listing who they are. And then that actually brings me quite nicely to the second systems element, which is about the tensions or the conflicts. So you've got your people and their priorities, but then you might need to think about where do those create tension or where do they pull against each other?
And it's important. When I often say this when I'm working with clients or groups, I often get a little bit kind of uncomfortable and say, oh, you know, do we want to lift the lid on that? Do we want to actually see where the conflicts are? But I think it's really important because it's not to say that the program's going to fail or it's not well designed, but it is about being honest about what are those conditions that we're operating in. Yeah. Which nicely brings me to our third system element, and that is external influences.
So that relates to what's happening around the program that we can't control.
Rebecca: Like, oh, your funding just changed. Oh, there's a global pandemic.
Marnie Yeah, exactly like that. And then the fourth system element is assumptions. So I think this was raised by a few people prior to... Prior to this webinar, which is very important. So assumptions are essentially things that we're hoping are true, that we may not have tested or we may not have even put on the table. So you might have quite a different assumption to what I have. And for logic models, where we make those really explicit, we put them on the table, and then when we're evaluating, we can actually go and test whether they hold, rather than kind of finding out halfway through the delivery of the program or the evaluation.
Rebecca: So how do we apply that in practice to logic models?
Marnie Yeah, good. So what I actually think works best, and I find easiest, is to create an example together. So I'm going to ask everyone to pull into their happy place and bring out their imagination. And what do you think about a program? Let's call it Roadside Community Health. So this is essentially a mobile health screening service that drives around and it visits remote and rural communities on a rotating schedule.
Rebecca: Cool. So I'm like, I'm picturing a van.
Marnie Yes, and that is absolutely right. We've got a van. We've got a couple of clinical staff. We've got some basic medical equipment. And the idea is pretty simple. We're going to bring health checks to people in areas that don't have easy access to a GP or a clinic. So coming back to our logic, if we look at this diagram, we can see that on paper, the logic model looks pretty clean. We've got the van, comes to the community, people come and get screening, they get their results and then they're referred on if needed.
So pretty much fits nicely into that inputs, activities. Outputs and outcomes, they all line up nicely. They do.
Rebecca: But do they line up too nicely? Is this the sort of thing that our audience has mentioned, if it's too nice? Straight forward, it might not reflect the way that all those outcomes are shaped by all these different things in the system, right?
Marnie Yes, very nice foreshadowing there, Rebecca. Indeed. Exactly. So, yeah, that's a limitation of linear logic, right? Yeah. If we think about that, if we don't really put the thought in and think about the system elements, there's lots of unanticipated risks. Yeah. So it might be that the van schedule doesn't line up with when people are free. There might already actually be a health provider who doesn't particularly love a van coming and competing with their territory. Or it might be that there's brilliant uptake in one town and then the next no one comes.
And why? We don't know. So next step then is to really think about in this context, what are our four systems elements? So if we start with number one, that's people in the system. So can we do a little brainstorming? Who do you think are the key people in our roadside community health system?
Fiona: Well, funders come to mind. I'd imagine they're wanting value for money and to hit their reach targets. And clinical staff sound to me pretty important because they'll be running the screenings, right? So they'll be focused on workload and training.
Rebecca: Sure, and if you're coming into community, you'll have some sort of community coordinators involved. You know, they'd be building trust, they'd be thinking about those local relationships.
Marnie Yeah, exactly. And of course, there's the participants themselves. So their priorities would be privacy, convenience, and of course, getting the healthcare that they need.
So what I like... about this exercise is that as soon as we start thinking about the different people in the system, we can already see that they don't all necessarily want the same thing and that's where our tensions start coming up. So if I think about the examples that you both gave, our funders, as you said, they want reach, they want people coming straight through the door and getting value for their investment. But then that coordinator that you mentioned, Rebecca, they might want to slowly take time and build the trust in the community.
So those things can kind of pull against each other and create a bit of tension. Any thoughts about what some other tensions might be?
Fiona: Well, as you just mentioned, I think it's reach versus depth. Yeah. So do you visit more towns or spend longer and fewer?
Marnie Good.
Rebecca: Yeah, and coming back to that sort of one model fits all thing, like do you roll out a standard model or do you take extra time to adapt it to different contexts and different communities?
Marnie Yeah, that's a great one. And those tensions show that our traditional inputs to outcomes model, you know, it's limiting. And so we really need to... Stop and actually ask where is that friction and where is it likely to be?
Rebecca: Yeah.
Marnie So quite an important one is external influences. So we think about roadside community health, what are some things that might happen outside of the programme that we don't necessarily have any control over?
Rebecca: The weather and possibly the roads.
Marnie Yep. Yep, no, you're right. So in rural New Zealand, as we all know, the weather is not exactly cooperative all the time, particularly in winter.
Fiona: And funding cycles too. If the contract changes, does that mean the whole model collapses?
Marnie Yeah, really good point. And both of those are a good reminder that many of the risks in the program aren't actually within its control. But yet we do need to be aware of what's in the broader environment and how that might impact on it.
Fiona: Cool.
Marnie And then our last one's assumptions. So any thoughts about things we might be hoping are true in this situation, but we might not have tested?
Fiona: Okay, so I think we're assuming that people trust a van they've never seen before. Yeah.
Marnie Yeah, it's also assuming that once that van leaves, that there's going to be somewhere that people can go for follow-up care if they've found something out. Yeah, I mean, that's a big one. Essentially, this whole program rests on the assumption that if we've got the mobile clinic that's meant to bring access to services, it won't work. If it screens a lot of people, identifies a lot of need for secondary care, But then there's no services to refer those people on to.
Fiona: Yeah. So how does this all come together, Mo?
Marnie I am glad you asked. So this on the screen is what it looks like where we bring it all together. So you can see that we've got that same skeleton, the core logic model running through the middle and then nothing's changed there. But what we have added is what's wrapped around it. So right at the top, we've got the external factors that we mentioned. That's like the road conditions. It might be workforce shortages, particularly in a health context, or funding cycles that change. Around the outside, we've got our key people there.
So that's our funders, our clinical staff, our community coordinators and our participants. And we've been really clear about what we think their motivations and priorities are going to be.
Rebecca: Right, cool.
Marnie Then we come to my favourite bit, our tensions. And that's sitting right in the pathway itself. So in that diagram, I used lightning bolts to represent those, but some people use thunderclouds or anything that looks kind of a bit scary. Or weather-y. And when we're planning the service model, we really do need to work through those tensions. So here we particularly prioritised the tension of standardisation versus flexibility to meet the needs of the different populations and areas. And you mentioned fees, so reach versus depth.
Do we go to lots of towns or do we go to a few and do them really well? So we place those in the logic model between the activities and the outputs to kind of show where that friction is going to manifest.
Rebecca: Right, because that's sort of where the frictions affect the connective tissue that means that the body can't move differently.
Marnie You are on board with my coffee metaphor. No, I'm not. I love it. No, I'm not. And then underneath it all there was our assumptions. So that's basically what the foundations that the whole model is resting on. And so we put assumptions there that, as you said, people will trust and engage with the service, that the referral pathway is gone, and a few more like staff have the capability to work across diverse communities. So that really shows that we've taken our skeleton, we've built around it. And we've got our living system now.
Rebecca: All right. And it does look cool. Like it's a fun looking map, at least to me it is. And I guess having something mapped out visually does perhaps help with another issue that our audience raised, which is about it can be difficult getting others on board with the logic in the model and like how you use it.
Marnie Yeah, it is. And actually that came up at the ANZIA conference too. Right, yeah. So I had some discussions with attendees about how having those systems elements laid out in a visual diagram can help make really something concrete that you can talk through with your client or your steering group or the program implementers rather than just trying to talk about these concepts in abstract. It's also actually a really good way to help get program designers to test those assumptions or tensions. So when you've got them there, they're documented in the model, people can see them and say, actually, I disagree, or we haven't captured that correctly, or add a new one of their own.
And that helps us to just work through before we've actually gone out and started delivering.
Rebecca: Yeah, it's a useful prompt to have something to actually point at and look at to get it the way.
Marnie Yeah, 100%. That said as well, the other point that came up in the ANZIA conference was about tailoring how much of the systems aspects that you use. And that very much depends on the program or who you're working with or the audience. So sometimes it makes sense to go full out. In the roadside community health example, we used all the elements and they were fully mapped. But actually, sometimes you just need a lighter touch. So if it's a small program or you're working with a group that's quite new to logic models, in that instance, you might just want to add the assumptions under a fairly standard logic model.
Or it might be that the main value is actually thinking about those contextual factors, and that's what you really need to focus on. So it's sort of horses for courses.
Fiona: So it's not a checklist then that you have to complete every time?
Marnie No, absolutely not. So that's the nice thing about this approach. You don't need to build an elaborate systems diagram in every instance. It's more like take what you need, leave the rest.
Fiona: Okay, cool.
Rebecca: You could, but you don't have to. Yeah, nice. So Fiona, you were also at the ANZIA conference. Did you have anything else that you wanted to add from there?
Fiona: Yeah, look, I was really interested in how often logic models and outcomes were discussed in sessions. And I think this was partly in the context of increased pressure to provide numbers. What some would view as hard evidence. About the difference a program is making, including social return on investment calculations.
Rebecca: Okay, so we're still talking a lot about that, eh?
Fiona: We're still talking a lot about that. It was a hot topic. It was a hot topic. Yeah. Yeah. Yeah. So one of the questions was really about the emphasis on outcomes. Okay. And many outcomes, as we know, can't be easily measured, certainly not quantitatively. And that's quite a challenge, I think, for many organisations. I'm not going to go into it here.
Rebecca: We could. We could, yeah.
Fiona: I won't. I'll resist. But I've observed, one of my observations is that many organisations, they seem to focus on the outcomes.
Marnie Yeah.
Fiona: But I think that's a distraction.
Marnie Oh, hold up, hold up. That's a controversial opinion. So if organisations aren't focusing on outcomes, what should they be focusing on or what should they be doing instead?
Fiona: Yeah, I get that it sounds controversial, but I see outcomes as a by-product. It's an indication of whether the activities and outputs are working as intended. And the outcomes, I see them as acting as a feedback loop that provides an indication of whether things are tracking as intended or whether the activities need to be tweaked or changed. So I'm not saying they're not important. But I think when we fixate on them, we're missing the important thing, which is actually it's what you're doing that will actually enable them to be achieved or not.
Marnie Yeah, that's a really good point. And I think what you mentioned there as well, it reminds me of another systems element, which is around feedback loops. So that's another thing that you actually can document in your logic model. What feedback loops are occurring within the system? In what ways can people learn? And in particular what does that mean in terms of changes or tweaks to the program or the policy as it's being delivered.
Fiona: I agree and I think also we have to remember that when you achieve some outcomes they're not actually static, they're state. So the feedback loops also can indicate to you whether they're actually increasing over time or decreasing because in actual fact quantity and quality of outcomes actually changes over time. So it's not a tick box, hey, I've achieved that and on you go. You have to keep tracking it and keep making sure.
Marnie And how does that work with the persona approach then, Fiona? Like if you've got different outcomes that you're tracking and different feedback loops for different persona, that sounds like it's getting a little bit complex, is it?
Fiona: It is getting pretty complex, but I think going back to what I'd said, I think what you can use the persona to do is actually to test whether the outcomes are actually equitable. So when you look at those feedback loops and there's changes in the system, is it negatively or positively affecting certain groups? Because what often happens in programs is those who wasn't actually really designed for are the ones who get the most benefits. And that's not uncommon. And those who it was designed for can actually be falling off.
So it enables you to test things.
Marnie Okay, that makes sense. One more question. Do you use personality reporting?
Fiona: Not normally. I mean, I've used them heavily in social return on investment calculations to weight the outcomes. But you could use them in reporting. Because if I was going to do that, I'd probably talk about the population group. So it would be woven from... I like knitting, so often things about weaving. And I think about weaving it into the findings so that they actually become part of it. So you would want to say, oh... This sort of group is not getting the support they need. And you'd want to try and, through an evaluation, identify why that might be the case.
It's a bit like back to your systems things about what might be working really well. well in urban and not in a rural. What might be working really well in a rural, but actually doesn't apply in an urban.
Rebecca: Yeah. Yeah, yeah.
Fiona: Because they're different communities.
Rebecca: We have used them in some of the more sort of social research style work that we've done as well, which does show, again, like the impact of certain system elements on different groups of people in society.
Fiona: Yeah, and you can use them to track somebody's journey through and to see if that pathway is getting stuck somewhere.
Rebecca: Cool. Yeah. And, yeah, like sometimes the act of evaluating something and finding out an outcome, as you say, that's the feedback loop where it goes back to that might then change how they do the activity, which then, you know, like you say, it's live, isn't it? It's not static.
Fiona: No, it's not. Yeah.
Rebecca: Cool. So here's what I'm hearing between the two of you, right? So Fiona has been saying you don't need to build a logic model around a single average typical participant, right? You can build causal pathways that different people follow, let your evaluation activities, including potentially your SROI weighting, reflect all of those distinct pathways, make it more realistic.
Fiona: Yeah.
Rebecca: Cool. And Marnie's been talking about how you can add more nuance to a theory of change. You can map those people, tensions, forces, assumptions, all the things that sit around it that actually influence what happens within it.
Fiona: Yeah, got it. Yeah, totally. I think that's spot on.
Rebecca: Yeah.
Fiona: But I just want to chip in here and say that sometimes a simple logic... Model is all that's needed and you shouldn't over egg things unnecessarily.
Rebecca: Also fair. Good. Cool. So This is just about the end of our kind of main bit that we were going to discuss, but we can get on to some questions. Before we do that, we have four main takeaway ideas from today that you might want to apply in some of your work. So firstly, look at your current logic model, identify the different pathways to change that it's assuming. Think about those assumptions. Think about which groups of people are likely to experience a different outcome or perhaps to reach that outcome in a different way.
Speaker 2: Yeah.
Rebecca: Okay. Second point, try building a persona for each of those different groups based on what you know about them. Now, this could be what you know from your engagement, from your existing evidence that you hold, or from the informed judgments you can make about your program and the populations you work with. The third thing to consider is the system elements, right? So who's got a stake? What do they want? What are these unresolved tensions? What's outside anyone's control? And which causal links are we assuming, perhaps, without having tested them?
Cool. So we're thinking about all that. And the fourth thing we were going to suggest that you try is a visual model. So include those systems elements can be a starting point for discussion, for planning, right? It can let stakeholders see some of the relationships, the context, the tensions and the assumptions, which you can then test, you can debate them, you can refine them and make it really work for people.
Speaker 2: Yeah, absolutely.
Rebecca: Yeah, cool. So we've put something about both of these methods into a... takeaway tool for after the webinar. It's a thing called a persona canvas and a systems checklist, which you'll be able to download after today's discussion.
Cool. So that's it from us. We would like to hear from you. Feel free to drop your questions in the chat, as we mentioned. Fiona and Marty will take as many as we can get through, and we'll see what we've got to kick off. All right. So as noted, we did have some pre-submitted questions. So we wanted to... Look at a couple of the main themes coming through that perhaps we hadn't talked to as much in the presentation already. So one of the key themes was about communicating your logic model so it can be used.
People talked about, well, how do you get others on board? How do you get teams to understand them? How do you make it feel real for the people who are delivering the services?
Marnie So I can start with that one. So at a very basic level, it can be really useful to actually get people in the room while you're developing the model in the first place. So I think where it doesn't work and we have some of those challenges is where an evaluation team kind of goes away, locks themselves in a dark room and ta-da, comes up with a logic model. So I would heavily suggest bringing people into the room, having those discussions about your system elements, your persona, and actually making them co-create the model.
And that, in my experience, absolutely helps bring people on board. Sure.
Fiona: Yeah, and I would just like to say I've seen it in project. projects where actually I've developed a theory of change with some people, like with a service. And when you get them all on board, it actually gets another level of usefulness for them because in actual fact they're now able to see their service on a page, not only just what it's going to achieve but actually what they're doing. And so some of the really positive feedback I've had from one group at least was that one of the most beneficial things of going through the process was they could actually see
parts of their service where they need to put more focus on than what they were previously. So it really mapped it all out. So I totally agree with Marnie. I think you want to test it at least and get as many people involved as possible. Sometimes with compressed funding, et cetera, you're not able to co-create, which is... preferable, but if you can't do that, try and at least test it with people and seek their feedback, get them involved in the process so that actually they kind of actually understand how it does, how it works.
Rebecca: And as you said with that group you worked with, you know, the action of actually helping to plan out a theory of change helped them to think about how they did their service and what they
Fiona: Yeah, totally. And I've also used colour coding. So when I've got different groups and there's going to be different pathways through, then I actually colour code the different parts so that you can see those pathways coming through visually. And I don't just have them all the same colour.
Rebecca: Yeah, it's a good idea. Yeah. Yeah. Yeah, love a good visual. Yeah. Cool. So one of the other main points that was raised that, oh man, we get it. It's the challenges of attribution when you're in a complex system. Right? So someone said attributing outcomes to individual organisations when they're part of a wider system. Or, you know, how do you articulate those outcomes when there's a provider level, there's a service level, there's a population level? It's complex, right? How do you do it?
Marnie Yeah, well, I don't want to rain out anyone's parade, but attribution is probably a goal that we will not reach. But what we can do is focus on contribution, and theory of change comes in really useful there. Yeah. Fiona, I know you've used that approach.
Fiona: I have used it, and I think there's a number of ways to use it, but one of the really great things is that contribution analysis is used to actually try and work out the attribution. So you're not looking at causation, you're looking at what is attributed and how likely is that to the different outcomes. And you need a theory of change for that because it's based on the short-term or medium-term outcomes that you've actually identified. You can actually use research even, so a few well-selected pieces of literature to identify the base level, to identify what happens otherwise.
But also what it asks you to do and think about is how else could this have been achieved? What are the contributing factors that might have actually contributed to it? So it gives you that broader picture and it steps us also into what we call the counterfactual. So the counterfactual is what would happen if that program didn't exist? That's the question. It is a little bit theoretical because it's, of course, We're not allowed ethically to do those sorts of experiments generally. But you're thinking about if it didn't happen, what would the result be otherwise?
And so I think that's a really useful way. I find it really useful for identifying all the different contributing factors, other possible ways, and exploring actually, has the program really, can it be really attributed to that initiative? Sometimes I'm not sure.
Rebecca: Yeah.
Fiona: It's marginal.
Rebecca: Yeah.
Fiona: Can be.
Rebecca: And that's awkward to write about, but it's sometimes true, yeah.
Fiona: Well, it's really awkward because if things are funded and you're saying it's marginal, then the question is a big old funding question, especially when people are very invested. That's really tough.
Rebecca: Yeah, yeah, it is, eh? But, I mean, that logic model should at least help you articulate the logic for what level of contribution has probably... Could you reasonably assume that their contribution has contributed to the outcome?
Marnie Yeah, a really simple way of thinking about it is you've got your logic model, your theory of change, you gather your evidence, does it match the theory? If there's some alignment, you can assume that there's some contribution.
Rebecca: Sure, yeah. No, that's great. Okay, we have a few live questions, which we're going to have a look at. So we have a question from Grant about the persona you were talking about. So by using the plural, are you specifically implying the model must use multiple perspectives?
Fiona: I wouldn't say must.
Rebecca: Yeah.
Fiona: But I would say in most likelihood, because you're trying to get away from a single population group, that most likely you would be using more than one. Because the idea is to reflect multiple population groups rather than a single person.
Rebecca: Yeah, yeah.
Fiona: But you don't absolutely have to, but that's how I've used it.
Rebecca: Right.
Fiona: So, yeah, it's...
Rebecca: Yeah. And relatedly from Garth, for each persona, or persona, For each persona, would you have a different logic model or a different theory of change?
Fiona: Yeah, so fortunately, I think on the screen it was really hard to show the nested approach. It's really hard to show it. But what I've done is actually developed one for each persona. So for each population group, I've developed a different one. And then I've taken them all to have an overarching, let's say. But you might end up with some physical disabilities. If you think about it, to have one of improved life satisfaction where you've got people with a degenerative disease who are losing their independence, who may have had a profession beforehand, to say that it's going to improve when actually their health is actually deteriorating, is probably, for me, it doesn't really ring.
It's highly likely.
Rebecca: Right.
Fiona: So in that case, I might have two different groups. I might have one group which has got improved life satisfaction, improved physical health, and I might colour code and have another strand which is capturing that other group specifically where it would be more about maintaining that physical health for as long as possible, maintaining the life satisfaction for as long as possible. There's some other things in there in the short term, like settling into the home thing that would be the same. It starts rolling out in that sense for the longer term in that example.
Rebecca: Sure, because realistically people will have different outcomes and that's, you know, you can only influence so much.
Fiona: Yeah, but it can be an end end. So I've done ones where I've had it, say, for a particular ethnicity, such as Māori. And you might have some outcomes specifically for them, but then you want to point out that actually, like I've done it around ecosystems and conservation, and you say, yeah, actually, that's them, but also it's an end-end. They're also contributing to these other outcomes. So you want to capture that.
Rebecca: Yeah, yeah. Okay, cool. We have another question. This is from Jeff from Marnie on the logic model. So they're great for planning, but they're limited for complex evaluation because the world diverges from the pre-drawn model, which is what we've kind of talked about, eh? But if you stick to that logic model to define what to evaluate, are you not evaluating what you... Thought the world works, not what the world's telling you about how it really works.
Marnie Yeah, absolutely, Jeff. So I would say that logic models are a tool. They're not the be-all, end-all. And a few things that we tend to do to make sure that we're not kind of drawing a model, set in stone, we're going to absolutely evaluate that, is first of all make it a, I'm going to go back to my, I get to go back to my living systems metaphor. So exactly as people with skeletons and bodies change over time, so should a logic model. So as we evaluate, we're learning. And we're finding out, actually, geez, there's another assumption that we never thought of.
That's when we go back to our logic model and amend it. And it may mean we also have to amend our evaluation questions and our evaluation criteria. But in a complex... system evaluation, you should absolutely be doing that. Yeah. Second point as well is, also when you develop your evaluation questions, many of them will be testing the logic model, but many of them will also be identifying, for example, are there any unanticipated outcomes? Yes. You may have a question on... What works for what population groups and what circumstances.
And that means that while you're using your logic model as a guide, you're not necessarily kind of thinking of it as a set in stone thing that really guides. And fracture evaluation.
Fiona: But also, if I can just add to that and chip in here, is that I don't treat them as static. I encourage clients and I encourage people to treat them as a live thing, that you adapt and change as things become more apparent and as you learn more. And so I see it as an iterative process that's never really finished. So I'd actually argue, yep, they're great for planning, I agree. But in fact, often where I start with when I'm going into evaluations or other things is actually reviewing the logic model and thinking, is it fit for purpose?
When was the last time it was updated? And does it actually really bear with whatever initial early information is coming through?
Marnie I would completely agree with that.
Rebecca: And it's sometimes even about, I mean, as... Jeff mentioned there, you know, sometimes you don't know what you don't know, right? There's contextual factors you might not have thought about until you get into it. But while you might be structuring, say, your interviews or your data collection around, research questions. I think part of it's about, you know, bearing in mind that there is emergence in a complex system, just keeping your ears and eyes open for things you weren't expecting or things that come up that weren't necessarily in your research plan, but if it keeps coming up, it might be worth looking into.
Marnie Yep, completely.
Rebecca: Yeah. Okay, here's another question from Pippa. Do you have views on how the logic model approach and evaluation frameworks are used in policy development? So like the first stages of testing interventions and monitoring the systems and regulatory regimes?
Marnie Yeah, I do. So I've always found it a little bit puzzling that logic models have often been thought of as a tool of evaluation. I mean, they're certainly useful for that, but in actual fact, logic models, intervention logics, theories of change, really at the heart, they essentially are a program planning tool. Well, they kind of are, yeah. So I strongly advocate, as you said there, Pippa, to use at least that approach to program planning. So while you have things like business cases and possibly regulatory impact assessments and all those other tools that are part of the policy or planning process,
if you can put it in a logic model format as well, that can often surface some of those assumptions or tensions that you might not have thought about.
Rebecca: Sure. Yeah, yeah. Now, do we have anything else to say about feedback loops? Because Catherine's asked if we have any other tips about how do we figure out the feedback loops.
Fiona: That's a good question. Yeah. Sometimes it becomes, it's really in the thinking. When I build a model, I actually think about what the interrelationships are. So I start mapping out what those interrelationships are. I don't find logic models are very good actually for showing that. It gets all too complex and messy. But I do start thinking about what's going to influence what. And I also try to identify things like what I call enabling outcomes. So those are things that you need to achieve first sometimes to unlock other outcomes.
If they're not achieved, the other things won't come to fruition. They're not likely to come to fruition, not to the extent that they could. So I think it's more you track the data, you kind of are monitoring things and looking at what can, you know, and mapping out what can impact something else, which in complex theory means chaos, to be honest, because you move one thing and it can change lots of things unexpectedly. Yeah, yeah. And so it does get really complex, but I think at a high level trying to get a sense of...
If this changes, what would that mean would be helpful.
Rebecca: And also observing this did change and here's what it did mean. And now what does that do for the rest of our logic of what might happen next?
Fiona: Yeah, yeah, exactly.
Rebecca: Yeah. Final question? Do we have time for a final question? Yep. Okay. So this is another one from Jeff, but this is for Fiona this time. So if we focus on activities, not outcomes. Will we not see changes in behaviour of our participants in response to things outside our programme? So it'd be great learning for us if we found things outside our activities that influence the outcomes that we're dreaming of.
Fiona: Oh, look, I totally agree. But I think, next separate off, when I made that statement, I was really talking about delivery itself.
Rebecca: Yeah, okay.
Fiona: And sometimes I find organisations in delivery, but in delivery, that doesn't mean you don't keep track of your outcomes. It's not to say that outcomes aren't really important, and I think it would be great. I think we should be actually tracking from the outcomes back again. So it's kind of a... It's messy. I can't say more than that. But I guess what I've seen is sometimes people are getting quite obsessed with that who are involved in delivery rather than, so that feedback I have from an organisation who could say, oh, I see the outputs and these initial outcomes, and they take it for me about how I'm going to deliver this service.
But I think it's a great question. I think absolutely we should be looking for and thinking about things outside of our activities that might be influencing them. Probably brings me back to contribution now, something, what else could be influencing things here? So I wouldn't like to suggest people just focus back on activities and outputs because we've known that that's actually been problematic, but I think it's different from evaluations.
Rebecca: Yeah, I mean, you need some boundaries, don't you? Or you'll be looking at everything. But keep your eyes open for everything else that's happening that people might tell you about.
Fiona: Well, yeah, and I think we need to discern the difference between actually hands-on delivery or something versus... measuring performance over time or actually evaluations themselves. So separate things.
Marnie Yeah.
Fiona: Interconnected but separate.
Marnie Sure. Yeah.
Rebecca: We have a slightly more technical question from Sophie. So what software do the team use to create the logic models and the mapping of the relationships and systems?
Marnie Yeah, so when we're doing the initial part, I actually really like Miro. Oh, yeah. So that's like an online whiteboard where you can put your little virtual post-it notes. That's right. These days we do a lot of workshops online. And they're great because they can be a co-creation tool where people can sort of share and move their post-it notes around and get nice and messy. Yep.
When you're actually doing the logic model, it really depends on, as we said, horses for courses. So sometimes you can actually just use a PowerPoint slide and create a really simple one using the tools and shapes and images that are available in PowerPoint. Sometimes you can use tools like ChatGPT, which is a great image creator. The Roadside Community Health Fund was done with the assistance of ChatGPT to make it look nice and do the design elements.
Rebecca: But what did you put it in first before you gave it to ChatGPT?
Marnie So that was first using Miro and then gave that to Chat.
Fiona: Yeah, and we've used our in-house graphic designer as well to help really with some of these things too.
Marnie Yeah, I mean I would say don't get caught up in design, it's much more important to get the content right. Yeah. But if you do have as a communication tool, if you can bring a bit of design elements in, that certainly helps with that comms piece.
Rebecca: All right. Well, it might be about time to finish up. So thank you, everyone, for staying on and for all your questions. We really like being able to have these discussions. We hope it's been valuable. Now, remember, if any of today's webinar has given you some more ideas or some thoughts for your own work, we're always happy to continue the conversation. So please feel free to reach out. Thanks again for joining us. We appreciate it. We hope to see you in the next webinar. Otherwise, have a great rest of your day, everyone.
Bye.
