Helping every child succeed: Scaling support that works

Susanna Loeb (00:00):

We've been trying to think about how you get students across the country access when they need it.

Denise Pope (00:09):

Welcome to School's In. Your go to podcast for cutting-edge insights in learning. From early education to lifelong development, we dive into trends, innovations, and challenges facing learners of all ages.

(00:25):

I'm Denise Pope, senior lecturer at Stanford's Graduate School of Education and co-founder of Challenge Success.

Dan Schwartz (00:32):

And I'm Dan Schwartz. I'm the Dean of the Graduate School of Education and the Faculty Director of the Stanford Accelerator for Learning.

Denise Pope (00:42):

Together, we bring you expert perspectives and conversations to help you stay curious, inspired, and informed.

(00:51):

Hi, Dan.

Dan Schwartz (00:53):

Hi Denise. How are you?

Denise Pope (00:55):

I am okay. How are you doing?

Dan Schwartz (00:57):

I'm good. Today we're going to talk about how you bring sort of knowledge and discovery from the university to scale, but before we do that, I thought I'd ask you what is scale anyway?

Denise Pope (01:10):

What is scale? Okay. So to me, scale means to take something that is kind of little or just homegrown and make it bigger, have more influence in the world. So an example would be like maybe you start a little mom and pop kind of restaurant chain or whatever, and then all of a sudden it becomes like McDonald's. That's a huge example, but taking something small and then getting it bigger. Would you agree? Am I close?

Dan Schwartz (01:42):

I might refine it a little bit, but it sounds like scale is sort of the number of people you get it to. More people is better.

Denise Pope (01:49):

Yeah, like if you have a good idea and it really helps people, you would want more people to benefit from it. Yeah.

Dan Schwartz (01:56):

We do have the world's expert on how you scale knowledge, so I want to jump straight to it. So this is Professor Susanna Loeb. She is at the Graduate School of Education. She was one of our best faculty and one of the things she's doing is she's really leading an initiative that is providing new models for how you scale knowledge from the university. Guess what its name is?

Denise Pope (02:19):

Scale.

Dan Schwartz (02:24):

Yes.

Denise Pope (02:24):

Okay. I cheated because I knew. I knew.

Dan Schwartz (02:24):

So welcome Susanna. Thank you for joining us.

Denise Pope (02:26):

Welcome.

Susanna Loeb (02:27):

It's a pleasure to be here. Thank you both.

Dan Schwartz (02:29):

So your model of scaling is probably different than what most people have in mind. Most people think of translation, sort of like they discover a drug and then how do you get it into all the pharmacies or something like that. Yours is different where you are scaling both the dissemination of knowledge but also the gathering of knowledge. Could you talk about this a little bit?

Susanna Loeb (02:50):

Yeah. So when we think of scale, sometimes you'll think like, "Oh, I have a product and I want to get it to very many people." But a lot of the scaling that we do is really something that we think is really important for students to have or schools to have like individual attention, something like that. And we think, "Okay, how do we really affect the world at scale?" And in order to do that, we really have to think about the world that we're trying to affect and who those decision makers are, who gets to select whether or not a student gets access to some resource that we found is really effective or that the world has found, the researchers have found is really effective? And so we think about who all those decision makers are and in some countries in the world, there's just kind of one group that's making all the decisions.

(03:42):

But in the US, we have devolved a lot of the decision making to school districts and that's 12 to 14,000 decision makers there and then they devolve some of that to schools. So we have to think about how to reach a lot of different people. And so that's one aspect of it. And then they're each operating in different contexts. So they might have these opportunities that differ across all sorts of different places in how they get this resource to students. So we really need to collect data to understand how you can do this thing that research has told us is important to think about doing in all these different places. So it's kind of context specific studies of effectiveness. We have to think about what resources are available. There's some place where it's really easy to hire people with expertise and maybe you need people with expertise.

(04:36):

And so in those places they can just hire and we can build an approach to getting to scale by that kind of hiring. But in other places you can't do with that. And so maybe you need to think virtually or you need to think in other ways about how you would get those resources. So I think there's all of this knowledge even when we know something is good, that you need to have about how you do things effectively across contexts for different kinds of students. And also the kind of how to get it implemented well when you're talking about places that have lots of different opportunities.

Denise Pope (05:12):

Wait, I have to just say something because

Dan Schwartz (05:13):

-fine, fine.

Denise Pope (05:13):

It's totally different from my McDonald's example. When you think of scale, and the way I thought of it is sort of mass production. We take something and then we mass produce it to scale and everybody gets the same Big Mac and fries and whatever, and they all taste the same across, you know, the globe. And what I love about what Susanna is doing with Scale is understanding that you don't want to produce a bunch of Big Macs, right? You want to actually make it work for the very different contexts that you're going to use this intervention in. So I think that's brilliant. I just wanted to say that.

Dan Schwartz (05:54):

Just to defend McDonald's, I think in Hawaii they have a pineapple hamburger.

Denise Pope (05:58):

Yeah, they do. They do. I know, in different places. Okay, defend McDonald's. But for the most part there's this level of mass production. And I love that Susanna understands that in education we're not just making widgets and we are literally having to work within context. So I think that's brilliant.

Dan Schwartz (06:16):

Yeah, no, I agree. So the concrete example here is tutoring, helping people- helping districts figure out how to use tutoring. So there's so many different contexts. I mean, are you exhausted? How do you do this?

Denise Pope (06:30):

Well, yeah. And just also explain what you mean by tutoring cause I think what comes to mind is hire a tutor for my kid's math grade to go up. So what do you mean by tutoring?

Susanna Loeb (06:41):

Yeah, I think in some ways it comes back to hiring a tutor to get my kid's math. So that if we look across what's happened in the world for many, many years is that lots of kids have gotten tutoring, but those have tended to be kids from families who could go hire a tutor for their kid when they were struggling. And across the board, kids have times when they are just working really well and what's going on in school, and then times when they struggle. And if you can hire that tutor, they can help them get over that struggle and get right back on track. But if you're in a place where you can't get that resource to your child, maybe you don't have the resources or for some reason, other reason you don't have access to it, then you're not getting that and it really creates a lot of inequality across students in that kind of access.

(07:30):

And so we've seen from the research that this kind of individual attention focused on academics, but with a consistent tutor who also builds motivation and makes it fun to do it, that's the most effective way that we know of accelerating students' learning and we have really unequal access to it. So we've been trying to think about how you get students across the country access to that when they need it. And again, this is a place where there are going to be different ways of doing that. Some places, there are a lot of adults already in schools and you can reallocate those adults to provide some of that one-on-one or small group attention to students over time.

(08:11):

In other places, you'll need to go out and pull in new people who can do that, whether they're local college students or other community members, whether it's a new group of paraprofessionals that come into the school. There are all sorts of different ways of doing this, but the real fundamentals that you want to make sure you hold onto is that students are getting high-quality instruction intensively on just what they need so that they can quickly pick up the material that they've been struggling with and then get right back on track and feel really like they can be successful as a student.

Dan Schwartz (08:47):

So you see there is the same burger patty in every burger.

Denise Pope (08:54):

But the bun is different, is that what you're saying? How it's done is different.

Susanna Loeb (08:58):

I don't want to compare it to a hamburger. Okay, we'll take the tomato or something. The tomato is always in there.

Denise Pope (09:03):

But this is complex. This is not easy.

Susanna Loeb (09:06):

It's interesting to try to figure out how to do this too. It's complex, but if you can do it in partnership, I think the most exciting part of our work is that we create these strong partnerships with districts and schools across the country. Not every school, but we get this real variation so we can understand these differences in how schools might pull this off and then with that variation that can apply to lots of different places. And those partner schools help us get access to other schools that are interested and we also can just create materials that help other places that we're not in such close contact with implement it because we've thought about, "Oh, you are a school that has these resources but not those resources. So try it this way."

Dan Schwartz (09:56):

So God, I have so many questions for you. Let me go with a context one, have you reduced it to four or five contextual variables that you focus on? So context could be sort of infinite in the level of variation that you look at.

Susanna Loeb (10:10):

Yeah, so I think it's less that we've kind of put the boxes into four, but we do have these processes schools go to try to understand how they want to do it. So one of our tools is a district playbook that kind of walks districts through creating first, for example, a landscape analysis of what are your real needs. So some districts really need help in early reading. So that's what they identify that as the area. And if you need help in early reading, it's actually quite a different kind of tutor that you need than if your area of need is, for example, in algebra or in high school mathematics and so you want to kind of figure that out. And you want to figure out whether this is really for all students.

(10:58):

Often in early reading for example, tutoring can be super useful for every student in the class because so many students need help, individual help, they missed this one sound or this one identification in phonics that will help them get to the next stage and so that kind of individual attention is really important there. As you go up in the grades, maybe not all students needed at all times and so then you have a different kind of approach. So this landscape analysis is a way of identifying first, your area of need and then what resources that are available near you can you use. And sometimes what districts do is they say, "Oh, okay, I have this labor force I can pull on. I have a university nearby." Something like that. Others say, "I don't have time to do this and I really want to find a program that can provide all of this to me."

(11:52):

And so there are a bunch of programs around the country that districts can partner with to provide tutoring. And those programs already have really good curriculum for different age groups in different subject areas, and they do some of the hiring and you can actually get them to do everything or just the things that the district is struggling with. So there's a process of identifying that. But a lot of those things that go into the district playbook, we've learned from research. So one of the things we learned, for example is that in the high school, probably the biggest barrier is the master schedule. And so we really have them think about the master schedule in how they're going to get this resource to students. That's much less of an issue in the early grades. So the research has really informed these tools that go out to the districts.

Dan Schwartz (12:44):

You're kind of doing this research, right? You like, convince districts to try two different ways, and then you look at the kids' test scores or... How do you collect data on which of the approaches given the contextual variable mattered?

Susanna Loeb (12:58):

Yeah, so that's a really great question. So some of the research is just observing in districts, where do they run into problems? But a lot of the research we do are these what we call randomized control trials where we systematically vary some things. So we'll try one-on-one versus one-on-two tutoring, or we'll try tutoring with extra coaching for tutors. Often, we've tried one program just versus nothing. So how important is it that students get access to this? But these randomized control trials are really nice because they give you the precise difference between very similar students who got this resource and these very similar students who didn't. So you get a really nice way of identifying the effect of it.

Dan Schwartz (13:44):

This is such a great virtuous cycle. So then you get this information and you then sort of update your guidance, your document or your decision tree for the districts?

Susanna Loeb (13:57):

That's just right.

Dan Schwartz (13:57):

Yeah, no, that's such a great virtuous cycle. It's really good.

(14:05):

So Denise, where do you think you've had the most success in scaling? So you do different things with challenge success, which is your main vehicle for scaling. Which aspect of it do you think is the most successful at going to scale?

Denise Pope (14:21):

Okay, that is a really good question. I would say the number of schools that we've actually worked with that either took our survey and then we give right back that information. We have a talk with them, we talk about what they should do and the number of schools who have come to our conferences, because I think those are the two areas where we make the most impact. We have a much smaller group of schools that we work go really deep with. But if we're talking about scale and influence and thinking about student well-being and engagement and belonging, I'd say the over 500 schools that have surveyed with us.

Dan Schwartz (15:00):

Interesting.

Denise Pope (15:02):

How would you answer?

Dan Schwartz (15:03):

Well, I'm glad you asked. So I think my most successful scaling besides my students that have gone out and done a lot more than I have is, since I've been dean, we've replaced almost 60% of the faculty.

Denise Pope (15:18):

Wait, is that because so many people were retiring?

Dan Schwartz (15:21):

Yeah, mostly retirements. And so that's sort of hiring with the other faculty, the knowledge makers, and so maybe that's the place of the biggest scaling effect. It's a big multiplier effect.

Denise Pope (15:34):

Because you're also building a new building and I would say that's going to last forever, ever, ever. But the scale of making a difference is through these people who are going to then pass on that knowledge to more students and so on and so on and so on.

Dan Schwartz (15:46):

Unless the building has a hell of a lobby and it just changes everybody's view of education.

Denise Pope (15:52):

Just from the lobby alone. Before we return to the conversation, we have a quick invitation for all of you.

Sam Wineburg (16:01):

Pretty much everyone struggles to separate fact from fiction online. That's why digital literacy isn't just nice to have, it's essential. I'm Sam Wineburg, Professor Emeritus at Stanford and co-founder of the Digital Inquiry Group, a new nonprofit that teaches students to make wise choices about what to believe online. I'll be joining Schools' In for a live taping at Stanford on May 21st. We'll dive into digital literacy in the AI era. How young people, educators, and all of us can better navigate the internet and avoid tumbling down rabbit holes.

Denise Pope (16:35):

So any listeners who happen to be in Palo Alto in May, please register now using the link in the episode description. We are so excited to see you there.

Dan Schwartz (16:49):

Susanna, the thing that I always stumble at is this is a lot of partnerships to hold and it sounds like each one, you actually have a broker, an interpersonal relationship with this district, you find someone that may not be the superintendent. Is this correct? And then if so, can you bring it to parents because you can't hold relationships with all the parents?

Susanna Loeb (17:13):

Yeah, though those are great questions and I think actually one of the innovations of our initiative is that we have part of it called ERPN, Education Research Partnership Network. And I think the idea is that we do hold these relationships and we have some people at Stanford who hold those relationships with districts around the country with some partner- with some umbrella organizations like Chiefs for Change or the Council of Great City Schools, which holds partnerships themselves and have trusting relationships with a number of districts so that we can see what those districts are and if their interests align with what we think would be useful in skills goals of getting the most promising practices to reach as many students as possible. We have access to that. It builds on our more traditional research practice partnerships where we get really, really deep relationships and it takes that idea of trust but really tries to do it so that we can get information across a large number of contexts in the US.

Dan Schwartz (18:18):

And then parents, you have a website where they can go and learn what they might think and do?

Susanna Loeb (18:24):

Yeah, most of scale is really focused on school districts and at the state level and somewhat at the federal level, those policy decisions and how those policy decisions can help things scale. But there are so many people, particularly families and parents, that can be helpful in bringing these approaches to the minds of the decision makers, to the minds of these district and state leaders. And so we do have materials on our website about how families can advocate for their students to get access to these kinds of things, how teachers and teachers unions can help advocate for these kinds of things. So we are in this really complex world of decision makers and we've really been trying to think about how we can get the full range of the decision makers and stakeholders to speak in a similar language and communicate clearly so that we have the best chance of getting this to as many students as we can.

Dan Schwartz (19:26):

Oh, that's awesome. The idea of focusing on all decision makers throughout the system as opposed to just at the top government level. I think this is- and your solution's really interesting.

Denise Pope (19:38):

Especially for parents. I mean, I think as parents you often feel powerless because, "Well, what can I do to change a policy at my school?" So can you give a concrete example of something a parent might do to advocate for this? Is it like vote for a school board member who's for this? Give an example.

Susanna Loeb (19:57):

Yeah, I mean I do think just some of the decisions are made at the school level where principals decide how they allocate the time of the different adults in the school. And so you can talk to your teacher and your principal about how much your own student would benefit from these kinds of supports, how you've seen it done in other places and the materials we give, we'll give you examples of where it's done so that you can talk to the people that are closest to you and that you might see more regularly but certainly then being able to advocate more collectively at school board meetings through letters and things that you could send to state officials. All of those are good, but I think thinking locally and thinking where you have access regularly is kind of the first step.

Dan Schwartz (20:48):

So I'm going to switch topics but not that much. So Susanna, your work was really meet the moment and it's on a well-defined topic, tutoring. So now the moment, moment is AI, right? It's like, all AI all the time. So are you taking that on? Are you going to give advice about what chatbot to use? How do you do this?

Susanna Loeb (21:13):

Yes. It was a really interesting time after the pandemic because students had gone through so much upheaval and there was a real interest in how can we meet students where they are and help them learn all of the things, or at least the most important things that they'd missed during the pandemic and if you looked at the research, there was this great research on how effective tutoring could be. And so that's why we started NSSA, which is the National Student Support Accelerator, and to kind of take that knowledge and move it forward. This AI example is our second area that we're moving into, and we're in a very different state there.

(21:53):

There isn't the evidence that this particular approach, this one AI tool versus this one is most effective, but we do have huge demand from schools, from districts, from states to figure out what is the best approach for moving forward. And so it really does align with this same process of doing research that helps us understand effectiveness and use and how to get the best stuff into schools. And we're doing that kind of research. And then again, we're going to be creating these materials for all the different stakeholders so they can decide most effectively how to leverage generative AI to meet their goals for the students in their schools.

Denise Pope (22:38):

Okay. This sounds a little bit interesting here because there's people who will say, "Wait, I want the human tutor. I don't want the computer as tutor. That doesn't sound like that's going to be as warm or friendly or have the insight or whatever." How are you going to deal with that?

Susanna Loeb (22:54):

Yeah, I mean that's just why research is so useful here because there's such an excitement about AI tutors, but all the evidence that we've seen so far is that the current AI tutors may be able to give the student an idea of what the underlying math is or help them get to the next level of math, but they don't have that motivating and engaging part that the human tutor brings. And so we actually don't think that schools should be shifting fully to AI tutors, but who knows where it will go in the future. So I think that's why you need to have this research to see that very few students tend to engage for extended periods of time with the AI tutor to realize that that's not going to be an effective approach right now to helping students learn. And it's not a one-time thing, it's something we're going to have to do as these tools develop over time.

Denise Pope (23:52):

Can I ask a question? There's lots of ways to get districts to do this, right? And one way that I know that you're very familiar with is through legislation. There's a reason why all schools have to have doorways that are wide enough for wheelchairs to get through and you can't just decide as a school that you're not going to do that. So how do you see policy and policy changes and legislation fitting into this scale in general and particularly with this initiative?

Susanna Loeb (24:22):

Yeah, that's a great question. So I gave you an example of what we provide to districts. We take the research that we're doing and give this really detailed playbook of how districts could go about choosing what they do and implementing what they do. We take a kind of similar approach at the state level or at the federal level for policy makers, but there you really need to get it down to kind of one or two pages. What are sample policies that could really affect what's going on?

(24:50):

And just as an example, in tutoring, there were resources coming out of the pandemic that could be used for tutoring and some states might say use this for tutoring, but we got some example language in there that described guardrails that could be put on that so that the tutoring was much more likely to be intensive and with a consistent tutor so you got that engagement, that kind of mentoring part of it in there. So those kinds of sample state policies are one approach that we take. And then there's also thinking about what are the funds available that would help make this better, help make more people take up the most promising approaches. And so we think about having states both put money towards that and advise districts on how they could use the funding that they already have to get students access to the most effective approaches.

Denise Pope (25:49):

So I mean, it's sort of mind-boggling what your center does, Susanna, because you are both working at a local level, a very intense student to tutor level in some sense. I know you're not doing it, but you're brokering those, right? But you're also working at this massive level of the federal level and the state level and then the district level. And you're kind of saying at each piece of this system, what is the role that we should play to make sure that kids are getting these research-based interventions? And it's just- it's mind-blowing sort of how intense that is, how different it is to work at the federal level versus with an individual school. And so it's really impressive and gives me hope for others to learn from you how to really take the things we know from research and get them out into the world. We've been doing research for so long in schools, and schools look so sadly, often the same as 20 years ago, 30 years ago, 40 years ago, right. So I don't know, it gives me hope.

Dan Schwartz (26:51):

What you were saying, Denise, made me wonder is Susanna's approach to scaling, it's self-scalable? So can anybody else do this besides you, right? I mean, it's pretty amazing what you've pulled off. Can you give me a little playbook for how to do this or is my personality just wrong?

Denise Pope (27:10):

Wait, don't answer that question, Susanna. Let's go with the playbook.

Susanna Loeb (27:14):

Okay, I like that second question better. Oh, I definitely think other people can do this. I think it's really a recognition that we have so many different decision makers and that it can be really helpful if people are speaking the same language so that if we can share the way that we're talking about something, we're much more likely to come to a similar agreement. Certainly, we vary on what we value or what approach we would really like to get out there, but I think a lot of it comes down to not really communicating in the ways that are most productive for getting students what we want. And so we think that this kind of cross decision maker look where we make it so that they each understand each other better is actually really productive and that other places can do that as well.

Dan Schwartz (27:59):

That's pretty amazing.

Denise Pope (28:00):

That's awesome. So Dan, putting you on the spot as always, what's one takeaway from today's show?

Dan Schwartz (28:08):

Susanna said this to me a long time ago that the way she thinks of policy is trying to decide who are the right decision makers in the system and then how do you get them the information. I think this is different than what most people think of policy. And so the fact that she's built this amazing network, it's pretty impressive. It's a different theory of decision making and policy and then an execution on that theory. So it's amazing.

Denise Pope (28:35):

I agree. And I also think one very key takeaway for me is this is a different kind of scaling. This is not mass production. Everyone's going to use it the same way. Even if it's like McDonald's and they have a little different hamburger, Susanna's not even doing that. Susanna's really helping to contextualize and make it work at the local level, which is I think the only way that you're going to get real school change. So kudos to you, Susanna. Thank you so much for being on the show. I think we all learned a lot and thank all of you for joining this episode of School's In. Be sure to subscribe to the show on Spotify, Apple Podcasts, or wherever you tune in. I'm Denise Pope.

Dan Schwartz (29:15):

And I'm Dan Schwartz.