[00:00:00] 

Intro

Kate Grandbois: [00:00:00] Welcome to SLP Nerd Cast. I'm 

Amy Wonkka: Kate. And I'm Amy. And we appreciate you tuning in. In our podcast, we review and provide commentary on resources, literature, and we discuss issues related to the field of speech, language 

Kate Grandbois: pathology. You can use this podcast for ASHA Professional Development. For more information about us and CER certification maintenance hours, you can go to our website, www.slpnerdcast.com.

SLP nerd Cast is brought to you by listeners like you. You can support our work by going to our website or social media pages and contributing every little bit counts. You can also find permanent products, notes and other handouts, including a handout for this episode. Site up. Some items are free, others are not, but everything is always affordable.

Check us out and leave us a review wherever you get your [00:01:00] podcasts. Find us on Instagram or Facebook or email us anytime at [email protected]. 

Amy Wonkka: So just a quick disclaimer, the contents of this episode are not meant to replicate clinical advice. SLP Nerd Cast. Its hosts and guests do not represent or endorse specific products or procedures that may be mentioned during our episodes, unless otherwise stated.

We are not PhDs, but we do research our material. We do our best to provide a thorough review and fair representation of each topic that we tackle. That being said, it's always likely that there's an article we've missed or another perspective that we haven't shared. If you have something to add to the conversation, please email us.

We'd love to hear from you, 

Kate Grandbois: and that's so true. We love hearing from our listeners. We do. Every time one of us gets a little message or email, we have this like flurry of text messages about how lovely it is to hear from you guys. So please keep the information coming. Um, as a quick reminder, we have a.

Something very exciting in the works. Coming up, we have our call for [00:02:00] papers or as we call it pods, call for pod pers pods. See what we did there? That's right. So clever. Um, our call for papers opens on September 2nd, 2020. The process is going to be similar to any other conference except you will present your work here with us.

So if you have an old presentation or if you're interested in presenting, go ahead and start writing your learning objectives and abstract and head on over to our website to submit. Uh, once you submit, your submission is going to be reviewed by us in conjunction with our advisory board. So we have an advisory board who keeps us in check to make sure we don't work in a vacuum, and they're a diverse group of, of wonderful people who can, um, give us feedback if it's.

If you've submitted something outside our area of expertise, so don't be shy. Um, presentations will be chosen for production based on evidence-based practice, learning objectives, disclosures, and other metrics that you can find on our website. So that's gonna be really exciting. 

Amy Wonkka: Looking forward to it. I'm very much looking forward to seeing 

Kate Grandbois: people's, [00:03:00] um, proposals.

It's gonna be awesome. I'm, as we've discussed many times, there's a lot that I don't know, the, I don't know, I know a lot about a little tiny thing. So I am super excited to, uh, learn from all of our peers. Uh, we also have financial and non-financial disclosures. I really feel like this is so dry. We just need to, like, our financial and non-financial disclosures are always listed on the website, but it is ethical for us to.

Say them before we speak about a topic. Uh, so, um, I am the owner and founder of Grand Block Therapy and Consulting, LLC. I'm also co-founder of SLP Nerd Cast. Um, Amy Wa Amy is an employee of a public school system and also a co-founder of SLP Nerd Cast. We're both members of Asha Sig 12. We both serve on the a a C advisory group for Massachusetts Advocates for Children.

I'm a member of the Berkshire Association for Behavior Analysts for Behavior Analysis and Therapy, otherwise known as Babbitt, mass. Abba, the Association for Behavior Analysis International and the [00:04:00] corresponding Speech Language Pathology and Applied Behavior Analysis special interest group. I tried to just go through that so fast.

I think that that was, that was admirable. I was just, it's, it's just so much. Maybe we need to make that shorter. Anyway, um, thank you for joining us today. Thank you for sitting through that tremendously painful intro that maybe we need to make shorter 'cause it was full of a lot of really boring words.

Um, you are tuning in with us today to hear about assessment. Amy, why did we choose this topic? 

Amy Wonkka: So, you know, we're reviewing and giving commentary on all of these different topics, and a listener checked in with us and asked us a super good question, which was, I've been asked to do an assessment and I don't have access to standardized tests.

Can I still do the 

Kate Grandbois: assessment? Which was such a good question. It gave me pause and I thought if it gives me pause. There's obviously more to the story and more that we need to unpack. Um, so after our great listener, um, [00:05:00] our participant asked us this question, we started to do a little bit of reading. Um, we went down a few research holes, um, and we realized that, you know, there is a common.

Theme or a common conception that standardized assessments are a requirement of assessment. Um, and so we wanted to learn a little bit more about that and share that with all of our listeners. This also, this topic also comes up a lot in my, um, mentorship with mentees and recent graduates that I, that I mentor.

Um, when do you use a standardized assessment? When do you, you know, what is your standardized assessment actually telling you? Is it telling you what you need to know? Um, what do you do when you have to make accommodations to the standardized assessment and all of those things? So, um, we feel like this is a, it's a topic that can apply to, um, a variety of different settings and every single speech and language pathologist does assessments, right?

So it's, it's relevant to 

Amy Wonkka: all of us. So today we are going to talk [00:06:00] primarily about standardized assessments. And within standardized assessments we're going to focus on norm reference tests. So those are tests in which a student's performance or a client's performance is compared against others of the same age and other demographic groups potentially.

Um, criterion reference tests are also considered standardized tests. They are. We are gonna focus on them more in depth in a subsequent podcast. So today, when we're talking about standardized tests, we're really just talking about those norm reference tests. So our learning objectives for today, as we, as we did the deep dive into all of the different resources and trying to figure out the, the best and most thoughtful clinical response to this question, uh, we.

Wanted to review the ethical underpinnings of assessment and what, what are the overall purposes of assessments? Um, we wanna identify the fundamental components of assessment. Our third learning [00:07:00] objective is going to be to identify the components of a standardized assessment and how to report use of any modifications.

Kate Grandbois: Yes, and as we, for those of you who are, um, frequent listeners to our program, we have a lot to say about a lot of things. Sometimes we can't. Squeeze it all in. So, um, in doing this research, we realized that there is a lot to cover, so stay tuned for a future episode where we go more, do a deep dive into informal assessment measures and talk a little bit more, um, in depth about the assessment process with with informal assessment.

Um, okay, so talking about our first learning objective, which is review the ethical underpinnings of assessment and the overall purpose of assessment. So this is a really dense topic, right? Assessment can look like a lot of different things for a lot of different people. Uh, but thinking of what we've tried to do as we were doing this research was keep a big picture [00:08:00] perspective.

So why do we evaluate our clients and students? Why do we do it in why, what's the point? Why do we do this in the first place? So there's an obvious first answer because it's ethical. So principle one K of the ethical code states individuals who hold the certificate of clinical competence shall evaluate the effectiveness of services provided.

So there's, there's more to that principle, but the word evaluation is in the ethical code. So to answer that question, why do we assess reason number one, because we are ethically obligated to evaluate the effectiveness of our treatment. Principle one, A individuals shall provide all clinical services and scientific activities competently.

And if you think about it, you can't really provide competent service without assessment. We go ahead. And we've 

Amy Wonkka: talked about this before, how you know that that really is what drives your treatment on an ongoing basis. And there are different types of assessment, but [00:09:00] overall that's, that's kind of something that's happening 

Kate Grandbois: all the time, right?

And we talk so much about how we have to meet our clients, where they are in treatment, you know, the zones of proximal development and all of that. This is, this is just one of those, it just makes sense moments. So how can you design treatment to meet your client where they are? If you don't know where they are at baseline.

So if you don't measure it, how do you know how to make it better? I don't know. You don't, you, you have to measure it. It's sort of like a duh, like you, it just makes sense. You have to, yes. I just said, duh. In a professional academic podcast, we're just, I'm gonna gloss over that, but, um, it seems obvious. So, you know, our entire field based, you know, our, the entire medical field, you go to a doctor with symptoms, they evaluate you, they give you a doc, a diagnosis, they give you medicine or PT or whatever it is the same framework.

And fun is fundamental across healthcare and it applies to us too. So it's ethical to evaluate, it just makes sense. We [00:10:00] do it to improve the treatment and, you know, identify areas of deficit to, to improve the lives of our clients. Evaluation drives treatment. Right. Um, 

Amy Wonkka: okay, so we do it for ethical reasons and to drive our treatment.

Why else do we do it? Like why do we do. Why do we do formal assessment? Why do we do this? 

Kate Grandbois: Is this is the, this is the whole purpose of this podcast. That's a very big question. Um, and I think we need to unpack that step by step because there's a lot of different components that go into thinking about why we use eva, why we use standardized tests, um, in our.

In our, in our assessments. And if the whole purpose is using evaluation to drive treatment, what role does, uh, do standardized tests have in that assessment process? Um, we are going to talk a little bit about, in an article that we've reviewed in a previous episode. Um, an [00:11:00] article by Emily deem called Writing Measurable and Academically Relevant IEP goals with 80% accuracy over three consecutive trials.

We reviewed this in our, um, writing goals and objectives episode a few weeks back. Um, and while the article is geared towards writing goals and objectives, the author makes a very important point that goal writing begins with assessment. Um, and regardless of your setting. That should be the case. So it doesn't necessarily mean formal assessment, it doesn't necessarily mean standardized assessment.

Um, but a lot of us use standardized assessment. So if you're in the schools, you're probably only conducting standardized eligibility testing every three years. But it does mean that you're currently, you're consistently analyzing your students' progress, um, and current performance and taking a repertoire of their skills.

Amy Wonkka: And when you think about like, really any work environment that you happen to be in, whether you're outpatient, whether you're private [00:12:00] practice, private pay, everything, you, you still really do need to be informed by something, right? You need to have that baseline information. And so it's true that, you know, I don't, I don't know how you extract goal writing.

Without some form of assessment. It's like writing goals for somebody you haven't met. If you 

Kate Grandbois: didn't assess, then you would just be blindly choosing things. Well, I don't know, let's just do this 'cause that sounds like fun. Or therefore this is what four year olds work on. That sounds like a good idea.

Right. So the assessment component is a real cornerstone, um, of, of everything that we do moving forward. So it makes a lot of sense. We use eva, we use standardized tasks. We use evaluation in general to give us ideas for what to work on. We're not arbitrarily, we are not arbitrarily making decisions about what to work on based on age.

We're doing it based on the data that we collect in our assessments. Um, this gives us an opportunity to think about the role of the [00:13:00] standardized assessment in the broader scope of why we assess in the first place. So deem does a really great job of this in this article. The point that she makes, and this is a super important point, is that norm referenced assessments are designed to compare individuals communication skills to those of aged matched, typically developing peers.

That's it. That is the whole point of a standardized assessment. That is why they exist. I'm gonna say it again. Norm reference assessments are designed to compare an individual individual's communication skills to those of age match, typically, typically developing peers. So that is the whole reason that they exist.

So again, circling back to that bigger question, do we need to use. Standardized assessments in our, in our evaluations. Well, what are you trying to get out of your evaluation? There are pros and cons to everything, right? So the general pros of standardized assessment, they're really good for determining where an individual [00:14:00] falls compared to, or.

Compared to age match, typically developing peers, which could be used for meeting criteria for classroom accommodations, an IEPA 5 0 4, they can be really appropriate for first time assessments if you're concerned that someone you know may or may not have a delay of some sort. Um. Individuals where the clinician is clinician isn't sure if there is a deficit.

So maybe it's not a first evaluation, but there are some concerns and you really need to know. Part of your question asking in that assessment is, how does this person compare to age matched, typically developing peers? Again, I know I'm like beating a dead horse here, um, but that's why they exist and that is why we use them.

There are limitations to standardized assessments. Um, so standardized assessments, they might not tell you what you need to know. I work with super complex communicators, um, and this is very much the flavor of what [00:15:00] happens in my clinical work when I use assessments. Um, so. They're not first time. The, the students that I work with are not getting an assessment for the first time.

They're middle schoolers or high schoolers. I don't need an assessment to tell me that there are more than one standard deviation below the mean and, you know, have a delay when compared to age match peers, I know that already they are in a substantially separate classroom. Um, they have significant needs, um, and they're complex learners.

So, you know, providing a, a administering a standardized. Assessment does not drive my treatment. It does not give me the information that I need to know. 

Amy Wonkka: And I think what we're talking about here is, is specific, very specific to norm referenced standardized assessments. So when we think about standardized assessments, sort is the big umbrella.

Yes. Within that are the norm referenced tests, which are those standardized tests that really, you know, that is, that's their whole design. They're comparing. You know, one person to a [00:16:00] similarly matched group. I mean, that, that is a whole, there's, there's a reason for those tests. Those tests exist, you know, to fill a specific niche, but on their own, they give you very specific but limited information.

So when we're thinking about those norm reference assessments. 

Kate Grandbois: So Dean makes this going back to this deem article, Dean makes a really good point about this, and she agrees that because Norm referenced assessments do this one thing, they are not. They are not useful in informing individual individualized treatment decisions.

She also references the work of other authors, power Defer and Flynn from 2012. Um, this was a great article. It's listed on our website. It was published in Perspectives on school-based issues. And these authors go even further from the deem article and inform us and say that use of standardized [00:17:00] assessments is not useful for driving individualized treatment.

However, it is extremely commonplace in our field. And this was a really interesting thing to, to read in literature because it is so commonplace and we know that it's, you know, ca colloquially, we sort of know that it has these limitations and yet we use them all the time. 

Amy Wonkka: No, I totally, I, I totally agree.

When you raised this point, when you had read all these articles in preparation for this and we were talking about it, I think, you know, stepping back and acknowledging that kind of everybody too, like, like being in school environments or outpatient environments, you really do feel like that's the driver.

Of eligibility. And so, you know, you try and pick through the item analysis Yeah. You know, of the I test or something. But for years and years it's not, that was 

Kate Grandbois: like the bulk of my assessment work was doing an item analysis self. Mm-hmm. That, that was my, mm-hmm. That was my report. Mm-hmm. Um, and [00:18:00] so, you know, this is, these two things are very much.

In juxtaposition of each other, right? We use them all the time. They're extremely commonplace, but they don't, they, according to the literature that we read, they are not. Their purpose is to not drive individual treatment decisions, even though the purpose of assessment is to drive individual treatment decisions.

So we sort of wondered what the history was there and asked the question, why, why are these assessments so often used when they're not necessary? And we're not, they're not designed for helping us make individualized treatments decisions. Um, why are they so popular? So we went back in time. We got our time machine.

We went back to 19, we went back to 1991. And we found, um, an article called Methods of Language Assessment, A Survey of California Public School Clinicians by Wilson Blackman Hall and El Shots. I'm gonna refer our listeners to the webpage to see how that is spelled and hope that I didn't totally butcher the last name like I always do.

Anyway, [00:19:00] in this article, they describe that the majority of clinicians, remember this is 1991, so this is a bajillion years ago when probably a lot of our listeners were not born yet. Um, I was definitely rocking band listening to Nirvana and enjoying my, I don't know. Between years. Um, so in this article, a majority of clinician used standardized assessments combined with other informal assessments including language sampling.

So in 1991, the majority of clinicians that they sampled heavily relied on standardized assessments. So since that time, clinicians have been heavily, according to the literature, clinicians have been really relying on standardized assessments. The authors make the point in 19, this 1991 article, that it was related to state mandates.

So the state mandated assessment had a huge impact on clinical assessment methods. Um, and another reported reason was time. So clinicians at that time said that they used a, a [00:20:00] standardized assessment because it was more efficient and less time consuming than doing a language sample where they had to collect a hundred utterances and then analyze the data.

Amy Wonkka: Well, and while I was not a speech pathologist in 1991, I can imagine that things were similar for people working in the public schools that state mandates mm-hmm. Are something you have a lot of and time is something that you have very little of. Yes. Um, so I mean, those two drivers make a lot of sense that we found out in a previous episode, you know, that those are things that are driving service delivery choices, like there so.

That makes sense, right? 

Kate Grandbois: So that was happening in 1991. It was a real thing. Let's fast forward to an article we found in 2009 that essentially says the same thing. This article by Fulcher Rude Castella, Earls and Higginbotham title, uh, diagnostic Decisions and Child Language Assessment findings from a Case Review, um, assessment Task, and that was published in Language, speech, and Hearing Services in school.[00:21:00] 

Um, this article is a pretty dense article, and we're not gonna get into the details, but the takeaway is that they sampled a, a relatively small group of SLPs 14 SLPs, and analyzed their clinical decision making. And they came to the consensus that, um, SLPs still heavily rely on standardized assessment measures.

Um, and. This is, it's, you know, if you think about it, going back to the big picture from 1991 until now, we still know they, they have made the same observations that the majority of clinicians continue to use standardized assessment over other assessment measures. Even though we should note that the articles, um, make note that they use it in combination with other assessment measures, but standardized, norm referenced assessment measures were the more heavily relied upon tools.

Amy Wonkka: Well, and I think too, you know, it bears mention that both of these articles are specific to people who are working in the school environment, but having worked in a range of different outpatient environments in different places in the [00:22:00] country, I can say that that's also consistent with my personal experience as a clinician.

Totally. Uh, you're, you're doing a standardized, so you're, you're grabbing it from the shared closet of all the different assessments or, or what have you, but the. The idea that the standardized, the norm referenced assessment is what's sort of driving the assessment process, I think is probably pretty pervasive, at least with pediatric language clients and probably also in other settings and with other groups of people as well.

Yes, that is 

Kate Grandbois: also my experience. Um, so okay, that's the landscape. It's been this way for a long time, at least in the literature documented for the past 20 years, so. Going back to that bigger question, it, does this even matter? Do we even care? And that DM article says, yes, this is a problem. She out outlines some pretty specific concerns related to relying heavily on standardized assessments.

The first [00:23:00] one is that when you rely heavily on standardized assessments, you end up writing goals and objective that teach the test. So you're designing your goals and objectives based on the responses from standardized assessments, which then. Teaches to the test when you go to reevaluate in the future to document progress.

The issue she brings up, the second issue she brings up is that increasing performance space on a test item might be academically irrelevant. So you're choosing something that the test told you was a problem, but that might not have, you know, improving that particular skill as identified by the test.

Might have absolutely no functional or positive impact on the life or academic wi, you know, academic or otherwise of the student. 

Amy Wonkka: And when we think about, you know, that article again is talking about clinicians in a school environment and in a school environment you are really looking at educational access.

So you're looking at teaching skills that enable a student to access the curriculum. Um, similarly though, if you're, if you have your outpatient hat on [00:24:00] and you're working at night in your outpatient setting and you end up, you know, doing a lot of structured, receptive, expressive. Picture pointing drills because that's what the, that's what the client had trouble with on the self or what have you.

You're also not helping develop person centered goals and objectives that are going to be meaningful and translate into actual functional gains outside of your structured therapy room. So again, I think this is something that, although we're talking in a lot of school-based terms, it's relevant. Across environments, no cross 

Kate Grandbois: settings.

Absolutely. And D makes the article, you know, makes this argument about choosing goals and objectives, writing goals and objectives that are academically irrelevant. But you know, me working in private practice, I'm gonna take that one step further and say that you might end up working on, on skills that are functionally irrelevant or that have no meaning or add no quality of life or are not valuable to the stakeholders.

Um, you know. Relying heavily on standardized assessments to drive your treatment can [00:25:00] really point you in the wrong direction in terms of what to work on, and I think that that is an unbelievably important point. 

Amy Wonkka: I agree. I think it's super important, and I think that it's another downfall of over-reliance on norm referenced assessments as your primary or only driver of your assessment is you, you, you're not make, you're not, you don't have the right information to be forming the right clinical question to create those meaningful goals and objectives.

Right, 

Kate Grandbois: exactly. So moving along sort of. Thinking again about our bigger, you know, big picture question. If your goal of the of assessment is to determine if a person falls within a certain developmental norm, they're having an evaluation for the first time, if they benefit from a repeat of that assessment to compare against, against age match peers or their own previous scores.

Um, if your workplace requires it for some reason, or it's required by a third party, third party funding source, then yes, you should possibly probably be using a standardized [00:26:00] assessment. If you're in a position for some reason where standardized assessments are not required or beneficial for any of those reasons, um.

Do you still need to use it? Maybe It depends on a lot of the above factors, but, and, but I think, you know, the takeaway here is that if you don't use the standardized assessment, or even if you do, we would encourage you to use language in your, in your report to justify why you either omitted or included a standardized measure.

So justifying what measure do you use in your assessment is definitely best practice in terms of evaluation writeup. And I know, you know, we're not. You know, get into the organization of evaluations or writing them up. We know they're time consuming. We've been practicing for entirely too long. It, they could be like, they're like the bane of my existence sometimes, because they're such a time suck.

So we understand that you, you know, need to be efficient in your writing, but in terms of justifying why you chose what tests, that is definitely something that should be included and is important. [00:27:00] 

Amy Wonkka: And I think when you're trying to ask yourself the question that our listener was asking us, do I need to use the standardized assessment?

And in this case, meaning a norm referenced assessment where you're comparing to a normative sample. It, it also goes back to what questions are you trying to answer through your assessment? Are you trying to answer where does this person's skill shake out compared to a normative sample? Well, then, yeah, that's, that's a very good tool for that.

I would say not in isolation, in, in conjunction with other things that we'll talk about in a, in a short minute. Um, but ask yourself the question, what am I trying to do here? What am I try, do I need to demonstrate? Skills relative to a normative sample for eligibility like that, that is the thing that connects you back to the norm reference.

So I think 

Kate Grandbois: you've, I think what you just said is a really good segue into our second learning objective, which is identify the fundamental components of assessment. So now we know what a standardized assessment is good for. I think it's good to talk about what other fundamental components of assessment, um, are [00:28:00] out there and why are we using them.

So there's a lot of different ways to think about this. And we are gonna, I know we are gonna, you know, have a second episode that, um, coming out later on that goes, does like a deeper dive into this topic. But we really need to think about, again, the purpose of assessment is that measurement and data collection piece that helps us drive our goals and objectives.

Um, you can't engage in effective or ethical treatment without assessment and you can't evaluate the effectiveness of treatment without data. So it makes sense. Measurement and data are fundamental components of assessment. This is another like that just makes sense moment. It does. Um, and remember that both qualitative and quantitative data should go into this process, and it's so much more than just percent corrected.

MLUI feel like MLU and language sample is, is a common theme that we saw in the literature while we were preparing for this episode. It's a common theme amongst, um, my coworkers and colleagues of my own work. But there's latency data, there's descriptive data, there's [00:29:00] observations. There are so many other things that go into, um, a, you know, a component, the components of assessment.

Amy Wonkka: And I think along with that there is the other piece of standardized assessments, which is the Criterion Reference test, which is a bit more, you know, it's a standardized test, but instead of measuring you against a normative sample of your peers, it's measuring against some set criteria where you obtain a score that's related to that criteria.

Uh, so that can be super helpful. That is a standardized test, but it's not. The norm reference test that many of us think of when we think of standardized assessment. Think along with that, you know, super qualitative stuff like interviews, right? Interviews are very, very meaningful. Getting information from the family, the teachers, you know who, whoever is an important person who can speak to us about their goals and objectives for this person, the client themselves is an incredibly important person to talk to.

So there's all of that together. Meshes. You guys can't see, but I'm mesh your hands 

Kate Grandbois: together. Like I'm bringing 

Amy Wonkka: [00:30:00] my hands together. 

Kate Grandbois: I 

Amy Wonkka: dunno, it's not pre hands. What 

Kate Grandbois: is this? 

Amy Wonkka: Meshing fingers? I don't know. It's, it's all the components of assessment coming together. So hold hands with each other. 

Kate Grandbois: Yes. You're holding hands with yourself.

That makes, that makes 

Amy Wonkka: total sense. So that's, um, classic example about quality assessment, 

Kate Grandbois: but I mean, you know, the quality assessment, I think you bring in really good point to the table about, you know. Interviews and all of these other kinds of data that you collect can collect in a quality assessment and sort of thinking outside the box.

So is our quality assessment just the, just those pieces of measurement? So if you measure it, is it just the score? What, no, no. Is it just the score? Absolutely not. Amy is, right. So let's say you have a whole bunch of scores. You measured MLU, you did a norm reference assessment, you did a criterion reference assessment.

You went wild, you had all this time, you took and collected all of these data. Is that it? No, the answer is no. Um, and there we found some, sorry. That sounded very condescending. If you were feeling [00:31:00] like that was just mean. Apologize. I didn't, I just got, it was just an enthusiastic, no, it wasn't a, it wasn't a condescending, I can't walk that back.

Anyway, so I found an article that agreed with me, so this power defer in Flynn from 2012 article. Make this point as well. So you have so much more than just your data. You have, and this is maybe the descriptive component. So how did they respond across different prompting methods? How did, how did they respond to different aspects of a teaching procedure that you did during an informal assessment?

How did you know, how many gesture prompts did you use? Did they respond well to phonemic cues? Um, what kind of environmental manipulations were helpful and important? What, what was their response to other kinds of accommodations, to the instructions that you gave? Um, are you, you know. Do you need to put descriptions of those modifications in your report?

What was their behavioral performance? What's their sensory status? What are some of the other variables that have an impact on their [00:32:00] communication skills? All of those pieces of information. I personally would argue this is just me. Have such a bigger impact on your, on your treatment moving forward.

It's not so much what they can do, but how do they learn? So what are the things that you can do? What are the accommodations that you can provide for that student or individual to support them where they are?

Amy Wonkka: And I think that some of this, you know, is, is not necessarily. You're, you're not all gonna write this, you're not gonna write all of these components up in your evaluation. I, I think back to when, you know, I was in my CF and I was working outpatient, and I had a client come in for one session and that was the evaluation, right?

So there's no way you're going to get that level of comprehensive data. Out of a session, no matter how amazing of an interviewer you are and how you know it's efficien, you're with your time. It's not. But I think that it, being aware of those different things and making note of them [00:33:00] whenever you can, and even making note of the fact that this is something that moving forward people should explore in treatment.

So if you've done your outpatient evaluator person came in one time, you did. Maybe a norm reference, maybe a criterion reference, maybe an interview and brief observation. You're gonna write those components up, but you might also, in your recommendations, put in components about things that should be explored further through therapy.

Right. So perhaps you didn't get a chance to see what happens if I give a phonemic cue. But back to your, your great point earlier about, you know, what would happen if the person moved away? What is it? What is relevant for other people to know maybe what's relevant? Um. It's just those questions. Here's some thing, you know, 'cause you know, I love to think on questions, but here are some things to think about in the future.

So whoever's reading this evaluation and going to use it and maybe move forward with some type of treatment, um, here are some things that. Might be worth 

Kate Grandbois: exploring a bit more in depth. Yes. And you, you just touched on something that I say to my mentees all the time. So what [00:34:00] goes into a quality assessment, just in general, right?

Because they're so individualized. If you can use the exercise and pretend that your student or client is moving to a different state or a different district, and you're writing this assessment, what does that other clinician need to know besides their developmental test scores? That is so tremendously important.

And if you can just, you're right, we don't have a million years to write these assessments and we don't have a million years to evaluate. But if you can encapsulate that in a little nugget of like, okay, yes, this is the data, this is where they are, but this is what you need to know about the student to help them learn.

That is like the perfect, here I'm doing my hands again. Making my fingers mess. You're 

Amy Wonkka: you're doing the comprehensive assessment fingers. 

Kate Grandbois: Comprehensive assessment fingers. Ah, that's the weirdest thing I've ever heard. I hope someone out there can do a mental image of what is now deemed comprehensive assessment fingers, which is just your two hands braiding your fingers [00:35:00] together.

This is just getting weirder and weirder. We're just gonna, we're just gonna move on, but. You know, to date we haven't got any feedback about this weirdness that we bring to the table. So that's good. That is great. So going back to the point of that is the nugget of combining where they actually are at baseline in their developmental norms maybe, but also what their actual skill set is combined with how they learn, is really the best of all the worlds in terms of, of, of what goes into a quality assessment.

Um. Learning objective number three is maybe it, maybe now it's a good time to jump into learning. Objective number three, identify the components of a standardized assessment, how to report use of any modifications and special considerations for standardized assessment use. And again, we're that learning objective says standardized assessment.

We're really talking mostly about, um, norm referenced assessments here. So we've talked a whole lot about, you know, what norm reference assessments [00:36:00] are good for, um, what they're not good for, what the history is of why we use them, what other things you need to consider to make sure that your evaluation is a quality assessment.

Um, and. I think it's really important, you know, now that we know what standardized assessments or norm referenced assessments specifically are good for, you know, what they do, we need to know what they are. So, you know, when you're using as just as like a quick refresher when you're using a norm referenced assessment and you're trying to interpret ROS scores, standards, scores, percentile, ranks, you know, what does that even do you know what, what is the, we're not gonna get into the math here, but you know, what do, what do those scores actually mean?

And how you, how can you interpret them to make good decisions about your student or your client? And when you make accommodations to the tests, how can you interpret those inco, you know, report on those accommodations? 

Amy Wonkka: And I think another really important thing to bring up here is that although the [00:37:00] concepts of standard score percentile rank, we'll talk more about them in a few minutes.

Although those are a consistent mental construct, the actual standardization process is not consistent from norm referenced tool to norm. Reference tool. So as a speech language pathologist who's administering a norm reference tool, it is really important that you read that part of the manual that tells you about the normative sample that tells you exactly who they conducted this assessment on and pulled information out about.

And I think it's an important part. It is an 

Kate Grandbois: important part, and I think that this is worth saying is that. I hate statistics. I am a speech pathologist. I'm medium plus good at my job. I think I've been around for a while. I, I don't know. Uh, it feels weird to pat yourself on, on your back, on your back, but like, you know, but I, I don't, I don't, I don't do statistics.

I'm not a statistician and I think that that's representative of clinicians at large. Not to say that I'm like one of many, but I don't, you're the only one of the only [00:38:00] SLPs I know that just like gets dirty in the data all the time and just, you know. Maybe that's a mug. Get dirty in the data. The data, you know, but you know, that like really enjoys the statistics of it.

There's a lot of us out there who don't, and yet it is so tremendously important to understand that not all norm reference tests are created equal. That norm reference, you know, norm reference is the normative sample is a thing that you need to understand what it is and not forget it once you took your stats class.

So let's talk about, quickly about what it is just to like make sure everybody's on the same page. Um. Standardized norm reference tests mean that they take a sample of people that represent a population and then they give them the test, and that those, those scores from that sample are analyzed so that the test taker your student scores can be compared to the sample of individuals.

So let's say, um, you're using the test to determine where a person falls, it's where they fall [00:39:00] in relation to that one sample. And that is called, that is what the normative sample is. And this presents a handful of problems depending on who you are using the test to evaluate. Um, most English tests have a normative sample of individuals who speak only English, and she a relatively homogenous cultural background that.

That is a problem, right? 

Amy Wonkka: That's a problem. 

Kate Grandbois: And we, it's really important I think, in this culture of we over rely on standardized assessments. We sort of, oh, you know, with a stats class, thank God I passed. I don't have to deal with that anymore. But this is really, it's a really important thing to think about when you're choosing your assessment measures.

That brings up major concerns of li, of linguistic bias and use of standardized assessments, particularly with linguistically diverse clients. Um, and we're gonna touch on that a little bit more, uh, later in the episode, but just sort of like park that in your brain for a second. So we can get through this dredges of the statistics.

You should be the one reading all of these things because understand, I know, I'd like to [00:40:00] know. This is much better than me, this fantastic. Um, but so just keeping in mind that those scores that are analyzed generally yield specific measurements that we've seen in test manuals of bajillion times. So raw scores, standard scores, percentile ranks, and age equivalencies.

Um, I'm gonna let Amy take the rest of this since she understands it so much better than me. This full disclaimer. We're not statisticians. We're not PhDs. If you wanna learn more about the deep, weird, what did I say? Dirty data. If you wanna get down and dirty, that's what, that's our mug. SLP nerd cast.

Getting down and dirty data. Yes. So if you wanna learn about more of those things, we've been trying to think of something for a mug forever. Um. Go to your stats book, go to the Wikipedia page maybe, and do some reading. But for the purposes of this episode, we're just gonna review raw score, standard score, percentile, rank, and age equivalency, 

Amy Wonkka: right?

So these are all scores that. You are going to get, if you do a norm referenced assessment, [00:41:00] your raw score is just the total number of items that were scored, correct on the test. So you may see raw scores. Raw scores don't tell you anything about the relationship to the normative sample until you do things with them.

So you scroll to the back of the book, you look at your person's age, you look at their raw score, then you're going to come up with something called the standard score. This standard score is plotting where that person fell in relationship to the normative sample. And we're thinking about it on a bell curve.

So we're thinking about, you know, the standard bell curve. You get standard deviations from the mean. Often that is tied to eligibility. So when you think about eligibility, in some places you might have to be two standard deviations below the mean to qualify as eligible for a specific service. This is a place where we see norm reference assessments used a lot.

So standard score that's giving you. In relationship to the normative sample percentile. Rank is also giving you a [00:42:00] relationship to the normative sample, but it's the percentage of scores that were equal to or less than the test taker score. Um, age equivalency is super interesting and we have an article up on our website that I would recommend that people check out.

It is by Maloney and Ravi. Ravi, I hope so, Linda. Um, and it is titled Limitations of Age Equivalent Scores in Reporting the Results of Norm Reference Tests. This article, first of all, gives you a really nice breakdown of the terminology we're talking about here, raw score, standard score, all of those things.

Uh, but they talk a bit about age equivalency, which I remember from grad school, a professor saying, don't use an age equivalent because it's a poor, it's, it's a poor measure. And essentially that's, that's what the whole article is about. So age equivalency is a comparison to the score of age groups whose average score is in the same range.

Uh, but it doesn't take into account any those other 

Kate Grandbois: variables. 

Amy Wonkka: Right, and, and we all know that average, average is the average of a number of different normal. [00:43:00] Percent like presentations of a skill, right? So when we think about the average age that people develop a certain, you know, adult production of a sound, it doesn't mean that if you don't hit it right on the nail, you're there, there's something deviant.

Um, so age equivalent and generally speaking, there are a lot of issues with the mathematics behind standardized assessments. Uh, Kate found a great article that I'll let her tell you about, but I think one other key point to remember about standard scores. Percentile rank age equivalent is that. That standardized group is variable from assessment measure to assessment measure too.

Yes. Not all. So you need to always check that. 

Kate Grandbois: That's exactly what this article says. So not all norm referenced assessments are created equal. Right? We go about using like, oh, we pull the self and the PLS and the Goldman Bristow and like we're gonna pull all of our favorite tests out and just assume that like, oh yeah, these are representative.

Like, no, that's not how it works. Um, there's a really great article by Catherine Alazar, um, called Common Problems with [00:44:00] Standardized Tests and How to Cope with Them. This article, I like the title. Yeah, but this article is my jam 'cause it's a page and a half. It's like quick and dirty. It has a very clear language.

It's not deep in like thick, you know, heavy stuff. It just gives you a very quick overview of what to look for when you open, when you like. Go through the drudges and you're like, Ugh, I gotta look in the manual. And you like open it and you're trying to find the page. I hate test manuals for some reason.

They just, for me, there are times suck. It's like you gotta, you have one question and you gotta like look and look and look and look. So you're in there. If you read this article, it's a page and a half. You can have like a handful of things to look for. To make sure that you have a good understanding of the norm reference sample that the test is and all of the other variables that go into what you need to know about a test.

So it's a great article. Katherine, Katherine Bal Alazar. Um, it's up on our website. Um, and I think that sort of brings us to other things that you need to consider about the normative sample. So [00:45:00] there's the math stuff. Um, the, the dirty data, it's just gonna be like that forever. Um, so there's, there's the math stuff and then there are other things that you really need to consider when you're looking at the normative sample.

And this brings us to the platform of discussing norm reference assessments and culturally and linguistic linguistically diverse individuals. This is a whole huge area in our field. There is a lot of literature out there on this. This is another like, full disclosure. People spend their entire careers studying this.

We are not gonna do that. That's not the purpose of this episode. This is meant as like a skimming of the top, um, of, you know, what some like key takeaways in terms of considerations for when you are evaluating culturally and linguistically diverse individuals. Um. I think one of the biggest takeaways is something that we've already touched on, um, and that, you know, most English speaking [00:46:00] tests are normed on English speakers with very homogenous backgrounds, and that is a deficit, a huge deficit to using them when you are evaluating culturally and linguistically like diverse individuals.

Um, we found another great article by Ebert and Fam 2017. And those authors state that a vast majority of English tests don't include bilingual children in the sample. Even the and, and which we already know. But SLPs still rely heavily on English tests to evaluate bilingual children. And that was in 2017.

So here we have this, again, clinical fact about how norm reference tests are not appropriate for evaluating linguistically and diverse, you know, diverse individuals. And the SLP, this norm of heavily relying on standardized tests to evaluate individuals. And so again, we have two things that are just, that are counterintuitive.

Amy Wonkka: I think to add a third thing to that, um, the assessment tools, techniques, and data [00:47:00] sources entry under clinical topics on the ASHA website kind of echoes that point. That when you are using a. Standardized assessments with that, that don't reflect the normative sample of the individual who you're assessing, then you shouldn't be reporting standardized scores.

Right. If that, so, so you have to go back to the manual as much as, I mean, I don't, yeah, you wanna like do a manuals control F and like search for it. You have to use them. You have to use the manual. You have, you have to know if the normative sample adequately represents the person that you're assessing.

Because if it doesn't, you shouldn't be reporting standardized scores. Right? You 

Kate Grandbois: shouldn't be reporting standardized scores. But also that brings us to the next article that sort of underlines this point. Um, there is an article by, um. Scott Pra, um, that was published in October, 2019. Um, and I, I don't wanna read the whole thing 'cause I, I feel like we're gonna lose momentum on the point that I'm trying to make.

But when you use this [00:48:00] normative, you know, these norm reference tests with a homogenous group, with culturally and linguistically diverse individuals, this article by P makes the point that there is a high, there is a problem with misdiagnosis of bilingual children as language impaired because of. This issue, um, the difference between the demographics that we serve and the lack of bilingual SLPs is also a contributing factor, and there's a great need for easily accessible information about the difference between language influence and language disorders.

Um, and our heavily reliance on norm referenced assessments does us no favors in this area as a field. 

Amy Wonkka: And you should be looking at using available standardized assessments for PE for clients who speak languages other than English, 

Kate Grandbois: which I don't know. I don't know what those are because I have not reached researched that yet.

Amy Wonkka: I'm sure they, well, and that's, I mean, that's part of, that is part of being culturally competent. That's [00:49:00] part of having bilingual service delivery. I think a big takeaway is for people who. Are assessing people who are not native English speakers, um, and who are not adequately, adequately represented in the normative sample, right?

You need to be aware of these things and you need to do some education. You need to do some research. Um, and it's not as simple as administering a test and not reporting the score. If you translate a standardized assessment, you're invalidating the results. So you need, you need to, to dig deep. Asha does have some resources online.

Um. But this is, this is something that everybody needs to be aware of and thoughtful about. Yes, 

Kate Grandbois: de definitely. Um, so, you know, we're gonna refer, we again, we could talk about this for hours. We're not gonna talk about this for hours. It's not the point of this episode, but we will direct you to the resources on our site, particularly these articles by Scott PR and Ebert and Fam 2017.

Those are two separate articles. Um, and to do some more reading about this as well as, aren't you posting some [00:50:00] online resources for Asha Practice portal? No. 

Amy Wonkka: Yeah, yeah. Yes. We'll have some ash resources, we'll have some more 

Kate Grandbois: resources on there about this to read more about it. Um, so again, big takeaways.

Know your tests, know your samples. Um, Amy mentioned some, you know, not reporting standardized scores. There are a lot of populations that require non-standard administration. I, I know I work with more emergent communicators with complex behavioral needs across the lifespan. Um, a lot of my students and clients.

I have a really hard time participating in standard administration of tests. I need to know my tests. I need to go into the manual that gives me hives and find out can I repeat a stimulus item? Can I use a gesture to draw their visual attention to the stimulus materials? Can I ask a caregiver? Can I? You know, some tests will allow you to take caregiver reports, some won't.

What are appropriate prompts I can use? Can I use the prompt more than [00:51:00] one time? Can I use a latency prompt? Am I supposed to pause for five to 10 seconds? Every test is different and I think that we, and I'm speaking for myself in my own experience. I guess I shouldn't say we, I should say I. Get very, um, into the habit and into the flow of what I'm doing.

Um, and sometimes we forget some of these subtle nuances and that completely invalidates the whole, I don't, you know, going back to like, why are you using this standardized test? To begin with, why are you assessing to begin with? It's to drive your treatment. You're using a norm reference test. It's for one of the reasons that we've already stated.

If you're making accommodations, that is okay. That might even be good. You can learn a lot from giving these accommodations. So let's say you give the PPBT to a really complex learner and they need massive amount of support to participate. You can't report standard scores. You can't report percentiles.

You probably shouldn't use age equivalency since the re, re references that we've looked at give, you know, indicate that statistically they're not a [00:52:00] very accurate description. Maybe you're comparing a raw score to a different raw score or a raw score from a previous. Assessment period. Fine. What else can you learn?

How many phoning cues did they need? How much prompting did they need? What kind of prompting did they respond to best? What are some other idiosyncratic, you know, observations that you made about their learning style and their participation in the test? If you're in a position where you have to give a standardized assessment to somebody who needs a lot of these accommodations, that's okay.

You can still do that, but you have to qualify what it is that you did. You have to describe the accommodations that you made, and then you have to make sure that you are familiar enough with the test to know what standardizations you broke from. 

Amy Wonkka: Yeah, exactly. I mean, I, I think, yeah, I think that that's a big takeaway in general to sort of distill it all down.

Norm referenced assessments have a place. Yes. Be aware of what you're using them for, and that is to get [00:53:00] information about how a client performed relative to a normative sample. So the information that comes out of that norm reference assessment includes a standard score that's about being relative to a normative sample.

It's, it's. It comes outta the raw score, but it tells you how many standard deviations you are from the mean for your normative sample. You have your percentile rank. That's also relative to a norm sample. It's telling you, you know how that client is in comparison with a normative sample relative to the number of people who scored above and below them on this test within their normative sample.

And then similarly, you have age equivalents, which are. Related to the average of the normative sample. So all of these things are connected back with that normative samples. So we've learned there's a place, there's a time for these to answer specific questions, norm reference assessments, and also we've learned you have to go back in the manual as much as you don't wanna go into the manual.

You have to go into the manual to find out important things about number one, who [00:54:00] is in the normative sample. And that helps inform whether you should or shouldn't even report these standard scores, right? Or percentile ranks. Is your person part of that normative sample? Well, if they're not, then you're not reporting any of these scores that you obtain that are related to the normative sample.

You also need to go in for all the things that you just listed, which I think are completely true. Any non-standard administration. Which varies from test to test also means that you may not be able to report any of these scores that are connected back to that normative sample, right? So there could be multiple reasons that you're not able to report the scores that are obtained relative to a normative sample.

And 

Kate Grandbois: I, I mean, just for the sake of full disclosure, I'm guilty of that. I remember I called you like five or six years ago and I was like, so what would you think if so-and-so? And you were like, well, I don't know why you're reporting those scores. You can't do that. I. Oh, I mean, I had been practicing for like 10 years.

We get really in a, we get in a, in a, in a [00:55:00] groove, in a rut. And I had drifted. I knew that manual when I graduated. I knew that manual in my first job. But I had given it and given it and given it and given it and given it and drifted from the standard administration. And fortunately I had you to correct me, give me the feedback and bring me back online.

Um, and I think, you know, just sort of going one step further from that. Thinking about that normative sample and tying it into why you're using it in the first place. Right? So, you know, are you using it for one of the reasons that we've described for, you know, because of your work setting, because of a funding source?

Because you have clinical questions about a developmental milestone that matches the normative sample, um, because of, you know, some other specific reason. Eligibility criteria for an IEP or a 5 0 4. Um, and you know, what, what other things should you be considering outside of your, of your norm referenced assessment?

What is the purpose of assessment to begin with? What are [00:56:00] you doing in your assessment that's going to drive your treatment? Please do not use norm referenced assessments to drive your treatment because it is not best practice. There is, there is literature out there that states it is not best practice for teaching to the test, for making clinical decisions that aren't relevant to academic life or functional life or quality of life and all of those other kinds of things.

Um. Norm reference assessments are great. We overuse them, we rely on them too much. We have for the last 20 years because we went in our time machine and, and we went back and looked at it and started doing like a whole, a whole recap here. Oh yes. Consistency with the time machine noise. That's excellent.

Amy Wonkka: I got a 

Kate Grandbois: sound effects guy, right? So, excuse me. So, you know, these are all really important things to consider and think about. 

Amy Wonkka: I think connected with that too is just a little bit of thinking about your work environment. I also think that probably when we ask the question, why are you doing a norm referenced standardized assessment?

It may be because it is tied to eligibility. It probably is. It should be because it's [00:57:00] tied to eligibility. So you also, as the clinician, might need to do some education for your administrators in your different environments and help them understand, and I would write this, I would suggest thinking about writing it explicitly into your assessment.

You may administer a standardized assessment. But you're not reporting standardized scores and be transparent about that. I'm not reporting standardized scores. This test was administered. I obtained this qualitative and quantitative information through the process, but standard scores are not being reported because A, either you had to utilize a non-standardized administration and it was not allowed when you went back and took the time to look through the manual, so you're not reporting those standardized scores.

Or perhaps the, the standardization, um, population is not representative of your client, so you're also going to be transparent about that. Yes, I'm not reporting standardized scores because of reasons X, Y, and Z, and have those conversations and you may find some of the literature and some of the references that are up on our website could be helpful in having those [00:58:00] conversations with people who are.

In an administrative capacity, if there is confusion around why you're not reporting a standardized score and how that's connected with eligibility somehow. Um, but because somebody has the expectation that they receive a standardized score does not mean that it is clinically appropriate and the right choice to make.

That's a good point to provide one of those. Yes. That's a good 

Kate Grandbois: point. Um, our time is up. I feel like we talked really fast for a long time, so I hope we did. I hope all of it made sense. Someone listened to this and learned something. 

Amy Wonkka: Yes, that's, I feel like today we talked, we talked a lot about assessment that is connected with eligibility and relationship to a normative sample.

And in our subsequent podcast we, and we know that that's not enough to drive clinical decision making about goals and objectives. In our subsequent assessment podcast, we're gonna talk a lot more about additional standardized and non-standardized measures. So formal and informal measures that can help really provide.

[00:59:00] Those robust interconnections of a complete assessment. I'm gonna not make the hands again, but join us for that. Yes, because that, that one will be my fun. Join us for that. Stay 

Kate Grandbois: tuned coming up soon. Um. Thank you for joining us, everyone, and come back soon. Okay.