The Living Schoolbook

Our faculty have settled into a 1-4 rating scale for EVERYTHING that we do in the School of Education, regardless of the activity or the standard being measured in our assessment system of Goal Aware tools and OSP.

In our system, a rating of a 3 or 4 is considered "meeting expectations" or "exceeds expectations" and are the norm for most student evaluations.
In a wonderful world, everyone would be getting 3's and 4's and our system would be pretty boring. However, this sort of rating system makes it particularly well suited for finding cases where a student is running into problems. A few 1's or 2's on a student would throw up a flag to faculty that there is an issue with that student that needs to be addressed. Its an attention grabber.
Similarly, if a program coordinator sees a class in the program that is getting 1's and 2's, it suggests that they spend some time looking at the design of the course or the program.

While this is great for our use case, we know that this is not what everyone has in mind for Goal Aware tools. There is likely a set of best practices that needs to fall out discussion within the teaching and assessment communities in Sakai and OSP. I posted this to the portfolio and pedagogy list a couple of weeks ago but didn't get any response. Is this idea that far out of whack with what teachers think about? Or is the community seriously lacking in this area (probably not good for a community building teaching tools!)?

I'll restate my post here:

Each goal that is linked to an activity (assignment) can be rated on a scale. How you define that scale (sort of/kind of related to scopes of rubrics) is really important.

Rating Scales could be configurable:

1. At the Goal Set

• The rating scale would apply to all of goals in that set
• all activities that are linked to goals in this goal set would use the same rating scale
• these activities may span worksites (courses) so the rating scale would apply to all of them.
• this provides a very consistent experience and is pretty well suited for the use case at SU.
• this makes it easy to spot “problems” with reports as the rating experience is normalized across all classes linking to a Goal Set.

2. At the Goal

• Each Goal would have its own rating scale
• activities that are linked a specific goal would use the same rating scale
• these activities may span worksites (courses) so the rating scale would apply to all of them.
• this provides a consistent rating experience across a program for a particular learning outcome, but allows for differences between the way that ratings are done for different outcomes.
• perhaps a little more difficult to spot "problems" since there is more variability in the rating "signal".

3. At the worksite

• The rating scale would apply for all ratings done in a worksite
• activities in the worksite use the same scale regardless of the goal, goal set or activity being rated.
• not a lot of "pros" for this....but, this is how it is in the 0.7 release that we use at SU! The rating scale picked in the Assignment tool config is used in all of the data points and assignments in the site.
• allows for flexibility of rating methodology between classes
• may result in a very inconsistent rating experience across sites and a lot of variability in signal across sites.
• if you use a consistent rating scale across sites anyway (like we do) this will (and does) support your use case.

4. At the tool placement

• The rating scale would apply for all activities done in that tool
• all assignments (or data points) in a worksite would use the same rating scale
• allows a different rating scale for each tool (activity type)
• activities may be linked to different goals, from different goal sets, from different work sites
• allows for flexibility of rating methodology between classes
• may result in a very inconsistent rating experience across sites and generate a really noisy signal for identifying problems.

5. At the link

• Each time an instructor links an activity to a goal, they can specify the rating scale that will be used to evaluate performance.
• the ultimate in flexibility!
• the noisiest signal...would this be useful at all for program evaluation?
• very good for individual teacher control of their classroom.

So..I think that this is a spectrum of sorts.

As I mention in my intro to Goal Aware tools:

Course Management Systems often enable central administration of classes from a technical perspective. However, faculty and students tend to remain isolated from the program in which the class takes place. IT systems seem to perpetuate the traditional “closed door” classroom.

#5 is pretty close to the "Closed Door" class. Program Assessment is difficult.
#1 is good for broad assessment and standardization.