The Living Schoolbook

It may be that the free form portfolio already in OSP will meet some of our needs. The areas surrounded by the green area could be supported by the portfolio tool. The areas outside of that need to be developed.

Another lesson I learned from the OSP 2.5 meeting in Ann Arbor is that there seems to be renewed excitement over the “Free form” style portfolio. LaGuardia Community College is piloting the use of OSP to recreate the portfolio experience that they currently are providing for their students with a commercial tool.

I have to admit, the free form tool always seemed to have the least amount of potential to me and I had dismissed it as “child’s play”. However, after seeing what LaGuardia is doing and thinking about it in the context of the “Class Journal” interaction style, I think that it may have a lot of potential! A few things off the top of my head that I know need to be done to support that interaction style would be to allow students to display structured data on a page of the freeform portfolio.

Functionally, we need to be able to display structured data elements like forms, wizards (and hopefully, assignment data in the future) on a portfolio page. This functionality is not supported yet in the freeform portfolio. If this were rWiki, I could envision a macro that would run an XSL transform against known data types and yield an HTML snippet that we could use in the page.

The ability for students to tag a wizard page with one or more “Goals” is also important to support this interaction style, as is the ability to receive informal assessments (ratings) and feedback from their teachers/peers/friends on each page of the portfolio.

Our interaction style model also states that this is an event that is initiated by the teacher. This is also an important gap that would need to be addressed. Inevitably, teachers would want to schedule multiple portfolio reviews during the course of a semester and have the feedback and ratings attached to those teacher-initiated events.

Ever since we conceptualized and built the Sakai Goal Management tool we have heard a need for a rubric built into the tool. However, it has not been very clear how we should manage and apply rubrics. Today, in the OSP 2.5 planning meeting, I heard of a possible way we could design the tools that made some sense to me. However, I think that the implication that I drew from the conversation makes we wonder if the idea is implementable.

Our current design for “Goal Aware” tools allows a user to create (and centrally manage) a tag (a Goal, in our parlance) for each learning outcome for a class. Another type of tag that schools might find a lot of value in would be a managed “Stage” type tag. As it was explained to me, for program assessment, there needs to be a managed system of “Outcomes”, “Stages” and rubrics for each combination of the two.

While we allow multiple “Goals” to be attached to any one activity (an assignment for example), we wouldn’t expect activities to apply to more than one “Stage”. As it was stated in this meeting, there would be the expectation that at any one stage, only one rubric should be used when evaluating student performance (ratings, in our case) for an outcome of the activity. This would help support the investigation into inter-rater reliability when more than one person is rating students.

It looks like this:

So far, so good. That makes sense to me, if the rubrics are simple rating scales that inform the "rating" process (ie: result in a single number), but my understanding is that they are not. The usual definition of a rubric is yet another matrix of "performance indicators" and a "rating scale" for each of those indicators. Typically the results of each of the ratings figure into the overall "grade" of an assignment, but in this case only result into a single rating one just one goal that an activity was supposed to address.

Since I am not a teacher, I can't say for sure, but this seems like a lot of work to do to assess a single student. My feeling is that teachers/faculty will largely oppose that level of granularity when doing assessment.

A couple of days ago, Barbara Shelly (one of the original "founders" and a former director of the LSB) came in for a visit. She and I spoke for a while about how she noticed that the adoption rate of education tools by teachers and students was rather low. At least, that's what I think we were talking about. :)

At one point she compared it to the explosive adoption of IM, cell phones, MySpace or Facebook. I think that she really pointed out an important difference between those technologies that put the user in the driver seat and allow them to network with whomever they want, whenever they want, for whatever purpose they want. That network effect and personal sense of control is something that isn't being replicated in the courseware space.

In courseware, it isn't easy to create a small collaboration space for you and your peers. Implementations usually manage student access to course spaces by the bang of the semester drum. There isn't much of a point to pouring a lot of your time into creating content that may be archived and put on a shelf or deleted altogether. What is it about these "closed audience" systems that keep users from having the same excitement about the tools as they do for their MySpace account? Steven Gilbert's blog post showed up for me after a quick Google for "myspace personal publishing blackboard". He notes that the social aspect makes a big difference.

Courseware allows teachers to "manage" a class and to provision services to the students. Adaptive release of content; provisioned users; enterprise integration; and integration with content repositories put a lot of power in the hands of an educator. In contrast, MySpace doesn't try to manage much of anything. You can make and break relationships when you want; put up or take down content when you want to express yourself; and what you put on MySpace won't be deleted by anyone but you. You have the control.

Portfolio tools try to do that too. Students should be able to put together a portfolio and show it to whomever they please.

I think that the dream here is that students are going to get really into putting together their portfolio, just like they do when they "dress up" their MySpace page and chat with their academic friends about all the things they learned in class. I can see that happening to some extent, but I can also see peer review "flame wars" happening through that communication too. I wonder how excited the institution would be to have that sort of content out there, with their logo on it, even if the discussions that happened there were the most productive ones that the students had. What forms of controls would make sense for that sort of environment without killing the sense of spontaneity and personal ownership that might make it appealing?

Better yet, can we build two systems that talk to each other? The courseware server where classes are managed and the official records are kept; and the personal "portfolio" platform which talks to the courseware system and to all of the student's (or teacher's) "friends".

A two system approach would make use of messaging of education related content between the different systems. When I login to my student system, I receive content from the classes I am subscribed to. New learning outcomes, activity descriptions (such as assignments, discussion topics, portfolio reviews, etc.) and assessment data are sent to me. I can turn in my assignments, post to discussions and share my portfolios through a content "port" on this channel and I can receive shared content (their discussion posts, and portfolios for example) from others on that channel as well.

The social aspect of the system occurs through alternative channels that users create and allow their friends to subscribe to. The content through these channels doesn't go through the courseware server.

The Resource tool also needs to be able to support the task of grouping a number of pieces of content with their associated metadata and allowing the student to create a new piece of content containing all of the selected items and a new section that describes the relationship between those items and what was learned by the student as well as any new ideas. The student will be able to tag this new piece of content and reuse it just like any other piece of content as well as designate it as part of a portfolio presentation.

Just as WebCT has a “Content Manager” section, Sakai has a folder like view of content called “Resources”. The current resources tool allows students to browse their files and classify and move them about in a hierarchical file structure. Other tools often make use of the resources tool, but few activities start there. The cognitive activity of classifying and selecting resources that they would like to showcase or discuss in their portfolio is different in that it makes most sense to start from the unordered pile of resources to classify, group and explain the connections between them. These reflective acts are at the heart of portfolio development.

Switching the resource view from a strict hierarchy to a view that supports a “free tagging” metaphor would allow students to see how faculty have classified their content in relation to institutional and class outcomes. From this same interface, they should be able to create new “outcome” tags and apply them to their content. This would allow students to create multiple views of their content based on various emerging themes.

Content Collection – Assignments

The assignment tool is arguably the most used tool in courseware packages and faculty and students expect it to behave a certain way with certain rules. The student work handed in through a traditional assignment tool currently is “stuck” in that tool and subsequently is unavailable for students to use in their portfolios without a lot of extra steps that duplicate their data in their personal content area. The assignment tasks will need to be changed slightly to accommodate this new task:

1. Establish an event (an assignment) that describes the work and deliverables that students are expected to deliver by the assignment due date.
2. Describe (via a tagging mechanism) the expected learning outcomes from the event. Along with this description will come a description of the rubric that will be used to assess student work.
3. Review student work and assess the student using the established rubric.
4. Release student grades and ratings to the students.
5. Deposit the student work in each student’s personal content area as a piece of standardized content (with outcome and assessment metadata) that can be used in portfolios.

Content Collection – Discussions

The discussion tool is also a common tool used in courses to share ideas and content in a threaded forum. This content “lives” in the discussion tool and is managed with the class and, as such, is not made easily available to students for reuse in their portfolios. While discussions are usually difficult to assess, the subject matter is rich material for personal reflection and, as such, participation in class discussions should be included as a means for content collection for portfolio development. The tasks associated with a class discussion may also need some modification to ensure that the discussion content is made available to the portfolio author”

1. Establish a forum that describes the topic to be discussed and whether or not the material will be released to students as material for their portfolios.
2. Describe (via a tagging mechanism) the expected learning outcomes from the discussion.
3. Either the faculty or students will start discussion threads and add posts to the discussions threads and reply to each other.
4. At the completion of the discussion, deposit the discussion threads/posts/forums that have been designated as “releasable” in each student’s personal content area as a piece of standardized content that can be used in portfolios.

Anyone involved in a survey of portfolio packages will quickly learn that the term “portfolio” is a loaded term. Part of the problem for the OSP software development team is that the designers of OSP 2.0 decided to try to accommodate several portfolio development and assessment strategies. While they were very broad with their definition of portfolios and developed their own stack of software tools to try to meet those needs, they disregarded the tasks involved in moving student created content from the traditional courseware tools in their classes to their portfolios.

The opportunity to use tools that teachers and students are already extremely familiar with as an entry point for building their portfolios would build on their mental model about what “courseware” is (a set of tools that support existing pedagogical practice such as giving assignments and discussing topics). Working from and expanding this understanding will help faculty that have been reluctant to experiment with portfolios to understand and use the toolset. This is an important difference from the shift in thinking that faculty and students have been asked to make when exchanging their traditional teaching techniques in favor of using portfolio systems that have not been well integrated into existing systems.

We also want to add a sense of student ownership to learning management systems that will make these systems more valuable to students. When seen simply as “courseware” that supports classes that start and end each semester, it makes sense to remove access to student-authored content after each academic period. While the evaluation hasn’t been performed, we believe that students would find more value in a system in which they could build off of the coursework they have done, retain their own personal copies of relevant coursework materials and continue to draw connections and new meaning out of that content. Sharing this new knowledge with fellow learners and educators and getting their feedback is key to providing a student-centered learning experience.

Our use of portfolios in the School of Education is similar to many other groups around the world. We want students to engage in a process that allows them to:

1. Collect content that they have authored in their class assignments and discussions;
2. Select evidence of their learning and classify their own ideas as they relate to institutional and personal learning outcome themes;
3. Reflect on the evidence and experiences;
4. Share (publish) their work with others;
5. Receive feedback and assessment of their progress.   

This is written as a bit of preparatory material for the Ann Arbor, Michigan OSP 2.5 Design and development meeting. I'm including a bit of history here because I think it reflects a case study on project management and product design that the Sakai and OSP communities are facing as they go forward. I welcome any and all corrections to my historical account of the story.

The rapid pace of development has been (thus far) lead by a few schools and institutions that have local interests in mind and an "itch to scratch". I think that the state of affairs is going to be changing soon, with an emerging "User Experience" working group focusing on heuristics and evaluations of new and existing tools to determine their overall usability. I expect that the OSP community and Sakai courseware tools developers will be able to form a new working group that can articulate a vision and design a system that provides a new experience for portfolio development.

Sakai is an open source effort to create a platform for collaboration and learning. An initial grant from the Mellon Foundation in 2003 funded Stanford, MIT, University of Michigan and Indiana University to combine their knowledge and expertise in software development to collaborate on this project. Over the past several years, the Sakai project has transitioned from grant funding to a foundation model, funded by the contributions its paying member organizations, which include primarily higher education institutions and a few vendors that sell support services around open source software. The egalitarian, meritocratic nature of this large open source community presents challenges to software design and development that are often not experienced in more structured environments.

As the Sakai community has evolved, the focus of the community and the definition of what Sakai is have shifted. As recently as 18 months ago, I think I recall that Sakai was referred to as a framework upon which tools could be built. By using the Sakai service APIs provided by the framework, tool developers could focus on the specific functionality of their tool and not have to worry about basic authentication, file storage and security issues typical of web based application development.

The original four institutions had built several tools (Assignment, Discussions, Resources, etc) on the framework to replace their own aging courseware systems on their respective campuses, mainly based on code from the University of Michigan. These tools had a common “look and feel”, used the same template presentation technology (Velocity) and were considered “core” tools for a Sakai distribution. Furthermore, the developers of these tools were working towards the goal of replacing essentially commodity services for which they had several good examples and that the user base had a lot of familiarity with.

During this period, Indiana University and rSmart (a commercial affiliate of the Sakai Foundation) were funded by the Carnegie Foundation for the Advancement of Teaching partnered to port the already successful “Open Source Portfolio” software developed by the University of Minnesota to Sakai. They decided to use the Java Spring framework and Java Server Pages as their presentation layer for the suite of tools that would collectively be known as OSP 2.0. It is important to note that portfolio systems have not worked their way into the culture of most institutions and the implementations of these systems vary widely from institution to institution. As such, there are no strong mental models to work from since the user base expectation of what a portfolio should be has not been firmly established.

The design team charged with redesigning OSP for the Sakai based system decided to make several changes and extensions to the Minnesota system. Portfolios were to be assembled from XML content snippets authored by students and presented by an XSL processing engine to achieve a web-based portfolio. The system is extremely configurable; so much so that “out of the box” the software does almost nothing and requires a great deal of XML design and development in order to customize it for the specific implementation requirements. I believe that the system needs to be drastically simplified and the workflow streamlined for particular uses. One primary use of portfolio software (and one we are very interested in promoting as a School of Education) is to support students to reflect on their learning and to solicit feedback on their progress toward their own learning objectives.

I think that these two development efforts were happening in isolation from each other, the team developing the “courseware” side of Sakai (Assignments and Discussions) had very little connection with the OSP designers and developers. When Syracuse University began using the Open Source Portfolio software in 2005 we realized that there should be a strong connection between the courseware tools (in which students create the bulk of their content) and the portfolio tools if they were to support an emerging assessment strategy in the college that was focused on assessment of student portfolios. Presentation of the concepts at Sakai conferences and in discussion lists proved that our simple idea made a lot of sense to educators. While no formal “evaluations” were done, we definitely heard the experts in the field respond positively to the general idea. The “infinitely configurable” design of the OSP software was widely seen as an impediment to the adoption and use of the software and the community is currently looking for suggestions for improving the design of the software to support a few basic workflows and teaching strategies or “interaction styles” between teachers and learners.

In the following few blog posts I will explain the organizational tasks that I think need to be accomplished by teachers and students using this software during the various stages of one style of portfolio development, the formative journal.

A Class Journal is a formative, reflective, emergent portfolio.

This data flow diagram illustrates the how teachers and students would interact to create, discuss and assess a private (student to teacher) class journal. Periodic review of student blogs (chronologic, personal, reflective portfolios) is likely to be a common interaction style that many teachers would find useful as an venue for formative feedback.

Blogs frequently allow readers (in this case the course instructor) to leave text comments (which this would also support). However, this interaction style would leverage the ability of students and instructors to identify learning outcomes for the journalling process and allow assessment of each blog entry on an ad hoc basis for formative assessment purposes.

Student blog posts would appear as messages in the instructor's inbox. Instructor's would periodically read the blog posts and send back either plain comments or comments with assessments (as related to tagged goals). An important feature of this interaction style is that it supports the idea that "knowledge is emergent" by allowing students to add new learning outcomes to their Goals and to tag their blog posts with those tags. Assessment of student learning, then, can be done in response to student Goals and Faculty Goals (related to the original activity description).

Again, once the course ends or the student leaves the institution, they have the option to keep a copy of all of these posts, assessments and comments for their own use, much like they do with email.

An important assumption is that a system that can engage in this sort of messaging becomes standardized, allowing students to remix and reuse their content from other activities in multiple classes at multiple institutions.