The Living Schoolbook - Interaction Styles

Goal Aware Free Form OSP Portfolio Pages

Sean Keesler — Fri, 18 May 2007 17:48:30 -0400

It may be that the free form portfolio already in OSP will meet some of our needs. The areas surrounded by the green area could be supported by the portfolio tool. The areas outside of that need to be developed.

Another lesson I learned from the OSP 2.5 meeting in Ann Arbor is that there seems to be renewed excitement over the “Free form” style portfolio. LaGuardia Community College is piloting the use of OSP to recreate the portfolio experience that they currently are providing for their students with a commercial tool.

I have to admit, the free form tool always seemed to have the least amount of potential to me and I had dismissed it as “child’s play”. However, after seeing what LaGuardia is doing and thinking about it in the context of the “Class Journal” interaction style, I think that it may have a lot of potential! A few things off the top of my head that I know need to be done to support that interaction style would be to allow students to display structured data on a page of the freeform portfolio.

Functionally, we need to be able to display structured data elements like forms, wizards (and hopefully, assignment data in the future) on a portfolio page. This functionality is not supported yet in the freeform portfolio. If this were rWiki, I could envision a macro that would run an XSL transform against known data types and yield an HTML snippet that we could use in the page.

The ability for students to tag a wizard page with one or more “Goals” is also important to support this interaction style, as is the ability to receive informal assessments (ratings) and feedback from their teachers/peers/friends on each page of the portfolio.

Our interaction style model also states that this is an event that is initiated by the teacher. This is also an important gap that would need to be addressed. Inevitably, teachers would want to schedule multiple portfolio reviews during the course of a semester and have the feedback and ratings attached to those teacher-initiated events.

A "two system" approach to courseware and student owned portfolios

Sean Keesler — Sat, 12 May 2007 02:55:13 -0400

A couple of days ago, Barbara Shelly (one of the original "founders" and a former director of the LSB) came in for a visit. She and I spoke for a while about how she noticed that the adoption rate of education tools by teachers and students was rather low. At least, that's what I think we were talking about. :)

At one point she compared it to the explosive adoption of IM, cell phones, MySpace or Facebook. I think that she really pointed out an important difference between those technologies that put the user in the driver seat and allow them to network with whomever they want, whenever they want, for whatever purpose they want. That network effect and personal sense of control is something that isn't being replicated in the courseware space.

In courseware, it isn't easy to create a small collaboration space for you and your peers. Implementations usually manage student access to course spaces by the bang of the semester drum. There isn't much of a point to pouring a lot of your time into creating content that may be archived and put on a shelf or deleted altogether. What is it about these "closed audience" systems that keep users from having the same excitement about the tools as they do for their MySpace account? Steven Gilbert's blog post showed up for me after a quick Google for "myspace personal publishing blackboard". He notes that the social aspect makes a big difference.

Courseware allows teachers to "manage" a class and to provision services to the students. Adaptive release of content; provisioned users; enterprise integration; and integration with content repositories put a lot of power in the hands of an educator. In contrast, MySpace doesn't try to manage much of anything. You can make and break relationships when you want; put up or take down content when you want to express yourself; and what you put on MySpace won't be deleted by anyone but you. You have the control.

Portfolio tools try to do that too. Students should be able to put together a portfolio and show it to whomever they please.

I think that the dream here is that students are going to get really into putting together their portfolio, just like they do when they "dress up" their MySpace page and chat with their academic friends about all the things they learned in class. I can see that happening to some extent, but I can also see peer review "flame wars" happening through that communication too. I wonder how excited the institution would be to have that sort of content out there, with their logo on it, even if the discussions that happened there were the most productive ones that the students had. What forms of controls would make sense for that sort of environment without killing the sense of spontaneity and personal ownership that might make it appealing?

Better yet, can we build two systems that talk to each other? The courseware server where classes are managed and the official records are kept; and the personal "portfolio" platform which talks to the courseware system and to all of the student's (or teacher's) "friends".

A two system approach would make use of messaging of education related content between the different systems. When I login to my student system, I receive content from the classes I am subscribed to. New learning outcomes, activity descriptions (such as assignments, discussion topics, portfolio reviews, etc.) and assessment data are sent to me. I can turn in my assignments, post to discussions and share my portfolios through a content "port" on this channel and I can receive shared content (their discussion posts, and portfolios for example) from others on that channel as well.

The social aspect of the system occurs through alternative channels that users create and allow their friends to subscribe to. The content through these channels doesn't go through the courseware server.

Reflection – Resources

Sean Keesler — Fri, 11 May 2007 01:16:09 -0400

The Resource tool also needs to be able to support the task of grouping a number of pieces of content with their associated metadata and allowing the student to create a new piece of content containing all of the selected items and a new section that describes the relationship between those items and what was learned by the student as well as any new ideas. The student will be able to tag this new piece of content and reuse it just like any other piece of content as well as designate it as part of a portfolio presentation.

Content Selection - Resources

Sean Keesler — Fri, 11 May 2007 01:14:32 -0400

Just as WebCT has a “Content Manager” section, Sakai has a folder like view of content called “Resources”. The current resources tool allows students to browse their files and classify and move them about in a hierarchical file structure. Other tools often make use of the resources tool, but few activities start there. The cognitive activity of classifying and selecting resources that they would like to showcase or discuss in their portfolio is different in that it makes most sense to start from the unordered pile of resources to classify, group and explain the connections between them. These reflective acts are at the heart of portfolio development.

Switching the resource view from a strict hierarchy to a view that supports a “free tagging” metaphor would allow students to see how faculty have classified their content in relation to institutional and class outcomes. From this same interface, they should be able to create new “outcome” tags and apply them to their content. This would allow students to create multiple views of their content based on various emerging themes.

Portfolio Development - content collection through coursework

Sean Keesler — Fri, 11 May 2007 01:08:08 -0400

Content Collection – Assignments

The assignment tool is arguably the most used tool in courseware packages and faculty and students expect it to behave a certain way with certain rules. The student work handed in through a traditional assignment tool currently is “stuck” in that tool and subsequently is unavailable for students to use in their portfolios without a lot of extra steps that duplicate their data in their personal content area. The assignment tasks will need to be changed slightly to accommodate this new task:

Establish an event (an assignment) that describes the work and deliverables that students are expected to deliver by the assignment due date.
Describe (via a tagging mechanism) the expected learning outcomes from the event. Along with this description will come a description of the rubric that will be used to assess student work.
Review student work and assess the student using the established rubric.
Release student grades and ratings to the students.
Deposit the student work in each student’s personal content area as a piece of standardized content (with outcome and assessment metadata) that can be used in portfolios.

Content Collection – Discussions

The discussion tool is also a common tool used in courses to share ideas and content in a threaded forum. This content “lives” in the discussion tool and is managed with the class and, as such, is not made easily available to students for reuse in their portfolios. While discussions are usually difficult to assess, the subject matter is rich material for personal reflection and, as such, participation in class discussions should be included as a means for content collection for portfolio development. The tasks associated with a class discussion may also need some modification to ensure that the discussion content is made available to the portfolio author”

Establish a forum that describes the topic to be discussed and whether or not the material will be released to students as material for their portfolios.
Describe (via a tagging mechanism) the expected learning outcomes from the discussion.
Either the faculty or students will start discussion threads and add posts to the discussions threads and reply to each other.
At the completion of the discussion, deposit the discussion threads/posts/forums that have been designated as “releasable” in each student’s personal content area as a piece of standardized content that can be used in portfolios.

Portfolio Development - an overview

Sean Keesler — Fri, 11 May 2007 00:57:46 -0400

Anyone involved in a survey of portfolio packages will quickly learn that the term “portfolio” is a loaded term. Part of the problem for the OSP software development team is that the designers of OSP 2.0 decided to try to accommodate several portfolio development and assessment strategies. While they were very broad with their definition of portfolios and developed their own stack of software tools to try to meet those needs, they disregarded the tasks involved in moving student created content from the traditional courseware tools in their classes to their portfolios.

The opportunity to use tools that teachers and students are already extremely familiar with as an entry point for building their portfolios would build on their mental model about what “courseware” is (a set of tools that support existing pedagogical practice such as giving assignments and discussing topics). Working from and expanding this understanding will help faculty that have been reluctant to experiment with portfolios to understand and use the toolset. This is an important difference from the shift in thinking that faculty and students have been asked to make when exchanging their traditional teaching techniques in favor of using portfolio systems that have not been well integrated into existing systems.

We also want to add a sense of student ownership to learning management systems that will make these systems more valuable to students. When seen simply as “courseware” that supports classes that start and end each semester, it makes sense to remove access to student-authored content after each academic period. While the evaluation hasn’t been performed, we believe that students would find more value in a system in which they could build off of the coursework they have done, retain their own personal copies of relevant coursework materials and continue to draw connections and new meaning out of that content. Sharing this new knowledge with fellow learners and educators and getting their feedback is key to providing a student-centered learning experience.

Our use of portfolios in the School of Education is similar to many other groups around the world. We want students to engage in a process that allows them to:

Collect content that they have authored in their class assignments and discussions;
Select evidence of their learning and classify their own ideas as they relate to institutional and personal learning outcome themes;
Reflect on the evidence and experiences;
Share (publish) their work with others;
Receive feedback and assessment of their progress.

The Sakai and OSP projects: different communities, same platform

Sean Keesler — Wed, 9 May 2007 14:04:02 -0400

This is written as a bit of preparatory material for the Ann Arbor, Michigan OSP 2.5 Design and development meeting. I'm including a bit of history here because I think it reflects a case study on project management and product design that the Sakai and OSP communities are facing as they go forward. I welcome any and all corrections to my historical account of the story.

The rapid pace of development has been (thus far) lead by a few schools and institutions that have local interests in mind and an "itch to scratch". I think that the state of affairs is going to be changing soon, with an emerging "User Experience" working group focusing on heuristics and evaluations of new and existing tools to determine their overall usability. I expect that the OSP community and Sakai courseware tools developers will be able to form a new working group that can articulate a vision and design a system that provides a new experience for portfolio development.

Sakai is an open source effort to create a platform for collaboration and learning. An initial grant from the Mellon Foundation in 2003 funded Stanford, MIT, University of Michigan and Indiana University to combine their knowledge and expertise in software development to collaborate on this project. Over the past several years, the Sakai project has transitioned from grant funding to a foundation model, funded by the contributions its paying member organizations, which include primarily higher education institutions and a few vendors that sell support services around open source software. The egalitarian, meritocratic nature of this large open source community presents challenges to software design and development that are often not experienced in more structured environments.

As the Sakai community has evolved, the focus of the community and the definition of what Sakai is have shifted. As recently as 18 months ago, I think I recall that Sakai was referred to as a framework upon which tools could be built. By using the Sakai service APIs provided by the framework, tool developers could focus on the specific functionality of their tool and not have to worry about basic authentication, file storage and security issues typical of web based application development.

The original four institutions had built several tools (Assignment, Discussions, Resources, etc) on the framework to replace their own aging courseware systems on their respective campuses, mainly based on code from the University of Michigan. These tools had a common “look and feel”, used the same template presentation technology (Velocity) and were considered “core” tools for a Sakai distribution. Furthermore, the developers of these tools were working towards the goal of replacing essentially commodity services for which they had several good examples and that the user base had a lot of familiarity with.

During this period, Indiana University and rSmart (a commercial affiliate of the Sakai Foundation) were funded by the Carnegie Foundation for the Advancement of Teaching partnered to port the already successful “Open Source Portfolio” software developed by the University of Minnesota to Sakai. They decided to use the Java Spring framework and Java Server Pages as their presentation layer for the suite of tools that would collectively be known as OSP 2.0. It is important to note that portfolio systems have not worked their way into the culture of most institutions and the implementations of these systems vary widely from institution to institution. As such, there are no strong mental models to work from since the user base expectation of what a portfolio should be has not been firmly established.

The design team charged with redesigning OSP for the Sakai based system decided to make several changes and extensions to the Minnesota system. Portfolios were to be assembled from XML content snippets authored by students and presented by an XSL processing engine to achieve a web-based portfolio. The system is extremely configurable; so much so that “out of the box” the software does almost nothing and requires a great deal of XML design and development in order to customize it for the specific implementation requirements. I believe that the system needs to be drastically simplified and the workflow streamlined for particular uses. One primary use of portfolio software (and one we are very interested in promoting as a School of Education) is to support students to reflect on their learning and to solicit feedback on their progress toward their own learning objectives.

I think that these two development efforts were happening in isolation from each other, the team developing the “courseware” side of Sakai (Assignments and Discussions) had very little connection with the OSP designers and developers. When Syracuse University began using the Open Source Portfolio software in 2005 we realized that there should be a strong connection between the courseware tools (in which students create the bulk of their content) and the portfolio tools if they were to support an emerging assessment strategy in the college that was focused on assessment of student portfolios. Presentation of the concepts at Sakai conferences and in discussion lists proved that our simple idea made a lot of sense to educators. While no formal “evaluations” were done, we definitely heard the experts in the field respond positively to the general idea. The “infinitely configurable” design of the OSP software was widely seen as an impediment to the adoption and use of the software and the community is currently looking for suggestions for improving the design of the software to support a few basic workflows and teaching strategies or “interaction styles” between teachers and learners.

In the following few blog posts I will explain the organizational tasks that I think need to be accomplished by teachers and students using this software during the various stages of one style of portfolio development, the formative journal.

Interaction Style 3: Class Journal

Sean Keesler — Fri, 27 Apr 2007 13:56:21 -0400

A Class Journal is a formative, reflective, emergent portfolio.

This data flow diagram illustrates the how teachers and students would interact to create, discuss and assess a private (student to teacher) class journal. Periodic review of student blogs (chronologic, personal, reflective portfolios) is likely to be a common interaction style that many teachers would find useful as an venue for formative feedback.

Blogs frequently allow readers (in this case the course instructor) to leave text comments (which this would also support). However, this interaction style would leverage the ability of students and instructors to identify learning outcomes for the journalling process and allow assessment of each blog entry on an ad hoc basis for formative assessment purposes.

Student blog posts would appear as messages in the instructor's inbox. Instructor's would periodically read the blog posts and send back either plain comments or comments with assessments (as related to tagged goals). An important feature of this interaction style is that it supports the idea that "knowledge is emergent" by allowing students to add new learning outcomes to their Goals and to tag their blog posts with those tags. Assessment of student learning, then, can be done in response to student Goals and Faculty Goals (related to the original activity description).

Again, once the course ends or the student leaves the institution, they have the option to keep a copy of all of these posts, assessments and comments for their own use, much like they do with email.

An important assumption is that a system that can engage in this sort of messaging becomes standardized, allowing students to remix and reuse their content from other activities in multiple classes at multiple institutions.

Interaction Style 2: Discussion

Sean Keesler — Tue, 24 Apr 2007 13:26:09 -0400

Discussion data flow in a distributed system looks a lot like email. It should!

This data flow diagram illustrates the how students and teachers could interact in a distributed LMS to engage in a group discussion.

A discussion tool is another typical tool found in courseware. The discussion tool is used to allow a teacher to set of "forums" or "threads" of conversation related to the course material. Like the assignment tool, it also lives and dies with the course worksite.

An alternative approach would be to distribute the messages between the courseware system and a learner centered platform, much like email systems do. The difference between the a distributed leaning management system discussion and traditional email systems would be that this content would be stored in a standardized format that allowed users to easily re-use the content in their other courses or portfolios.

Again, once the course ends or the student leaves the institution, they have the option to keep a copy of all of these discussions for there own use, much like they do with email.

Again, there is a lot of duplicated data in distributed system like this which may end up causing issues for storage. However, this does allow each learner to retain a record of all of the discussions that occurred in their classes...forever if they wish to. Similarly, this frees the institution from maintaining the course indefinitely "just in case" students need access to their data.

An important assumption is that a system that can engage in this sort of messaging becomes standardized, allowing students to remix and reuse their content in multiple classes at multiple institutions.

Interaction Style 1: Graded/Rated Assignments

Sean Keesler — Wed, 18 Apr 2007 20:40:33 -0400

Assignment data flow IF a system was distributed, allowing students to truly own their data

This data flow diagram illustrates the how students and teachers would interact in a distributed LMS to carry out the traditional graded assignment workflow.

A frequent use of courseware is to assign work to students, collect that work from students, assess the work and release the grades to students. In most courseware systems, this is all done in one tool. The Assignment tool (or the assignment and gradebook tools) "live" in a course. The design of courseware is mainly to facilitate the instructor's ability to manage the course.

An alternative approach would be to distribute the various transactions between two systems, the courseware system and a learner centered platform. Each set of tools would be designed and configured with the primary user's interest in mind.

The student would receive a message in their "inbox" with the assignment instructions, learning outcomes and rubric for the assignment.

The student would then do the work, saving their "drafts" in their own system and eventually sending back a response to the assignment with their work attached. Note that a copy is sent to the instructor, with the original content still on the student system.

The teacher would receive the student work in her "inbox" and (assuming that there is to be no formative feedback) evaluate the work against the rubric and releasethe grades back to all of the students.

There is a lot of duplicated data in a system like this which may end up causing issues for storage. However, this does allow each learner to retain a record of all of the assignments, work and assessment data that occurred in their classes...forever if they wish to. Similarly, this frees the institution from maintaining the course indefinitely "just in case" students need access to their data.

Workflow for coursework and portfolios

Sean Keesler — Tue, 27 Mar 2007 21:47:27 -0400

Data Flow Diagram of SyrCLE:

Courseware and portfolio systems are usually designed as relational databases to allow a minimum of data duplication while giving users different roles and permissions to interact with their content and data. Students and faculty have expectations about how different tools should be used to organize data into familiar structures with implicit rules for interacting with the data in those tools. The design of these tools have not not allowed a lot of interplay between the tools. It is difficult to publish a portfolio that reuses classroom content (ie: assignments and discussion threads) that the student otherwise has access to view and manipulate in the limited manner the tool allows.

Furthermore, if a student enrolls in another institution or another class that doesn't use the same instance of the courseware, the data from the different classes can not be combined to create new content.

What is needed is a student centered content management system that serves as the student's virtual backpack and their main platform for lifelong learning. The interface between their content management system and the classroom management tools deployed in a LMS has not yet been designed.

This data flow diagram is intended to describe the flow of information into, out of and within a distributed, networked LMS that has two main components: course content and student content management tools.

A couple days ago I shared some "reconceptualizing" Sakai and OSP based on some feedback from a couple of our faculty and the problems that Dr. Joseph Shedd (our PI) has shared in our meetings between Weber/UMich/Portland/Virginia Tech.

From these meetings I think I am hearing the following (my own opinions are sprinkled all over here):

Dr. Shedd wants to be able to run a pilot portfolio program in Syracuse high schools that students can take with them to college. In my opinion, this shouldn't merely be an export of some files or presentations....it should be richer than that...way richer...it should bring along everything the student did in their coursework. All the content, semantics and assessments that they ever got. Anything less is, well, lame.
The attendees at the last Goal Aware BOF were talking about the next "Goal Aware" tool. It sounded like it should be something really general and blog-like to me...something that should allow the student to select some of that old content, tag it with their own goal (and other) metadata and reflect on it, creating a running journal of content, reflection, meta-reflection, etc...this is some folks idea of a great learning tool.
Our current crop of Sakai tools have been independently designed and developed as silos. There is no communication between assignments, discussions, portfolios. Jim Pease has been sort of stymied and frustrated even thinking about how to embed the Goal management and rating interface into these tools. The result is functional but far from optimal from a usability perspective. The metaphor of "bolting on" helper tools to add functionality results in needless extra clicks to accomplish tasks. If I were king I would want a suite of simple tools that just work to support a few interaction styles. Less is more. In Sakai it seems like more is more.
The current usability issues with OSP and Sakai seem to me to be due to one of a couple of things:

Either the design tries to accomodate too many things
or the design was done around a workflow that doesn't match what we really want.

I started to create a diagram (that is NOT done yet, but I would love feedback) to get my head around what I think would be a better workflow for a system designed to create portfolios.