The Living Schoolbook

A while back on the Sakai lists there was some talk about making Sakai a place that integrated with other, new Web 2.0 applications.

When I build and deploy Sakai, I often wonder to myself i Sakai is getting too big for its britches. Its great that anyone can build a tool for this platform, but that has negative consequences too. My sense is that the bulk of the resources are still devoted to development. Tools are released way too early (IMHO) without enough QA. Reviews like the one I just did for the OSP matrix tool are getting done after the tool is already out there being used.

When it comes to building a freeforn portfolio authoring tool, OSP hasn't put forth the resources to do it yet. I recently wrote the list about considering external authoring environments.

I was looking at Google Presentations tonight. It would make a pretty nice portfolio platform. For those of us looking for an easy to use freeform portfolio tool that allows users to author a simple presentation and share it with others, this is pretty attractive. It allows collaborative editing and chat by viewers of the portfolio (what a fun way to get feedback!). If you aren't interested in assessment tools in your portfolio authoring environment, it might be attractive.

If I remember what LaGuardia wanted to do, they wanted to be give out a "stub" portfolio starter that would get the portfolio creation process rolling. Google just let me import and then edit a PowerPoint file.

Wouldn't it be great if a student could "turn in" their URL for their Google "portfolio" (or a MySpace/FaceBook page, etc.) in OSP so that a portfolio evaluator could view and assess it?

I love the idea of separating the student owned authoring and portfolio storage space from the institutional system. If OSP could retrieve and save a copy of the student's portfolio, it would allow the institution to get a copy of the student's work while allowing the student to REALLY OWN their stuff.

With OSP, its the opposite. Students author their stuff in Sakai/OSP and then wonder, "How will I get my stuff out of there so I can use it after I leave here?" The IMS portfolio spec seems to assume that students will want to move their stuff from portfolio system to portfolio system. Export from one place -> Import somewhere else. I would prefer to author and publish external to all institutional systems and show them where MY stuff is.

I think OSP excels as a content management system with a workflow built in for assessment. Prompting students for certain types of information and presenting that information to a board of evaluators consistently is a process that needs to have an information system specifically for that. OSP forms, wizards and templates support that.

We haven't done very well at supporting freeform portfolios. I'm wondering if we should even be in that business.

Sean

I sent an email out to a few people in the OSP and developer community in Sakai with the following contents. I need to flush this idea out a little more. It sounds a bit preachy and a probably fails to convey the idea I have in my head.

The Sakai community has developed a Java based system that allows developers to deploy services and tools for use in collaboration, teaching and learning. The Sakai developer community has, with input from faculty, developed many tools for use in teaching practice. Many of these tools are modeled against existing tools present in other learning management systems. Common workflows found in other systems' tools are captured and embedded in tools like the Sakai assignment, discussion and test tools (for example). To faculty and students, the UI and workflow that are presented by these tools ARE Sakai. To them, the tools are the services that Sakai offers.

Sakai (and many other Learning Management Systems) present a challenge to institutions, teachers and learners. Communities of practice that desire to adopt Sakai for their own use are often forced to adapt their practice to Sakai tools or to develop/modify tools to meet their own needs. As the local community changes their practice over time they have to continuously modify tools to accommodate the changes. This is a costly process that may impede adoption and acceptance by teachers and learners.

The faculty and students at adopting institutions has to learn to skip from tool to tool to support their practice. For example, if an instructor wants to present some reading material to students, ask them to write an assay about the reading assignment and engage the students in group discussion about their learning, this sequence requires no fewer than three Sakai tools. If the teacher wanted to have the students peer review each others work and then reflect on the entire assignment, the workflow gets very frustrating to manage.

A movement in the Sakai community is growing around the frustration behind the user experience. This community consists largely of designers and Human Computer Interaction specialists that plan to spend a lot of time in the field doing heuristic evaluations of existing tools and ethnographic studies of how local communities of practice work. These experts will be able to make incremental improvements to the existing tool design and model new tools that will be able to support certain workflows that occur within those communities of practice. The success of these new or improved tools to meet everyone's needs depends on how well those local workflows reflect universally accepted practice. Surely, experimental changes will be require further local development to provide new features. In general, the "business logic" of a community of practice will be encapsulated in each tool design.

As an instructional technologist, who sits somewhere between the faculty and the developers, I see a gap between the UX group's effort and the developer's technical knowledge of system architecture. There is a need for a new type of community participant that engages the pedagogy group to gather requirements around services that make sense to teachers and learners, and enlists the expertise of the developer and UX communities in the creation of an abstract layer of pedagogical services and basic UI components (not tools) that provide a selection of discoverable services in Sakai that will allow communities of practice to model their own teaching and learning practice independent of specific tool design.

The Goal Management project that started at Syracuse University began as a single tool that allowed a user (likely a teacher or program administrator) to articulate learning outcomes. We made a modification to the standard assignment tool to allow the teacher to identify the learning outcomes (from the goal management tool) that each assignment addresses and then later assess students' work in terms of those outcomes. The original plan was to make a number of other tools (discussion, tests, OSP) "goal aware" as well. This is very difficult to do well for a variety of technical reasons. Technical problems aside, a pedagogical problem exists with this approach. Once the tool is made "goal aware", that functionality exists everywhere. We have added a configuration to turn it on or off, but it lies at the tool placement level. This means that the creator of the site needs to make a decision to enable a feature when they create the site or later go back and turn it on. What seems like a better approach would be to establish "planning" and "assessment" services that would always available to a user in Sakai, regardless of the tool being used.

We can envision a scenario where a workflow is authored by a teacher where they invoke a "planning" service to describe the next activity, complete with deadlines, outcomes and rubrics. The teacher then invokes the "assignment" service and deploys their activity. Students log into Sakai and automatically invoke a service where they see that their teacher has just asked them to write an essay. They write the essay and invoke the "turn in" service and upload their file. The teacher sees the student has turned in the assignment and invokes the "assessment" service that prompts them with their planning information, the student's work and a means to assess the student. They see the student needs help on this assignment and invoke the "private discussion" service and begin a week long dialogue with that student about how they could improve their understanding on the topic. As the private discussion progresses, the student makes some notes in their reflective journal about something important that they just realized. The student later turns in their assignment and the teacher reassesses the work.

The above scenario can be supported now with existing Sakai tools. However, there is no thread that ties the different activities that would occur across several tools together. Further, this scenario would currently require some significant overhead and planning on both parties. Jumping from tool to tool; discovering that new content was available; and realizing that the content in one tool related to previous activities in another tool all contribute to the problem. Moveover, if a new service was created for a locally created need, it too could be invoked and inserted into the workflow without changing the other service designs.

I believe that there may be a large percentage of faculty needs that could be met by thinking in terms of services that have meaning to our primary audience, rather than thinking in terms of tools. A great number of us understand the frustration that exists when we talk to faculty about how to use Sakai or how best to develop a tool. We have to keep in mind that these faculty are the tip of the iceberg, the early adopters that elect to engage a highly technical group because of the recognition of the potential. The majority of our audience lies outside of this group, waiting for us to start talking their language. Let us make it our mission to learn to do so.

It may be that the free form portfolio already in OSP will meet some of our needs. The areas surrounded by the green area could be supported by the portfolio tool. The areas outside of that need to be developed.

Another lesson I learned from the OSP 2.5 meeting in Ann Arbor is that there seems to be renewed excitement over the “Free form” style portfolio. LaGuardia Community College is piloting the use of OSP to recreate the portfolio experience that they currently are providing for their students with a commercial tool.

I have to admit, the free form tool always seemed to have the least amount of potential to me and I had dismissed it as “child’s play”. However, after seeing what LaGuardia is doing and thinking about it in the context of the “Class Journal” interaction style, I think that it may have a lot of potential! A few things off the top of my head that I know need to be done to support that interaction style would be to allow students to display structured data on a page of the freeform portfolio.

Functionally, we need to be able to display structured data elements like forms, wizards (and hopefully, assignment data in the future) on a portfolio page. This functionality is not supported yet in the freeform portfolio. If this were rWiki, I could envision a macro that would run an XSL transform against known data types and yield an HTML snippet that we could use in the page.

The ability for students to tag a wizard page with one or more “Goals” is also important to support this interaction style, as is the ability to receive informal assessments (ratings) and feedback from their teachers/peers/friends on each page of the portfolio.

Our interaction style model also states that this is an event that is initiated by the teacher. This is also an important gap that would need to be addressed. Inevitably, teachers would want to schedule multiple portfolio reviews during the course of a semester and have the feedback and ratings attached to those teacher-initiated events.

Ever since we conceptualized and built the Sakai Goal Management tool we have heard a need for a rubric built into the tool. However, it has not been very clear how we should manage and apply rubrics. Today, in the OSP 2.5 planning meeting, I heard of a possible way we could design the tools that made some sense to me. However, I think that the implication that I drew from the conversation makes we wonder if the idea is implementable.

Our current design for “Goal Aware” tools allows a user to create (and centrally manage) a tag (a Goal, in our parlance) for each learning outcome for a class. Another type of tag that schools might find a lot of value in would be a managed “Stage” type tag. As it was explained to me, for program assessment, there needs to be a managed system of “Outcomes”, “Stages” and rubrics for each combination of the two.

While we allow multiple “Goals” to be attached to any one activity (an assignment for example), we wouldn’t expect activities to apply to more than one “Stage”. As it was stated in this meeting, there would be the expectation that at any one stage, only one rubric should be used when evaluating student performance (ratings, in our case) for an outcome of the activity. This would help support the investigation into inter-rater reliability when more than one person is rating students.

It looks like this:

So far, so good. That makes sense to me, if the rubrics are simple rating scales that inform the "rating" process (ie: result in a single number), but my understanding is that they are not. The usual definition of a rubric is yet another matrix of "performance indicators" and a "rating scale" for each of those indicators. Typically the results of each of the ratings figure into the overall "grade" of an assignment, but in this case only result into a single rating one just one goal that an activity was supposed to address.

Since I am not a teacher, I can't say for sure, but this seems like a lot of work to do to assess a single student. My feeling is that teachers/faculty will largely oppose that level of granularity when doing assessment.

A couple of days ago, Barbara Shelly (one of the original "founders" and a former director of the LSB) came in for a visit. She and I spoke for a while about how she noticed that the adoption rate of education tools by teachers and students was rather low. At least, that's what I think we were talking about. :)

At one point she compared it to the explosive adoption of IM, cell phones, MySpace or Facebook. I think that she really pointed out an important difference between those technologies that put the user in the driver seat and allow them to network with whomever they want, whenever they want, for whatever purpose they want. That network effect and personal sense of control is something that isn't being replicated in the courseware space.

In courseware, it isn't easy to create a small collaboration space for you and your peers. Implementations usually manage student access to course spaces by the bang of the semester drum. There isn't much of a point to pouring a lot of your time into creating content that may be archived and put on a shelf or deleted altogether. What is it about these "closed audience" systems that keep users from having the same excitement about the tools as they do for their MySpace account? Steven Gilbert's blog post showed up for me after a quick Google for "myspace personal publishing blackboard". He notes that the social aspect makes a big difference.

Courseware allows teachers to "manage" a class and to provision services to the students. Adaptive release of content; provisioned users; enterprise integration; and integration with content repositories put a lot of power in the hands of an educator. In contrast, MySpace doesn't try to manage much of anything. You can make and break relationships when you want; put up or take down content when you want to express yourself; and what you put on MySpace won't be deleted by anyone but you. You have the control.

Portfolio tools try to do that too. Students should be able to put together a portfolio and show it to whomever they please.

I think that the dream here is that students are going to get really into putting together their portfolio, just like they do when they "dress up" their MySpace page and chat with their academic friends about all the things they learned in class. I can see that happening to some extent, but I can also see peer review "flame wars" happening through that communication too. I wonder how excited the institution would be to have that sort of content out there, with their logo on it, even if the discussions that happened there were the most productive ones that the students had. What forms of controls would make sense for that sort of environment without killing the sense of spontaneity and personal ownership that might make it appealing?

Better yet, can we build two systems that talk to each other? The courseware server where classes are managed and the official records are kept; and the personal "portfolio" platform which talks to the courseware system and to all of the student's (or teacher's) "friends".

A two system approach would make use of messaging of education related content between the different systems. When I login to my student system, I receive content from the classes I am subscribed to. New learning outcomes, activity descriptions (such as assignments, discussion topics, portfolio reviews, etc.) and assessment data are sent to me. I can turn in my assignments, post to discussions and share my portfolios through a content "port" on this channel and I can receive shared content (their discussion posts, and portfolios for example) from others on that channel as well.

The social aspect of the system occurs through alternative channels that users create and allow their friends to subscribe to. The content through these channels doesn't go through the courseware server.

The Resource tool also needs to be able to support the task of grouping a number of pieces of content with their associated metadata and allowing the student to create a new piece of content containing all of the selected items and a new section that describes the relationship between those items and what was learned by the student as well as any new ideas. The student will be able to tag this new piece of content and reuse it just like any other piece of content as well as designate it as part of a portfolio presentation.

Just as WebCT has a “Content Manager” section, Sakai has a folder like view of content called “Resources”. The current resources tool allows students to browse their files and classify and move them about in a hierarchical file structure. Other tools often make use of the resources tool, but few activities start there. The cognitive activity of classifying and selecting resources that they would like to showcase or discuss in their portfolio is different in that it makes most sense to start from the unordered pile of resources to classify, group and explain the connections between them. These reflective acts are at the heart of portfolio development.

Switching the resource view from a strict hierarchy to a view that supports a “free tagging” metaphor would allow students to see how faculty have classified their content in relation to institutional and class outcomes. From this same interface, they should be able to create new “outcome” tags and apply them to their content. This would allow students to create multiple views of their content based on various emerging themes.

Content Collection – Assignments

The assignment tool is arguably the most used tool in courseware packages and faculty and students expect it to behave a certain way with certain rules. The student work handed in through a traditional assignment tool currently is “stuck” in that tool and subsequently is unavailable for students to use in their portfolios without a lot of extra steps that duplicate their data in their personal content area. The assignment tasks will need to be changed slightly to accommodate this new task:

1. Establish an event (an assignment) that describes the work and deliverables that students are expected to deliver by the assignment due date.
2. Describe (via a tagging mechanism) the expected learning outcomes from the event. Along with this description will come a description of the rubric that will be used to assess student work.
3. Review student work and assess the student using the established rubric.
4. Release student grades and ratings to the students.
5. Deposit the student work in each student’s personal content area as a piece of standardized content (with outcome and assessment metadata) that can be used in portfolios.

Content Collection – Discussions

The discussion tool is also a common tool used in courses to share ideas and content in a threaded forum. This content “lives” in the discussion tool and is managed with the class and, as such, is not made easily available to students for reuse in their portfolios. While discussions are usually difficult to assess, the subject matter is rich material for personal reflection and, as such, participation in class discussions should be included as a means for content collection for portfolio development. The tasks associated with a class discussion may also need some modification to ensure that the discussion content is made available to the portfolio author”

1. Establish a forum that describes the topic to be discussed and whether or not the material will be released to students as material for their portfolios.
2. Describe (via a tagging mechanism) the expected learning outcomes from the discussion.
3. Either the faculty or students will start discussion threads and add posts to the discussions threads and reply to each other.
4. At the completion of the discussion, deposit the discussion threads/posts/forums that have been designated as “releasable” in each student’s personal content area as a piece of standardized content that can be used in portfolios.

Anyone involved in a survey of portfolio packages will quickly learn that the term “portfolio” is a loaded term. Part of the problem for the OSP software development team is that the designers of OSP 2.0 decided to try to accommodate several portfolio development and assessment strategies. While they were very broad with their definition of portfolios and developed their own stack of software tools to try to meet those needs, they disregarded the tasks involved in moving student created content from the traditional courseware tools in their classes to their portfolios.

The opportunity to use tools that teachers and students are already extremely familiar with as an entry point for building their portfolios would build on their mental model about what “courseware” is (a set of tools that support existing pedagogical practice such as giving assignments and discussing topics). Working from and expanding this understanding will help faculty that have been reluctant to experiment with portfolios to understand and use the toolset. This is an important difference from the shift in thinking that faculty and students have been asked to make when exchanging their traditional teaching techniques in favor of using portfolio systems that have not been well integrated into existing systems.

We also want to add a sense of student ownership to learning management systems that will make these systems more valuable to students. When seen simply as “courseware” that supports classes that start and end each semester, it makes sense to remove access to student-authored content after each academic period. While the evaluation hasn’t been performed, we believe that students would find more value in a system in which they could build off of the coursework they have done, retain their own personal copies of relevant coursework materials and continue to draw connections and new meaning out of that content. Sharing this new knowledge with fellow learners and educators and getting their feedback is key to providing a student-centered learning experience.

Our use of portfolios in the School of Education is similar to many other groups around the world. We want students to engage in a process that allows them to:

1. Collect content that they have authored in their class assignments and discussions;
2. Select evidence of their learning and classify their own ideas as they relate to institutional and personal learning outcome themes;
3. Reflect on the evidence and experiences;
4. Share (publish) their work with others;
5. Receive feedback and assessment of their progress.   

This is written as a bit of preparatory material for the Ann Arbor, Michigan OSP 2.5 Design and development meeting. I'm including a bit of history here because I think it reflects a case study on project management and product design that the Sakai and OSP communities are facing as they go forward. I welcome any and all corrections to my historical account of the story.

The rapid pace of development has been (thus far) lead by a few schools and institutions that have local interests in mind and an "itch to scratch". I think that the state of affairs is going to be changing soon, with an emerging "User Experience" working group focusing on heuristics and evaluations of new and existing tools to determine their overall usability. I expect that the OSP community and Sakai courseware tools developers will be able to form a new working group that can articulate a vision and design a system that provides a new experience for portfolio development.

Sakai is an open source effort to create a platform for collaboration and learning. An initial grant from the Mellon Foundation in 2003 funded Stanford, MIT, University of Michigan and Indiana University to combine their knowledge and expertise in software development to collaborate on this project. Over the past several years, the Sakai project has transitioned from grant funding to a foundation model, funded by the contributions its paying member organizations, which include primarily higher education institutions and a few vendors that sell support services around open source software. The egalitarian, meritocratic nature of this large open source community presents challenges to software design and development that are often not experienced in more structured environments.

As the Sakai community has evolved, the focus of the community and the definition of what Sakai is have shifted. As recently as 18 months ago, I think I recall that Sakai was referred to as a framework upon which tools could be built. By using the Sakai service APIs provided by the framework, tool developers could focus on the specific functionality of their tool and not have to worry about basic authentication, file storage and security issues typical of web based application development.

The original four institutions had built several tools (Assignment, Discussions, Resources, etc) on the framework to replace their own aging courseware systems on their respective campuses, mainly based on code from the University of Michigan. These tools had a common “look and feel”, used the same template presentation technology (Velocity) and were considered “core” tools for a Sakai distribution. Furthermore, the developers of these tools were working towards the goal of replacing essentially commodity services for which they had several good examples and that the user base had a lot of familiarity with.

During this period, Indiana University and rSmart (a commercial affiliate of the Sakai Foundation) were funded by the Carnegie Foundation for the Advancement of Teaching partnered to port the already successful “Open Source Portfolio” software developed by the University of Minnesota to Sakai. They decided to use the Java Spring framework and Java Server Pages as their presentation layer for the suite of tools that would collectively be known as OSP 2.0. It is important to note that portfolio systems have not worked their way into the culture of most institutions and the implementations of these systems vary widely from institution to institution. As such, there are no strong mental models to work from since the user base expectation of what a portfolio should be has not been firmly established.

The design team charged with redesigning OSP for the Sakai based system decided to make several changes and extensions to the Minnesota system. Portfolios were to be assembled from XML content snippets authored by students and presented by an XSL processing engine to achieve a web-based portfolio. The system is extremely configurable; so much so that “out of the box” the software does almost nothing and requires a great deal of XML design and development in order to customize it for the specific implementation requirements. I believe that the system needs to be drastically simplified and the workflow streamlined for particular uses. One primary use of portfolio software (and one we are very interested in promoting as a School of Education) is to support students to reflect on their learning and to solicit feedback on their progress toward their own learning objectives.

I think that these two development efforts were happening in isolation from each other, the team developing the “courseware” side of Sakai (Assignments and Discussions) had very little connection with the OSP designers and developers. When Syracuse University began using the Open Source Portfolio software in 2005 we realized that there should be a strong connection between the courseware tools (in which students create the bulk of their content) and the portfolio tools if they were to support an emerging assessment strategy in the college that was focused on assessment of student portfolios. Presentation of the concepts at Sakai conferences and in discussion lists proved that our simple idea made a lot of sense to educators. While no formal “evaluations” were done, we definitely heard the experts in the field respond positively to the general idea. The “infinitely configurable” design of the OSP software was widely seen as an impediment to the adoption and use of the software and the community is currently looking for suggestions for improving the design of the software to support a few basic workflows and teaching strategies or “interaction styles” between teachers and learners.

In the following few blog posts I will explain the organizational tasks that I think need to be accomplished by teachers and students using this software during the various stages of one style of portfolio development, the formative journal.