The Living Schoolbook - Sakai

What's in Your Portfolio?

Sean Keesler — Sat, 15 Dec 2007 00:24:04 -0500

Did you ever see those Capital One commercials where a band of marauders advances on an innocent customer as he makes a purchase with his credit card? The marauders are stopped just before they strike when they realize that this guy was smart enough to choose a credit card that didn't have ridiculous interest rates. The slogan "What's in your wallet?" begs the watcher to make an either/or decision to get slaughtered by the marauders, or to go with their low interest card.

As I read Trent Batson's Campus Technology article, The ePortfolio Hijacked, I thought of hordes of accreditors descending on innocent students, ready to take control of their portfolios and destroy their opportunities to recast themselves as educated, enlightened members of society. He really makes it clear that an assessment system portfolio is not a "learning portfolio". He has drawn a line in the pedagogical sand and proclaimed that there are the assessment system portfolios and then there are REAL portfolios.

Here at Syracuse, the assessment system that is used as a tool to gather data for program review and accreditation needs is not readily available to students. Assessments are mapped to program proficiency standards and faculty rate student work on a 1-4 scale in a system that the students do not have access to. Most of the faculty do not have access to the database that stores that info. Data is collected through spreadsheets and fed into the assessment system by hand by just a few people.

An alternative approach for deploying an assessment system might open possibilities for reflection and formative feedback that we are currently missing. If the data was turned around so that students could see what ratings they received on each of the standards and why, it could stir up some really rich discussion between teachers and students. The Goal Management project that we started here was meant to make transparent the relationship between the program standards, classroom assignments and the data collection process that was being performed. Trent might say that this data only has an audience with deans, accreditors and department heads but I think that it may be the spark that starts students down the road to reflective thinking.

I'd like to think of an assessment system as a means to start a dialogue between teachers and learners around some common themes and a compliment to formative assessment and "assessment for learning", rather than a competitor to both. It is really how you design and use the system, not an inherent property of it. My guess is that there are ways to implement assessment systems that would encourage reflection and result in ideas that a student might take a lot of pride in. The key is in the teachers' ability to engage the student in conversation and seize the teachable moments that such a system makes possible.

Are "e-portfolios" too ambitious?

Sean Keesler — Mon, 26 Nov 2007 02:52:37 -0500

The faculty and students have been "doing" portfolios in the Inclusive Elementary Special Education program since before I came here in 2000. They used to have a catered dinner at "Drumlins" and sit at round tables while 3 or 4 students would show their binder-based portfolio to a host teacher and a faculty member while nibbling on hors d'oevres and ham sandwiches at the end of the semester. The event was mostly celebratory. Even if some advice was dispensed during one of these sessions, it seemed like it was too late to me, a newcomer to all of this.

Several years later, as the school began experimenting with "e" portfolios in the form of Dreamweaver and Netscape Composer authored web sites, more time was spent teaching technical skills to the future elementary teachers so that they could showcase the highlights of the semester to their peers and teachers. As a member of a tech group that created and maintained a home made course management system called "Dialogue", which was designed around the idea that EVERY opportunity for assessment/review was also an opportunity to begin a thread of formative discussion, I thought the portfolio idea had a lot of potential to be a great learning opportunity.

The technical barrier to authoring one of these portfolios was substantial, even when we shifted gears and began starting students with a PowerPoint template, with stubbed out pages suggesting that the students address our emerging "School of Education Proficiencies" in their portfolios. How to resize images, scan documents that proved that they were "proficient" and embed video snippets that played on multiple platforms remained issues that were dealt with during the last few weeks at the tail end of the semester. The rest of the semester was spent doing the real work of the school, training students to be excellent teachers.

For the past two years we have been using the web based Open Source Portfolio software. While OSP has improved markedly in that short time, the other barriers remain. As such, our student's portfolios are usually not worked on (or read) by most of the teachers in the school until the end of the semester. In that regard, nothing has changed. It seems that coursework is coursework and the portfolio remains this "other" thing that gets tacked on the end of the semester.

I have this vision of faculty sitting down in the morning with a cup of coffee and reading their students' daily musings (just like I check a pile of blogs on a regular basis), writing back a few thoughts and using that information to inform the next week's lessons.

I wonder if e-portfolios are way too ambitious. A blog with a tagging feature, an few rss feeds and a class blog "aggregator" might be all we need to embed "portfolio thinking" into classes.

Educational Portfolio #1 – The outcomes based accreditation or program review portfolio

Sean Keesler — Wed, 21 Nov 2007 16:29:25 -0500

Disclaimer: For those of you who do not know me, I am not a teacher. I provide technical support for grant funded educational projects in the School of Education at Syracuse University. Take this perspective with that grain of salt.

Over the past three years of working with the Open Source Portfolio and Sakai communities, I have heard lots of people talk about how they plan to implement the portfolio tools. Some of the uses highlight the ability of the tools to create a standard presentation (like a resume or curriculum-vitae) by using a template. In a later entry, I’ll be walking you through the steps I would follow in order to create a set of forms, wizards and a template that will allow users to pop out a presentation like that. This entry isn’t about those types of uses. I am sure that everyone understands the purpose of a portfolio like that. I want to take a moment to talk about the types of portfolios that can be useful in education settings. This entry doesn't really talk about how to imlpement an accreditation portfolio. I want to set the stage first.

The outcomes based accreditation or program review portfolio

Some schools need a means to review student learning/progress/achievement of ALL of the students in their programs in a very comprehensive way. The prerequisite to a system like this is that the faculty has decided upon an assessment framework that will be applied to all of their students and all of their classes. A “portfolio” effort like this has several components.

Establishment of a framework for evaluation

In order to set this up, a set of outcomes must be agreed to by the faculty. This is a process that many schools apparently do not engage in very often. I haven’t heard of a faculty that spontaneously decides to engage in this process. They usually have more interesting things to do than debate the purpose of the programs in which they teach. Usually it is a mandate from an accrediting body that initiates the process. It was for our faculty.

Identification of key program assessments

A plan has to be designed for measuring the performance/achievement of the students in the program (and in so doing, the effectiveness of the program). A natural tool to use for this purpose is the assessments already in place in the program’s courses. Chances are, a subset or cross section of the work the students already do in their classes would provide the information needed to see how the students and the program are doing. By highlighting the assessments already in place in the program’s curriculum and matching them to the outcomes that they address we can aggregate student work aggregated into FACULTY DESIGNED portfolios that serve the purpose of the program.

Assessment of student work in light of the program outcomes

Once the a faculty member has identified an assessment (an assignment usually) in their course that is intended to assess (at least in part) one or more of the program outcomes, the student’s work will need to be assessed against each of those outcomes. Whether the instructor of the course performs that assessment while grading papers or a "program review panel" performs the assessment later on, the process requires standardized rating scales and rubrics to ensure that all of the faculty are on the same page when assessing the student work. Establishing standard rubrics that ensure inter-rater reliability is another huge task that takes place outside of the “system”. The axiom “garbage in – garbage out” applies here.

Analysis of the Data

It may surprise you to see the multiplicative effects that adding new assessments, outcomes and students to this structure has on the amount of data that there is to look at and analyze. A cohort of 100 students going through a program with 30 highlighted assessments each identified as addressing on average of 2 outcome each provides 6000 pieces of data for analysis. If assessment is continuously performed by instructors (rather than by a “panel” at the end of the semester) it might be helpful if both students and the faculty could have real time access to this assessment portfolio data to help identify problems and trends in student performance and hopefully correct the problem with relevant formative feedback, counseling, etc. If the data analysis is done at the end of the semester, the possibility of formative feedback may be forfeit and the focus on the data analysis would then be to get an overview of the performance of various aspects of the program in order to improve it.

Implementing Change

Once the data analysis is completed, some issues may be identified. The faculty may decide to change its rubrics to tune the assessment process, rephrase the program outcomes, restructure the program curriculum and/or pick different assessments for review. Accreditation bodies are particularly interested in this process for overall review and data driven decision-making.

Without going into details about the implimentation of a solution in Sakai and OSP, it should be clear enough that a mandate to engage in this level of program review requires changes in the organization of a school. In our school, a new full time position was created to manage the assessment process. The assessment coordinator is charged with setting up the assessment framework, collecting the data and running reports.

Goal Management

When the LSB started using OSP to address our program review portfolio needs, we quickly identified a gap between the tool set's capabilities and design philosophy. Most everything in OSP is geared towards making the process initiated by students. In the OSP matrix tool, the STUDENT gets to pick the items that meet the criteria of the matrix. While we were interested in any additional material that students felt was relevent to present as part of their own evidence of mastery of the program outcomes, we knew that asking them to resubmit all of the work that the faculty had identified was a process that was unreliable and unnecessary.

The Goal Management tool released as a “contrib” tool for the 2.4 release was an effort to provide a means for a user to articulate multiple sets of outcomes (goals) and to share those goals with a number of class worksites in Sakai. Instructors who create assignments in their classes can identify the outcomes that pertain to each assignment and rate student work against each. The outcomes and rating data are stored in the Sakai database and can be aggregated and reported.

While we do not have funding right now to continue developing the idea, the basic premise has rung true with many other institutions and is likely to be further developed by Indiana University (hopefully with helpful input by the rest of the community) to be an integrated component of versions of OSP in future releases of the software.

In the interest of full disclosure, our own implementation of Goal Management in Sakai is NOT the production system that our assessment coordinator uses to collect data. Our implementation is a research project that lacks some critical features that would make it an acceptible substitute for the current production system. Discussions are in progress about the cost-benefit of building the necessary features and maintaining a Sakai instance as compared to continuing to use the current system.

Open Source Portfolio Tools - a portfolio development kit

Sean Keesler — Fri, 16 Nov 2007 01:09:28 -0500

The Open Source Portfolio (http://www.osportfolio.org) is often difficult to explain to folks. There are some aspects to the design of the software that seem to be in contradiction with one another. There is a tension between the belief that portfolios are "student owned" and the technical reality that some of the tools are far too complex for students to use. As a result, implementers of OSP use the flexibility of the tools to attempt to craft an environment that meets the needs of all of the users. The simple fact that the experience is designed and implemented by users who may be two degrees from the users make this a challenging piece of software to work with.

I plan to write a bit more about this "suite" of tools in the coming weeks to help identify some ways that I think implementors can get productive fast, but read on for a brief intro to the tools of the Open Source Portfolio.

The Forms tool

The form tool allows a user (probably someone who is wise in the ways of XML) to define data input screens (forms) that will be used to prompt users and structure data entry into the system that will later be used to construct a portfolio. The user gives the form a name, types in some instructions and then identifies an XML Schema Definition (XSD) file that actually describes what the fields are in the form, the validation rules, and some special display information. Optionally, the form designer may further want to craft an alternative view for the form and will identify XSL (Extensible Stylesheet Language) files that will transform a default rendering of a form to one of design.

Wizard pages

While there is no “wizard page” tool, this structure is used as a building block in both the Wizards and the Matrix tools. A wizard page has a unique name and will present guidance (instructions, rationale and examples) to the portfolio owner. When constructing a wizard page, the designer will be able to add the following “pieces” to the page:

Forms (from the above mentioned tool) for the portfolio owner to fill out.
A reflection form for the owner to use.
An evaluation form for an evaluator to use when assessing the page.
A feedback form for a portfolio reviewer to use to provide informal comments.

When describing a wizard page, all of these structured data input forms are completely optional. At a very minimum portfolio owners will always be able to upload and attach files to the page.

The Wizards tool

The Wizards tool allows a user to create either a series of screens that further "scaffold" the user through the process of entering the data that will used to create their portfolio. There are two types of wizards in OSP:

Sequential Wizards can be constructed as a set of pages that are presented one after the other.
Hierarchical wizards present a nested tree of categories and pages to users.

In either case a the pages of the wizard can be constructed to include guidance (instructions, rationale and examples) to the portfolio user. The wizard author can choose forms (from the above mentioned tool) that will prompt the portfolio author to provide pieces of information relevant to the portfolio. The wizard author also has the opportunity to prompt the user to add reflections along the way, receive feedback from reviewers and evaluators.

The Matrix tool

The Matrix tool allows the designer to construct grids of one-page wizards wizard pages (the same wizard pages as described above). The rows and columns of the matrix are described as “criteria” and “levels” in the tool. A matrix “cell” can be set to be “unlocked” (ready for the portfolio owner to use) or “locked” (unavailable to the portfolio owner). Matrix cells can be set to unlock (top to bottom, left to right, or as the instructor deems appropriate) as the owner submits the cells for evaluation.

The Style tool

The Style tool allows the designer to designate an uploaded css file as a “style”.

The Layouts tool

The Layouts tool allows a designer to designate an xhtml file as a template for a free-form portfolio page. Page layouts can have many different designs (Title over text, two columns, Title, an image and a text area, etc.). Two page layouts come with the software.

The Portfolio Template tool

The Portfolio Template tool allows a designer to prompt the portfolio owner for various wizards, matrices, forms and files that they have and create a polished presentation of all of it. The designer can specify css stylesheets, JavaScript files, images and logos to create a branded look for their users. The designer needs to define all of the pieces that go into making a complete portfolio of this type and designate an XSL stylesheet to reorganize the XML mashup of all of that structured (forms, wizards and matrices) and unstructured data (raw attached files) into a presentation.

The Portfolio Tool

The portfolio tool allows the portfolio owner to either create a portfolio from a template (see above) or to design her own portfolio.

If the owner decides to build their own portfolio, they can choose a style (see the style tool above) and begin constructing a series of pages, each with a predefined layout (see above layouts tool).

If the owner wants to build a template-based portfolio, they are prompted to select appropriate data structures (as defined by the template designer) to fill out the portfolio.

When a portfolio is created, the owner can specify who gets to see it (a user, a group of users, guest users) and for how long or they can make their portfolio public to the world.

Does that sound complicated? It is! Trying to figure out how all of these tools work together to provide a workable solution is daunting...and that is just half of it. Once you have scraped your knuckles for a while, you have to actually design the actual forms, the matrices, wizards, styles, layouts and templates that will produce the kind of product that you think your “client” will accept and portfolio authors will find valuable.

Rethought Goal Management - OSP 2.6 Planning meeting in Indianapolis

Sean Keesler — Tue, 23 Oct 2007 17:28:06 -0400

A quick model of what I thought I heard about the future of Goal Management.

The big things that are different here:

Goal sets include both Goals and Levels. In essence a Goal Set actuallly will be a matrix. The illustration belies the multi-dimensional aspect that will result from a hierarchical set of goals.
Program chairs or assessment coordinators will be able to "stub out" assessment placeholders and link them to cells (a goal and a level).
Linking of activities (like assignments) can be semi-automated when faculty are prompted to identify their activity with an assessment placeholder.
The assessment placeholder would allow the chair or corrdinator to present a consolidated set of fields that will be used to collect ratings. Each field will need to have rules that disseminate the input ratings up to the linked cells.

Education mashup of the "tool set du jour" - a strategy to pursue

Sean Keesler — Mon, 1 Oct 2007 19:02:28 -0400

Last week I had the privilege of attending a brown bag lunch seminar given by Professor Emeritus Donald Ely on the subject of distance learning.

The discussion began with Dr. Ely putting an emphasis on assessment but seemed to diverge quickly to participant discussions about how it is difficult to create the virtual environment necessary to teach tacit skills like public speaking and medicine. Questions like, "How do I assess active listening skills over distance" were typical of the initial discussion.

It was noted that some LMS's seemed to have certain features that filled certain functional gaps and some of the participants made suggestions/observations that many tools that are widely used today (AIM, Skype, Google Docs) may help fill some perceived gaps in learning mangement tools such as Blackboard and WebCT.

It was a good discussion that was accompanied by the excited enthusiasm that is typical of sharing sessions about the new tools. Markedly absent from the entire discussion, however, was the realization that there is going to be a huge gap left as faculty and students move from a provisioned set of tools (their centrally managed, institutional courseware systems) to a distributed set of tools. Keep in mind that the accountability movement almost demands a centralized system from which we query about student, class and program performance.

It strikes me that next generation LMS's may be more like a mashup view of the "tool set du jour". Faculty and students will continue to find and use new tools that better suit their pedagogical/learning needs. The institution will want to gather and analyze data created in those tools for a variety of reasons (program evaluation, best practice identification, etc.) What is missing is the common language that these new tools will need to speak before we can use them. Sure Google Docs is great as a place for students to work collaboratively, but the institution needs a means to harvest that document for its own purposes. Student portfolios can be created on Facebook, but how does the college compare the student's portfolio over time? Someone may build a killer 3rd party assignment tool, but can we realistically consider it if the institutiuonal system can't interface with it?

I think that this is a strategic direction where a community like Sakai could weigh in heavily. Rather than emulating what we already see out there in the courseware market, let's distinguish ourselves and anticipate where the market is going.

Different ways to manage rubrics

Sean Keesler — Mon, 1 Oct 2007 15:11:14 -0400

I spoke this morning with John Gosney from IUPUI about their plans to move the Goal Management project forward. One of the functional areas that seems to need a little attention was the need to have multiple rubrics. A while ago I emailed the OSP portfolio list this bit:

Rubric needs GM-122

During our time in Ann Arbor we talked about the need for managed rubrics.

I don't know if anyone has any experience with other rubric management or creation tools out there...I know rubristar (http:// rubistar.4teachers.org) because of another project I was involved in a few years back. After revisiting the rubristar site and talking with Joe Shedd about his expectations, I thought I would bounce some ideas off the community about the requirements for GM-122 (http:// jira.sakaiproject.org/jira/browse/GM-122).

A rubric is usually represented by a matrix of goals/outcomes (usually shown on the left side) and a rating scale (shown across the top) and in each cell is some statement about performance indicators that would inform an evaluator how to choose a score for each outcome. I think that a common use of the OSP matrix has been to set up a structure where the goals are the "Criteria" and the "Scales" and statements are embedded in the evaluation forms in each cell of the matrix. The horizontal axis of the matrix is often thought of as a "time" or "stage" dimension.

When I think of this in terms of Goal Management, I see the linking of Goals with a Rating Scale as setting up a rubric structure. The current implementation provides a blank text area where the author of the assignment/datapoint/wizard page/matrix cell can write in plain text how they plan to score student work. This works ok if there is no programatic effort to organize how the rating scales are used, but the next step is obviously to support such an effort. I believe that there are a few such efforts that we are hearing that we would like OSP to support. Note, from here on out, when I say "rubric" I am using it to describe the descriptive text that would go in each cell of just one row of a "real" matrix (performance indicators as they relate to just one Goal).

Here are some of the rubric organization scenarios I have heard about in my conversations:

Managed assessments: Some rubrics are rather specific to (and must be tied to) a particular assessment item and must be approved by an "assessment coordinator" for educational QA purposes as part of a larger assessment system strategy. Changing the assessment/rubric in this case involves more than just the teacher. (Saginaw, Syracuse, RINET)

Generally reusable (but unchangeable) rubrics: Some rubrics may be general purpose rubrics that are NOT tied to an assessment, but the dissemination of these approved rubrics may be a strategy of an institution to push forward an agenda of best practice for assessment by providing a handy reference library of general purpose writing, mathematics and science rubrics (for example). While the choice whether or not to use one of these "off the shelf" rubrics (and which one) is left to the teacher, providing some information to the teacher about the schools expectations of its students at different stages (and perhaps suggesting an appropriate rubric for this grade level/stage of development) would make this service more valuable. (Indiana?)

Reusable rubric templates: Similar to the above, but the library of "off the shelf" rubrics are merely starting points. There is not a priority to ensure that everyone is doing assessment the exact same way. When a teacher uses one of these rubrics, they can easily edit the performance indicators to suit their needs and create a new rubric, just for their new assignment. (Rubristar approach)

Sharing of rubrics: This is a bottom up approach to establishing "best practice". As the teachers create their own rubrics against goals, they have the opportunity to publish them as part of the "reusable" library so other teachers can use/edit/republish them. (Someone?)

Do these make sense? Which efforts do you think we should place a priority on? Should they all be supported eventually?

OSP opinion for Academic Impressions Web Conference

Sean Keesler — Thu, 27 Sep 2007 19:30:07 -0400

I was asked if I could provide "just a paragraph about your view of both OSP/Sakai, the state of eportfolios in the country, and a sentence or two about where Syracuse with ePortfolios for the upcoming Academic Impressions: ePortfolios for Learning and Assessment :: Web Conference. Here is what I wrote...

The Open Source Portfolio community understands that portfolio systems are often implemented to fulfill a business need as a key component of a program or institution wide assessment system. Portfolio proponents have long lamented that this taints the portfolio system and inhibits its use as a platform for storytelling and making meaning out of a diverse set of ideas.

We also understand that accreditation boards and external agencies that advocate for standardized data collection and quantitative analysis of student performance are not likely to be receptive to the subjective feedback and guidance that should accompany student owned portfolios.

I think that we generally DISAGREE with the widely help belief that one system can not fulfill both the needs of the institution/program for standardized review/ evaluation AND the pedagogical needs for a student owned portfolio. Over the past two years we have come to recognize that by incorporating the institutional portfolio needs for standardization and data collection into the courseware tools in Sakai, we can free the portfolio tools for student reflection and presentation of their ideas to their peers, their teachers or professional contacts.

In the School of Education at Syracuse University we have been piloting the “Goal Aware” courseware tools in one of our programs. While the tools are functional, the use of the standardized data collection features has not yet been institutionally mandated and (not surprisingly) have only seen limited use. We require our students to publish a portfolio that addresses our “five proficiencies” by completing an OSP matrix column two or three times during their education here. My sense is that while some students find the experience valuable, the back loaded timing of the portfolio review each semester prevents the type of ongoing feedback loop that makes all of the difference. Many students find the additional work to create media rich portfolios technically challenging (given the compressed time frame that they have to complete the work) and also have a difficult time switching their mindset from “giving the right answer” to “providing evidence of learning” and “reflection” at the end of the semester.

I’d enjoy the opportunity to discuss our work with anyone interested with it. Feel free to share my contact info with your audience.

I wanted the recipient to know that I was "I am NOT faculty nor in a decision making role in our college. I manage the technical development, implementation and support of the system. My perspective comes from discussions with faculty and students in the course of doing my job. I tend to be critical of our local implementation while remaining VERY optimistic of what others could do with the Open Source Portfolio system."

OSP's freeform portfolio

Sean Keesler — Fri, 21 Sep 2007 10:19:33 -0400

A while back on the Sakai lists there was some talk about making Sakai a place that integrated with other, new Web 2.0 applications.

When I build and deploy Sakai, I often wonder to myself i Sakai is getting too big for its britches. Its great that anyone can build a tool for this platform, but that has negative consequences too. My sense is that the bulk of the resources are still devoted to development. Tools are released way too early (IMHO) without enough QA. Reviews like the one I just did for the OSP matrix tool are getting done after the tool is already out there being used.

When it comes to building a freeforn portfolio authoring tool, OSP hasn't put forth the resources to do it yet. I recently wrote the list about considering external authoring environments.

I was looking at Google Presentations tonight. It would make a pretty nice portfolio platform. For those of us looking for an easy to use freeform portfolio tool that allows users to author a simple presentation and share it with others, this is pretty attractive. It allows collaborative editing and chat by viewers of the portfolio (what a fun way to get feedback!). If you aren't interested in assessment tools in your portfolio authoring environment, it might be attractive.
If I remember what LaGuardia wanted to do, they wanted to be give out a "stub" portfolio starter that would get the portfolio creation process rolling. Google just let me import and then edit a PowerPoint file.
Wouldn't it be great if a student could "turn in" their URL for their Google "portfolio" (or a MySpace/FaceBook page, etc.) in OSP so that a portfolio evaluator could view and assess it?
I love the idea of separating the student owned authoring and portfolio storage space from the institutional system. If OSP could retrieve and save a copy of the student's portfolio, it would allow the institution to get a copy of the student's work while allowing the student to REALLY OWN their stuff.
With OSP, its the opposite. Students author their stuff in Sakai/OSP and then wonder, "How will I get my stuff out of there so I can use it after I leave here?" The IMS portfolio spec seems to assume that students will want to move their stuff from portfolio system to portfolio system. Export from one place -> Import somewhere else. I would prefer to author and publish external to all institutional systems and show them where MY stuff is.
I think OSP excels as a content management system with a workflow built in for assessment. Prompting students for certain types of information and presenting that information to a board of evaluators consistently is a process that needs to have an information system specifically for that. OSP forms, wizards and templates support that.
We haven't done very well at supporting freeform portfolios. I'm wondering if we should even be in that business.
Sean

Rethinking Sakai in terms of Pedagogical Services

Sean Keesler — Tue, 19 Jun 2007 15:29:55 -0400

I sent an email out to a few people in the OSP and developer community in Sakai with the following contents. I need to flush this idea out a little more. It sounds a bit preachy and a probably fails to convey the idea I have in my head.

The Sakai community has developed a Java based system that allows developers to deploy services and tools for use in collaboration, teaching and learning. The Sakai developer community has, with input from faculty, developed many tools for use in teaching practice. Many of these tools are modeled against existing tools present in other learning management systems. Common workflows found in other systems' tools are captured and embedded in tools like the Sakai assignment, discussion and test tools (for example). To faculty and students, the UI and workflow that are presented by these tools ARE Sakai. To them, the tools are the services that Sakai offers.
Sakai (and many other Learning Management Systems) present a challenge to institutions, teachers and learners. Communities of practice that desire to adopt Sakai for their own use are often forced to adapt their practice to Sakai tools or to develop/modify tools to meet their own needs. As the local community changes their practice over time they have to continuously modify tools to accommodate the changes. This is a costly process that may impede adoption and acceptance by teachers and learners.
The faculty and students at adopting institutions has to learn to skip from tool to tool to support their practice. For example, if an instructor wants to present some reading material to students, ask them to write an assay about the reading assignment and engage the students in group discussion about their learning, this sequence requires no fewer than three Sakai tools. If the teacher wanted to have the students peer review each others work and then reflect on the entire assignment, the workflow gets very frustrating to manage.
A movement in the Sakai community is growing around the frustration behind the user experience. This community consists largely of designers and Human Computer Interaction specialists that plan to spend a lot of time in the field doing heuristic evaluations of existing tools and ethnographic studies of how local communities of practice work. These experts will be able to make incremental improvements to the existing tool design and model new tools that will be able to support certain workflows that occur within those communities of practice. The success of these new or improved tools to meet everyone's needs depends on how well those local workflows reflect universally accepted practice. Surely, experimental changes will be require further local development to provide new features. In general, the "business logic" of a community of practice will be encapsulated in each tool design.
As an instructional technologist, who sits somewhere between the faculty and the developers, I see a gap between the UX group's effort and the developer's technical knowledge of system architecture. There is a need for a new type of community participant that engages the pedagogy group to gather requirements around services that make sense to teachers and learners, and enlists the expertise of the developer and UX communities in the creation of an abstract layer of pedagogical services and basic UI components (not tools) that provide a selection of discoverable services in Sakai that will allow communities of practice to model their own teaching and learning practice independent of specific tool design.
The Goal Management project that started at Syracuse University began as a single tool that allowed a user (likely a teacher or program administrator) to articulate learning outcomes. We made a modification to the standard assignment tool to allow the teacher to identify the learning outcomes (from the goal management tool) that each assignment addresses and then later assess students' work in terms of those outcomes. The original plan was to make a number of other tools (discussion, tests, OSP) "goal aware" as well. This is very difficult to do well for a variety of technical reasons. Technical problems aside, a pedagogical problem exists with this approach. Once the tool is made "goal aware", that functionality exists everywhere. We have added a configuration to turn it on or off, but it lies at the tool placement level. This means that the creator of the site needs to make a decision to enable a feature when they create the site or later go back and turn it on. What seems like a better approach would be to establish "planning" and "assessment" services that would always available to a user in Sakai, regardless of the tool being used.
We can envision a scenario where a workflow is authored by a teacher where they invoke a "planning" service to describe the next activity, complete with deadlines, outcomes and rubrics. The teacher then invokes the "assignment" service and deploys their activity. Students log into Sakai and automatically invoke a service where they see that their teacher has just asked them to write an essay. They write the essay and invoke the "turn in" service and upload their file. The teacher sees the student has turned in the assignment and invokes the "assessment" service that prompts them with their planning information, the student's work and a means to assess the student. They see the student needs help on this assignment and invoke the "private discussion" service and begin a week long dialogue with that student about how they could improve their understanding on the topic. As the private discussion progresses, the student makes some notes in their reflective journal about something important that they just realized. The student later turns in their assignment and the teacher reassesses the work.
The above scenario can be supported now with existing Sakai tools. However, there is no thread that ties the different activities that would occur across several tools together. Further, this scenario would currently require some significant overhead and planning on both parties. Jumping from tool to tool; discovering that new content was available; and realizing that the content in one tool related to previous activities in another tool all contribute to the problem. Moveover, if a new service was created for a locally created need, it too could be invoked and inserted into the workflow without changing the other service designs.
I believe that there may be a large percentage of faculty needs that could be met by thinking in terms of services that have meaning to our primary audience, rather than thinking in terms of tools. A great number of us understand the frustration that exists when we talk to faculty about how to use Sakai or how best to develop a tool. We have to keep in mind that these faculty are the tip of the iceberg, the early adopters that elect to engage a highly technical group because of the recognition of the potential. The majority of our audience lies outside of this group, waiting for us to start talking their language. Let us make it our mission to learn to do so.

Goal Aware Free Form OSP Portfolio Pages

Sean Keesler — Fri, 18 May 2007 17:48:30 -0400

It may be that the free form portfolio already in OSP will meet some of our needs. The areas surrounded by the green area could be supported by the portfolio tool. The areas outside of that need to be developed.

Another lesson I learned from the OSP 2.5 meeting in Ann Arbor is that there seems to be renewed excitement over the “Free form” style portfolio. LaGuardia Community College is piloting the use of OSP to recreate the portfolio experience that they currently are providing for their students with a commercial tool.

I have to admit, the free form tool always seemed to have the least amount of potential to me and I had dismissed it as “child’s play”. However, after seeing what LaGuardia is doing and thinking about it in the context of the “Class Journal” interaction style, I think that it may have a lot of potential! A few things off the top of my head that I know need to be done to support that interaction style would be to allow students to display structured data on a page of the freeform portfolio.

Functionally, we need to be able to display structured data elements like forms, wizards (and hopefully, assignment data in the future) on a portfolio page. This functionality is not supported yet in the freeform portfolio. If this were rWiki, I could envision a macro that would run an XSL transform against known data types and yield an HTML snippet that we could use in the page.

The ability for students to tag a wizard page with one or more “Goals” is also important to support this interaction style, as is the ability to receive informal assessments (ratings) and feedback from their teachers/peers/friends on each page of the portfolio.

Our interaction style model also states that this is an event that is initiated by the teacher. This is also an important gap that would need to be addressed. Inevitably, teachers would want to schedule multiple portfolio reviews during the course of a semester and have the feedback and ratings attached to those teacher-initiated events.

Application of rubrics to Goal Aware Activities

Sean Keesler — Tue, 15 May 2007 21:24:48 -0400

Ever since we conceptualized and built the Sakai Goal Management tool we have heard a need for a rubric built into the tool. However, it has not been very clear how we should manage and apply rubrics. Today, in the OSP 2.5 planning meeting, I heard of a possible way we could design the tools that made some sense to me. However, I think that the implication that I drew from the conversation makes we wonder if the idea is implementable.

Our current design for “Goal Aware” tools allows a user to create (and centrally manage) a tag (a Goal, in our parlance) for each learning outcome for a class. Another type of tag that schools might find a lot of value in would be a managed “Stage” type tag. As it was explained to me, for program assessment, there needs to be a managed system of “Outcomes”, “Stages” and rubrics for each combination of the two.

While we allow multiple “Goals” to be attached to any one activity (an assignment for example), we wouldn’t expect activities to apply to more than one “Stage”. As it was stated in this meeting, there would be the expectation that at any one stage, only one rubric should be used when evaluating student performance (ratings, in our case) for an outcome of the activity. This would help support the investigation into inter-rater reliability when more than one person is rating students.

It looks like this:

Application of rubrics to Goal Aware activities
		"Stage" type tags
		Stage 1	Stage 2	Stage 3
"Goal" Type Tags	Outcome 1	Stage 1 Rubric for Outcome 1	Stage 2 Rubric for Outcome 1	Stage 3 Rubric for Outcome 1
	Outcome 2	Stage 1 Rubric for Outcome 2	Stage 2 Rubric for Outcome 2	Stage 3 Rubric for Outcome 2
	Outcome 3	Stage 1 Rubric for Outcome 3	Stage 2 Rubric for Outcome 3	Stage 3 Rubric for Outcome 3
	Outcome 4	Stage 1 Rubric for Outcome 4	Stage 2 Rubric for Outcome 4	Stage 3 Rubric for Outcome 4
	Outcome 5	Stage 1 Rubric for Outcome 5	Stage 2 Rubric for Outcome 5	Stage 3 Rubric for Outcome 5

So far, so good. That makes sense to me, if the rubrics are simple rating scales that inform the "rating" process (ie: result in a single number), but my understanding is that they are not. The usual definition of a rubric is yet another matrix of "performance indicators" and a "rating scale" for each of those indicators. Typically the results of each of the ratings figure into the overall "grade" of an assignment, but in this case only result into a single rating one just one goal that an activity was supposed to address.

Since I am not a teacher, I can't say for sure, but this seems like a lot of work to do to assess a single student. My feeling is that teachers/faculty will largely oppose that level of granularity when doing assessment.

A "two system" approach to courseware and student owned portfolios

Sean Keesler — Sat, 12 May 2007 02:55:13 -0400

A couple of days ago, Barbara Shelly (one of the original "founders" and a former director of the LSB) came in for a visit. She and I spoke for a while about how she noticed that the adoption rate of education tools by teachers and students was rather low. At least, that's what I think we were talking about. :)

At one point she compared it to the explosive adoption of IM, cell phones, MySpace or Facebook. I think that she really pointed out an important difference between those technologies that put the user in the driver seat and allow them to network with whomever they want, whenever they want, for whatever purpose they want. That network effect and personal sense of control is something that isn't being replicated in the courseware space.

In courseware, it isn't easy to create a small collaboration space for you and your peers. Implementations usually manage student access to course spaces by the bang of the semester drum. There isn't much of a point to pouring a lot of your time into creating content that may be archived and put on a shelf or deleted altogether. What is it about these "closed audience" systems that keep users from having the same excitement about the tools as they do for their MySpace account? Steven Gilbert's blog post showed up for me after a quick Google for "myspace personal publishing blackboard". He notes that the social aspect makes a big difference.

Courseware allows teachers to "manage" a class and to provision services to the students. Adaptive release of content; provisioned users; enterprise integration; and integration with content repositories put a lot of power in the hands of an educator. In contrast, MySpace doesn't try to manage much of anything. You can make and break relationships when you want; put up or take down content when you want to express yourself; and what you put on MySpace won't be deleted by anyone but you. You have the control.

Portfolio tools try to do that too. Students should be able to put together a portfolio and show it to whomever they please.

I think that the dream here is that students are going to get really into putting together their portfolio, just like they do when they "dress up" their MySpace page and chat with their academic friends about all the things they learned in class. I can see that happening to some extent, but I can also see peer review "flame wars" happening through that communication too. I wonder how excited the institution would be to have that sort of content out there, with their logo on it, even if the discussions that happened there were the most productive ones that the students had. What forms of controls would make sense for that sort of environment without killing the sense of spontaneity and personal ownership that might make it appealing?

Better yet, can we build two systems that talk to each other? The courseware server where classes are managed and the official records are kept; and the personal "portfolio" platform which talks to the courseware system and to all of the student's (or teacher's) "friends".

A two system approach would make use of messaging of education related content between the different systems. When I login to my student system, I receive content from the classes I am subscribed to. New learning outcomes, activity descriptions (such as assignments, discussion topics, portfolio reviews, etc.) and assessment data are sent to me. I can turn in my assignments, post to discussions and share my portfolios through a content "port" on this channel and I can receive shared content (their discussion posts, and portfolios for example) from others on that channel as well.

The social aspect of the system occurs through alternative channels that users create and allow their friends to subscribe to. The content through these channels doesn't go through the courseware server.

Reflection – Resources

Sean Keesler — Fri, 11 May 2007 01:16:09 -0400

The Resource tool also needs to be able to support the task of grouping a number of pieces of content with their associated metadata and allowing the student to create a new piece of content containing all of the selected items and a new section that describes the relationship between those items and what was learned by the student as well as any new ideas. The student will be able to tag this new piece of content and reuse it just like any other piece of content as well as designate it as part of a portfolio presentation.

Content Selection - Resources

Sean Keesler — Fri, 11 May 2007 01:14:32 -0400

Just as WebCT has a “Content Manager” section, Sakai has a folder like view of content called “Resources”. The current resources tool allows students to browse their files and classify and move them about in a hierarchical file structure. Other tools often make use of the resources tool, but few activities start there. The cognitive activity of classifying and selecting resources that they would like to showcase or discuss in their portfolio is different in that it makes most sense to start from the unordered pile of resources to classify, group and explain the connections between them. These reflective acts are at the heart of portfolio development.

Switching the resource view from a strict hierarchy to a view that supports a “free tagging” metaphor would allow students to see how faculty have classified their content in relation to institutional and class outcomes. From this same interface, they should be able to create new “outcome” tags and apply them to their content. This would allow students to create multiple views of their content based on various emerging themes.

Portfolio Development - content collection through coursework

Sean Keesler — Fri, 11 May 2007 01:08:08 -0400

Content Collection – Assignments

The assignment tool is arguably the most used tool in courseware packages and faculty and students expect it to behave a certain way with certain rules. The student work handed in through a traditional assignment tool currently is “stuck” in that tool and subsequently is unavailable for students to use in their portfolios without a lot of extra steps that duplicate their data in their personal content area. The assignment tasks will need to be changed slightly to accommodate this new task:

Establish an event (an assignment) that describes the work and deliverables that students are expected to deliver by the assignment due date.
Describe (via a tagging mechanism) the expected learning outcomes from the event. Along with this description will come a description of the rubric that will be used to assess student work.
Review student work and assess the student using the established rubric.
Release student grades and ratings to the students.
Deposit the student work in each student’s personal content area as a piece of standardized content (with outcome and assessment metadata) that can be used in portfolios.

Content Collection – Discussions

The discussion tool is also a common tool used in courses to share ideas and content in a threaded forum. This content “lives” in the discussion tool and is managed with the class and, as such, is not made easily available to students for reuse in their portfolios. While discussions are usually difficult to assess, the subject matter is rich material for personal reflection and, as such, participation in class discussions should be included as a means for content collection for portfolio development. The tasks associated with a class discussion may also need some modification to ensure that the discussion content is made available to the portfolio author”

Establish a forum that describes the topic to be discussed and whether or not the material will be released to students as material for their portfolios.
Describe (via a tagging mechanism) the expected learning outcomes from the discussion.
Either the faculty or students will start discussion threads and add posts to the discussions threads and reply to each other.
At the completion of the discussion, deposit the discussion threads/posts/forums that have been designated as “releasable” in each student’s personal content area as a piece of standardized content that can be used in portfolios.

Portfolio Development - an overview

Sean Keesler — Fri, 11 May 2007 00:57:46 -0400

Anyone involved in a survey of portfolio packages will quickly learn that the term “portfolio” is a loaded term. Part of the problem for the OSP software development team is that the designers of OSP 2.0 decided to try to accommodate several portfolio development and assessment strategies. While they were very broad with their definition of portfolios and developed their own stack of software tools to try to meet those needs, they disregarded the tasks involved in moving student created content from the traditional courseware tools in their classes to their portfolios.

The opportunity to use tools that teachers and students are already extremely familiar with as an entry point for building their portfolios would build on their mental model about what “courseware” is (a set of tools that support existing pedagogical practice such as giving assignments and discussing topics). Working from and expanding this understanding will help faculty that have been reluctant to experiment with portfolios to understand and use the toolset. This is an important difference from the shift in thinking that faculty and students have been asked to make when exchanging their traditional teaching techniques in favor of using portfolio systems that have not been well integrated into existing systems.

We also want to add a sense of student ownership to learning management systems that will make these systems more valuable to students. When seen simply as “courseware” that supports classes that start and end each semester, it makes sense to remove access to student-authored content after each academic period. While the evaluation hasn’t been performed, we believe that students would find more value in a system in which they could build off of the coursework they have done, retain their own personal copies of relevant coursework materials and continue to draw connections and new meaning out of that content. Sharing this new knowledge with fellow learners and educators and getting their feedback is key to providing a student-centered learning experience.

Our use of portfolios in the School of Education is similar to many other groups around the world. We want students to engage in a process that allows them to:

Collect content that they have authored in their class assignments and discussions;
Select evidence of their learning and classify their own ideas as they relate to institutional and personal learning outcome themes;
Reflect on the evidence and experiences;
Share (publish) their work with others;
Receive feedback and assessment of their progress.

The Sakai and OSP projects: different communities, same platform

Sean Keesler — Wed, 9 May 2007 14:04:02 -0400

This is written as a bit of preparatory material for the Ann Arbor, Michigan OSP 2.5 Design and development meeting. I'm including a bit of history here because I think it reflects a case study on project management and product design that the Sakai and OSP communities are facing as they go forward. I welcome any and all corrections to my historical account of the story.

The rapid pace of development has been (thus far) lead by a few schools and institutions that have local interests in mind and an "itch to scratch". I think that the state of affairs is going to be changing soon, with an emerging "User Experience" working group focusing on heuristics and evaluations of new and existing tools to determine their overall usability. I expect that the OSP community and Sakai courseware tools developers will be able to form a new working group that can articulate a vision and design a system that provides a new experience for portfolio development.

Sakai is an open source effort to create a platform for collaboration and learning. An initial grant from the Mellon Foundation in 2003 funded Stanford, MIT, University of Michigan and Indiana University to combine their knowledge and expertise in software development to collaborate on this project. Over the past several years, the Sakai project has transitioned from grant funding to a foundation model, funded by the contributions its paying member organizations, which include primarily higher education institutions and a few vendors that sell support services around open source software. The egalitarian, meritocratic nature of this large open source community presents challenges to software design and development that are often not experienced in more structured environments.

As the Sakai community has evolved, the focus of the community and the definition of what Sakai is have shifted. As recently as 18 months ago, I think I recall that Sakai was referred to as a framework upon which tools could be built. By using the Sakai service APIs provided by the framework, tool developers could focus on the specific functionality of their tool and not have to worry about basic authentication, file storage and security issues typical of web based application development.

The original four institutions had built several tools (Assignment, Discussions, Resources, etc) on the framework to replace their own aging courseware systems on their respective campuses, mainly based on code from the University of Michigan. These tools had a common “look and feel”, used the same template presentation technology (Velocity) and were considered “core” tools for a Sakai distribution. Furthermore, the developers of these tools were working towards the goal of replacing essentially commodity services for which they had several good examples and that the user base had a lot of familiarity with.

During this period, Indiana University and rSmart (a commercial affiliate of the Sakai Foundation) were funded by the Carnegie Foundation for the Advancement of Teaching partnered to port the already successful “Open Source Portfolio” software developed by the University of Minnesota to Sakai. They decided to use the Java Spring framework and Java Server Pages as their presentation layer for the suite of tools that would collectively be known as OSP 2.0. It is important to note that portfolio systems have not worked their way into the culture of most institutions and the implementations of these systems vary widely from institution to institution. As such, there are no strong mental models to work from since the user base expectation of what a portfolio should be has not been firmly established.

The design team charged with redesigning OSP for the Sakai based system decided to make several changes and extensions to the Minnesota system. Portfolios were to be assembled from XML content snippets authored by students and presented by an XSL processing engine to achieve a web-based portfolio. The system is extremely configurable; so much so that “out of the box” the software does almost nothing and requires a great deal of XML design and development in order to customize it for the specific implementation requirements. I believe that the system needs to be drastically simplified and the workflow streamlined for particular uses. One primary use of portfolio software (and one we are very interested in promoting as a School of Education) is to support students to reflect on their learning and to solicit feedback on their progress toward their own learning objectives.

I think that these two development efforts were happening in isolation from each other, the team developing the “courseware” side of Sakai (Assignments and Discussions) had very little connection with the OSP designers and developers. When Syracuse University began using the Open Source Portfolio software in 2005 we realized that there should be a strong connection between the courseware tools (in which students create the bulk of their content) and the portfolio tools if they were to support an emerging assessment strategy in the college that was focused on assessment of student portfolios. Presentation of the concepts at Sakai conferences and in discussion lists proved that our simple idea made a lot of sense to educators. While no formal “evaluations” were done, we definitely heard the experts in the field respond positively to the general idea. The “infinitely configurable” design of the OSP software was widely seen as an impediment to the adoption and use of the software and the community is currently looking for suggestions for improving the design of the software to support a few basic workflows and teaching strategies or “interaction styles” between teachers and learners.

In the following few blog posts I will explain the organizational tasks that I think need to be accomplished by teachers and students using this software during the various stages of one style of portfolio development, the formative journal.

Interaction Style 3: Class Journal

Sean Keesler — Fri, 27 Apr 2007 13:56:21 -0400

A Class Journal is a formative, reflective, emergent portfolio.

This data flow diagram illustrates the how teachers and students would interact to create, discuss and assess a private (student to teacher) class journal. Periodic review of student blogs (chronologic, personal, reflective portfolios) is likely to be a common interaction style that many teachers would find useful as an venue for formative feedback.

Blogs frequently allow readers (in this case the course instructor) to leave text comments (which this would also support). However, this interaction style would leverage the ability of students and instructors to identify learning outcomes for the journalling process and allow assessment of each blog entry on an ad hoc basis for formative assessment purposes.

Student blog posts would appear as messages in the instructor's inbox. Instructor's would periodically read the blog posts and send back either plain comments or comments with assessments (as related to tagged goals). An important feature of this interaction style is that it supports the idea that "knowledge is emergent" by allowing students to add new learning outcomes to their Goals and to tag their blog posts with those tags. Assessment of student learning, then, can be done in response to student Goals and Faculty Goals (related to the original activity description).

Again, once the course ends or the student leaves the institution, they have the option to keep a copy of all of these posts, assessments and comments for their own use, much like they do with email.

An important assumption is that a system that can engage in this sort of messaging becomes standardized, allowing students to remix and reuse their content from other activities in multiple classes at multiple institutions.

Rhode Island implementation in Education Week

Sean Keesler — Fri, 27 Apr 2007 12:48:45 -0400

In an Education Week article this week entitled "Accountability, or Mastery?" the author mentions the pioneering assessment work being done by Rhode Island. Rhode Island uses the rSmart CLE with a modified Goal Aware Assignment tool to support this multi-dimensional assessment of students.

Interaction Style 2: Discussion

Sean Keesler — Tue, 24 Apr 2007 13:26:09 -0400

Discussion data flow in a distributed system looks a lot like email. It should!

This data flow diagram illustrates the how students and teachers could interact in a distributed LMS to engage in a group discussion.

A discussion tool is another typical tool found in courseware. The discussion tool is used to allow a teacher to set of "forums" or "threads" of conversation related to the course material. Like the assignment tool, it also lives and dies with the course worksite.

An alternative approach would be to distribute the messages between the courseware system and a learner centered platform, much like email systems do. The difference between the a distributed leaning management system discussion and traditional email systems would be that this content would be stored in a standardized format that allowed users to easily re-use the content in their other courses or portfolios.

Again, once the course ends or the student leaves the institution, they have the option to keep a copy of all of these discussions for there own use, much like they do with email.

Again, there is a lot of duplicated data in distributed system like this which may end up causing issues for storage. However, this does allow each learner to retain a record of all of the discussions that occurred in their classes...forever if they wish to. Similarly, this frees the institution from maintaining the course indefinitely "just in case" students need access to their data.

An important assumption is that a system that can engage in this sort of messaging becomes standardized, allowing students to remix and reuse their content in multiple classes at multiple institutions.

Interaction Style 1: Graded/Rated Assignments

Sean Keesler — Wed, 18 Apr 2007 20:40:33 -0400

Assignment data flow IF a system was distributed, allowing students to truly own their data

This data flow diagram illustrates the how students and teachers would interact in a distributed LMS to carry out the traditional graded assignment workflow.

A frequent use of courseware is to assign work to students, collect that work from students, assess the work and release the grades to students. In most courseware systems, this is all done in one tool. The Assignment tool (or the assignment and gradebook tools) "live" in a course. The design of courseware is mainly to facilitate the instructor's ability to manage the course.

An alternative approach would be to distribute the various transactions between two systems, the courseware system and a learner centered platform. Each set of tools would be designed and configured with the primary user's interest in mind.

The student would receive a message in their "inbox" with the assignment instructions, learning outcomes and rubric for the assignment.

The student would then do the work, saving their "drafts" in their own system and eventually sending back a response to the assignment with their work attached. Note that a copy is sent to the instructor, with the original content still on the student system.

The teacher would receive the student work in her "inbox" and (assuming that there is to be no formative feedback) evaluate the work against the rubric and releasethe grades back to all of the students.

There is a lot of duplicated data in a system like this which may end up causing issues for storage. However, this does allow each learner to retain a record of all of the assignments, work and assessment data that occurred in their classes...forever if they wish to. Similarly, this frees the institution from maintaining the course indefinitely "just in case" students need access to their data.

A walkthrough through portfolios in SyrCLE

Sean Keesler — Wed, 28 Mar 2007 16:52:18 -0400

I made a screencast of the OSP setup we have here in the School of Education. I wasn't able to interview a student so that I could show off their portfolio, but it really does look pretty good.

Instead, you get to see some test data... Ipsum lorem...

The file is pretty big (56MB) and it doesn't "stream" well. I used IShowU to make it and for $20, its a pretty good little app! I just wish it would create a "fast start" movie. Maybe I missed something in the setup...

Regardless, I tried to take the viewer through the following:

Drivers behind our pilot
Our plan for organizing students into cohorts
The types of sites we use in our instance of Sakai
Some documentation and a conceptual model I wrote about Portfolio Templates (on confluence).
Some preliminary web designs for what we wanted the portfolios to look like.
A description of the different data structures created to support our plan and their availability through the Community Library. (See the attachments to that page of confluence).
A demonstration of what evaluators get to see when they look at student portfolios published using our "Portfolio Review" Template .
A description of our alternative method for assessing student portfolios using the Data Point tool.

Workflow for coursework and portfolios

Sean Keesler — Tue, 27 Mar 2007 21:47:27 -0400

Data Flow Diagram of SyrCLE:

Courseware and portfolio systems are usually designed as relational databases to allow a minimum of data duplication while giving users different roles and permissions to interact with their content and data. Students and faculty have expectations about how different tools should be used to organize data into familiar structures with implicit rules for interacting with the data in those tools. The design of these tools have not not allowed a lot of interplay between the tools. It is difficult to publish a portfolio that reuses classroom content (ie: assignments and discussion threads) that the student otherwise has access to view and manipulate in the limited manner the tool allows.

Furthermore, if a student enrolls in another institution or another class that doesn't use the same instance of the courseware, the data from the different classes can not be combined to create new content.

What is needed is a student centered content management system that serves as the student's virtual backpack and their main platform for lifelong learning. The interface between their content management system and the classroom management tools deployed in a LMS has not yet been designed.

This data flow diagram is intended to describe the flow of information into, out of and within a distributed, networked LMS that has two main components: course content and student content management tools.

A couple days ago I shared some "reconceptualizing" Sakai and OSP based on some feedback from a couple of our faculty and the problems that Dr. Joseph Shedd (our PI) has shared in our meetings between Weber/UMich/Portland/Virginia Tech.

From these meetings I think I am hearing the following (my own opinions are sprinkled all over here):

Dr. Shedd wants to be able to run a pilot portfolio program in Syracuse high schools that students can take with them to college. In my opinion, this shouldn't merely be an export of some files or presentations....it should be richer than that...way richer...it should bring along everything the student did in their coursework. All the content, semantics and assessments that they ever got. Anything less is, well, lame.
The attendees at the last Goal Aware BOF were talking about the next "Goal Aware" tool. It sounded like it should be something really general and blog-like to me...something that should allow the student to select some of that old content, tag it with their own goal (and other) metadata and reflect on it, creating a running journal of content, reflection, meta-reflection, etc...this is some folks idea of a great learning tool.
Our current crop of Sakai tools have been independently designed and developed as silos. There is no communication between assignments, discussions, portfolios. Jim Pease has been sort of stymied and frustrated even thinking about how to embed the Goal management and rating interface into these tools. The result is functional but far from optimal from a usability perspective. The metaphor of "bolting on" helper tools to add functionality results in needless extra clicks to accomplish tasks. If I were king I would want a suite of simple tools that just work to support a few interaction styles. Less is more. In Sakai it seems like more is more.
The current usability issues with OSP and Sakai seem to me to be due to one of a couple of things:

Either the design tries to accomodate too many things
or the design was done around a workflow that doesn't match what we really want.

I started to create a diagram (that is NOT done yet, but I would love feedback) to get my head around what I think would be a better workflow for a system designed to create portfolios.

Rating Scales for Goal Aware tools

Sean Keesler — Sat, 24 Feb 2007 10:35:35 -0500

Our faculty have settled into a 1-4 rating scale for EVERYTHING that we do in the School of Education, regardless of the activity or the standard being measured in our assessment system of Goal Aware tools and OSP.

In our system, a rating of a 3 or 4 is considered "meeting expectations" or "exceeds expectations" and are the norm for most student evaluations.
In a wonderful world, everyone would be getting 3's and 4's and our system would be pretty boring. However, this sort of rating system makes it particularly well suited for finding cases where a student is running into problems. A few 1's or 2's on a student would throw up a flag to faculty that there is an issue with that student that needs to be addressed. Its an attention grabber.
Similarly, if a program coordinator sees a class in the program that is getting 1's and 2's, it suggests that they spend some time looking at the design of the course or the program.

While this is great for our use case, we know that this is not what everyone has in mind for Goal Aware tools. There is likely a set of best practices that needs to fall out discussion within the teaching and assessment communities in Sakai and OSP. I posted this to the portfolio and pedagogy list a couple of weeks ago but didn't get any response. Is this idea that far out of whack with what teachers think about? Or is the community seriously lacking in this area (probably not good for a community building teaching tools!)?

I'll restate my post here:

Each goal that is linked to an activity (assignment) can be rated on a scale. How you define that scale (sort of/kind of related to scopes of rubrics) is really important.

Rating Scales could be configurable:

1. At the Goal Set

The rating scale would apply to all of goals in that set
all activities that are linked to goals in this goal set would use the same rating scale
these activities may span worksites (courses) so the rating scale would apply to all of them.
this provides a very consistent experience and is pretty well suited for the use case at SU.
this makes it easy to spot “problems” with reports as the rating experience is normalized across all classes linking to a Goal Set.

2. At the Goal

Each Goal would have its own rating scale
activities that are linked a specific goal would use the same rating scale
these activities may span worksites (courses) so the rating scale would apply to all of them.
this provides a consistent rating experience across a program for a particular learning outcome, but allows for differences between the way that ratings are done for different outcomes.
perhaps a little more difficult to spot "problems" since there is more variability in the rating "signal".

3. At the worksite

The rating scale would apply for all ratings done in a worksite
activities in the worksite use the same scale regardless of the goal, goal set or activity being rated.
not a lot of "pros" for this....but, this is how it is in the 0.7 release that we use at SU! The rating scale picked in the Assignment tool config is used in all of the data points and assignments in the site.
allows for flexibility of rating methodology between classes
may result in a very inconsistent rating experience across sites and a lot of variability in signal across sites.
if you use a consistent rating scale across sites anyway (like we do) this will (and does) support your use case.

4. At the tool placement

The rating scale would apply for all activities done in that tool
all assignments (or data points) in a worksite would use the same rating scale
allows a different rating scale for each tool (activity type)
activities may be linked to different goals, from different goal sets, from different work sites
allows for flexibility of rating methodology between classes
may result in a very inconsistent rating experience across sites and generate a really noisy signal for identifying problems.

5. At the link

Each time an instructor links an activity to a goal, they can specify the rating scale that will be used to evaluate performance.
the ultimate in flexibility!
the noisiest signal...would this be useful at all for program evaluation?
very good for individual teacher control of their classroom.

So..I think that this is a spectrum of sorts.

As I mention in my intro to Goal Aware tools:

Course Management Systems often enable central administration of classes from a technical perspective. However, faculty and students tend to remain isolated from the program in which the class takes place. IT systems seem to perpetuate the traditional “closed door” classroom.

#5 is pretty close to the "Closed Door" class. Program Assessment is difficult.
#1 is good for broad assessment and standardization.

Goal Aware Profile

Sean Keesler — Tue, 6 Feb 2007 23:14:58 -0500

Multidimentional learning outcome profile

How do students come to understand how well they are doing in a program? Certainly, they can look at their grades in their courses and see their GPA. Within a specific class, they know what their grade for each assignment.

But students are probably used to playing games where they see different aspects of themselves. As they play the game, they accrue more "points" in different areas throughout the game. It defines them as game players.

I think that courseware (specifically courseware with next generation assessment ideas like the Goal Aware idea built in) should allow students to see "profiles" of themselves in this manner.

LSB History - Part 6

Sean Keesler — Thu, 18 Jan 2007 17:39:19 -0500

The following is a segment of a larger story that I wrote to attempt to summarize the history of the Living SchoolBook, a research group in the School of Education at Syracuse University. In this entry I begin to talk a bit about our involvement in the Sakai network and our contribution to that group. If you haven't read any of the previous articles about the history of the LSB, you may want to look at the previous entries related to this topic.

Our First Sakai Tool

Since we had no experience developing in java or on Sakai, a request was made for funding for consulting services from rSmart in order to ensure that we could design and build a prototype “Goal Management” tool by the next conference. The LSB and the office of the CIO shared the cost for of a contract for 40 expert hours of rSmart developer time. rSmart developers met with the LSB team to review the requirements for the tool and made several helpful suggestions. The rSmart developer believed that the tool would take us one man-month worth of effort to build the tool. However, LSB developer’s lack of experience with Sakai was acknowledged as a significant challenge. The remainder of the consultation contract was to be held in reserve to be used on an as needed basis to answer LSB developer questions.

Two LSB developers (Jim Pease and Huan Yang) worked on the tool development for approximately 3 months with occasional questions to the Sakai community development lists and rSmart developers. rSmart's Chris Coppola and John Ellis met with us to review the requirements document for the project on a couple of occasions. By the end of that development period and we were able to more fully describe the impending business need for this tool as well as demonstrate a prototype tool at the Austin Sakai conference.

It should be noted that we felt that having a relationship with a Sakai commercial affiliate like rSmart would help us the risk of NOT being able to meet our own goals. In retrospect, the community lists and existing documentation were very useful to our developers and the contracted time with rSmart was barely scratched. We only used about 3 hours of the 40 we had budgeted for.

Other contributions - both ways

Engaging the community in discussions about what a portfolio should be and how to use the software to get what yu want out of it is a valuable part of the open source model for software development. I believe that the information I took from the development, testing and documentation efforts for Sakai/OSP 2.0 and 2.1 and helped to shape and change our own expectations of our own assessment system. Our efforts may have helped to nudge the community in a sllghtly different direction when thinking about portfolios as well. It should be noted that as of the writing, OSP 2.4 requirements state that matrix cells and wizard pages will be "goal aware". The University of Michigan, Indiana University and rSmart appear interested in this new approach to portfolios.

LSB History - Part 5

Sean Keesler — Thu, 18 Jan 2007 11:20:08 -0500

The following is a segment of a larger story that I wrote to attempt to summarize the history of the Living SchoolBook, a research group in the School of Education at Syracuse University. In this entry I begin to talk a bit about our involvement in the Sakai network and our contribution to that group. If you haven't read any of the previous articles about the history of the LSB, you may want to look at the previous entries related to this topic.

New CIO

In the winter of 2004 Syracuse University appointed a new CIO. He had just come from the University of Rhode Island where he was involved with the Open Source Portfolio Initiative, a multiple university effort to build open source portfolio software on Sakai (an open source framework). A brief introduction made it clear that the CIO’s interest in open source technology and portfolio software was very much in alignment with the LSB’s goals and beliefs. A proposal was written to the LSB PI’s to further investigate the OSP software as a new approach to solving our development issues. Whiile the LSB’s initial evaluation of the OSP software led us to believe that it would be an adequate platform to develop our own system, a great deal of the decision to abandon our own development in favor of the new system was fueled by a desire for the School of Education’s work in education systems relevant in a larger arena as well as the potential to work in cooperation with the central IT group.

An unanticipated benefit from the decision came when other IT groups on campus were drawn into the project. The LSB was no longer responsible for maintenance and setup of the servers, network and storage systems that were going to be important for a successful implementation. Eventually two new servers were purchased by ITS and connected to the university SAN and placed on the monitoring system to ensure system uptime and reliability. This allowed the LSB to focus efforts on GAP analysis and beta tests of the new software.Not only were hardware and personnel made available for the project. The CIO made a financial commitment to the Sakai Educational Partners Program.

Learning Sakai-OSP & Gap Anaylsis

In order to quickly get up to speed as to the capabilities of the yet unfamiliar software, the LSB began participating in the scripted testing effort for OSP 2.0 in the spring of 2005 during which several bugs were reported to OSP developers. A beta test was performed with a small Maymester class in 2005, which evaluated the OSP and Sakai courseware tools. The open communication between experienced OSP developers and users at organizations such as rSmart, Indiana, DePaul and Portland State Universities. The value of the community discussion could not be overstated as in a crucial step in shaping our next steps. It became clear that the culture and beliefs already developed at the LSB would be a good fit for this larger open source community.

During testing and experimentation of the toolsets, the LSB began to recognize that many of the tools in the OSP software mapped well to ideas expressed in our own previous requirements discussions. Most of the courseware tools that were present in the Sakai toolset looked much like those in commercial courseware packages. It was clear that they were not developed with the same constructivist pedagogy that the LSB developed Dialogue for. However, we also noticed that our own faculty were using less and less of the group conversation features of Dialogue and instead used the system as an assignment service. Instead, the faculty were asking students to do more reflection in PowerPoint based portfolios that were beginning to be used as summative assessments at the end of each semester.

A key component that was missing from the Sakai/OSP solution was a toolset for faculty to use that would allow them to identify which School of Education Proficiencies were addressed by each assignment and subsequently rate student performance for each identified proficiency. This tool set would allow be useful as a means for evaluation of student learning in relation to goals, accomplishing curriculum mapping activities, evaluating program effectiveness. In addition, the tools would provide a means for embedding the new program goals into the daily business of faculty. The new “goal aware” system may even be an ideal platform for performing research on the effectiveness of portfolio building as a teaching and learning strategy.

The conceptual separation of the “Institutional Portfolio” and the “Student Reflective Portfolio” could be accomplished by performing activies that were to be examined for program evaluation purposes using the “goal aware” courseware tools derived from the Sakai toolset. As students prepared their personal portfolio in order to illustrate their ability to explain their practice, they would be able to self select any evidence that they saw as relevant to the proficiencies using the OSP toolset. A list of completed coursework that had been identified by faculty as designed to teach the proficiencies would serve as an input to the self selection process.

The “goal aware” concept was presented to the Sakai and OSP communities at the Baltimore and Austin conferences where it received some attention from rSmart president Chris Coppola who encouraged the LSB to develop it further.

Teaching, Assessment and Responsive Pedagogy

Sean Keesler — Sat, 9 Dec 2006 09:51:46 -0500

It was late at night at the Atlanta Sakai conference and I was thinking a little more about making pithy statements about the practice of teaching. At the tech demo table for Goal Aware tools I was asked whether I thought that faculty would be resistant to using Goal Aware tools to markup their assignments with the learning outcomes that they are supposed to be addressing and then thoughtfully rate students' performance towards learning outcomes. I am not qualified to answer (I am not a teacher) but that won't stop me from thinking a little about it...I'll even think out loud. It helps ground me in the rationale for the Goal Aware tools idea.

The movement toward standardization of learning outcomes in general is met with resistance. Whenever an outside agency steps in to regulate or inspect or make change in an existing system, those vested in that system are going to resist. The accrediting body for the School of Education doesn't force the schools that it inspects to adopt their standards, however. NCATE requires that the school articulate their own expectations of students. While not at all a trivial task, this makes the pill easier to swallow (doesn't it?). Even though the consensus building effort is extremely laborious (or so I have heard), I know that during the process faculty know that this is probably good medicine.When that consensus building process finishes its first iteration, I think you can hear the sigh of relief in surrounding hills and towns. There is real sense of accomplishment.

They don't say, "Hooray! Now we can get to work measuring student progress toward our outcomes; finding ways to improve our classes; adjusting our programs and our curriculum. After that maybe we can set new goals and outcomes for round two!"

Nope, you won't get a sigh of relief about that. This is endless work. Iterative and continuous work that conscientious teachers and administrators have to struggle with. The time and energy to generate good assessment data that can be relied upon to draw conclusions about the program and the classes will take away from the time spent doing other things such as research, planning new lessons and talking to students. To the less insightful it may seem counterproductive. Plenty of arguements can (and have) been made against it.

As a student in classes, I have experienced plenty of instances in my programs (not in the SOE) where I suspected that the faculty weren't on the same page. I realized that there wasn't a flow in the program. Assumptions were made about what I would know before I came into the class. I recall taking exams where scores were abysmally low, but corrected by grading on a curve. I've always thought there was something wrong with that as a student.

Imagine a class where students were supposed to be taught to swim. The faculty in the program expect students to emerge from the class being able to swim a variety of strokes and to be able to tread water. In this imaginary class the instructor sees that students are not learning to swim, for whatever reason. Maybe his methods aren't very good. Maybe his assumptions about what students already were capable of were off. Maybe students are distracted by too much chlorine in the water or they have poor eating habits. A reasonable action to take would be to adjust and respond to this signal. However, this teacher decides to lower the water level in the pool until the normal distribution of student height in the class yields the appearance of a class of students treading water and swimming across the pool. Progress as expected. Pity these students and the teacher of the next course in the sequence of the program: Lifesaving 101.

That's a silly example, isn't it? We know that good teachers need to be responsive to the results of the assessments they give. In the most fortuitous cases, this means that the teacher will be able to make up the gap between their own assumptions about students and learning in order to meet the goals of their class. In the less fortuitous case, goals must be adjusted. Communication has to happen outside the classroom. Programs may need tweaking. That is more work. It is essential work. In many cases it may be unappreciated work.

Blackboard BOF

Sean Keesler — Wed, 6 Dec 2006 18:37:36 -0500

When we presented our original Goal Aware ideas at the Baltimore Sakai Conference, I spoke with Deborah Everhart in the halls after our presentation. She told me that Blackboard was working on requirements for a similar system. I had followed up with Deb on the phone after the conference. She had someone in their product development group call me at Syracuse. I described our ideas to her and we really weren't sure how to proceed. At that time, I felt that the intellectual property issues around the idea was an impediment to collaboration.

While our Goal Aware installation of Sakai in the School of Education is our center for innovation in pedagogy and technology to support that, our enterprise courseware installation is Blackboard. It would be great if "Goal Awareness" could be baked into that installation. It would be even better if Blackboard could facilitate that.

Today, Chris Coppola asked the Bb reps what they have done to date to contribute to open source and Sakai. They weren't able to respond with much. At this BOF I re-introduced the Goal Aware idea to the Blackboard representatives and suggested that contributing some development or design work to build in interoperable components in Bb that could work with our development effort in Sakai. We didn't get technical. I just threw it out as a potential way for Bb to demonstrate that they are contributing to innovation and open source efforts.