The Living Schoolbook - Sakai Porch

Educational Portfolio #1 – The outcomes based accreditation or program review portfolio

Sean Keesler — Wed, 21 Nov 2007 16:29:25 -0500

Disclaimer: For those of you who do not know me, I am not a teacher. I provide technical support for grant funded educational projects in the School of Education at Syracuse University. Take this perspective with that grain of salt.

Over the past three years of working with the Open Source Portfolio and Sakai communities, I have heard lots of people talk about how they plan to implement the portfolio tools. Some of the uses highlight the ability of the tools to create a standard presentation (like a resume or curriculum-vitae) by using a template. In a later entry, I’ll be walking you through the steps I would follow in order to create a set of forms, wizards and a template that will allow users to pop out a presentation like that. This entry isn’t about those types of uses. I am sure that everyone understands the purpose of a portfolio like that. I want to take a moment to talk about the types of portfolios that can be useful in education settings. This entry doesn't really talk about how to imlpement an accreditation portfolio. I want to set the stage first.

The outcomes based accreditation or program review portfolio

Some schools need a means to review student learning/progress/achievement of ALL of the students in their programs in a very comprehensive way. The prerequisite to a system like this is that the faculty has decided upon an assessment framework that will be applied to all of their students and all of their classes. A “portfolio” effort like this has several components.

Establishment of a framework for evaluation

In order to set this up, a set of outcomes must be agreed to by the faculty. This is a process that many schools apparently do not engage in very often. I haven’t heard of a faculty that spontaneously decides to engage in this process. They usually have more interesting things to do than debate the purpose of the programs in which they teach. Usually it is a mandate from an accrediting body that initiates the process. It was for our faculty.

Identification of key program assessments

A plan has to be designed for measuring the performance/achievement of the students in the program (and in so doing, the effectiveness of the program). A natural tool to use for this purpose is the assessments already in place in the program’s courses. Chances are, a subset or cross section of the work the students already do in their classes would provide the information needed to see how the students and the program are doing. By highlighting the assessments already in place in the program’s curriculum and matching them to the outcomes that they address we can aggregate student work aggregated into FACULTY DESIGNED portfolios that serve the purpose of the program.

Assessment of student work in light of the program outcomes

Once the a faculty member has identified an assessment (an assignment usually) in their course that is intended to assess (at least in part) one or more of the program outcomes, the student’s work will need to be assessed against each of those outcomes. Whether the instructor of the course performs that assessment while grading papers or a "program review panel" performs the assessment later on, the process requires standardized rating scales and rubrics to ensure that all of the faculty are on the same page when assessing the student work. Establishing standard rubrics that ensure inter-rater reliability is another huge task that takes place outside of the “system”. The axiom “garbage in – garbage out” applies here.

Analysis of the Data

It may surprise you to see the multiplicative effects that adding new assessments, outcomes and students to this structure has on the amount of data that there is to look at and analyze. A cohort of 100 students going through a program with 30 highlighted assessments each identified as addressing on average of 2 outcome each provides 6000 pieces of data for analysis. If assessment is continuously performed by instructors (rather than by a “panel” at the end of the semester) it might be helpful if both students and the faculty could have real time access to this assessment portfolio data to help identify problems and trends in student performance and hopefully correct the problem with relevant formative feedback, counseling, etc. If the data analysis is done at the end of the semester, the possibility of formative feedback may be forfeit and the focus on the data analysis would then be to get an overview of the performance of various aspects of the program in order to improve it.

Implementing Change

Once the data analysis is completed, some issues may be identified. The faculty may decide to change its rubrics to tune the assessment process, rephrase the program outcomes, restructure the program curriculum and/or pick different assessments for review. Accreditation bodies are particularly interested in this process for overall review and data driven decision-making.

Without going into details about the implimentation of a solution in Sakai and OSP, it should be clear enough that a mandate to engage in this level of program review requires changes in the organization of a school. In our school, a new full time position was created to manage the assessment process. The assessment coordinator is charged with setting up the assessment framework, collecting the data and running reports.

Goal Management

When the LSB started using OSP to address our program review portfolio needs, we quickly identified a gap between the tool set's capabilities and design philosophy. Most everything in OSP is geared towards making the process initiated by students. In the OSP matrix tool, the STUDENT gets to pick the items that meet the criteria of the matrix. While we were interested in any additional material that students felt was relevent to present as part of their own evidence of mastery of the program outcomes, we knew that asking them to resubmit all of the work that the faculty had identified was a process that was unreliable and unnecessary.

The Goal Management tool released as a “contrib” tool for the 2.4 release was an effort to provide a means for a user to articulate multiple sets of outcomes (goals) and to share those goals with a number of class worksites in Sakai. Instructors who create assignments in their classes can identify the outcomes that pertain to each assignment and rate student work against each. The outcomes and rating data are stored in the Sakai database and can be aggregated and reported.

While we do not have funding right now to continue developing the idea, the basic premise has rung true with many other institutions and is likely to be further developed by Indiana University (hopefully with helpful input by the rest of the community) to be an integrated component of versions of OSP in future releases of the software.

In the interest of full disclosure, our own implementation of Goal Management in Sakai is NOT the production system that our assessment coordinator uses to collect data. Our implementation is a research project that lacks some critical features that would make it an acceptible substitute for the current production system. Discussions are in progress about the cost-benefit of building the necessary features and maintaining a Sakai instance as compared to continuing to use the current system.

Open Source Portfolio Tools - a portfolio development kit

Sean Keesler — Fri, 16 Nov 2007 01:09:28 -0500

The Open Source Portfolio (http://www.osportfolio.org) is often difficult to explain to folks. There are some aspects to the design of the software that seem to be in contradiction with one another. There is a tension between the belief that portfolios are "student owned" and the technical reality that some of the tools are far too complex for students to use. As a result, implementers of OSP use the flexibility of the tools to attempt to craft an environment that meets the needs of all of the users. The simple fact that the experience is designed and implemented by users who may be two degrees from the users make this a challenging piece of software to work with.

I plan to write a bit more about this "suite" of tools in the coming weeks to help identify some ways that I think implementors can get productive fast, but read on for a brief intro to the tools of the Open Source Portfolio.

The Forms tool

The form tool allows a user (probably someone who is wise in the ways of XML) to define data input screens (forms) that will be used to prompt users and structure data entry into the system that will later be used to construct a portfolio. The user gives the form a name, types in some instructions and then identifies an XML Schema Definition (XSD) file that actually describes what the fields are in the form, the validation rules, and some special display information. Optionally, the form designer may further want to craft an alternative view for the form and will identify XSL (Extensible Stylesheet Language) files that will transform a default rendering of a form to one of design.

Wizard pages

While there is no “wizard page” tool, this structure is used as a building block in both the Wizards and the Matrix tools. A wizard page has a unique name and will present guidance (instructions, rationale and examples) to the portfolio owner. When constructing a wizard page, the designer will be able to add the following “pieces” to the page:

Forms (from the above mentioned tool) for the portfolio owner to fill out.
A reflection form for the owner to use.
An evaluation form for an evaluator to use when assessing the page.
A feedback form for a portfolio reviewer to use to provide informal comments.

When describing a wizard page, all of these structured data input forms are completely optional. At a very minimum portfolio owners will always be able to upload and attach files to the page.

The Wizards tool

The Wizards tool allows a user to create either a series of screens that further "scaffold" the user through the process of entering the data that will used to create their portfolio. There are two types of wizards in OSP:

Sequential Wizards can be constructed as a set of pages that are presented one after the other.
Hierarchical wizards present a nested tree of categories and pages to users.

In either case a the pages of the wizard can be constructed to include guidance (instructions, rationale and examples) to the portfolio user. The wizard author can choose forms (from the above mentioned tool) that will prompt the portfolio author to provide pieces of information relevant to the portfolio. The wizard author also has the opportunity to prompt the user to add reflections along the way, receive feedback from reviewers and evaluators.

The Matrix tool

The Matrix tool allows the designer to construct grids of one-page wizards wizard pages (the same wizard pages as described above). The rows and columns of the matrix are described as “criteria” and “levels” in the tool. A matrix “cell” can be set to be “unlocked” (ready for the portfolio owner to use) or “locked” (unavailable to the portfolio owner). Matrix cells can be set to unlock (top to bottom, left to right, or as the instructor deems appropriate) as the owner submits the cells for evaluation.

The Style tool

The Style tool allows the designer to designate an uploaded css file as a “style”.

The Layouts tool

The Layouts tool allows a designer to designate an xhtml file as a template for a free-form portfolio page. Page layouts can have many different designs (Title over text, two columns, Title, an image and a text area, etc.). Two page layouts come with the software.

The Portfolio Template tool

The Portfolio Template tool allows a designer to prompt the portfolio owner for various wizards, matrices, forms and files that they have and create a polished presentation of all of it. The designer can specify css stylesheets, JavaScript files, images and logos to create a branded look for their users. The designer needs to define all of the pieces that go into making a complete portfolio of this type and designate an XSL stylesheet to reorganize the XML mashup of all of that structured (forms, wizards and matrices) and unstructured data (raw attached files) into a presentation.

The Portfolio Tool

The portfolio tool allows the portfolio owner to either create a portfolio from a template (see above) or to design her own portfolio.

If the owner decides to build their own portfolio, they can choose a style (see the style tool above) and begin constructing a series of pages, each with a predefined layout (see above layouts tool).

If the owner wants to build a template-based portfolio, they are prompted to select appropriate data structures (as defined by the template designer) to fill out the portfolio.

When a portfolio is created, the owner can specify who gets to see it (a user, a group of users, guest users) and for how long or they can make their portfolio public to the world.

Does that sound complicated? It is! Trying to figure out how all of these tools work together to provide a workable solution is daunting...and that is just half of it. Once you have scraped your knuckles for a while, you have to actually design the actual forms, the matrices, wizards, styles, layouts and templates that will produce the kind of product that you think your “client” will accept and portfolio authors will find valuable.