The Living Schoolbook

Did you ever see those Capital One commercials where a band of marauders advances on an innocent customer as he makes a purchase with his credit card? The marauders are stopped just before they strike when they realize that this guy was smart enough to choose a credit card that didn't have ridiculous interest rates. The slogan "What's in your wallet?" begs the watcher to make an either/or decision to get slaughtered by the marauders, or to go with their low interest card.

As I read Trent Batson's Campus Technology article, The ePortfolio Hijacked, I thought of hordes of accreditors descending on innocent students, ready to take control of their portfolios and destroy their opportunities to recast themselves as educated, enlightened members of society. He really makes it clear that an assessment system portfolio is not a "learning portfolio". He has drawn a line in the pedagogical sand and proclaimed that there are the assessment system portfolios and then there are REAL portfolios.

Here at Syracuse, the assessment system that is used as a tool to gather data for program review and accreditation needs is not readily available to students. Assessments are mapped to program proficiency standards and faculty rate student work on a 1-4 scale in a system that the students do not have access to. Most of the faculty do not have access to the database that stores that info. Data is collected through spreadsheets and fed into the assessment system by hand by just a few people.

An alternative approach for deploying an assessment system might open possibilities for reflection and formative feedback that we are currently missing. If the data was turned around so that students could see what ratings they received on each of the standards and why, it could stir up some really rich discussion between teachers and students. The Goal Management project that we started here was meant to make transparent the relationship between the program standards, classroom assignments and the data collection process that was being performed. Trent might say that this data only has an audience with deans, accreditors and department heads but I think that it may be the spark that starts students down the road to reflective thinking.

I'd like to think of an assessment system as a means to start a dialogue between teachers and learners around some common themes and a compliment to formative assessment and "assessment for learning", rather than a competitor to both. It is really how you design and use the system, not an inherent property of it. My guess is that there are ways to implement assessment systems that would encourage reflection and result in ideas that a student might take a lot of pride in. The key is in the teachers' ability to engage the student in conversation and seize the teachable moments that such a system makes possible.

The faculty and students have been "doing" portfolios in the Inclusive Elementary Special Education program since before I came here in 2000. They used to have a catered dinner at "Drumlins" and sit at round tables while 3 or 4 students would show their binder-based portfolio to a host teacher and a faculty member while nibbling on hors d'oevres and ham sandwiches at the end of the semester. The event was mostly celebratory. Even if some advice was dispensed during one of these sessions, it seemed like it was too late to me, a newcomer to all of this.

Several years later, as the school began experimenting with "e" portfolios in the form of Dreamweaver and Netscape Composer authored web sites, more time was spent teaching technical skills to the future elementary teachers so that they could showcase the highlights of the semester to their peers and teachers. As a member of a tech group that created and maintained a home made course management system called "Dialogue", which was designed around the idea that EVERY opportunity for assessment/review was also an opportunity to begin a thread of formative discussion, I thought the portfolio idea had a lot of potential to be a great learning opportunity.

The technical barrier to authoring one of these portfolios was substantial, even when we shifted gears and began starting students with a PowerPoint template, with stubbed out pages suggesting that the students address our emerging "School of Education Proficiencies" in their portfolios. How to resize images, scan documents that proved that they were "proficient" and embed video snippets that played on multiple platforms remained issues that were dealt with during the last few weeks at the tail end of the semester. The rest of the semester was spent doing the real work of the school, training students to be excellent teachers.

For the past two years we have been using the web based Open Source Portfolio software. While OSP has improved markedly in that short time, the other barriers remain. As such, our student's portfolios are usually not worked on (or read) by most of the teachers in the school until the end of the semester. In that regard, nothing has changed. It seems that coursework is coursework and the portfolio remains this "other" thing that gets tacked on the end of the semester.

I have this vision of faculty sitting down in the morning with a cup of coffee and reading their students' daily musings (just like I check a pile of blogs on a regular basis), writing back a few thoughts and using that information to inform the next week's lessons.

I wonder if e-portfolios are way too ambitious. A blog with a tagging feature, an few rss feeds and a class blog "aggregator" might be all we need to embed "portfolio thinking" into classes.

Disclaimer: For those of you who do not know me, I am not a teacher. I provide technical support for grant funded educational projects in the School of Education at Syracuse University. Take this perspective with that grain of salt.

Over the past three years of working with the Open Source Portfolio and Sakai communities, I have heard lots of people talk about how they plan to implement the portfolio tools. Some of the uses highlight the ability of the tools to create a standard presentation (like a resume or curriculum-vitae) by using a template. In a later entry, I’ll be walking you through the steps I would follow in order to create a set of forms, wizards and a template that will allow users to pop out a presentation like that. This entry isn’t about those types of uses. I am sure that everyone understands the purpose of a portfolio like that. I want to take a moment to talk about the types of portfolios that can be useful in education settings. This entry doesn't really talk about how to imlpement an accreditation portfolio. I want to set the stage first.

The outcomes based accreditation or program review portfolio

Some schools need a means to review student learning/progress/achievement of ALL of the students in their programs in a very comprehensive way. The prerequisite to a system like this is that the faculty has decided upon an assessment framework that will be applied to all of their students and all of their classes. A “portfolio” effort like this has several components.

Establishment of a framework for evaluation

In order to set this up, a set of outcomes must be agreed to by the faculty. This is a process that many schools apparently do not engage in very often. I haven’t heard of a faculty that spontaneously decides to engage in this process. They usually have more interesting things to do than debate the purpose of the programs in which they teach. Usually it is a mandate from an accrediting body that initiates the process. It was for our faculty.

Identification of key program assessments

A plan has to be designed for measuring the performance/achievement of the students in the program (and in so doing, the effectiveness of the program). A natural tool to use for this purpose is the assessments already in place in the program’s courses. Chances are, a subset or cross section of the work the students already do in their classes would provide the information needed to see how the students and the program are doing. By highlighting the assessments already in place in the program’s curriculum and matching them to the outcomes that they address we can aggregate student work aggregated into FACULTY DESIGNED portfolios that serve the purpose of the program.

Assessment of student work in light of the program outcomes

Once the a faculty member has identified an assessment (an assignment usually) in their course that is intended to assess (at least in part) one or more of the program outcomes, the student’s work will need to be assessed against each of those outcomes. Whether the instructor of the course performs that assessment while grading papers or a "program review panel" performs the assessment later on, the process requires standardized rating scales and rubrics to ensure that all of the faculty are on the same page when assessing the student work. Establishing standard rubrics that ensure inter-rater reliability is another huge task that takes place outside of the “system”. The axiom “garbage in – garbage out” applies here.

Analysis of the Data

It may surprise you to see the multiplicative effects that adding new assessments, outcomes and students to this structure has on the amount of data that there is to look at and analyze. A cohort of 100 students going through a program with 30 highlighted assessments each identified as addressing on average of 2 outcome each provides 6000 pieces of data for analysis. If assessment is continuously performed by instructors (rather than by a “panel” at the end of the semester) it might be helpful if both students and the faculty could have real time access to this assessment portfolio data to help identify problems and trends in student performance and hopefully correct the problem with relevant formative feedback, counseling, etc. If the data analysis is done at the end of the semester, the possibility of formative feedback may be forfeit and the focus on the data analysis would then be to get an overview of the performance of various aspects of the program in order to improve it.

Implementing Change

Once the data analysis is completed, some issues may be identified. The faculty may decide to change its rubrics to tune the assessment process, rephrase the program outcomes, restructure the program curriculum and/or pick different assessments for review. Accreditation bodies are particularly interested in this process for overall review and data driven decision-making.

Without going into details about the implimentation of a solution in Sakai and OSP, it should be clear enough that a mandate to engage in this level of program review requires changes in the organization of a school. In our school, a new full time position was created to manage the assessment process. The assessment coordinator is charged with setting up the assessment framework, collecting the data and running reports.

Goal Management

When the LSB started using OSP to address our program review portfolio needs, we quickly identified a gap between the tool set's capabilities and design philosophy. Most everything in OSP is geared towards making the process initiated by students. In the OSP matrix tool, the STUDENT gets to pick the items that meet the criteria of the matrix. While we were interested in any additional material that students felt was relevent to present as part of their own evidence of mastery of the program outcomes, we knew that asking them to resubmit all of the work that the faculty had identified was a process that was unreliable and unnecessary.

The Goal Management tool released as a “contrib” tool for the 2.4 release was an effort to provide a means for a user to articulate multiple sets of outcomes (goals) and to share those goals with a number of class worksites in Sakai. Instructors who create assignments in their classes can identify the outcomes that pertain to each assignment and rate student work against each. The outcomes and rating data are stored in the Sakai database and can be aggregated and reported.

While we do not have funding right now to continue developing the idea, the basic premise has rung true with many other institutions and is likely to be further developed by Indiana University (hopefully with helpful input by the rest of the community) to be an integrated component of versions of OSP in future releases of the software.

In the interest of full disclosure, our own implementation of Goal Management in Sakai is NOT the production system that our assessment coordinator uses to collect data. Our implementation is a research project that lacks some critical features that would make it an acceptible substitute for the current production system. Discussions are in progress about the cost-benefit of building the necessary features and maintaining a Sakai instance as compared to continuing to use the current system.

The Open Source Portfolio (http://www.osportfolio.org) is often difficult to explain to folks. There are some aspects to the design of the software that seem to be in contradiction with one another. There is a tension between the belief that portfolios are "student owned" and the technical reality that some of the tools are far too complex for students to use. As a result, implementers of OSP use the flexibility of the tools to attempt to craft an environment that meets the needs of all of the users. The simple fact that the experience is designed and implemented by users who may be two degrees from the users make this a challenging piece of software to work with.

I plan to write a bit more about this "suite" of tools in the coming weeks to help identify some ways that I think implementors can get productive fast, but read on for a brief intro to the tools of the Open Source Portfolio.

The Forms tool

The form tool allows a user (probably someone who is wise in the ways of XML) to define data input screens (forms) that will be used to prompt users and structure data entry into the system that will later be used to construct a portfolio. The user gives the form a name, types in some instructions and then identifies an XML Schema Definition (XSD) file that actually describes what the fields are in the form, the validation rules, and some special display information. Optionally, the form designer may further want to craft an alternative view for the form and will identify XSL (Extensible Stylesheet Language) files that will transform a default rendering of a form to one of design.

Wizard pages

While there is no “wizard page” tool, this structure is used as a building block in both the Wizards and the Matrix tools. A wizard page has a unique name and will present guidance (instructions, rationale and examples) to the portfolio owner. When constructing a wizard page, the designer will be able to add the following “pieces” to the page:

• Forms (from the above mentioned tool) for the portfolio owner to fill out.
• A reflection form for the owner to use.
• An evaluation form for an evaluator to use when assessing the page.
• A feedback form for a portfolio reviewer to use to provide informal comments.

When describing a wizard page, all of these structured data input forms are completely optional. At a very minimum portfolio owners will always be able to upload and attach files to the page.

The Wizards tool

The Wizards tool allows a user to create either a series of screens that further "scaffold" the user through the process of entering the data that will used to create their portfolio. There are two types of wizards in OSP:

• Sequential Wizards can be constructed as a set of pages that are presented one after the other.
• Hierarchical wizards present a nested tree of categories and pages to users.

In either case a the pages of the wizard can be constructed to include guidance (instructions, rationale and examples) to the portfolio user. The wizard author can choose forms (from the above mentioned tool) that will prompt the portfolio author to provide pieces of information relevant to the portfolio. The wizard author also has the opportunity to prompt the user to add reflections along the way, receive feedback from reviewers and evaluators.

The Matrix tool

The Matrix tool allows the designer to construct grids of one-page wizards wizard pages (the same wizard pages as described above). The rows and columns of the matrix are described as “criteria” and “levels” in the tool. A matrix “cell” can be set to be “unlocked” (ready for the portfolio owner to use) or “locked” (unavailable to the portfolio owner). Matrix cells can be set to unlock (top to bottom, left to right, or as the instructor deems appropriate) as the owner submits the cells for evaluation.

The Style tool

The Style tool allows the designer to designate an uploaded css file as a “style”.

The Layouts tool

The Layouts tool allows a designer to designate an xhtml file as a template for a free-form portfolio page. Page layouts can have many different designs (Title over text, two columns, Title, an image and a text area, etc.). Two page layouts come with the software.

The Portfolio Template tool

The Portfolio Template tool allows a designer to prompt the portfolio owner for various wizards, matrices, forms and files that they have and create a polished presentation of all of it. The designer can specify css stylesheets, JavaScript files, images and logos to create a branded look for their users. The designer needs to define all of the pieces that go into making a complete portfolio of this type and designate an XSL stylesheet to reorganize the XML mashup of all of that structured (forms, wizards and matrices) and unstructured data (raw attached files) into a presentation.

The Portfolio Tool

The portfolio tool allows the portfolio owner to either create a portfolio from a template (see above) or to design her own portfolio.

If the owner decides to build their own portfolio, they can choose a style (see the style tool above) and begin constructing a series of pages, each with a predefined layout (see above layouts tool).

If the owner wants to build a template-based portfolio, they are prompted to select appropriate data structures (as defined by the template designer) to fill out the portfolio.

When a portfolio is created, the owner can specify who gets to see it (a user, a group of users, guest users) and for how long or they can make their portfolio public to the world.

Does that sound complicated? It is! Trying to figure out how all of these tools work together to provide a workable solution is daunting...and that is just half of it. Once you have scraped your knuckles for a while, you have to actually design the actual forms, the matrices, wizards, styles, layouts and templates that will produce the kind of product that you think your “client” will accept and portfolio authors will find valuable.

A quick model of what I thought I heard about the future of Goal Management.

The big things that are different here:

• Goal sets include both Goals and Levels. In essence a Goal Set actuallly will be a matrix. The illustration belies the multi-dimensional aspect that will result from a hierarchical set of goals.
• Program chairs or assessment coordinators will be able to "stub out" assessment placeholders and link them to cells (a goal and a level).
• Linking of activities (like assignments) can be semi-automated when faculty are prompted to identify their activity with an assessment placeholder.
• The assessment placeholder would allow the chair or corrdinator to present a consolidated set of fields that will be used to collect ratings. Each field will need to have rules that disseminate the input ratings up to the linked cells.