Prototyping & User Testing in Learning Design

Sketch of classroom

By: rorydraw used under Creative Commons license.

Prototyping and user testing are common practices in many design fields, yet when designing courses and curricula within higher education, developing prototypes and testing them with users seems to be a rarity. How might prototyping and user testing help us improve our designs for learning?

Prototypes help designers discover and address design challenges and constraints early in the design process through creating low-cost and often low-fidelity mock-ups. There is nothing like building a prototype to uncover the real challenges in a project. Prototypes can lead to solutions that could not have been imagined at the start of the design process, but through repeated efforts to improve a design, a beautiful solution is found.

Prototyping becomes even more helpful when we get our prototypes out into the world and test them with the people who will ultimately use what we are designing. Through user testing, we get incredible feedback on our design solutions. User testing gives us real answers to questions such as: Is it easy to use? Does it meet users needs? Did we make good assumptions about how users will approach our design? Through user testing, we get a view into the mind of our users so we can evolve and further develop our prototypes to better meet their needs.

When designing courses and curricula in higher education, we do have some practices that might be considered prototypes. Syllabi, curricular maps, and learning activity designs evolve through many drafts in the design process. Yet, are we treating these drafts as prototypes or are we refining a single solution? By developing multiple prototypes, each representing different approaches to a design challenge, designers take on a frame of divergent thinking-expanding and exploring many aspects and possible solutions to the challenge before moving to a convergent thinking mode to narrow and refine ideas. Yet, how often in learning design do we develop multiple concepts as possible design solutions? Often we skip exploring multiple approaches to a challenge and move immediately into narrowing and refining an early concept. Additionally, the process of developing multiple concepts requires a time commitment. Yet in learning design, do we give enough time to the prototyping process to allow designs to evolve and improve through the iterative building and refining of multiple prototypes?

While we may do some prototyping in course and curricular design work, I rarely see prototypes tested with students before a course or a curriculum is launched. There may be some good reasons for this, as many curricula take students several years to complete. User testing something so large prior to launch could be improbable or impossible for many institutions. Even at the course level user testing can seem impractical, trial running a semester long course for the purposes of user testing could be challenging for many schools.

Should we just skip user testing all together and make adjustments over time as we run the course with students year-over-year? This seems be to a standard practice in course design. We put together a syllabus, launch it, then gradually tweak and improve it each semester or year. Of course we should use student feedback to improve our designs over time, but don’t students in the early runs of a new course design deserve our best design efforts? How might user testing give us feedback to improve our designs prior to launch so that a course might be great on the first offering rather than having to wait a few years for it to be refined?

If doing a full semester or multi-year trial seems improbable, how might we do smaller, modular user tests? Could we develop user tests that allow us to test prototypes of particular learning activities or core resources? Could we find ways to test curricular structures and sequences with students without going through the whole sequence?

It seems we could learn a great deal in the design process and produce better designs more quickly by involving more prototyping and user testing in the process. So how do we do it? What have you seen that works well?


Crowdfunding Higher Education

Dr. Leslie Jensen-Inman and Jared Spool, of the Unicorn Institute, have started a Kickstarter campaign to help fund their startup User Experience Design School, Center Centre. The campaign surpassed its initial target within a few hours and within several days it has raised more than $65,000 from more than 650 backers.

Seeking donations to support programs is nothing new to higher education institutions. However in this case, the school is foregoing traditional fundraising pathways, using instead technology supported social networks and crowdfunding tools to mobilize support for the program.


Visual Course Design

Close up of sticky notes being used for course design.

Designing the scope and sequence to learning activities for a new course can be a challenging process. In the early stages of course design, my brain is full ideas and concepts for learning activities, possible readings and resources to assign, and presentation techniques I might use.  It can be difficult to get that jumble of ideas into a meaningful course structure while trying to maintain a good balance of work for my students and me during the term.  Until recently, I would rough out the scope and sequence for a new course in a word processor as a part of writing my syllabus, but I’ve been finding that the building the first draft in a word processor makes it difficult to make quick changes. Additionally, at the early stages of course design, I am trying to form connections between concepts and ideas and drafting in the word processor forces a linearity to my thinking that makes it harder to see those connections. To better get the ideas out of my head and into a place where I can work with them to build connections, I have recently been using a visual method to help me sort out all of the pieces of a new course in the early stages of the design process.

On a stickFoam core and sticky notes used for course planningy-note, I write out a number for each week of the term and attach them to a large piece of foam core. These help me see the calendar constraints for my course.

Using different colored sticky notes, I write out the weekly activities, assignments, and student projects and add them to the board under the number of the week that the activities will take place. I write out topic and content themes for the assignments. The theme notes help me to develop a thematic progression through the units.  I will also make notes, either on the board or somewhere else, about the amount of time I expect students to work on the assignments or projects.

On other sticky notes, I write out the reading selections, noting the main themes of the reading and the page quantity for the selection.  The theme notes help me to match reading and assignment themes. The page numbers help me create a balanced work load through the weeks of the term.

Having the entire semester of assignments and activities posted on a single board helps me to see the flow of the course in a way I can’t when the schedule is spread over multiple pages in a document. The large format helps me to make connections on unit structures and themes and to quickly see if the work load is spread evenly through each week of the term. If I want to make a change, it is fast and easy to move notes around the board or to write out a new note. Once I feel I have a good scope and sequence developed, I write out the schedule in my syllabus, make final adjustments, and add the information to my course in the LMS. By using an analog visual structure for drafting the scope and sequence of a new course I find it easier to get the jumble of ideas out of my head and into a space where I can easily move them around to form a coherent course design that has good thematic flow and a balanced work load for students.


Mental Models and Usability

Conceptual Models Diagram from Norman, D. A. (1988). The Design of Everyday Things. New York, Basic Books.

Conceptual Models Diagram from Norman, D. A. (1988). The Design of Everyday Things. New York, Basic Books.

In his book, The Design of Everyday Things, Donald Norman explores methods to design usable objects and systems using everyday objects such as door handles and faucet knobs as examples. Norman emphasizes the role of conceptual models in creating easy to use systems. Any system or device will be easier to use if the user has a good conceptual model of how the system works. It is the designer’s job to create a conceptual model for the system that aligns with users’ mental models or educates users on the model of the system (p.189).

Three different aspects of mental models must be distinguished: the design model, the user’s model, and the system image. The design model is the conceptualization that the designer has in mind. The user’s model is what the user develops to explain the operation of the system. Ideally, the user’s model and the design model are equivalent. However, the user and the designer communicate only through the system itself: its physical appearance, its operation, the way it responds, and the manuals and instructions that accompany it. Thus the system image is critical: the designer must ensure that everything about the product is consistent with and exemplifies the operation of the proper conceptual model (pp. 189-190).

In complex systems aligning these mental models can be challenging. Designers are likely to be experts in how the system works, whereas users may have a limited or incorrect understanding of the complex system. This can create a situation where designers create a system that reflects their expert understanding, not the novice mental model of the user. This type of design error can commonly be seen in websites in which the navigation of the site is based on the org. chart of the business—a system image based on insider view of the business rather than a system image based how website visitors think about the organization or  how they frame the tasks they want to accomplish.

In higher education, programs, curricula, and courses are just a few of the many complex systems that leaders are called to design. University leaders have an insider expert conceptual models of the educational system—a model that is likely very different from the conceptual model held by  students and other constituents of the university.  Through user-centered design practices such as those recommended by Norman, educational leaders can design learning systems that are understandable and user friendly for their students.

References

Norman, D. A. (1988). The Design of Everyday Things. New York, Basic Books.


Learning Space Design Specification

Classroom

Educause has developed a Learning Space Rating System to help schools develop active learning classrooms.

The Learning Space Rating System (LSRS) project provides a set of measurable criteria to assess how well the design of classrooms support and enable active learning activities. Noting the success of several architectural programs to promote sustainable building design, the LSRS provides a scoring system to serve as an indicator of how well a classroom’s design serves the goal of active learning. The LSRS criteria form the basis for a rating system that will allow institutions to benchmark their environments against best practices within the higher education community.

The rating system provides guidance in these areas.

  • Integration with the campus context
  • The design and planning process
  • Support and operations
  • Environmental quality
  • Layout and furnishings
  • Tools and technology

For background information on the project see the article in the Journal of Learning Spaces: The Case for a Learning Space Performance Rating System.


Apple, Design, Backward Design

Tim Cook, CEO of Apple, opened a product release event today by playing this video which emphasizes the role of design in Apple’s work. Cook began his presentation saying,

I really love that video…I wanted to open with it this morning because I think it does such an incredible job of talking about our values. It reflects the unique way that Apple creates what we believe are the very best products in the world. You’re going to see some amazing products this morning that could only have been developed in that unique way.

In this video and his presentation Cook highlights the central role of design in Apple’s work; a process that focuses first on what they want the users of their products to experience and feel.

A Mac loving New Testament Professor I know sent me a text message today, “Are you watching the Apple streaming video? Their opening video is basically about backward design.”

Backward design is a framework for designing learning experiences that begins with articulating what students should know and be able because of a learning experience, then moving to designing and creating the learning activities and experiences that will help students attain the desired learning (Wiggins and McTighe, 2005, Fink, 2013).

Cook made a clear statement today about the central role of design for Apple. Just as Apple’s design process focuses on how users will be affected by their products, backward design focuses on what students will learn in a course or program. In my experience backward design is often done at the level of individual courses, but not necessarily at the level of an entire program or school. Are there colleges or universities where backward design plays a core strategic role? Are there deans, provosts, or presidents who are talking about the strategic role of design in ways similar to how Tim Cook is talking about design at Apple?

If you know of any examples of where this is happening, please share.

References
Fink, L. Dee. (2013). Creating significant learning experiences, revised and updated. San Francisco: Jossey-Bass.

Wiggins, G. and McTighe, J. (2005). Understanding by design, expanded 2nd edition. Pearson.


Liz Coleman on reinventing Liberal Arts Education

Liz Coleman, then President of Bennington College, gave a phenomenal TED talk in 2009 discussing the role of liberal arts education in the advancement of the public good. She discussed how Bennington was redesigning its curriculum with critical political and social challenges such as health, education, and the use of force as organizing focal points for learning.

“We intend to turn the intellectual and imaginative power, passion, and boldness of our students, faculty, and staff on developing strategies for acting on the most critical challenges of our time.”

While the college would still work with the arts and sciences, there is to be an increased focus on integrative work and learning. The Bennington model would create an action oriented curriculum that focuses on developing abilities in areas such as rhetoric, design, mediation, improvisation, quantitative reasoning, and technology as core abilities central to addressing complex social problems.