With the explosion of online courses in higher education, many students are opting for the convenience of learning in a virtual environment. As a former instructional technologist for one of New Jersey's online education pioneers, Atlantic Cape Community College (ACCC), I was responsible for assisting in the development, implementation, and support of more than 90 Web-based course sections each semester. The experience of aiding online students (through telephone and e-mail support) spurred my interest in creating a tutorial product to guide new users through the course management system (CMS) and to suggest online study habits for success. The generation of this tutorial served a dual purpose: It constituted the thesis for my master's degree, and it provided orientation to the Web Course Tools (WebCT) CMS. By tracking the tutorial's development and plans for subsequent revisions, I hoped to streamline the process of improving this product and keeping the CD-ROM current and relevant as a resource for new virtual students. Even though I am now a full-time faculty member, responsibility for the tutorial still rests with me, and I am currently modifying the product based on insight gleaned from user feedback.

This article provides an overview of the development of the WebCT tutorial, with particular emphasis on the multiple feedback mechanisms that were crucial to assessing its overall effectiveness. By illustrating the importance of soliciting responses from a range of different parties—students, faculty members, and key administrators—this account may offer a helpful model for others who are currently in the process of designing similar online tutorials or training modules.

Preliminary Research and Objectives

While searching for literature related to CMSs and adaptation problems, I evaluated statistics from and solutions implemented by other higher education institutions. Of particular note was a study from Florida Gulf Coast University (Smith & Benscoter, 1999) that described an Internet tutorial, developed by faculty members, to be used by students at the onset of their Web-based courses. This tutorial addressed not only basic communication functions between students and faculty members but also technological troubleshooting issues. An article by Berger (1999) outlined pertinent concerns about technologically infused distance education and provided recommendations for diminishing new-user anxiety. Some of these suggestions were to create a Web page or site for operational procedures, to be specific when writing instructions, and to provide a method for becoming familiar with the technology to minimize initial cognitive overload. These studies and others like them provided the rationale and the impetus to develop a customized product for ACCC's students.

The objectives for the customized tutorial's construction process were to:

• identify what CMS utilities students considered cumbersome or challenging in terms of basic functionality;
• gather tips for good online study habits from faculty members and students with significant experience in virtual learning environments;
• compile solutions to these problems into a user-friendly CD-ROM product that would supplement the CMS and be available prior to the start of the semester;
• review research on multimedia principles, especially those related to user-end and contiguity and modality issues, and apply the synthesis of multimedia research to the construct of the product; and
• determine a model for product evaluation and develop a reasonable revision schedule.

Faculty Survey

In December 1999, I implemented a survey (Exhibit 1) to assess faculty members' understanding of the functionality of the CMS and obtain information about problems that new students have when using it. In addition, I wanted to solicit definitions of a good online student. Twenty-six out of 31 faculty members returned the survey. Some of the respondents merely filled out the questionnaire, but others embraced the opportunity to clearly identify user problems with the CMS and to candidly address desirable qualities of successful online students. As reported by the faculty, desirable traits for virtual students included time-management skills, self-motivation, the ability to adhere to deadlines, and prompt, effective communication with the instructor and fellow students. The faculty survey responses provided a preliminary set of data that could then be cross-referenced with a student needs analysis.

Student Needs Analysis

In February 2000, I implemented the student-level needs analysis in 13 courses using the built-in mail utility of the CMS. Because faculty feedback suggested that online testing fosters anxiety, I surveyed only those classes in which testing was conducted over the Internet with the testing utility of the CMS. The questions in the student survey were distilled from the faculty survey and reduced to three essential prompts:

1. List the qualities you think an online student must possess to be successful. (See Figure 1 for results.)
2. List the WebCT tools or areas of tools that you have had difficulty with, from most difficult to least difficult, and include a description of the problems you have encountered. (See Figure 2 for results.)
3. If you were issued a CD-ROM study-habit guide on how to be a successful online student, what kind of content would you expect to see? (See Figure 3 for results.)

Roughly 33% of the surveys were returned (Exhibit 2). Most students reported that although they quickly caught on to the functions of the CMS utilities, they sometimes had to learn the mechanisms of a specific task while performing it.

Putting the results in graph form provided a visual map of which areas needed concentration and helped me prioritize the content of the tutorial. I used Inspiration software to visualize the tutorial product in the style of a concept map (Exhibit 3). The dean of distance education and the director of instructional pedagogy reviewed this map and provided suggestions for language simplification and additional study habits. The final version of the product reflected their recommendations.

Product Development

The mode of delivery was my first consideration with regard to effectively relaying the tutorial's content. Westendorp (1996) investigated consumer online help and found that animation aided learner retention: Consumers retained instructions better when they were visually shown what to do. Issues of modality (i.e., methods of information delivery such as auditory files, animation, text, and still graphics) and contiguity (i.e., what modes should be paired together) also seemed to be important in determining how to deliver the instruction. In applying a generative theory of multimedia instruction, Mayer (1997) found that subjects retained more information when presented with both narration and animation than when presented with either of these forms alone. In a later study, Moreno and Mayer (1999) found that multimedia instruction was most effective when these two modes of information delivery occurred at the same time or were contiguous.

The development of the tutorial required a user-friendly end product. Unfortunately, local Internet speed connections, especially in some rural areas, forced me to consider CD-ROM delivery instead of Web delivery; ACCC's lack of RealServer access cemented this decision. Given that Macromedia's Flash player and Adobe's Acrobat Reader are common on many home computers—and available on all ACCC campus computers as prepackaged software or browser plug-ins—I used Flash 3 and portable document format (.pdf) files to create the interface and printable directions. I also used Lotus ScreenCam software, which neatly couples narration and animation, to demonstrate the CMS functions.

Steered by literature on multimedia navigation and popular metaphors such as travel and adventure (Dias, Gomes, & Correia, 1999), I adopted the popular theme of sailing for the graphical interface (Exhibit 4). This seemed an appropriate metaphor, as the college serves all of southern New Jersey's coastal residents. The vector graphics related to sailing were created in the native Flash environment so that they would "perform" optimally. The size of the interface did not exceed 640 x 480 pixels—as recommended by O'Hanlon (1999), who urged multimedia authors and Web designers to design to the lowest likely monitor setting. I titled the product "Smooth Sailing in Your WebCT Online Course."

I divided the tutorial into four key parts: an animated tour of the CD-ROM's contents, a lesson on navigating the disc and downloading viewer programs (Exhibit 5), a section on common CMS utilities (Exhibit 6), and a general guide to online study habits (Exhibit 7). Navigation options within the tutorial enabled users to access to any part of the product at any time.

Product Evaluation: Beasley's Life Cycle Design Approach

I based the evaluative phase of product development on Beasley's (1998-99) preference for a life cycle design approach that includes audience analysis in each stage of revision (Exhibit 8). Though Beasley's model is geared toward commercial products, the idea of feedback steering product development and revision seemed logical because the tutorial was designed to be a mass-distributed item.

In the first evaluation at the mapping stage, as noted above, the dean of distance education and the director of instructional technology at ACCC provided feedback. Both individuals had been involved in the early stages of the development and delivery of online courses at ACCC, and both had frequent contact with online faculty and new online students. Much of the second evaluation, a faculty review of the beta version of the tutorial (Exhibit 9), was positive in terms of the content and the graphical interface. Suggestions for revision included lowering the volume for certain elements of the tutorial, putting a screenshot of the console for animations in the liner notes, and providing special instructions on how to start the CD-ROM if it did not engage by itself. Less than 50% of the faculty members polled in the initial survey responded at this stage, but their comments prompted the immediate overhaul of the CD-ROM's liner notes to simplify operation of the tutorial. The new liner notes included a screenshot of the animation console operation, step-by-step instructions for setting the user's monitor to comfortably fit the Flash interface, and a quick overview of the most commonly used CMS communication utilities. The third evaluation, conducted at the student level, produced positive and encouraging feedback (Exhibit 10). Most students found the tutorial easy to use, simple to navigate, and especially helpful in advance of test-taking and sending attachments.

Every semester an overhaul of the CD-ROM is considered. To date, incremental upgrades of the CMS have yet to warrant a third renovation of the tutorial product. However, student feedback (still pending for 2003) as well as upgrade differences in the functions of the CMS will help direct any future modifications of the tutorial. Faster local connection speeds and new screen recording software may also impact the design and delivery of the product. At this time, the tutorial can be obtained through the ACCC bookstore, but eventual publication is the goal. The next version will be more (institutionally) generic to increase the likelihood of it being utilized by other colleges.

Conclusion

Through my experience with digital design and multimedia authoring, I have learned that preliminary assumptions and my own bias will not contribute to a viable product. It is the users' needs and feedback that can best steer development of a product and allow it to evolve from phase to phase to become more ideal. Though my initial intent was to find out which instructional modes promote the greatest learning, my research included the discovery that good multimedia courses, like a good Web page, are never truly finished. A multimedia author needs to research, solicit feedback, accept criticism, and be prepared to apply necessary revisions.

[Editor's note: This paper is modified from a presentation at the 2001 WebCT User Conference in Vancouver.]

References

Beasley, R. (1998-99). Interactive multimedia development: Predesign analysis. Journal of Educational Technology Systems, 27(1), 23-42.

Berger, N. (1999). Pioneering experience in distance learning: Lessons learned. Journal of Management Education, 23(6), 684-691.

Dias, P., Gomes, M., & Correia, A. (1999). Disorientation in hypermedia environments: Mechanisms to support navigation. Journal of Educational Computing Research, 20(2), 93-117.

Mayer, R. (1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32(1), 1-19.

Moreno, R., & Mayer, R. (1999). Cognitive principles in multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91(2), 358-368.

O'Hanlon, N. (1999). Web-based tutorial: Does course use differ from general use? Journal of Interactive Learning Research, 10(2), 217-228.

Smith, S., & Benscoter, A. (1999). Implementing a web-based Internet tutorial for web-based courses. The American Journal of Distance Education, 13(2), 74-80.

Westendorp, P. (1996). Learning efficiency with text, pictures, and animation in online help. Journal of Technical Writing and Communications, 26(4), 401-417.

puzzle gamestime management gamessimulation gamesshooter gamesbrain teaser gamesmarble popper gameskids games