July 1998 // Case Studies
The Flashlight Project: Tools for Monitoring the Progress of our Hopes and Fears about Technology in Education
by Stephen C. Ehrmann
Note: This article was originally published in The Technology Source (http://ts.mivu.org/) as: Stephen C. Ehrmann "The Flashlight Project: Tools for Monitoring the Progress of our Hopes and Fears about Technology in Education" The Technology Source, July 1998. Available online at http://ts.mivu.org/default.asp?show=article&id=1034. The article is reprinted here with permission of the publisher.

Since the 1950s, people have been trying to use computers, video, and telecommunications to revolutionize education. For decades, their vision was dominated by several ideas about how to use technology to extend the teachers' reach. For example:

  • using television to make some of the world's greatest teachers available to students at great distances or over time (e.g., for years after a great lecturer in philosophy had retired, one could still watch her lectures);
  • using television to educate remote students or groups of students who could not otherwise reach or afford a teacher;
  • using computers to provide self-paced, individualized instruction with tailored feedback that could help students learn a greater variety of topics with greater speed and effectiveness and yet at lower costs because of economies of scale. The wisdom and insight of great teachers could be used to help create the software.

Although there are many examples of such uses of technology, they have not been sufficient to constitute a revolution in the nature of education. The most important barriers to using technology to help teachers to extend themselves have been economic (e.g., the expense of developing courseware, computers advancing so fast that expensive courseware becomes obsolete before ever finding a market, etc. [Morris et. al. 1994]).

The original hopes for revolution of teacher augmentation and replacement have been stymied. Nevertheless, higher education does seem to be headed into a revolution, one that will use technology energetically but not primarily as a way of extending our best and brightest teachers. Flashlight is a flexible evaluation toolkit that can be used to study the kinds of progress that are actually happening in most places. It can also be used to check whether some of our most common fears about technology are coming true. As we'll see, Flashlight can be used to start studies at the level of classes, courses of study, services, and institutions, as well as enabling studies that cross institutional boundaries.

The Emerging Vision

The revolution in education that is actually emerging is based on uses of technology that are:

  • similar for on-campus and off-campus students (and for professionals in the larger world);
  • helping a wider range of students (and faculty) reach a wider range of intellectual tools and resources;
  • helping students learn real-world skills that are partly dependent on technology use (e.g., statistical ways of thinking that are more visual and more recursive than possible with paper-and-pencil techniques; ways of thinking about photography and graphic design that make use of computer-based image processing and printing);
  • helping to widen access to education (e.g., to students whose time on campus is limited, students with disabilities, etc.);
  • increasing the ability of an institution to compete for students;
  • used in ways that can improve faculty-student interaction, student-student interaction, active learning, time on task, and other practices that research shows are usually correlated with improved learning outcomes.

The software used to implement these visions is usually not the content-specific, mass-produced teaching machine of the older vision. Instead it is most often empowering tools or library-like resources, often the same as the tools and resources used outside the academy. The vision on which Flashlight is based, and the methods used to identify it, are described on The Flashlight Project Homepage.

The Flashlight Current Student Inventory

The Flashlight Current Student Inventory (CSI) can be used to collect facts and opinions from currently enrolled students about the uses of technology in the bulleted list above, as well as many other applications. The CSI is a toolkit of almost 500 indexed questions that can be used to draft surveys, questionnaires, and protocols for interviews and focus groups. Many of those questions focus on the ways in which technology is helping, or failing to help, implement research-based principles of good practice in education (Chickering & Ehrmann, 1996). It includes the Flashlight Evaluation Handbook (1997) of guidance on creating studies, readings, resources, and case studies. The Flashlight Current Student Inventory is available for site license from the TLT Group.

Flashlight has a variety of applications, including:

  • guiding improvement of classes and courses of study (e.g., majors, minors, freshman year skills development, writing across the curriculum) and strengthening the roles played by technology in such efforts;
  • evaluating major grant-funded projects;
  • improving technology-based services (e.g., libraries, computing services, telecommunications, and Internet connectivity) and their leverage in educational improvement;
  • supporting integration of strategic thinking about the curriculum and technology services;
  • preparing for accreditation;
  • helping faculty, departments, or institutions compare their uses of technology and outcomes; and
  • redesigning student evaluations of faculty.

For example, faculty in a department might collaborate to develop a standardized survey for their students (with some optional sections or questions for each course). They could also use the CSI to design a regular program of group interviews with students, to ask questions that are too open-ended to address with surveys. Over time, the information could help the department see whether its changing uses of technology for students on- and off-campus are helping (or interfering with) active problem-solving, student-faculty interaction, student-student collaboration, student time on task, and other key quality-related activities. This information will be valuable on its own and can also help explain why learning outcomes (as assessed by other means) are or are not improving. If there are no other means of comparing learning outcomes, then a finding that teaching and learning practices are improving could stand in as a measure of quality.

The faculty can also use their survey and interviews to monitor changes in other issues of interest (e.g., quality of training and support for technology and the extent to which students are using certain technologies in other courses, in their jobs, and in their leisure time). This kind of information, and the continuing discussion about the research tools and findings, can help faculty collectively improve their instructional program and teaching methods. The same findings can also show an accreditor how the department is using data to improve instruction and use technology more cost-effectively.

The CSI and its accompanying handbook are the first components of Flashlight to be released. Under development are a guide to cost modeling and an inventory of questions for gathering information from faculty. Other instruments are also planned.

The Flashlight Approach

The Flashlight approach to asking questions is based on two basic premises drawn from research on technology and learning.

First, technology per se does not determine learning outcomes (Educational Technology..., 1994). Rather, learning outcomes are influenced by the choices that faculty, students, and others make about the organization of teaching and learning, including their practices, choices about content, and so on (Ehrmann, March/April 1995). The role of modern technology is to expand the available choices (this is part of what we mean when we say that it is 'empowering'). Technology also makes some of those choices more feasible in certain circumstances. For example, all forms of presentation (e.g., lectures in a lecture hall with slides, lectures in a lecture hall with computer "overheads," reading the lecture material in a book, hearing the lecture one-on-one in a faculty office, reading the lecture material on the World Wide Web) have the same learning outcomes, but some of those technologies make it possible to reach people at a distance, some are more flexible for last minute changes, some are more adaptable to highly visual materials, and so on. For this reason the CSI has separate families of questions about teaching and learning practices (especially their prevalence), technology per se (e.g., how reliable or unreliable has it been), and most importantly, student reports on how frequently they use a specific technology to support a specific teaching and learning practice, and how useful and/or problematic they judge the technology to be when used for that purpose.

For example, Flashlight questions can be used to ask students how often they use the local e-mail system to do homework with other students, and what advantages and disadvantages it has relative to other means of working together.

A second premise underlying Flashlight's design: it is difficult and sometimes impossible to evaluate local uses of technology by comparing learning outcomes. There are many reasons for this, not the least of which is that the use of technology often involves changes in the goals of the program and thus in how student learning is assessed. Fortunately, research indicates that some teaching and learning practices tend to cause improvements in learning outcomes (e.g., increased time on task; student-student collaboration).

For that reason, Flashlight focuses many of its questions on such research-based principles of good practice. If Flashlight data indicates that good practices are more prevalent today than yesterday (or in a comparison group) and technology is judged to be useful in implementing those practices to this degree, then a case can be made that technology has probably helped improve learning outcomes, even in situations where such improvements cannot be directly observed.

When improvements in learning outcomes can be directly observed, Flashlight data can help indicate whether technology use played any role in that improvement (e.g., Brown, 1997).

Note: A table is also available which illustrates the ways in which Flashlight and the CSI compare with the components of the vision of higher education improvement and transformation summarized earlier.

What's Been Discovered So Far

Flashlight went into distribution in December 1997, so the only findings thus far are a side benefit of the beta test process. Early studies using the beta version of Flashlight uncovered instances where technology was not being used in ways that altered teaching or learning practices. Gary Brown of Washington State University (1997) reports on a lecturer who continued to lecture, but with better "slides" for example. Harrington (1997) reports on sections taught in computer networked classrooms that were encouraged to be taught in ways quite similar to sections taught in traditional classrooms; faculty wanted to be fair to students in the traditional classrooms. In such circumstances it is not likely that long-term learning outcomes improved.

On the other hand, Gary Brown used Flashlight to study a special seminar for at-risk students. He documented an improvement in freshman GPA and, with Flashlight, found evidence suggesting that technology use had helped implement superlative teaching and learning practices in these seminars, providing a plausible explanation for the improvement in grades.

Even these early uses of the beta version of Flashlight have had an impact on practice. Brown reports that "[t]he expansion of the Freshman Seminar program can be partially attributed to Flashlight findings from the pilot courses. The Flashlight report helped persuade key members of the Faculty Senate to support the new program, which passed by just a few votes."

Brown's findings strengthened prior research in the field that showed that the use of multimedia is unlikely to improve outcomes if it is not used to improve teaching and learning practices. As he noted,

A lecture, with or without multimedia, does not purport to help students manage large complex tasks, exercise creativity, or even discuss content with peers or experts inside or outside of educational institution... WSU, encouraged by findings surfaced and articulated through the use of the Flashlight CSI, has begun to reexamine some of our general education course objectives.

He goes on to say that "Several instructors who previously have used multimedia-enhanced lectures are introducing, incrementally as access to resources permits, more interactive and generative activities into their classes" (Brown, 1997).

There were practical implications from other Flashlight findings, as Brown reports:

For instance, the number of students who used the Internet to discuss subject matter beyond the classroom walls pleasantly surprised administrators of the Freshman Seminar program. Consequently, the revised syllabus now includes a project developed with partners in other WSU colleges and at other institutions... Further, findings from the Flashlight survey have helped to garner a $10,000 grant to develop a template version of [that] model (Brown, 1997).

Other benefits are more subtle. One of the original objectives of Flashlight was to provide the kind of information that would make it easier for educators to think strategically, and that seems to be happening. President Pamela Pease of the online International University writes that "International University has found the Flashlight Project a process that has engaged the entire institution toward its goal of improving student services, and teaching and learning within a 'virtual' learning environment" (personal communication, March 17, 1998). Tony Fiddes of Qantas Airways is preparing to evaluate their Web-based internal training system, Qantas College Online. He reports that Flashlight is "prompting managers to think more strategically about training and how they manage it—moving much more to the notion of having to discuss and manage individual training plans and looking at expected behavioural changes" (personal communication, March 9, 1998).

Flashlight use also seems to help individual faculty in thinking about instructional design and teaching issues. In describing her uses of Flashlight in working with faculty members at Indiana University-Purdue University Indianapolis (IUPUI), Elizabeth Rubens says,

For me, the most powerful effect of Flashlight is that it confronts faculty with a new set of criteria with which to judge the effectiveness of their teaching. After reviewing the items, faculty begin to think how they might move beyond simply packaging lectures via a new delivery mechanism and begin to help students structure experiences which promote the principles of good practice. Seeing the yardstick change from simple exam scores to thinking about student collaboration and creativity is a powerful motivator to promote curricular change (Personal communication, March 16, 1998).

Flashlight's History, Home, and Benefactors

Flashlight was created by the Annenberg/CPB Project and is now headquartered at the TLT Group (the Teaching, Learning, and Technology Affiliate of the American Association for Higher Education). Flashlight planning was supported by the Fund for the Improvement of Postsecondary Education (FIPSE). Development is being carried out by the Western Cooperative for Educational Telecommunications of WICHE and Indiana University-Purdue University Indianapolis (IUPUI). Other participating institutions are the Education Network of Maine, Maricopa Community Colleges, Rochester Institute of Technology, and Washington State University. The Flashlight Project is directed by Stephen C. Ehrmann, Ph.D., vice president of the TLT Group. The associate director of the Flashlight Project is Robin Etter Zuniga of the Western Cooperative for Educational Telecommunications.

Learning More about Flashlight

The best starting place on the Web to learn about Flashlight at the TLT Group's Web site. Sample Flashlight questions and a more detailed description of the Current Student Inventory are posted at the Web site of the Western Cooperative for Educational Telecommunications.

Flashlight has a free electronic newsletter that publishes information about workshops, availability of tools, and so on. To subscribe, send e-mail to listproc@listproc.wsu.edu with the one line message "SUBSCRIBE F-LIGHT your name".

If your institution, consortium, or system is interested in site licensing the Flashlight Current Student Inventory, obtaining a video (NTSC) about the project, sponsoring a talk or workshop about evaluation, technology and the Flashlight project, getting online training about evaluation and Flashlight, or obtaining consulting services from the Flashlight team, please contact the TLT group.


Brown, G. (1997). Flashlight at Washington State University. Pullman, WA: Washington State University Center for Teaching and Learning. Retrieved March 16, 1998 from the World Wide Web: http:www.ctl.wsu.edu/research/top-papers/flcases.htm.

Chickering, A. & Ehrmann, S. C. (1996). Implementing the seven principles: Technology as lever. AAHE Bulletin, October 1996, pp. 3-6. Retrieved March 15, 1998 from the World Wide Web: http://www.aahe.org/technology/ehrmann.htm.

Educational Technology Research and Development (1994). XLII: 2/3. Washington, DC: Association for Educational Communications and Technology.

Ehrmann, S. C., (March/April 1995). Asking the right questions: What does research tell us about technology and higher learning?. Change: The Magazine of Higher Learning, 27 (2), 20-27. Retrieved March 15, 1998 from the World Wide Web: http://www.learner.org/edtech/rscheval/rightquestion.html

Ehrmann, S. C. (July/August 1997). The Flashlight Project: spotting an elephant in the dark. Assessment Update, 9 (4), pp. 3, 10-11, 13. Retrieved March 16, 1998 from the World Wide Web: http://www.aahe.org/technology/elephant.htm

Ehrmann, S. C. & Zuniga, R. E. (1997). The Flashlight evaluation handbook, including the Flashlight Current Student Inventory, Washington, DC: The TLT Group.

Harrington, S. (1997). Case study: An introductory writing sequence at IUPUI: Investigation as the basis for change. In S. C. Ehrmann, S. C. and R. E. Zuniga, The Flashlight evaluation handbook, including the Flashlight Current Student Inventory. (pp. 2-41--2-52). Washington, DC: The TLT Group.

Morris, P., Ehrmann, S. C., Goldsmith, R., Howat, K. & Kumar, V. (1994). Valuable, viable software in education: Cases and analysis. New York: Primis Division of McGraw-Hill.

shooter gamescard gameshidden object gamespc game downloadsmarble popper gamesdownloadable gamestime management gamespc games
View Related Articles >