Computer-Based Testing and the Internet: Issues and Advances - Softcover

Softcover

ISBN 10: 047001721X ISBN 13: 9780470017210

Publisher: Wiley, 2005

Synopsis

Internationally there is a growing number of professionals administering psychological tests by computer. Computer Based Testing and The Internet highlights four main themes that define current issues, technical advances and applications: advances in computer based testing, including test designs; operational issues, including security and legal issues; new uses for tests in employment and credentialing; and future of computer based testing. Each contributor addresses issues of control, quality, security and technology within the subject matter of their particular chapters providing an excellent overview.

About this title

About the Author

Dave Bartram is Past President of the International Test commission and is heading ITC projects on international guidelines for standards in test use and standards for computer-based testing and the Internet. He is Chair of the British Psychological Society's Steering Committee on Test Standards and Convenor of the European Federation of Psychologists' Associations Standing Committee on Tests and Testing. He is President-Elect of the IAAP's Division 2.

Professor Bartram is Research Director for SHL Group plc. Prior to his appointment with SHL in 1998, he was Dean of the Faculty of Science and the Environment, and Professor of Psychology in the Department of Psychology at the University of Hull. He is a Chartered Occupational Psychologist, a Fellow of the British Psychological Society (BPS) and a Fellow of the Ergonomics Society. In 2004 he received the BPS award for Distinguished Contributions to Professional Psychology. His specialist area is computer-based testing and Internet assessment systems. Within SHL he is leading the development of their next generation of Internet-based delivery systems and the development of a multi-dimensional generic Competency Framework.

He has published large numbers of popular, professional and academic articles and book chapters, and has been the Senior Editor of the BPS Test Reviews. He has been an editor or co-author of several works including the 1992, 1995 and 1997 BPS Reviews of Psychometric Tests; Organisational Effectiveness: the Role of Psychology (with Ivan Robertson and Militza Callinan, published in 2002 by Wiley)and the BPS Open Learning Programme for Level A (Occupational) Test Use (with Pat Lindley, published by BPS Blackwell in 1994.

Ronald K. Hambleton holds the title of Distinguished University Professor and is Chairperson of the research and Evaluation Methods Program and Executive Director of the Center for Educational Assessment at the University of Massachusetts, Amherst, in the United States. He earned a B.A. in 1966 from the University of Waterloo in Canada with majors in mathematics and psychology, and an M.A. in 1967 and Ph.D. in 1969 from the University of Toronto with specialties in psychometric methods and statistics. Professor Hambleton teaches graduate-level courses in educational and psychological testing, item response theory and applications, and classical test theory models and methods, and offers seminar courses on applied measurement topics. He is co-author of several textbooks including (with H. Swaminathan and H. Jane Rogers) Fundamentals of Item Response Theory (published by Sage in 1991) and Item response Theory: Principles and App0licaitons (published by Kluwer in 1985), and co-editor of several books including International Perspectives on Academic Assessment ( with Thomas Oakland, published by Kluwer in 1995), Handbook of Modern Item response Theory ( with Wim van der Linden, published by Springer in 1997) and Adaptation of Educational and Psychological Tests for Cross-Cultural Assessment ( with Peter Merenda and Charles Spielberger, Published by Earlbaum in 2005). His research interests are in the areas of item response model applications to educational achievement and credentialing exams, standard-setting, test adaptation methodology, score reporting and computer-base testing. he has received several honors and awards for his more than 35 years of measurement research including honorary doctorates from Umea University in Sweden and the University of Oviedo in Spain, the 1994, National Council on Measurement in Education Career Award, the 2003Association of Test Publisher National Award for Contributions to Computer-Based Testing, and the 2005 E.F. Lindquist Award for Contributions to Assessment. Professor Hambleton is a frequent consultant to state department  of education, national government agencies  and credentialing organizations.

From the Back Cover

No topic is more central to innovation and current practice in testing and assessment today than computers and the Internet. This timely publication highlights for main themes that define current issues, technical advances and applications of computer-based testing.

• Advances in computer-based testing-new test designs, item selection algorithms, exposure control issues and methods, and new tests that capitalize on the power of computer technology.
• Operation issues-systems design, test security, and legal and ethical matters.
• New and improved uses-for tests in employment and credentialing.
• The future of computer-based testing-identifying potential issues, developments, major advances and problems to overcome.

Written by internationally recognized contributors, each chapter focuses on issues of control, quality, security and technology. These issues provide the basic structure for the International Test Commission's new Guidelines on Computer-Based Testing and Testing on the Internet. the contributions to this book have played a key role in the development of these guidelines.

Computer-Based Testing and the Internet is a comprehensive guide for all professionals, academics and practitioners working in the fields of education, credentialing, personnel testing and organizational assessment. It will also be of value of students developing expertise in these areas.

From the Inside Flap

No topic is more central to innovation and current practice in testing and assessment today than computers and the Internet. This timely publication highlights for main themes that define current issues, technical advances and applications of computer-based testing.

• Advances in computer-based testing-new test designs, item selection algorithms, exposure control issues and methods, and new tests that capitalize on the power of computer technology.
• Operation issues-systems design, test security, and legal and ethical matters.
• New and improved uses-for tests in employment and  credentialing.
• The future of computer-based testing-identifying potential issues, developments, major advances and problems to overcome.

Written by internationally recognized contributors, each chapter focuses on issues of control, quality, security and technology. These issues provide the basic structure for the International Test Commission's new Guidelines on Computer-Based Testing and Testing on the Internet. the contributions to this book have played a key role in the development of these guidelines.

Computer-Based Testing and the internet is a comprehensive guide for all professionals, academics and practitioners working in the fields of education, credentialing, personnel testing and organizational assessment. It will also be of value of students developing expertise in these areas.

Excerpt. © Reprinted by permission. All rights reserved.

Computer-Based Testing and the Internet

Issues and AdvancesBy Dave Bartram

John Wiley & Sons, (UK)

Copyright © 2005 Dave Bartram
All right reserved.
ISBN: 9780470017210

Chapter One

Testing on the Internet: Issues, Challenges and Opportunities in the Field of Occupational Assessment

Dave Bartram SHL Group plc, UK

This chapter starts by considering what the Internet is and what it can it offer in relation to testing and assessment in the work and organisational field. It then goes on to take a look into the future and consider a range of practical and good practice issues. In considering testing on the Internet we need to consider both the technical strengths and weaknesses of the Internet itself (as a transport medium) and the limitations that the WWW technology imposes on the design of tests and control over their delivery. Throughout the chapter, the emphasis will be on the use of computer-based and web-based testing in the field of occupational assessment. Other chapters in this volume consider applications in other fields, notably educational testing and testing for licensing and certification.

COMPUTER-BASED TESTING (CBT) BEFORE THE INTERNET

The main value of CBT historically has been in the area of report generation. Some of the earliest systems (back in the days before personal computers) were designed to automate the scoring and interpretation of instruments such as the MMPI. With the advent of the personal computer, we saw the development of computer-administered versions of paper and pencil tests. These provided some advantages over paper and pencil, in terms of control of administration, and some disadvantages (e.g. the need for sufficient hardware to test groups of people). They also raised the question of equivalence with their paper and pencil counterparts. Most research (see Bartram, 2005; Mead & Drasgow, 1993) has tended to show that equivalence was not a major problem so long as the tests were not speeded measures of ability.

Bartram (1997) commented on the fact that, despite the potential offered by technology for novel forms of assessment, the literature on computer-based assessment (CBA) within occupational assessment settings has been largely confined to a small number of issues. These have been dominated by the issues relating to the parallel use of computer-based and paper-based versions of the same tests and use of computers to generate descriptive and interpretative reports of test results (for reviews, see Bartram and Bayliss, 1984; Bartram, 1987b, 1989, 1993, 1994, 2005).

INNOVATION IN COMPUTER-BASED TESTING

Despite the increasing sophistication of computer-based assessment systems, within the field of occupational assessment the tests they contain are, typically, computer implementations of old paper-and-pencil tests. Nevertheless, there has been innovation in the field and the consequences of that innovation are increasingly finding their way into commercial practice. Tests can be innovative in a number of different ways. The most obvious is where the actual test content is innovative. However, innovation can also occur in less obvious ways. The process used to construct the test may be innovative and rely on computer technology and the nature of the scoring of the items may be innovative. In practice there is an interaction between these different aspects of innovation, in that some of the most interesting developments in test content also involve innovation in how that content is created.

For computer-based testing, the most obvious examples of content innovation can be found where tests use sound or video to create multi-media items. Drasgow, Olson-Buchanan and Moberg (1999) describe a full-motion interactive video assessment, which uses video clips followed by multiple choice questions. Simulations can also be run on computer to provide realistic work-sample assessments. Hanson, Borman, Mogilka, Manning, and Hedge (1999), for example, describe the development of a computer-based performance measure for air traffic control selection and Bartram (Bartram & Dale, 1983; Bartram, 1987a) describes the use of a simplified landing simulator for use in pilot selection.

Innovation in content also relates to the use of more dynamic item types, for example, where drag-and-drop or other familiar Windows-based operations are used rather than the simple point-and-click simulation of paper-and-pencil multiple-response. A review of this area of innovation in item types is presented by Drasgow and Mattern in Chapter 3.

Innovation in content, however, is often associated with novel methods of content generation. Item generation techniques have provided the potential for a whole host of new item types as well as more efficient production of conventional items. Bartram (2002a), and Goeters and Lorenz (2002) describe the use of generative techniques to develop a wide range of task-based and item-based tests for use in pilot selection. It is worth noting, however, that most of the developments in this area of innovation have occurred in areas where selection leads to very high cost training or into high risk occupations or both (as is the case for trainee military pilot selection). Innovation is expensive, and the sort of tests described in the papers referred to above have required extensive research and development programmes. However, as in all areas of testing, the lessons learned from this work will result in benefits in due course for the general field of occupational assessment.

Computer software provides for the recording of very detailed information about a test-taker's performance. In addition to the response given to an item, we can record how long the person took to respond. We can also record information about choices made and changed during the process of responding. For more complex item types we can track the performance of the person as they work their way through a task or series of subtasks. Bartram (2002b) reported validation data from the use of a set of computer-based ability tests that were administered without any time limit. These were designed for use in a diagnostic mode for people entering further education training courses. Time to respond was normed independently for each item and response latency was scored together with accuracy to produce a measure of efficiency. This efficiency score had higher validity than the traditional number correct score.

While there has been some experimentation in the use of response latency data for checking response stability (Fekken & Jackson, 1988) and socially desirable responding (Holden & Fekken, 1988; George & Skinner, 1990a, 1990b), these approaches have not been developed into practical applications for general use in personnel selection or other areas of I/O assessment as yet.

Item Response Theory (IRT) has been with us since the 1980s (Lord, 1980), but its application has tended to be confined to educational and some large-scale occupational uses. It has not been generally applied in the area of occupational assessment until relatively recently. IRT has the considerable advantage of approaching test construction from the item level. Its application to routine occupational assessment has become possible with the advent of better data collection and data management procedures. IRT has many advantages over traditional methods; however it also comes with some costs: the need for larger samples of data with which to determine the properties of items. Although computer technology has provided the possibility of implementing adaptive testing using (in particular where it is based on IRT models) the impact of this on general test practice has been slight in the occupational field. The main reason why traditional classical fixed-item-set tests have held sway for so long has been one of infrastructure. Neither paper and pencil nor PC-based testing is well suited to adaptive testing and the use of large item banks. The Internet potentially changes all of this. There are clear signs that attitudes to CBA are changing as people come to appreciate the real benefits of technology for assessment, and as the technological infrastructure needed to support these applications becomes increasingly ubiquitous.

Development of Computer-Based Testing and Growth of the Internet

The use of computer-based testing is increasing rapidly. It has been helped not only by the development of better interfaces, but by the dramatic increases in volume of and accessibility to hardware. Access to the Internet is now available to you in your home for a few hundred pounds of capital outlay. In addition we have seen the advent of email and restricted Internet services on digital TV systems. The new millennium heralded the appearance of the first generation of WAP mobile phones, with their ability to access the Internet in a wireless environment and the promise of broadband 3G systems becoming available in the next few years. However, the pattern of development is not uniform around the world. Even where the technology is present, some users are more conservative than others in their adoption of that technology.

Computer networks have existed for a long time. The first use of a hyperlinked network by the US military occurred in 1957. Academic institutions in the UK joined in 1973 when University College London set up the first connection. The first commercial UK IP network was set up in 1989. At the start of the 1990s, Tim Berners Lee proposed the idea of using a standard graphical browser and a communication standard to provide access to data from any source, and so 'invented' the World-Wide Web (WWW). The Mosaic browser, the first of the WWW browsers, appeared in 1992. In 1994, Netscape was founded and a year later, Microsoft embraced the Internet, having previously dismissed it.

In many ways we can look on 1995 as the real beginning of widespread use of the Internet, the time at which it started to become part of the fabric of many people's everyday lives. In the few years since then, the range of applications and volume of use have mushroomed. For all practical purposes, while the potential of the Internet has been known for many years, it has only just reached the stage of development at which that potential can begin to be realised. We are now at a significant watershed in its development for a number of reasons.

Within North America, Europe and Asia-Pacific, we now have widespread availability of inexpensive, high-powered computer systems. As the hardware has become more widespread, so the range of service providers has increased. Now it is as easy to get onto the net as it is to have a phone installed. Indeed, wherever a phone or a cable TV connection has been installed, an Internet connection can be made. Once on the net, you have access to information and services that were previously restricted to expert users or specialists. You can be your own travel agent; you can buy books and other goods from anywhere in the world; you can consult experts, read government reports, or find a new job.

The convergence towards common standards has made it commercially viable for service providers to offer users more and more sophisticated applications. The advances in technology have provided us with standard features we would hardly have dreamt of a few years ago: minimum screen resolutions of 1024 x 768 and 32-bit colour resolution; real-time animation, video and sound capabilities; multi-tasking and so on.

Finally, we have also witnessed an increase in reliability. This is the key to the use of computers in testing. Though computer systems are still prone to crashes, hang-ups and network failures, we are moving rapidly closer to the point where the user expectation is that computers should operate reliably.

Tests and documents are essentially 'soft' products. As such they can be downloaded over the Internet to users. This means that the Internet can be used as a complete commercial solution for test publishers. There is no longer any need for printing and production, warehousing and postal delivery services.

More significant for testing, however, is the shift in locus of control provided by the Internet from the 'client side' to 'server side'. For paper and pencil testing, publishers have had to provide users with test items, scoring keys and interpretation algorithms. As these are 'public', the danger of compromise and security breaches is high. Test users can (and do) pass these materials on to people who are not authorised to use them. All the test data also resides with the user. The process of developing norms, checking the performance of test items and carrying out validation studies is dependent upon costly procedures for recovering data from users. For the Internet this situation is reversed. The data and the intellectual property reside on the publisher's server. The user has access only to those parts of the process that they need.

Time for a Revolution!

The infrastructure is now being built to support a radical change in the way testing is done. We are seeing the widespread availability and acceptance of computers; standardisation on operating systems and interfaces and the growth of the Internet as the generic communication medium.

Ipsos-Reid in their annual 'Face of the Web' survey (reviewed by Pastore, 2001c) actually claim that the revolution is over and that we are entering a 'post-revolutionary' phase as growth of the Internet market starts to level off in most of the developed regions around the world. Global Internet population was estimated at around 350 million at the end of 2000. For 2005 it is estimated that it will be around 500 million. The US still has more people online than any other country, but its share of global users is shrinking. Western Europe plus the remainder of the English speaking world (UK, Australia, Canada and South Africa) now rivals the US as a bloc. In 2000, 65% of the population in Sweden used the Internet in a 30-day period, while in Canada it was 60%. This surpassed the US (59%), for which the percentage remained constant from 1999 to 2000. However, in a poll conducted in 2002 (Ipsos, 2002), the percentage of US Americans going online in a 30-day period had jumped to 72%. Very rapid growth was also noted in Korea, urban China, urban India, France, Germany and the UK (where usage increased from 35% in 2000 to 50% in 2002).

Awareness of the Internet is now almost universal in North America, Australia, Europe and Japan. While awareness levels are low in urban areas of China, India and Russia, they are increasing rapidly. The 2002 Ipsos study, for example showed an increase from 9 to 19% in urban India usage (over a 30-day period) and from 21 to 30% in urban China between 2000 and 2002. Internet use in Asia-Pacific is expected to surpass that in the US by 2005 (Pastore, 2001d, quoting an International Data Corp. study). Asia-Pacific (excluding Japan) is forecast to exceed 240 million users from the 2001 base of 64 million, with a fundamental shift in the position of China.

While predictions are always dangerous, by 2010 I believe that in the field of occupational assessment we will see computer-managed and computer-delivered assessment becoming the default and paper-and-pencil testing the exception.

WHAT CAN THE INTERNET OFFER NOW?

In this section some current areas of application of the Internet are considered, and illustrated. The applications covered include:

using the Internet to support test users

assessment for development

test practice and familiarisation

recruitment and selection

post-hire applications, including online 360-degree feedback and development.

Supporting Test Users

Most discussion of the Internet tends to focus around the delivery of tests to test takers online. However, the Internet also provides a new medium for distribution of test materials, reports and manuals, and for the automated collection of data. Even traditional paper and pencil materials can be delivered online as PDF format files using e-book publishing technologies.

Many publishers now use the Internet as a major source of support for test users. SHL, for example, delivers updates for its offline PC-based test systems over the Internet, together with norm updates and other technical data to support paper-and-pencil test users. As a medium, the Internet also provides a channel for users to communicate with each other (through user-groups) and with the publishers of materials. This communication is not just for traditional support purposes, but also provides the potential for supplier and client to have more of an ongoing dialogue through which the supplier can better target future research and development.

Self-Assessment and Practice

It is widely acknowledged (e.g. Kellett, 1991; Callan & Geary, 1994) that we should provide people who are to be assessed with the opportunity for practice, such that when the final assessment is taken they are all performing at or close to their asymptotic level. In the past, this has not been possible. Until recently, the only practical mechanism for giving people experience of testing has been through the use of practice tests or dissemination of information leaflets that provide examples of test items.

Continues...

Excerpted from Computer-Based Testing and the Internetby Dave Bartram Copyright © 2005 by Dave Bartram. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

Edition details

• Publisher: Wiley
• Publication date: 2005
• ISBN 10: 047001721X
• ISBN 13: 9780470017210
• Binding: Paperback
• Language: English
• Edition number: 1
• Number of pages: 272
• Editor: Bartram Dave, Hambleton Ron