Tuesday, March 17, 2009

Including Surveys with Courses

The most difficult part of online surveys is that they are so optional. If the survey is part of a required course, response rates can be raised as long as the end-users know that their survey responses are also required, and that there will be consequences of not answering (or rewards for answering).

One trick to get assessment data from courses with tests is to mix the survey questions in with the required test page(s). That is, if some of the questions on a test are survey questions (there is no "correct" answer), but the other questions are graded, there will be a significantly higher response rate to those questions. If your test/survey software only provides one question at a time, this feature can be bypassed by the end-users. Software like the ReadyGo Server Side Testing module can track pages with combinations of test and survey questions, and therefore get higher response rates.

Another trick is to make survey responses a requirement to be able to take the final exam in a required course. Incentives such as a raffle for survey participants provide a more positive form of invitation to take the survey.

Thursday, March 12, 2009

Assuring security when giving tests

The requirements of online testing and absolute security/verification are generally incompatible. As noted, the user and their assistant simply need to be at the terminal at the same time in order to cheat the system. If absolute verification that the individual is responsible for every answer is required, the only guaranteed solution is proctored testing with periodic checks using a government issued photo-identification card.
If the user is trying to cheat the on-line system, there are many ways they can bypass all the security measures proposed by the various vendors:
1. The registered user and their assistant can both be at the terminal/computer taking the test. Any biometric or password-based verification can be completed by the real user, while at the same time their assistant provides them the answers.
2. The registered user can share their password with their assistant, and thus any password-based system can be fooled.
3. A camera taking pictures of the end-user's environment will not pick up the blue-tooth headset through which they are being fed the answers.

Instead of trying to increase security, the approaches we have found to work well include:
1. Give the required questions multiple times throughout the course rather than just at the end. For example, after a page in the middle of a chapter, give a short quiz with the required question. Feed the answer to the student if they got it wrong. At the end of the chapter, repeat the question. Finally, in the final exam, give the question. Make all tests required, but only the last one counts. If, by the third attempt, the user doesn't answer correctly, you have a bigger problem with that employee, and alternative remediation will be necessary.
2. Require the user/employee to sign and send in an affidavit that their responses were their own work. This provides legal/regulatory compliance that the organization has done their due diligence.

Most end-users (especially those who cheat) will opt for the easier alternative. If the tests are comprehensive, but easier (and less annoying) than the effort to cheat, they will generally choose to just take the test. We have seen that a 45 minute PowerPoint presentation followed by a required test is a really bad way to present content. Users walk away until the automated part is finished, and then come back to just take the test. Instead, make it so that the user can navigate anywhere in the course. If they fail an exam, send them back to the content so that they have to pass the material.

If you make a reasonable effort, you should be able to satisfy the regulatory requirements. If requirements cannot be met this way, consider proctored examinations in a controlled environment.