Adaptive Quiz Plugin

Adaptive Quiz

The Adaptive Quiz is a new activity module that allows instructors to administer an adaptive style quiz. As the student attempts the quiz an algorithm uses a formula to choose questions and estimate the students ability within a certain amount of error. When adding/editing the adaptive quiz, instructors can define the activity parameters that will be used by the algorithm while the student is attempting the activity.

The adaptive quiz requires the use of both question bank categories and tags, in order to determine the questions that will be displayed during an individual user's attempt. Tags are used to assign the questions difficulty levels. Then categories of questions can be assigned to the adaptive quiz question pool.

The activity parameters allow the instructor to determine the difficulty levels of questions available for the quiz. A starting level is selected for the first question and then the algorithm selects the next difficulty level for a question to be chosen from. If the student answers a question correctly, a more challenging question is chosen next. If the student answers a question incorrectly, a less challenging question is chosen next.

During the start of an attempt the number of questions is undetermined. The amount of questions presented will be between a minimum and maximum number of questions set in the activity parameters. After the minimum number of questions have been answered, the quiz will stop when the user's ability level is determined to the required accuracy or the quiz stops when the maximum number of questions have been answered.

Adaptive Quiz Permissions

The Adaptive Quiz permissions are assigned to user via roles. To add/edit a role go to the Settings block > Site Administration > Users > Permissions > Define roles. To edit the permissions for an existing role select the edit icon or add a new role by selecting the Add a new role button at the bottom of the page.

The 3 Adaptive Quiz permissions are:

1. Add a new adaptive quiz: Create a role with this permission enabled to allow users to add a new adaptive quiz. For example, allowing this permission for a teacher role will give teachers the capability to create adaptive quizzes in their courses.
2. Review adaptive quiz submissions: This permission is not being used. The capabilities for this permission were added to the next permission, "View adaptive quiz reports".
   
3. View adaptive quiz reports: This permission gives users the capabilities to view adaptive quiz reports and to view quiz submissions.
   

Creating Difficulty Levels for Questions

The Adaptive Quiz uses questions that have a difficulty level. In order to assign questions a difficulty level tags must be created and assigned to the questions. A separate tag must be created for each difficulty level. Each tag can be used by multiple questions.

Users must be assigned a system level role with manage tags permissions allowed in order to add tags to the site. For example, a user assigned a teacher role in a course will not be able to create tags, but will be able to assign tags to questions.

To add a tag go to the Settings block > Site Administration > Appearance > Manage Tags. The tag name needs to be entered in a specific format to work with the adaptive quiz. Enter "adpq_difficulty level" for the tag name, the "difficulty level" should be a number between 1-999. For example, in the following image a tag is being created:

1. A tag name is entered using the format described above "adpq_200".
2. Select the Add official tags button to create the tag.
   

Official tags can be changed to Default tags by selecting the tag name and then select the Edit this tag link. Note that if you change an official tag to a default tag, the default tag will be deleted if no questions are assigned to it.

Once the tags are created they can be assigned to questions.

Assigning Tags to Questions

Next the tags can be assigned to questions to give the questions a difficulty level. The adaptive quiz will use the question difficulty levels to determine which questions can be included in the quiz. To demonstrate I will login as a teacher and edit a question in one of their courses.

To create or edit questions in a course go to the Settings block > Course Administration > Question bank > Questions. From the questions screen you can:

1. Select the category that you will add questions to or that you will edit questions from.
   
2. Select the Create a new question button to create a new question.
3. Select the Edit icon of an existing question to edit the question.

On the question create/edit screen the tags section is located at the bottom of the screen. There are two sections for tags, Official tags and Other tags.

1. Official tags can be selected from the multi-select drop down box. All the official tags will be listed, use the scroll bar to view the entire list. To assign an official tag to the question click on it, selected tags will be highlighted. To select multiple official tags for a question use Shift+Select or Control+Select.
   
2. Other tags is where default tags can be entered. Enter tag names separated by commas, e.g., adpq_201,adpq_202. There can be no spelling errors in this section or the tag assignment will not work.

The following image is showing the Tags section for a question. The official tag "adpq_101" has been assigned to the question.

Question Types

Questions used in the adaptive quiz must:

• be automatically scored as correct/incorrect
• be tagged with their difficulty that is within the range for the quiz
  

For questions that are setup in a way where some answers are partially correct, the algorithm will treat any partially correct answer as a correct answer. For example, if you have question where all five answers must be selected for full credit, the adaptive quiz algorithm will count the answer correct with only one answer selected.

The quiz will only count answers correct if they can be automatically graded. For example, the answer for an essay question will be seen as incorrect by the algorithm.

Adding an Adaptive Quiz

To add an Adaptive Quiz turn editing on and select the Add an activity or resource link.



Select the adaptive quiz from the list of activities popup window.



Adding the adaptive quiz:

1. Name and description: Add a name, the description is optional.
2. Display description on course page: Select this box to include the quiz description on the course page.
3. Attempts allowed: Set the attempts allowed, unlimited is the default. This determines the number of times a student can re-attempt the activity instance. If the student's attempts exceed the attempts allowed, they will no longer be able to take the test.
   
4. Required password: Enter a password for the quiz, this is an optional setting. Entering a password requires that students enter the password in order to attempt the test.
   
5. Browser security: Select yes/no for browser security. If enabled, the attempt will be displayed in a full screen popup using JavaScript. The student must be using a browser with JavaScript enabled.
   

6. Question pool: Select the question category(ies) where the activity will pull questions from during an attempt. To be included in the quiz, the questions must be tagged with a difficulty level between the lowest and highest level of difficulty set for the quiz. This is a multi-select field.
   
7. Starting level of difficulty: At the beginning of an attempt the activity will display a question with this difficulty level. A question must exist in the starting level otherwise the attempt will end automatically.
   
8. Lowest level of difficulty: During an attempt the activity will not display any questions with a difficulty level below this level.
   
9. Highest level of difficulty: During an attempt the activity will not display any questions with a difficulty level above this level.

10. Minimum number of questions: Enter the minimum number of questions the user must answer before the attempt can be stopped/completed. After the minimum number of questions have been answered the adaptive algorithm will stop the quiz when the acceptable amount of error has been achieved or when the maximum number of questions have been answered.
11. Maximum number of questions: Enter the maximum number of questions a user can answer during an attempt.
12. Standard error to stop: When the amount of error in the measure of the user's ability drops below this amount, the test will stop. Tune this value from the default value of 5% to require more or less precision in the ability measure. For a value of 5%, enter 5 in this field.
    
13. Group mode: Standard Moodle group visibility functionality.
14. Visible: Standard Moodle visibility functionality.
    
15. ID number: A name or number to identify the activity instance, in addition to the name field.
    

Select the one of the Save buttons when the form is complete.

Attempting an Adaptive Quiz

The student opens the quiz by selecting the quiz link from the course home page.



Then select the Start attempt button to start the quiz.



When the student starts the quiz the first question will have the difficulty level specified when adding the quiz, which is level 75 in this instance.



The student answered the previous question correctly so the quiz uses a formula to calculate the next level of difficulty a question should be chosen from. In this instance it is from level 98.



The level 98 question is answered incorrectly and then a level 87 question is selected. This continues until the quiz is completed. There are two ways to complete the quiz.

• After the minimum number of questions have been answered, the quiz will stop when the user's ability level is determined to the required accuracy. For this quiz the standard error that must be achieved is 10, which means the user's ability level or score must be accurate to within +- 10%.
  
• If the standard error isn't achieved, the quiz stops when the maximum number of questions have been answered. For this quiz the maximum number of questions that can be answered is 30.
  

Reports

There are two reports for the Adaptive Quiz:

• Attempts report
• Individual user attempts report
  

To view the Adaptive Quiz reports select the link to the adaptive quiz from the course home page.



Select the View report button to view the Attempts report.



The Attempts report has 3 columns, which can all be used to sort the list:

1. Name: The name links to the user's profile screen.
   
2. Number of attempts: The number of attempts the user has are listed here. Selecting this number links to the Individual user attempts report.
3. Score: This column shows the score and margin of error. The score is a number between the lowest and highest difficulty level of questions allowed for the quiz. This quiz used questions with difficulty levels between 1-40. If multiple attempts have been made by the user, the score of the first attempt will be listed in the Attempts report. Details of all the attempts can be viewed in the Individual user attempts report.
   

The Individual user attempts report has more detailed information about a user's attempts. The report includes:

1. State of attempt: For this user one attempt has been completed.
2. Reason for stopping the attempt: The quiz attempt was stopped because the standard error was within the limits imposed by the activity after answering 28 questions.
3. Sum of questions attempted: Amount of questions answered for the attempt.
4. Score: This column shows the score and margin of error. The score is a number between the lowest and highest difficulty level of questions allowed for the quiz. This quiz used questions with difficulty levels between 1-40.
   
5. Attempt start time
6. Attempt finish time
7. Review Attempt / Delete Attempt: Select the review attempt link to review the quiz taken by the user. The review screen includes a chart and detailed information about each question answered. Select the delete attempt link to delete the attempt from the reports.

Selecting the Review Attempt link, shown by (7) in the previous image, displays the following report.




There is also information about adaptive quiz attempts in the Recent activity block.