{"id":974321,"date":"2022-01-25T04:46:17","date_gmt":"2022-01-25T09:46:17","guid":{"rendered":"https:\/\/www.prime-wow.com\/?p=974321"},"modified":"2022-01-25T04:46:17","modified_gmt":"2022-01-25T09:46:17","slug":"creating-better-online-multiple-choice-questions","status":"publish","type":"post","link":"https:\/\/www.prime-wow.com\/?p=974321","title":{"rendered":"Creating better online multiple choice questions"},"content":{"rendered":"<p>In this blog post we explore good practices around creating online computing questions, specifically multiple choice questions (MCQs). Multiple choice questions are a popular way to help teachers and learners work out the next steps in learning, and to assess learning in examinations. As a case study, we look at some data related to learner responses to <a href=\"https:\/\/www.raspberrypi.org\/blog\/supporting-teachers-students-remote-learning-free-video-lessons\/\" target=\"_blank\" rel=\"noreferrer noopener\">computing questions on the Oak National Academy platform<\/a>.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"500\" height=\"333\" src=\"https:\/\/www.prime-wow.com\/wp-content\/uploads\/2022\/01\/nguyen-dang-hoang-nhu-qDgTQOYk6B8-unsplash-500x333-1.jpg\" alt=\"Someone fills in a standardised test with multiple choice questions using a pencil.\" class=\"wp-image-78046\" \/><\/figure>\n<\/div>\n<p>The case study illustrates the many things MCQ authors have to think about while designing questions, and that there is much more research needed to understand how to get an MCQ \u201cjust right\u201d.<\/p>\n<h2>Uses of multiple choice questions<\/h2>\n<p>Online auto-marked MCQs are now being integrated into classroom activities, set as homework, and used in self-led learning at home. Software products involving MCQs, such as Kahoot and Socratic, are easy to use for many, and have become popular in some learning contexts. MCQ may have become more prevalent due to increased online teaching and the availability of whole curricula through platforms such as the Oak National Academy.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"500\" height=\"333\" src=\"https:\/\/www.prime-wow.com\/wp-content\/uploads\/2022\/01\/homeschooling-5121262_1920-500x333-1.jpg\" alt=\"A girl does school work at a laptop at home.\" class=\"wp-image-58723\" \/><\/figure>\n<\/div>\n<p><a href=\"https:\/\/www.gwern.net\/docs\/spaced-repetition\/2021-yang.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">An international group of researchers from China, Spain, Singapore, and the UK<\/a> recently looked into the reasons why MCQ-based testing might improve learning. Chunliang Yang and his co-authors concluded that there are three main ways that MCQ tests help learners learn:<\/p>\n<ul>\n<li>They provide learners with additional exposure to learning content<\/li>\n<li>They provide learners with content in the same format that they will be later assessed in\u00a0<\/li>\n<li>They motivate learners, e.g. to prompt them to commit more effort to learn in general<\/li>\n<\/ul>\n<h2>What does the research say about creating multiple choice questions?<\/h2>\n<p>In recent research reviewing the use of MCQs, Andrew Butler from Washington University in St Louis looked at the <a href=\"https:\/\/www.learningscientists.org\/blog\/2017\/10\/10-1\" target=\"_blank\" rel=\"noreferrer noopener\">effectiveness of MCQs in relation to learning, rather than assessment<\/a>. Andrew gives the following advice for educators creating MCQs for learning:<\/p>\n<ul>\n<li>Think about the thinking processes the learner will use when answering the question, and make sure the processes are productive for their learning<\/li>\n<li>Don\u2019t make the question super easy or too difficult, but make it challenging \u2014 the difficulty needs to be \u201cjust right\u201d<\/li>\n<li>Keep the phrasing of the question simple\u00a0<\/li>\n<li>Ensure that <strong>all<\/strong> answers are plausible; providing three or four answers is usually a good idea<\/li>\n<li>Be aware that if learners pick the wrong answer, this can reinforce the wrong thinking<\/li>\n<li>Provide corrective feedback to learners who pick the wrong answer<\/li>\n<\/ul>\n<p>What I find particularly interesting about Andrew\u2019s advice is the need to make the difficulty of the MCQ \u201cjust right\u201d for learners. But what does \u201cjust right\u201d look like in practice? More research is needed to work this out.<\/p>\n<h2>The anatomy of a multiple choice question<\/h2>\n<p>When talking about MCQs, there are technical terms to describe question features, e.g.:<\/p>\n<ul>\n<li>Incorrect answers are called <strong>distractors<\/strong> (or lures)<\/li>\n<li>A distractor is defined as <strong>plausible<\/strong> if it\u2019s an answer a layperson would see as a reasonable answer<\/li>\n<li>Plausible distractors are called <strong>working<\/strong> distractors<\/li>\n<\/ul>\n<p>Here at the Foundation, we created MCQs for the Oak National Academy when we adapted <a href=\"https:\/\/www.raspberrypi.org\/blog\/developing-progression-for-teaching-computing\/\" target=\"_blank\" rel=\"noreferrer noopener\">our Teach Computing Curriculum classroom materials<\/a> into <a href=\"https:\/\/www.raspberrypi.org\/blog\/supporting-teachers-students-remote-learning-free-video-lessons\/\" target=\"_blank\" rel=\"noreferrer noopener\">video lessons and accompanying home learning content<\/a> to support learners and teachers during school closures. Data about what questions are attempted on the Oak platform, and what answer options are chosen, is stored securely by Oak National Academy. The Oak team kindly provided us with four months of anonymous data related to responses to the MCQs in <a href=\"https:\/\/teachcomputing.org\/curriculum\/key-stage-4\/data-representations\" target=\"_blank\" rel=\"noreferrer noopener\">the \u2018GCSE Computer Science \u2013 Data representations\u2019 unit<\/a>.<\/p>\n<p>Over this period of four months, learners on the platform made more than 29,000 question attempts on the thirty-five questions across the nine lessons that make up this data representation unit. Here is a breakdown of the questions by topic area:<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-blog-entry\"><a href=\"https:\/\/www.raspberrypi.org\/app\/uploads\/2022\/01\/image3.png\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" loading=\"lazy\" width=\"800\" height=\"318\" src=\"https:\/\/www.prime-wow.com\/wp-content\/uploads\/2022\/01\/image3-800x318-1.png\" alt=\"Data about responses to a set of multiple choice questions on the Oak Academy platform.of a multiple choice question on the Oak Academy platform.\" class=\"wp-image-78027\" \/><\/a><figcaption>Responses to MCQs in the GCSE Computer Science data representation unit on Oak National Academy, data from February 2021 to end of May 2021 (click to enlarge)<\/figcaption><\/figure>\n<\/div>\n<p>As shown in the table, more questions relate to binary arithmetic than to any other topic area. This was a specific design decision, as it is well-known that learners need lots of practice of the processes involved in answering binary arithmetic questions.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-blog-entry\"><a href=\"https:\/\/www.raspberrypi.org\/app\/uploads\/2022\/01\/image2-1.png\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" loading=\"lazy\" width=\"800\" height=\"339\" src=\"https:\/\/www.prime-wow.com\/wp-content\/uploads\/2022\/01\/image2-1-800x339-1.png\" alt=\"Part of the graph of learning objectives for the Teach Computing Curriculum unit GCSE Computer Science data representation.\" class=\"wp-image-78029\" \/><\/a><figcaption>Part of the graph of learning objectives for the Teach Computing Curriculum unit <a href=\"https:\/\/teachcomputing.org\/curriculum\/key-stage-4\/data-representations\" target=\"_blank\" rel=\"noreferrer noopener\">GCSE Computer Science \u2014 Data representations<\/a> (click to enlarge)<\/figcaption><\/figure>\n<\/div>\n<p>Let\u2019s look at an example question from the binary arithmetic topic area, with one correct answer and two distractors. The learning objective being addressed with this question is \u2018Perform addition in binary on two binary numbers\u2019.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"500\" height=\"227\" src=\"https:\/\/www.prime-wow.com\/wp-content\/uploads\/2022\/01\/image1-2-500x227-1.png\" alt=\"Screenshot of a multiple choice question on the Oak Academy platform.\" class=\"wp-image-78026\" \/><figcaption>One of the MCQs in the GCSE Computer Science data representation unit on the Oak National Academy, as displayed on the online platform<\/figcaption><\/figure>\n<\/div>\n<p>As shown in the table below, in four months, 1170 attempts were made to answer the example question. 65% of the attempts were correct responses, and 35% were not, with 21% of responses being distractor b, and 14% distractor c. These distractors appear to be working distractors, as they were chosen by more than 5% of learners, which has been suggested as a rule-of-thumb threshold that distractors have to clear to be classed as working.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><a href=\"https:\/\/www.raspberrypi.org\/app\/uploads\/2022\/01\/image2.png\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" loading=\"lazy\" width=\"712\" height=\"166\" src=\"https:\/\/www.prime-wow.com\/wp-content\/uploads\/2022\/01\/image2.png\" alt=\"Data about responses to a multiple choice question on the Oak Academy platform.\" class=\"wp-image-78028\" \/><\/a><figcaption>Example MCQ in the GCSE Computer Science data representation unit on the Oak National Academy, plus response data from February 2021 to end of May 2021 (click to enlarge)<\/figcaption><\/figure>\n<\/div>\n<p>However, because of the lack of research into MCQs, we cannot say for certain that this question is \u201cjust right\u201d \u2014 it may be too hard. We need to do further research to find this out.<\/p>\n<h2>Creating multiple choice questions is not easy<\/h2>\n<p>The process of creating good MCQs is not an easy task, because question authors need to think about many things, including:<\/p>\n<ul>\n<li>What learning objectives are to be addressed<\/li>\n<li>What plausible distractors can be used<\/li>\n<li>What level of difficulty is right for learners<\/li>\n<li>What type of thinking the questions are encouraging, and how this is useful for learners<\/li>\n<\/ul>\n<p>In order for MCQs to be useful for learners and teachers, much more research is needed in this area to show how to reliably produce MCQs that are \u201cjust right\u201d and encourage productive thinking processes. We are very much looking forward to looking at this topic in our research work.<\/p>\n<p>To find out more about the computing education research we are doing, you can browse our <a href=\"https:\/\/www.raspberrypi.org\/research\" target=\"_blank\" rel=\"noreferrer noopener\">website<\/a>, take part in our monthly <a href=\"https:\/\/www.raspberrypi.org\/computing-education-research-online-seminars\/\" target=\"_blank\" rel=\"noreferrer noopener\">seminars<\/a>, and read our <a href=\"https:\/\/www.raspberrypi.org\/research\/publications\" target=\"_blank\" rel=\"noreferrer noopener\">publications<\/a>.<\/p>\n<p>The post <a rel=\"nofollow\" href=\"https:\/\/www.raspberrypi.org\/blog\/better-online-multiple-choice-questions-education-edtech\/\">Creating better online multiple choice questions<\/a> appeared first on <a rel=\"nofollow\" href=\"https:\/\/www.raspberrypi.org\">Raspberry Pi<\/a>.<\/p>\n<p>&#013;<br \/>\n&#013;<br \/>\nSource: Raspberry Pi &#8211; <a href=\"https:\/\/www.raspberrypi.org\/blog\/better-online-multiple-choice-questions-education-edtech\/\" target=\"_blank\" rel=\"noopener\">Creating better online multiple choice questions<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this blog post we explore good practices around creating online computing questions, specifically multiple choice questions (MCQs). Multiple choice questions are a popular way to help teachers and learners work out the next steps in learning, and to assess &hellip; <a href=\"https:\/\/www.prime-wow.com\/?p=974321\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":974322,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[77,110],"tags":[5],"class_list":["post-974321","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-rpi","category-unfiltered-rss","tag-rpi"],"_links":{"self":[{"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=\/wp\/v2\/posts\/974321","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=974321"}],"version-history":[{"count":0,"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=\/wp\/v2\/posts\/974321\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=\/wp\/v2\/media\/974322"}],"wp:attachment":[{"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=974321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=974321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.prime-wow.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=974321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}