Confidence-based learning

Confidence-Based Learning, CBL, measures the correctness of a learner's knowledge and confidence in that knowledge. It is designed to increase retention and minimize the effects of guessing, which can skew the results of traditional single-score assessments. It distinguishes between what individuals think and actually know.

The measurement allows creating a customized learning plan for each learner. The process, similar to quality improvement processes such as Six Sigma, continues until the learner achieves total mastery – defined as validly achieving confidence and correctness for 100% of the content twice in a row. Mastery leads to putting the knowledge into practice.

History

The Confidence-Based Learning Methodology is a culmination of more than 70 years of academic, commercial, and governmental research into the connection between confidence, correctness, retention, and learning. The first academic paper on the subject was written in 1932 and asserted that measuring confidence and knowledge was a better predictor of performance than measuring knowledge alone, which can be prone to guesswork.^[1]

Extensive research and technological advances ultimately led to further development of the methodology in measuring confidence and correctness.

Research into confidence and knowledge

The framework for Confidence-Based Learning Methodology is based primarily around the research of Darwin Hunt, Dieudonne LeClerq, Emir Shuford, and James E. Bruno. Significant advances in the knowledge and confidence connection were made by all of four researchers, but Bruno brought their collective work together in a methodology that made it possible for knowledge and confidence to be effectively measured.

Fundamental research that helped form the framework for CBL includes:

• Dr. Darwin Hunt – New Mexico State University

Hunt conducted significant research with the US Navy and focused on the dimensions of knowledge, linking confidence and correctness with retention. His process involved a two-step approach – (1) answer the question (objective measurement of correctness), and then state your confidence in your answer (subjective confidence statement). According to Hunt, research shows that the retention of newly learned material is systematically related to "how sure" people are about the correctness of their answers when they learn it.

Hunt, in an article titled, "The Concept of Knowledge and How to Measure It," stated that "Knowledge has many dimensions and the importance of confidence in one's knowledge state cannot be overlooked." ^[2]

• Dr. Dieudonne LeClerq – University of Belgium

LeClerq focused on item bank testing - the process of using a pool of questions, from which questions are drawn and randomly delivered to learners to see how well they answer questions without pattern recognition or order influencing the process. The result is higher quality knowledge and information. The outcome was a report on the quality of information/knowledge that shows where misinformation exists.^[3]

• Dr. Emir Shuford – University of California at Los Angeles (UCLA)

Shuford developed a measurement algorithm that focused on the determinations of the reliability of someone's knowledge and how a learners' knowledge reliability was improved or diminished based on their level of doubt or confidence in it.^[4]

• Dr. James E. Bruno – University of California at Los Angeles (UCLA)

Bruno, a Professor of Education at UCLA, began his work on understanding the connection between knowledge, confidence, retention, and the quality of knowledge while consulting for the North Atlantic Treaty Organization (NATO) and the RAND Corporation in the 1960s. Bruno eventually turned his focus to the K–12 market and packaged his developments into practical application for the K–12 educational market in the early 1990s.^[5]

Bruno's research on the linkage between knowledge, confidence, and behavior led to the intuitive conclusion that knowledge alone is necessary but not sufficient to create behavior. Rather, it is the fusion of knowledge and confidence that leads to behavioral outcomes and empowers people to act. People who are confidently correct will take actions that are productive. The reverse is also true in that individuals who are confident about misinformation will take action, with consistently negative (and potentially dangerous) results.

This measurement of knowledge quality was initially called Information Reference Testing (IRT).^[6] The IRT process uses a unique two-dimensional assessment process in which a single answer for each question generates two metrics simultaneously – correctness and confidence.

Numerous research studies in the 1980s and 1990s established a link between correctness and confidence, but most of these testing approaches first asked learners to state what they believed to be the correct answer to a question, then asked them to state their confidence in the answer just selected. Even though these approaches help us understand the level of confidence a learner has in their answers, research shows that when learning and confidence measurements are two discrete actions, learners tend to overstate their confidence level. Confidence, in this setting, is a logical response to a question, not a spontaneous measurement of true emotion.

While developing the IRT methodology, Bruno pointed out something other researchers failed to address. When evaluating the accuracy of testing and assessment outcomes, one must take into account the level of honesty in each learner's answers. If a person is provided with several answer choices and does not know the answer, their only option is to guess. If they happen to guess correctly, they get credit for knowledge they don't have.

Bruno tested his IRT process among students, and claimed successes with juvenile offenders in the LA County Corrections System who, according to Bruno, had dropped out of mainstream society because they were unable to adapt and succeed in a traditional learning environment. He claims that with IRT, these juveniles were able to achieve academic success while enjoying the process. Bruno's focus on this audience led to the game-like simplicity of the IRT process, which encouraged honesty in the evaluation and learning process in a safe environment where the emphasis was on knowledge quality rather than a score. Bruno ultimately was able to develop a simple methodology for measuring knowledge quality that became the basis for Confidence-Based Learning (CBL).^[7][8]

Further development

Bruno continued in the research and development of Information Reference Testing by continuing to focus on measuring confidence and correctness in the workplace. The methodology was eventually automated to allow an individual to self-assess their knowledge (simultaneously measuring confidence and correctness) that led to feedback on how the learner was doing, what gaps exists in knowledge (based on confidence levels), and what the learner specifically needed to learn. The methodology for this simultaneous measurement of confidence and correctness was patented in 2005. Around that time, the name of the IRT process was changed to Confidence-Based Learning (CBL).^[9]

Process

Confidence-Based Learning begins the learning process by asking the learner a set of questions and then filling knowledge gaps with critical content, whereas most traditional online learning approaches deliver content first and then test to validate each learner's understanding of the content. The CBL approach is similar to the centuries-old Socratic Method, which despite an abundance of advanced technologies and approaches to delivering training, has proven to be one of the most effective methodologies ever developed when it comes to ensuring that learning takes place.

By combining a knowledge diagnostic and prescriptive learning into one process, the CBL System offers a contextually-smart learning environment that is akin to a rigorous quality improvement process with a focus on learning.

1. Diagnose – The system starts by diagnosing the true knowledge of learners (i.e., what they actually know vs. what they think they know.) This diagnosis is achieved through CBL's answer selection process which determines the knowledge quality for each learner their confidence in that knowledge. This knowledge quality is then categorized by objective into one of four knowledge quadrants in the Learning Behavior Model. The two top quadrants are likely to result in action because the learner's confidence in the knowledge is high. The two bottom quadrants are likely to result in inaction because the learner’s confidence in the information is low.
2. Prescribe – After the learner's knowledge gaps are identified through the diagnostic session, the CBL system immediately presents an individualized learning plan for the learner in the CBL Learning Center. The Learning Center provides a visual representation of the learner's results in the following order: areas in which the learner is misinformed, uninformed, contains doubt, and has achieved mastery. The prescriptive plan includes feedback on the learner's performance and the learning content needed to fill knowledge gaps.
3. Learn – Once the individualized learning plan is provided to the learner, they can begin filling knowledge gaps. The personalized learning program enables the learner to click on the highest priority items first (e.g., areas where they were misinformed). As the learner reviews each categorized question, they discover the correct answer, their answer, and the explanation as to why the correct answer is right and why the incorrect answers are wrong. The explanation contains the critical information necessary for the learner to obtain mastery of the content. It also contains key information explaining the risks associated with the incorrect knowledge and the consequences of taking the wrong action.

Continue... The process uses iterative learning sessions to present information to the learner; therefore learners will continue the diagnose-prescribe-learn process until all knowledge gaps are closed. CBL's individualized approach narrows down the content as the learner demonstrates mastery. This eliminates the need for users to spend time learning what they have already mastered and makes the learning process more efficient. Because the CBL learning process is based on an individual's knowledge and confidence, the number of learning sessions needed to achieve mastery for a module will vary per learner. It is the Confidence-Based Learning innovative methodology that enables the learner to quickly learn and retain the material with confidence.

Knowledge quadrants

The knowledge quadrants in the Learning Behavior Model and their associated learner behaviors are as follows:

• Misinformation—knowledge a learner confidently believes to be correct, but which is actually incorrect. Those who have confidence in wrong information (misinformation) will very likely make mistakes on the job, which puts companies at the most risk.
• Mastery—knowledge a learner knows confidently that is correct, and which will likely be applied correctly in practice. Learners who have correct knowledge and a high degree of confidence in their knowledge (mastery) are masters of that knowledge domain. These learners are likely to act and act correctly, resulting in higher performing and more productive learners who make fewer mistakes.
• Doubt—knowledge a learner believes to be correct, but an element of doubt exists that may cause the learner not to act on that knowledge. Someone who harbors doubt may be correct on a certification, but is likely to act with hesitation or not act at all.
• Uninformed—knowledge that a learner has not acquired yet. Someone who is uninformed is unlikely to act, which can result in a state of paralysis.

See also

• Concept inventory
• Psychometrics

References

1. A method of correcting for guessing in true-false tests and empirical evidence in support of it." Journal of Social Psychology, 3 (1932): 359-362.)
2. The Concept of Knowledge and How to Measure it, by Darwin P. Hunt; Journal of Intellectual Capital, 2003.
3. Item Banking: Interactive Testing and Self-Assessment Conference Proceedings by James Bruno and Dieudonne Leclerq; presented at the NATO Advanced Research conference, 1992.
4. Quantifying Uncertainty into Numerical Probabilities for the Reporting of Intelligence, by Emir Shuford & Thomas Brown; Rand report prepared for the Defense Advanced Research Projects agency, 1973.
5. Identifying and Addressing Information Deficits for undergraduate students in Science, by James Bruno, Michael Pavel & Steve Strand; Journal of Educational Technology Systems, 1995
6. Information Reference Testing (IRT) in Corporate and Technical Training Programs, by James Bruno; UCLA 1995. (paper-based)
7. Using Testing to Provide Feedback Support to Instruction: A reexamination of the role of assessment in educational organizations, by Dr. James Bruno; UCLA, 1992
8. Concurrent Validity of Information Referenced Testing Format Using MCW-APC Scoring Methods, by James Bruno & Jamal Abedi; Journal of Computer Based Instruction, 1993.
9. United States Patent: 6,921,268