An Assessment of Knowledge by Pedagogical Computation on Cognitive Level Mapped Concept Graphs Rania Aboalela PhD Defense Department of Computer Science Kent State University 1 The Outline The Problem Contribution The proposed model and theory. 1.The Cognitive Level Mapped Concept Graph (CLMCG) 2.Concept Mapped Test and Evaluation Method 3.Question Example 4.The Concept States and the Theory 5.The Assessment Analytics of the Theory TCS2 Connecting the Questions to the concept map CLMCG The procedure of the Assessment Theory TCS2 The Probability Computation to Estimate the Concept States of the Students The problem to find the accurate probability of knowing a concept Example of many questions asked about the concept-(Contradiction) Example of indirectly tested concepts-DS Example of the Variety of Errors Values According to the Type of the Question Evaluation of real test (The probability calculation) The Experiment to Validate the Methods of the Theory. 1.The validation of the theories-Correction 2.The accuracy of the theories (The comparison of the probability between the method, the direct response and the method computation) 3.The accuracy result of knowing the domain of the learning object in VS, DS, PS 4.The efficient-size of footprint 5.Some studied application) the evaluation of the instructor test) Can I put them in the appendix of the dissertation? 6.The website Conclusion & Future Works 2 The Problem
Can a techniques and model help us to objectively (algorithmically) understand, analyze, assess the learning states and skill levels of a student with respect to the conceptual contents, concepts and their relationship that defines a specific knowledge domain. Armed with the latest developments in graph representation techniques, computer/automated inference technology, and pedagogical theories we provide a framework towards achieving the above. 3 Contribution The methodology is precision and efficiency The precision The graph paradigm of a semantic/ontological scheme provides precise measurement in knowledge assessment by focusing on organizing all the concepts in one space The cognitive link between the existing concepts in one domain increases the accuracy of the estimation of the assessed concepts The proposed mathematical approaches provide an accurate estimation of the probabilities of knowing and not knowing the concepts in addition to the estimation probability of the concepts that is ready to be known 4 Contribution The efficiency Fast inference algorithm with minimum number of testing concepts The amount of estimated concepts increases, even though the amount of tested concepts may be minimized and eliminated under the conditions laid down by the target skill levels The precise computational analysis and classifying the assessment results in terms of concept states brings a new level of nuance in estimating student knowledge and in simplifying the knowledge assessment of learning
The graph paradigm of a semantic/ontological scheme simplifies the difficulty involved in the question-design process. The components of the methods contribute to new objective conceptcentric assessment (quantitative vs. qualitative) 5 The Concept Mapped Knowledge Domain 1.The syllabus relation [Area Knowledge Space]. The syllabus relation retains the occurrence of the concepts in a formal textbook such that it retains the chapter, section, sub-section, etc. 2.The ontological relation. The ontological relation links the concepts in termsAof class, part, and B instances of, relationships. Lk 3.The cognitive relation. The cognitive skill relation captures the relationship between concepts in L A B terms of prerequisite concept needed to be attain to know the target concept at certain skill level. k 6 The Syllabus Relation The Root The concepts in the course name or the book name The data structures with C using STL 0.0.0.0.0.0.n.p The Internal Nodes
The concepts in the titles -Chapter title The Introduction to Algorithm 3.0.0.0.0.0.n.p Selection Sort Simple Search The Internal Nodes The concepts in the titles -Section title Algorithm Simple search Algorithm 3.2.0.0.0.0.n.p Selection Sort Algorithm 3.1.1.0.0.0.n.p The Leaves Nodes The fine concepts Sorting Algorithm Begins with 3.1.1.0.1.2.v.p Sorting Algorithm 3.1.1.0.1.1.n.p Sequence of Items Sequence of
Items 3.1.1.0.1.3.n.p Use To order 3.1.1.0.1.5.v.s 3.1.1.0.1.7.v.p List Rearrangement 3.1.1.0.1.4.n.s 3.1.1.0.1.6.n.s Explicit Implicit Verb Concept Concept Node Hieratical Occur in link Strategy 3.1.1.0.1.8.n.s Implements 3.2.0.1.1.2.v.s Elements 3.1.1.0.1.9.n.s Algorithm 3.2.0.1.1.1.n.s Strategy
3.2.0.1.1.3..n.s Verb & Noun Occur in link 7 The Ontology Concept Mapped Knowledge Domain 8 The Cognitive Concept Mapped Knowledge Domain 9 The Cognitive Concept Mapped Knowledge Domain 10 The Syllabus Concept Mapped Knowledge Domain 11 Concept Mapped Testing & Evaluation Method In order to measure the student learning we set up a concept mapped testing and evaluation method. A test is a set of questions. Students are required to answer them based on their knowledge. Grader evaluates the student knowledge based on the answers and grades. In conventional evaluation a grader will grade the answers and assign a quantitative score for the student. We slightly modify the evaluation method where the grader instead of a numerical score, is asked to evaluate if there is evidence in the answer that the student has succeed or failed to attain learn a concept at a certain cognitive skill level. We called it concept mapped evaluation or grading. 12 Question Example
Show the order of elements in the [given] array after each pass of the Selection Sort Algorithm. int arr[6] = {5, 1, 8, 2, 7, 9} [write the final result in the array] In conventional, a grader will evaluate the answers and assign a quantitative score. Graders Evaluation: Grader 1 - Selection Sort Algorithm [OK], The Passes of selection sort algorithm [OK], array[OK], The Order of the element in the array[OK], Order [OK], Sort [OK]. Cognitive Analysis: To answer question #1 correctly the concepts Sort and The Order need to understood, The concept array need to be applied, the concept The Order of the element in the array needs to be applied ,The Passes of selection sort algorithm needs to be evaluated and applied and the concept selection sort algorithm needs to be applied 13 The Concept States of the Theory The three basic sets of the concept states: 1. Verified Skills (VS) 2. Derived Skills (DS) 3. Potential Skill (PS) After the assessed individual complete the assessment, his result will be in six concept states 1. Verified Known Skills (VKS) 2. Derived Known Skills (DKS) 3. Potential known skill (PKS) 4. Verified Not knowing Skills (VNS) 5. Derived Not knowing Skills (DNS) 6. Potential Not knowing skill (PNS) 14 Verified Skills set VS(k) If (Qi,Cx)Lk & Cx is correct answer Cx Verified skills at level k VS(k) Verified skills at level k VS(k) Cx Verified skills at level k VS(k) tested concepts.
Where, Lk C Qi x Qi Verified skills at level k VS(k) Test questions. Cx Verified skills at level k VS(k) Tested concepts. Lk Verified skills at level k VS(k) Bloom link of level k. VS(k) Verified skills at level k VS(k) Verified skills at level K 15 Verified Known Skills set VKS(k) If (Qi,Cx)Lk & Cx is correct answer Cx Verified skills at level k VS(k) Verified Known Skills at level k VS(k) Cx Verified skills at level k VS(k) completely correct answer concepts. Where, Qi Verified skills at level k VS(k) Test questions. Cx Verified skills at level k VS(k) Tested concepts. Lk Verified skills at level k VS(k) Bloom link of level k. VKS(k) Verified skills at level k VS(k) Verified Known skills at level K Correct answer Lk C x Qi To answer the question Qi correctly the Student Si should know the concept Cx at level Lk 16 The Negative Theories Verified Not knowing Skills VNS(k) If (Qi,Cx)Lk & Cx is incorrect answer Cx Verified skills at level k VS(k) Verified Unknown skills at level k
VNS(k) if Cx Verified skills at level k VS(k) Incorrect concepts answer then Cx Verified skills at level k VS(k) VNS(k) Where, Qi Verified skills at level k VS(k) Test questions. Cx Verified skills at level k VS(k) Tested concepts. Lk Verified skills at level k VS(k) Bloom link of level k. VNS(k) Verified skills at level k VS(k) Verified unknown skills at level K Incorrect answer Lk C x Qi To answer the question Qi correctly the Student Si should know the concept Cx at level Lk Thus if the student answer the question incorrect that means he doesnt know the concept at the tested level 17 Derived Skill Set DS The DS is the set of concepts in the prerequisite set at certain skill level of the tested concepts and they have never been directly tested. For example, Derived Skill (DS) at level skill level 2 is defined as where there is indirect evidence that the concept Ci is understood or (not understood) by the student, then it will belong to DS (K=2) Where, Qi Verified skills at level k VS(k) Test Questions. Cx Verified skills at level k VS(k) Verified Skill Set. Ci Verified skills at level k VS(k) Another concept in the CLMCG Lk Cx Lk ,Lm Verified skills at level k VS(k) Bloom labels. Qi Derived skill at level k=2 Lm
Ci Proof: The student knows Cx at level k. Ci must be known to master Cx at level k. Thus, Ci is known to the student. 18 Derived Known Skill Set DKS(k=2) Is a concept Ci DKS(2)? If Ci is not in verified set, but if there exists two links such that (Q i,Cx),Lk, & (Ci,Cx)Lm & Cx Verified skills at level k VS(k) VKS(k) that it is in Verified Skill Set and m= 2 or k= m , then Ci is in Derived Skill Set at level 2, i.e. Ci Verified skills at level k VS(k) DKS(2). Where, C Qi Verified skills at level k VS(k) Test Questions. Cx Verified skills at level k VS(k) Verified Skill Set. Ci Verified skills at level k VS(k) Another concept in the CLMCG Lk ,Lm Verified skills at level k VS(k) Bloom labels. Lk x Derived Known skill at level k=2 Qi Lm Ci 19 Derived Not Knowing Skill DNS(k=2) If Ci is in verified unknown set, but if there exists two links such that (Qi,Cx),Lk, & (Ci,Cx)Lm & Cx Verified skills at level k VS(k) VNS(k) that it is in Verified unknown Skill Set and m= 2 or k= m , then Ci is in Derived Unknown Skill Set at level 2, i.e. Ci Verified skills at level k VS(k) DNS(2).
C Lk x Derived Not Known skill at level k=2 Qi Lm Ci Qi Verified skills at level k VS(k) Test Questions. Cx Verified skills at level k VS(k) Verified Not known Skill Set. Ci Verified skills at level k VS(k) Another concept in the CSM Lk ,Lm Verified skills at level k VS(k) Bloom labels 20 Support Node & Support Set Let CA is a node. Let CB is another node from which there is a level k link to A. Then we call CB a level k support node of CA . Let S(CA ,k) is the set of all such CB nodes in the complete concept graph G. The S(CA ,k) is the level k Support set for CA . i.e. all concepts in this set must be learned to have a level k skill in A. CA L CY k
Lm CC Lm CD L L k k CB CA Support set at level k CY Support set at level M 21 Derived Known Skill Set DKS (k>2) Is Cy in DS(k) where k>2? If Cy is known i.e. it is in DKS(2) or VKS(2), and if all level k support nodes of C y i.e. S(Cy,k) are in VKS(2)DKS(2), then CDKS(2), then Cy will be considered as a latent skill at level k. If Cy Verified skills at level k VS(k) DKS(2) DKS(2), then C VKS(2) and S(Cy,k) is subset of DKS(2) DKS(2), then C VKS(2) Cy Verified skills at level k VS(k) LS(k) Lk2 Derived Known skill at level k C Lk2
Lk Qx Lk2 Cy Lk Lk C x C Lk2 CA Lk2 CB Derived known skills at level 2 22 Derived Not Known Skill Set DNS (k>2) If Cy is known unknown i.e. it is in DNS(2) or VNS(2), and if all level k support nodes of Cy i.e. S(Cy,k) are in VNS(2)DKS(2), then CDNS(2), then Cy will be considered as Derived unknown skill at level k. If Cy Verified skills at level k VS(k) DNS(2)DKS(2), then CVNS(2) and S(Cy,k) is subset of DNS(2) DKS(2), then C VNS(2) Cy Verified skills at level k VS(k) DNS(k) Lk2 Derived Not Known skill at level k
C Lk2 Lk Qx Lk2 Cy Lk Lk C x C Lk2 CA Lk2 CB Derived Not known skills at level 2 23 Potential Known Skill Set PKS(k) Is A a potential skill at level k? Let S(A,k) is the support set of A at level k. If every node in the S(A,k) is subset of VKSDKS(2), then CDKS at any level (doesnt matter-because we only want to guarantee that the set is known) i.e. S(A,k) VS() V DKS(), VS() V DKS(),
but there is no evidence that A is known, then A is in potential skill set PKS(k) i.e. A Verified skills at level k VS(k) PKS(k) L C L where Q Potential Known Skill L C C d , C x Verified skills at level k VS(k) VS A C C , CA , CB Verified skills at level k VS(k) DS L L L L Lk Verified skills at level k VS(k) Bloom link at level k C C C k d m x k x k k k-
k C A B Derived Known skills at level k 2 24 Potential Not Knowing Skill Set PNS(k) Is A a potential Not ready to be Known skill at level k? Let S(A,k) is the support set of A at level k. If every node in the S(A,k) is subset of VNSDNS at any level (doesnt matter-DNS at any level (doesnt matterbecause we only want to guarantee that the set is not known) i.e. S(A,k) VNS DNS, but there is no evidence that A is VNS DNS at any level (doesnt matter- DNS, but there is no evidence that A is known, then A is an element in Potential Not Known Skill set PNS(k) i.e. A Verified skills at level k VS(k) PNS(k) where Potential Not Knowing Skill C d , C x VNS VNS C C , CA , CB VNS DNS Lk VNS Bloom link at level k. Lk Lk d Lm C x Lk
Qx Lk A Lk Lk CC C CA Lk CB Derived Not knowing skills at level k 2 25 The Assessment Analytics of the Theory TCS2 Question Example Show the order of elements in the [given] array after each pass of the Selection Sort Algorithm. int arr[6] = {5, 1, 8, 2, 7, 9} [write the final result in the array] In conventional, a grader will evaluate the answers and assign a quantitative score. Graders Evaluation: Grader 1 - Selection Sort Algorithm [OK], The Passes of selection sort algorithm [OK], array[OK], The Order of the element in the array[OK], Order [OK], Sort [OK]. Cognitive Analysis: To answer question #1 correctly the concepts Sort and The Order need to understood, The concept array need to be applied, the concept The Order of the element in the array
needs to be applied ,The Passes of selection sort algorithm needs to be evaluated and applied and the concept selection sort algorithm needs to be applied 26 Cognitive Mapped Analytics of Students Knowledge: [We can show all hidden things the analytics can reveal] VKS(2)={ Order} VKS(3)={Selection Sort Algorithm , The Passes of selection sort algorithm, array, The Order of the element in the array} VKS(4)={The Order of the element in the array} VKS(5)={The Passes of selection sort algorithm} DKS(2)={"Selection Sort, Sort Process, The algorithm, Unsorted Order, The places step of selection search algorithm, Illustration example of the Selection Sort Algorithm, Selection Sort Function, The result List, Ascending Order, The smallest element, The largest Element, The elements , The Content , Iteration, The places step of Selection Sort, Traversal of the elements, Position, process, Places, Step, Selection, Search, illustration, Example, Ascending, Function, Result, Smallest, Largest, Sort} DKS(3)={Selection Sort, The result List, Ascending Order, The largest Element, The smallest element, Traversal of the elements, Iterations, The places step, Sort Process, Passes, The resulting List} PKS(3)={The index of smallest element, Sublist Array, Simple search algorithm, Radix Sort Algorithm, Heap Sort Algorithm", "Insertion Sort Algorithm, Sort} PKS(2)={Analyses of the algorithm, Algorithm performance, Running Time, Selection Sort running time O(n2) } 27 Labeling the Questions to the Concepts The analyzing method Lk C
QiA x To answer the question Qi A correctly the Student Si should know the concept Cx at level Lk 28 Labeling The Tested Concept with RBT To [Lk] B one must know A(basic theory1) Ex: To Apply Sort the student must know the Sorting Algorithm. To Analyze Sorting Algorithm the student must know running time. L A k B The Bloom relation is a direct link from the Parent A to the child B with link property that requires that to know the child with skill L k we need to know the parent. According to the theory 1, B couldnt be answered correctly unless A is known. 29 Connect Questions with Mapped Concepts 30 The Probability Computation to Estimate the Concept States of the Students The concept state CS of an individual: - Is a set of concepts in the domain that he/she is capable of answer its related questions - The concept state includes the concepts denoted by id number combined
with the skill level such as CS(Cj) = {Cj(L) VNS CS | Cj is a concept at skill Level L with index number j Verified skills at level k VS(k) positive integer number. - Example of a concept state of a student CS = {C1(L2),C2(L3),C3(L6),C4(L2)} 31 The Probability Computation to Estimate the Concept States of the Students The problem: The existence of the concepts in a complex domain such as there are variant prerequisite relations for the concept and the concept could be inferred by more than one concepts. The reflected evaluations of the concept such as the student contradiction in the answer of the same concept Solution: Using Bayes theorem to calculate the probability of knowing a concept even though the concept is evaluated based on reflected evaluations of the concept. Also, could be used to calculate the probability of knowing the concept even in the existence of the complex relation between the concepts in the concept space such as the relation between the concepts in the s of VS, DS and PS. 32 Example to Find the Probability of Knowing the Concepts in Complex Domain 33 Example to Find the Probability of Knowing the Concepts Asked by Many Questions Get Conflicted Evaluations Cc (L3) rq1 rq2 Cc(L3)
Tested Concept rq3 rq4 rqi qq5 rq6 rqi Incorrect Answer Correct Answer Many questions asked about one concept Cc(L3) 34 Examples to Find the Probability of Knowing The Concepts in Complex Domain Example to find the probability of knowing the concepts in the cases: 1. The concepts is evaluated by many questions 2. The concepts is indirectly evaluated Let the set of questions Q = {q1, q2}, asked about the set of concepts CS = {C1, C2, C3} such that question q1 asks about concepts c1 and c2, and question q2 asks about the concepts C1, C2, C3. If the answer is correct then the probability of knowing the concept c is = (1-) and the probability of not knowing the concept c is = . On the other hand, if the response to a question q is incorrect then the probability of knowing the concept c, which has been asked by the question q is = and the probability of not knowing the concept c is = (1-. The two constants, respectively called (careless) error probability and guessing probability at q. 35 Example to Find the Probability of Knowing the Concepts in Complex Domain
q1 c1 c4 q Question asked about the concepts c5 q2 c2 c6 c3 c7 c Tested Concept Directly asked by a question c Tested Concept Indirectly asked by a question 36 The Used Equation-Bayes Theorem P(|) = - denotes knowing the concept C number -P() is the unconditional probability of knowing the concept Cj, which is the initial probability of knowing the concept Cj. It is just the rate of the correct responses to the questions asked about the concept Cj
-is the set of the responses to the questions asked about the concept C j -P(R|Cj) is the probability the responses (evidences) on the condition of knowing the concept Cj - is the probability of the responses on the condition of not knowing the concept C j -is the unconditional probability of not knowing the concept C j, which is the initial probability of not knowing the concept Cj. It is just the rate of the incorrect responses to the 37 The Calculation of P ( R|C j )= ( C j) =1 To know the conditional probability of knowing the concept based on the evidences (responses), we used the assumption of the conditional probability of the responses in the dependency on C j is the multiplication of the response data given an assumption of knowing the concept C j at the evaluation R Thus, ,} the occurrence or non-occurrence of B are independent events in their conditional probability distribution given Y. In other words, R and B are conditionally independent given Y if and only if, given knowledge that Y occurs, knowledge of whether R occurs provides no information on the likelihood of B occurring, and knowledge of whether B occurs provides no information on the likelihood of R occurring 38 The Calculation of P ( R|C j )= ( C j) =1 To know the conditional probability of knowing the concept based on the evidences (responses), we used the assumption of the conditional probability of the responses in the dependency on C j is the multiplication of the response data given an assumption of not knowing the concept C j at the evaluation R Thus,
,} 39 The Problem Based on the given graph, we are examining the probability of knowing the concept Cj. Where, Ri is the set of evidences (responses), Ri = {Q1, Q2, Q3.Qn}, i = 1, 2, 3..,n. Each element of Ri is the response to the question that verifies knowledge about one or more concepts. There is a correct response denoted by Q r, and incorrect response denoted by , r is an integer number indicates index of the questions - R1 = Q12Q3 Q Q Q 11 22 33 - R2 = Q1Q3 - R3 = Q2Q3 - R4 = Q3 C11 C22 C 3 C 4 - P(Qr|) = g , when there is dependency between Qr and
- P(|) = m - P() = k 43 The Solution P(|) = OQ 1 OQ 2 C1 C2 C 3 C 4 OQ 3 - R1 = Q12Q3 - R2 = Q1Q3 - R3 = Q2Q3 -
R4 = Q3 - P(Qi|) = g , when there is dependency between Qi and - P(|) = m - P() = k - We use Bayes theorem to measure the probability of knowing the concept when we get an answer. So the evidence is the answer and the unknown is the probability of knowing the concept. - The cases when we get correct answer: 1. P(Qr|) is the probability of the correct answer accruing with not knowing the concept 2. P(|) is the probability of correct answer to question i accruing with knowing the concept C j - The cases when we get incorrect answer: 1. P(|) is the probability of incorrect answer to question r accruing with knowing the concept C j 2. P(|) is the probability of incorrect answer to question r accruing with not knowing the concept C j 44 The Used Equation-Bayes Theorem P(|i) = - denotes knowing the concept C number -P() is the unconditional probability of knowing the concept Cj, which is the initial
probability of knowing the concept Cj. It is just the rate of the correct responses to the questions asked about the concept Cj -is the set of the responses to the questions asked about the concept C -P(Ri|Cj) is the probability the responses (evidences) on the condition of knowing the concept Cj - is the probability of the responses on the condition of not knowing the concept C j -is the unconditional probability of not knowing the concept C j, which is the initial probability of not knowing the concept Cj. It is just the rate of the incorrect responses to the questions asked about the concept C . 45 The Calculation of P ( R i|C j )= ( C j) =1 To know the conditional probability of knowing the concept based on the evidences (responses), we used the assumption of the conditional probability of the responses in the dependency on C j is the multiplication of the response data given an assumption of knowing the concept C j at the evaluation Ri Thus, ,} 46 The Calculation of P ( R i|C j ) = ( C j) =1 To know the conditional probability of knowing the concept based on the evidences (responses), we used the assumption of the conditional probability of the responses in the dependency on C j is the multiplication of the response data given an assumption of not knowing the concept C j at the evaluation Ri
Thus, ,} 47 Explaining of the Conditional Probability P(|) - In the case of correct answer the given conditional probability is P(Qr|) = g - Based on the given probability value of P(Qr|) = g. Where g is a guessing. Also, we have e presents the error. Both g & e have the same values because the correct response happened with not knowing the concept C i by Lucky guessing. Where we have to consider the error e when we have the correct response with knowing the concept C i - So P(|) = 1- e where e = g P(|) = 1- g - In the case of incorrect answer the given conditional probability is P(|) = m - Based on the given probability value of (Qr|) = m. Where m is mistake/error. Also, we have e presents the error. Both m & e have the same values because the incorrect response happened with knowing the concept C i, by mistake. Where we have to consider the error e when we have the correct response with knowing the concept C i - So P(|) = 1- e where e = m
P(|) = 1- m 48 - P(|) - R1 = Q12Q3 P(Qr|) = g when there is dependency between Qr and P(|) = m P() = k P() = 1- P() = 1-k Since P(Qi|) = g. Where, g = e then P(|) = 1-e = 1-g - P(|) = - = - = - = - = -
= - = = = q11 q22 C11 C22 C 3 C 4 based on the structure of the relation between the questions and the concepts, Q3 is not related with C1 49 P(|) - R2 = Q1Q3 P(Qr|) = g when there is dependency between Qr and P(|) = m P() = k P() = 1- P() = 1-k Since P(Qi|) = g. Where, g= e then P(|) = 1-e = 1-g Since P(|) = m. Where, m= e then P(|) = 1-e =1-m - P(|) =
- = - = - = - based on the structure of the relation between the questions and the concepts, Q3 is not related with C q11 q22 C11 C22 C 3 C 4 = - = -
= - = 2-50 - R3 = Q2Q3 P(Qr|) = g when there is dependency between Qr and P(|) = m P() = k P() = 1- P() = 1-k Since P(Qr|) = g, where g = e then P(|) = 1- e = 1-g Since P(|) = m, where m= e then P(|) = 1-e =1-m - P(|) = - = - = - = - = -
= - = - = - = P(| q11 q22 C11 C22 C 3 C 4 based on the structure of the relation between the questions and the concepts, Q3 is not related with C 2-51 P(|) - R4 = Q3 P(Qr|) = g when there is dependency between Qr and
P(|) = m P() = k P() = 1- P() = 1- k Since P(Qr|) = g. Where, g= e then P(|) = 1-e = 1-g Since P(|) = m. Where, m= e then P(|) = 1-e =1-m - P(|) = - = - = - = - = - = - = - = (based on the structure of the relation between the questions and the concepts, Q3 is not related with C1)
q11 q22 C11 C22 C 3 C 4 2-52 P(|) - R1 = Q12Q3 P(Qr|) = g when there is dependency between Q1 and P(|) = m P() = k P() = 1- P() = 1- k Since P(Qr|) = g. Where, g= e then P(|) = 1-e = 1-g Since P(|) = m. Where, m= e then P(|) = 1-e =1-m - P(|) = - = - =
- = - = - = - = q11 q22 C11 C22 C 3 C 4 based on the structure of the relation between the questions and the concepts, Q1 and Q3 is not related with C 53 - R2 = Q1Q3 P(Qr|) = g when there is dependency between Q1 and
P(|) = m P() = k P() = 1- P() = 1- k Since P(Qr|) = g. Where, g= e then P(|) = 1-e = 1-g Since P(|) = m. Where, m= e then P(|) = 1-e =1-m - P(|) = - = - = - = - = - = - = - = P(|)
based on the structure of the relation between the questions and the concepts, Q1 and Q3 is not related with q11 q22 C11 C22 C 3 C 4 54 P(|) - R3 = Q2Q3 P(Qr|) = g when there is dependency between Q1 and P(|) = m P() = k P() = 1- P() = 1-k Since P(Qr|) = g. Where, g = e then P(|) = 1-e = 1-g Since P(|) = m. Where, m= e then P(|) = 1-e =1-m - P(|) = - = -
= - = - = - = - = - = q11 q22 C11 C22 C 3 C 4 based on the structure of the relation between the questions and the concepts, Q1 and Q3 is not related with C2
55 P(|) - R4 = Q3 P(Qr|) = g when there is dependency between Q1 and P(|) = m P() = k P() = 1- P() = 1- k - Since P(Qr|) = g. Where, g= e then P(|) = 1- e = 1- g Since P(|) = m. Where, m= e then P(|) = 1- e =1-m - P(|) = - = - = - = - = - =
q11 q22 C11 C22 C 3 C 4 based on the structure of the relation between the questions and the concepts, Q1 and Q3 is not related with C2 - 2-56 P(|) - R1 = Q12Q3 q11 q22 C11 C2 C 3 C 4 - In the case of concept C3, which supports to only one concept C2,
and there is another concept in the support set of the supported concept C1 - P(|) = P(|)P() - = P(|)k - = ( )*k 2-57 P(|) q11 q22 C11 C22 C 3 C 4 - In the case of concept C3, which supports to only one concept C2, and there is another concept in the support set of the supported concept C1 - P(|) = P(|)P() - = P(|)k - = *k 58 P(|) - In the case of concept C3, which supports to only one concept C2, q11
q22 and there is another concept in the support set of the supported concept C1 C11 C22 C 3 C 4 - P(|) = P(|)P() - = P(|)k - = *k 59 P(|) - In the case of concept C3, which supports to only one concept C2, and there is another concept in the support set of the supported q11 q22 C11 C22 C 3 C
4 concept C1 - P(|) = P(|)P() - = P(|)k - == k 60 P(|) - In the case of concept C4, which supports to more than one concept C1, and C2 - We know the probability of the supported concept C1 & C2 by the evidences R1. Thus, - P(|) = q11 q22 C C111 C2 C22 C 3 C 4 61 P(|) - In the case of concept c4, which supports to more than one concept C1, and C2 - We know the probability of the supported concept C1 & C2 by the evidences R1.
q11 q22 C11 C22 C 3 C 4 - P(|) = - = - = 62 P(|) q11 q22 C11 C22 C 3 C
4 - In the case of concept c4, which supports to more than one concept c1, and c2 - P(|) = - = - = 63 P(|) q11 q22 C11 C22 C 3 C 4 - In the case of concept c4, which supports to more than one concept c1, and c2 - P(|) = - = - = 64 P(|) q22 q11 C11
C22 C 3 C 4 - In the case of concept c4, which supports to more than one concept c1, and c2 - P(|) = - = 65 First Suggestion of (R|C) By the multiplication considering 0.5 as 0.5 Q# OQ 1 DQ 1 DQ 3 OQ 2 C11 C22 C 3 C
4 DQ 2 DQ 4 The columns in the experiments graph of the probability of knowing the concept Concept # Estimated Probability by Kent Method Estimated The accurate probability by probability by joint Direct Question evidences (using Bayes) QO1 Open Question C1 OQ1 Open Question C2 OQ2 Open Question C2 P(Cj|Ri) P(Cj|(OQ, DQ)) P(Cj|OQ) P(Cj|DQ) C1
0.9 0.9 1 C2 0.5 0.9 0.91 C3 0.92 0.1 0.56 0.1 .12 Evidence Table C Response The strength of evidence Type for each open question of the probability of knowing the concept P(OQ|Cj) The probability of knowing the concept by using evidences from all open questions P(Cj|Ri) = P(Cj|(rOQ1, rOQ2))
V 1 0.9 0.9 V 1 0.9 0.5 V 0 0.1 0.5 The Confirmation Test (Second Test of Direct Questions) Q# DQ1 DQ2 DQ3 DQ4 C4 Q Type C# Q Type
Direct Question Direct Question Direct Question Direct Question C# C Type Response The strength of evidence The probability of for each direct question of knowing the the probability of knowing concept by the concept evidences from OQ P(DQ|Cj) & DQ P(Cj|Ri) = P(C|(OQ, DQ)) C1 V 1 0.9 1 C2 V 1 0.9
0.91 C3 D 0 0.1 0.56 C4 D 0 0.1 0.12 The used equation to calculate the probability of knowing the concept by the computation is P(|) = 66 The explaining of computing the probability of knowing the concepts The estimated probability of knowing the concept c2 by evidence propagation of VS method (by only one test (OQ)) In the case of the concept C2, which is in the concept state VS and tested by more than one questions: . OQ1 & OQ2 P(C2|R) = Where, R= {OQ1, 2} rOQ1 is a correct response to the open question OQ1 is incorrect response to the open question OQ2
- P(OQ1 P(2 ) = 0+m= 0+0.1 = 0.1 - = = 0.9 = m = 0.1 = g = 0.1 = = 1- 0.1= 0.9 - P(C2) is the unconditional probability of the directly tested concept c2 P(C2) = 0.5 is calculated by the equation P(C2) = - P(R|Kc2) is the probability the responses (evidences) on the condition of knowing the concept c 2 P(R|C2) = = 0.9 0.1 = 0.09 - P(R|NKc2) is the probability of the responses on the condition of not knowing the concept c 2 P(R|) = = 0.1 0.9 = 0.09 Thus, P(C2|R) = 68 The Explaining of Computing the Probability of Knowing the Concept C 2 DQ 1 DQ 3 OQ 1 OQ
2 C11 C22 C 3 C 4 DQ 2 DQ 4 The estimated probability of knowing the concept c2 by computation of direct question DQ (The confirmation test) in the addition to the set of open questions OQ In the case of the concept C2, which is tested in the concept state VS and tested by the set of questions in OQ = {OQ1,} and confirmed by direct question DQ2 P(C2|R) = Where, R= {, DQ2} 0.9 = The unconditional probability of the directly tested concept C2, P(C2) = 0.67 Where it is calculated by the equation P(C2) P(R|C2) = = 0.5*0.9 = 0.45 P(R|) = = 0.5*0.1 = 0.05 Thus, = P(C2|R) 69 The Explaining of Computing the Probability of Knowing the Concept C3 The estimated probability of knowing the concept c3 by evidence propagation of DS method (By only one test) In the case of the concept C3 which is in the concept state DS and supports more than one tested concept in VS: C1 & C2 P(C3|R) =
Where, R= {C1, C2} The unconditional probability of the indirectly tested concept c 3, P(C3) = = 0.5 and P() = 0.5 P(OQ1|C1) P(OQ1,2|C2) = 0.5 P(OQ1|) = 0.1 P(OQ1,2|) = 1- P(OQ1,2|C2) = 1-0.5 =0.5 P(C1) = P(OQ1|C1) = 0.9 P(C2) = P(OQ1,2|C2) = 0.5 P() = P(OQ1|) = 0.1 P() = P(OQ1,2|) = 0.5 For C1 , C3 , which assumed dependent, the probability of both occurring is = = = P(C1) Similar to that we have = = = P(C2) = = () = )] )] )] )] = [(0.9 0.5) )] = [(0.9 0.5)] (0.5 0.5)] = 0.45 0.25= 0.11 Similar to that we have = = = )] )] = (0.1 0.5)(0.5 0.5) =0.05 0.25 = 0.01 Thus, = P(C3|R) 70 The explaining of computing the probability of the concept C3 DQ 1 DQ 3 OQ
1 OQ 2 C1 C2 C 3 C 4 DQ 2 D Q4 The estimated probability of knowing the concept c3 by the computation based on two tests (DQ & OQ) In the case of the concept C3, which is estimated by DS method and confirmed by direct question DQ 3 P(C3|R) = Were, R= {OQc1,c2, } P(OQc1,c2)|) = P(OQc1,c2) = 0.92 P()|) = 0.1 P(OQc1,c2)|) = 1-P(OQc1,c2)|) = 1- 0.92 = 0.08 P()| ) = 1- m = 1- 0.0.1 = 0.9 - The unconditional probability of the indirectly tested concept c 3, P(C3) = 0.5. Where it is estimated by the equation. P(C3) = = 0.5 - The estimated probability of knowing the concept c3 by OQ considers the counting of OQ question to be added to correct or incorrect questions. - = P(OQc1,c2| ) P( ) = 0.92 0.1 = 0.092 - = P(OQc1,c2|) P()= 0.072 P(C3|R) = 71
DQ 1 DQ 3 OQ 1 OQ 2 C1 C2 C 3 C 4 DQ 2 DQ 4 The estimated probability of knowing concept c4 by evidence propagation of DS method (By only one test) - In the case of the concept c4, which supports to only one concept c2, and there is another concepts in the support set of the supported concept C2 - The probability P(C4|C2) will be calculated using Bayes version 1 - = From Bayes version 1 = = 0.5 0.5 = 0.25
- The unconditional probability of the indirectly tested concept c 4 is P(C4) P(C4) From Bayes version 1 = 72 The Problem Based on the given graph, we are examining the probability of knowing the concept Cj at skill level L. Where, Ri is the set of evidences (responses), Ri = {Q1, Q2, Q3.Qn}, i = 1, 2, 3..,n. Each element of Ri is the response to the question that verifies knowledge about one or more concepts. There is a correct response denoted by Qr, and incorrect response denoted by , r is an integer number indicates index of the questions Find P(|Ri), k = 2,3,4,5,6 indicates the skill level of the concept C j, based on the following cases: OQ 4 DQ 5 DQ 6 DQ 7 L3 OQ 3 L4 C
4 L5 L4 L4 L2 C 7 L4 OQ 2 L2 L2 L6 C 3 C 2 C 1 L2 L2 C 6 OQ 1 L5
L5 C 5 L5 C 8 75 The Problem - R1 = Q12Q3 Q4Q6Q7 - R2 = Q1Q3 - R3 = Q2Q3 - R4 = Q3 - P(Qr|) = g , when there is dependency between Qr and - P(|) = m - P() = k 76 The estimated concepts based on the proposed theory OQ 4 DQ 5 DQ 6 DQ 7 L3 OQ
3 L4 C 4 L5 L4 L4 L2 C 7 L4 OQ 2 Relation Type L2 L2 L6 C 3 C 2 C 1 L2 L2 C 6
OQ 1 L5 L5 C 5 L5 C 8 V V V V D D D D D D D P P C# Level# Related OQ 1 2 3 4 5
6 7 6 7 6 7 8 8 6 2 3 4 2 2 2 5 4 5 4 5 5 1 2 3 4 2 3 3 3 3 77 Counting skills result according to the related questions in the figure Type of Questions
OQ 4 DQ 7 DQ 8 DQ 10 DQ 11 OQ 3 OQ 2 OQ 1 L3 L2 L4 L2 L5 L4 C 3 C 4 L4 C 7
L4 C 2 L2 L2 C 6 L5 L5 L2 DQ 14 L2 DQ 13 C 1 C 5 L5 C 8 L5 DQ 12 L2 DQ 6 L6 DQ
5 DQ 9 OQ & DQ OQ & DQ OQ & DQ OQ & DQ DQ DQ DQ DQ DQ Type of Bloom LinkConcepts Counting Estimated Skill V 2 1 V 3 1 V 4 1 V 6 1 D 2 3 D 3 1 D 4 1 D
5 1 P 5 1 The related questions that have to be matched Relation Type C# Level# Related OQ Related DQ V V V V D D D D D D D P P 1 2 3 4 5 6 7 6 7 6
7 8 8 6 2 3 4 2 2 2 5 4 5 4 5 5 1 2 3 4 2 3 3 3 3 5 6 7 8 9 13 14 10 11 9 & 10 11 & 13
9 &12 11&12 78 The Experiments to Validate the Concept States of the Theory TCS2 1. The validation of the proposed methods 2. The matching analyses (validation) 3. The accuracy of the Concept States 4. The Accuracy (The method and the real response of the concepts) 5. The Efficiency (Footprint) 79 The Experiments to Validate the Concept States of the Theory TCS2 The experiment analysis Two type of questions: 1. Direct Questions (DQ) The DQ tests the identical skill level, which has been tested by the instructor, but it directly specifies the level of the concept. 2. The Open Questions (OQ) The normal questions which prepared by the instructor By using the proposed assessment analytics of the theory we conclude which skill level was included in that open question and prepare the DQ
DQ are designed for directly verifying the matching of the related skills between OQ and DQ based on the relation within the three sets of concept states of VKS, DKS, and PKS Therefore, two types of questions are offered: (1) OQ. (2) DQ We calculate the percentage matching between the answers of assessed individuals to the two types of questions to validate the correctness and the accuracy of the methods 80 The Experiments to Validate the Concept States of the Theory TCS2 The experiment analysis The value of 1 is given to each correct answer for each tested skill by OQ and DQ. The value of 0 is given to each incorrect answer for each tested skill by OQ and DQ. Thus, if the students answer to the identical tested skill had the same value either 0 or 1 in DQ and OQ, then the matching value will be 1, otherwise it will be 0. Value of Answer to Open Question Value of Answer to Direct Question Matching Result Validating Theory 1 0 1 0 1 0
0 1 1 1 0 0 Correct (+) Correct (-) False (+) False (-) 81 Counting skills result according to the related questions in the figure Type of Questions OQ 4 DQ 7 DQ 8 DQ 10 DQ 11 OQ 3 OQ 2 OQ 1
L3 L2 L4 L2 L5 L4 C 3 C 4 L4 C 7 L4 C 2 L2 L2 C 6 L5 L5 L2 DQ 14 L2 DQ
13 C 1 C 5 L5 L2 DQ 6 L6 DQ 5 DQ 9 OQ & DQ OQ & DQ OQ & DQ OQ & DQ DQ DQ DQ DQ DQ The related questions that have to be matched C 8 L5 DQ 12 Type of Bloom LinkConcepts Counting
Estimated Skill V 2 1 V 3 1 V 4 1 V 6 1 D 2 3 D 3 1 D 4 1 D 5 1 P 5 1 Relation Type V V V V D D D D
D D D P P C# Level# Related OQ Related DQ 1 2 3 4 5 6 7 6 7 6 7 8 8 6 2 3 4 2 2 2 5 4 5
4 5 5 1 2 3 4 2 3 3 3 3 5 6 7 8 9 13 14 10 11 9 & 10 11 & 13 9 &12 11&12 82 Example to Find the Probability of Knowing the Concepts in Complex Domain 102 Connect Questions with Mapped Concepts 103 The Validation of VS -The matching
100% 90% 84% 90% 83% 83% 80% 70% 58% 60% 50% 40% 31% 30% 20% 13% 13% 12% 10% 5% 10% 2% 2% 4% 3% 2% 2% 2% 1% 0% 0% Level 2 Level 3 Level 4 Level 5 Level 6 False (-) 0-1 Correct (-) 0-0 False(+) 1-0 Correct(+) 1-1 The percentage of the matching of skills in the set
of VS 104 The Validation of DS-The Matching 100% 95% 90% 84% 81% 80% 71% 70% 67% 60% 50% 40% 31% 30% 23% 20% 10% 0% 15% 2% 2%
12% 1% 2% Level 2 Level 3 False (-) 0-1 False(+) 1-0 1% 5% Level 4 3% 0% 2% 1% 0% Level 5 Correct (-) 0-0 Level 6 Correct(+) 1-1 The percentage of the matching of skills in the set of DS 105 The Validation of PS-The Matching 100%
92% 89% 90% 86% 81% 80% 79% 70% 60% 50% 40% 30% 20% 10% 0% 0% Level 2 False (-) 0-1 3% Level 3 False(+) 1-0 Level 4 5% 3% 2%
Level 5 Correct (-) 0-0 Level 6 Correct(+) 1-1 The percentage of the matching of skills in the set of PS 106 The Percentage of Knowing the Concepts in DS Corresponding to the All Participants Comparison Between the Estimation by DS Method & by Real Response Y: shows the percentage of knowing the concept X the concept ID Shows the estimation of knowing the concepts by the method DS (Indirectly) Shows the estimation by real directly response 107 The Validation of the Estimation of Knowing the Concepts in DS Based on the Answer of the Perfect Student Comparison Between the Estimation of Knowing the Concept Based on the DS Method & Real Directly Response Y : the probability of knowing the concept X : the concept ID
Direct response is a response to the question asked directly about the concept at certain skill level Indirect response is a response to the question infers to the concept. It would be a question asked about the supported concept The estimation by DS method: is the estimation of knowing the concept based on the related tested concepts. Wherein the related concept is the supported concept by the estimated concept. In other words, the concept in DS is a prerequisite concept to the tested concept The estimated probability given 1 to the correct and 0 to incorrect response for either the direct 108 response on indirect response The Accuracy of the Estimation of Knowing the Concepts in DS Based on the Answer of the Perfect Student Comparison Between the Probability of Knowing the Concept Based on the DS Method & Real Directly Response
Y : the probability of knowing the concept X : the concept number Direct response is a response to the question asked directly about the concept at certain skill level The estimation probability of knowing the concept based on direct response is given 0.9 to the correct response and 0.1 to incorrect response If the type of question is a kind of multiple-choice, then an additional value of errors = 0.03 will be considered Thus, the estimation probability of knowing the concept based on direct response is given 0.88 to the correct response and 0.13 to incorrect response P(Kc) by the computation: is the probability of knowing the concept mathematically calculated by many observation (the two responses) P(Kc) by Direct response: is the probability of knowing the concept estimated by direct response P(Kc) by DS method: is the probability of knowing the concept is estimated by indirect response Indirect response is a response to the question infers to the concept. It would be a question asked about the supported concept The estimation probability of knowing the concept based on indirect response is given 0.8 to the correct response and 0.2 to incorrect response If the type of question is a kind of multiple-choice, then an additional value of errors = 0.03 will be considered Thus, the estimation probability of knowing the concept 109 based on direct response is given 0.77 to the correct The Validation of the Estimation of Knowing the Concepts in DS Based on the Answer of a Student Randomly Selected Comparison Between the Estimation of Knowing the Concept Based on the DS Method & Real Directly Response
Y : The probability of knowing the concept X : The concept ID Direct response is a response to the question asked directly about the concept at certain skill level Indirect response is a response to the question indirectly asked about the concept but infers to it. It would be a question asked about the supported concept The estimation by DS method: is the estimation of knowing the concept based on the related tested concepts. Wherein the related concept is the supported concept by the estimated concept. In other words, the concept in DS is a prerequisite concept to the tested concept The estimated probability given 1 to the correct and 0 to incorrect response for either the direct 110 response on indirect response The Accuracy of the Estimation of Knowing the Concepts in DS Based on the Answer a Student Randomly Selected Comparison Between the Probability of Knowing the Concept Based on the DS Method & Real Directly Response
Y : The probability of knowing the concept X : The concept ID Direct response is a response to the question asked directly about the concept at certain skill level The estimation probability of knowing the concept based on direct response is given 0.9 to the correct response and 0.1 to incorrect response If the type of question is a kind of multiple-choice, then an additional value of errors = 0.03 will be considered Thus, the estimation probability of knowing the concept based on direct response is given 0.88 to the correct response and 0.13 to incorrect response P(Kc|OQ,DQ) by the computation: is the probability of knowing the concept mathematically calculated based on many observation (the two responses) P(Kc|DQ) by Direct response: is the probability of knowing the concept estimated by direct response P(Kc|OQ) by DS method: is the probability of knowing the concept estimated by indirect response Indirect response is a response to the question infers to the concept. It would be a question asked about the
supported concept The estimation probability of knowing the concept based on indirect response is given 0.8 to the correct response and 0.2 to incorrect response If the type of question is a kind of multiple-choice, then an additional value of errors = 0.03 will be considered 111 Thus, the estimation probability of knowing the The Accuracy of the Estimation of Knowing the Concepts in DS Based on the Answer of Randomly Selected Student Comparison Between the Probability of Knowing the Concept Based on the DS Method & Real Directly Response Y : The probability of knowing the concept X : The concept ID Direct response is a response to the question asked directly about the concept at certain skill level The estimation probability of knowing the concept based on direct response is given 0.9 to the correct response and 0.1 to incorrect response If the type of question is a kind of multiple-choice, then an additional value of errors = 0.03 will be considered Thus, the estimation probability of knowing the concept based on direct response is given 0.88 to the correct response and 0.13 to incorrect response P(Kc|OQ,DQ) by the computation: is the probability of knowing the concept mathematically calculated based on
many observation (the two responses) P(Kc|DQ) by Direct response: is the probability of knowing the concept estimated by direct response P(Kc|OQ) by DS method: is the probability of knowing the concept estimated by indirect response Indirect response is a response to the question infers to the concept. It would be a question asked about the supported concept The estimation probability of knowing the concept based on indirect response is given 0.8 to the correct response and 0.2 to incorrect response If the type of question is a kind of multiple-choice, then an additional value of errors = 0.03 will be considered 112 Thus, the estimation probability of knowing the The Validation of the Estimation of Knowing the Concepts in PS Based on the Answer of a Perfect Student Comparison Between the Estimation of Knowing the Concept Based on the PS Method & Real Directly Response Y : The probability of knowing the concept X : The concept ID
Direct response is a response to the question asked directly about the concept at certain skill level Indirect response is a response to the question indirectly asked about the concept but infers to it. It would be a question asked about the supported concept The estimation by PS method: is the estimation of knowing the concept based on the related tested concepts. Wherein the related concept is the supported concept by the estimated concept. In other words, the concept in DS is a prerequisite concept to the tested concept The estimated probability given 1 to the correct and 0 to incorrect response for either the direct 113 response on indirect response The Accuracy of the Estimation of Knowing the Concepts in PS Based on the Answer of the perfect student Comparison Between the Probability of Knowing the Concept Based on the PS Method & Real Directly Response
Y : The probability of knowing the concept X : The concept ID Direct response is a response to the question asked directly about the concept at certain skill level The estimation probability of knowing the concept based on direct response is given 0.9 to the correct response and 0.1 to incorrect response If the type of question is a kind of multiple-choice, then an additional value of errors = 0.03 will be considered Thus, the estimation probability of knowing the concept based on direct response is given 0.88 to the correct response and 0.13 to incorrect response P(Kc|OQ,DQ) by the computation: is the probability of knowing the concept mathematically calculated based on many observation (the two responses) to direct and indirect response P(Kc|DQ) by Direct response: is the probability of knowing the concept estimated by direct response P(Kc|OQ) by PS method: is the probability of knowing the concept estimated by response to indirect related question Indirect response is a response to the question infers to the concept. It would be a question asked about the supported concept The estimation probability of knowing the concept based on indirect response is given 0.8 to the correct response and 0.2 to incorrect response If the type of question is a kind of multiple-choice, then an additional value of errors = 0.03 will be considered 114 Thus, the estimation probability of knowing the concept based on direct response is given 0.77 to the
The Accurate Probability of Knowing the Domain of Learning Object of the Perfect Student Based on the Evaluation Result of the Entire Concepts in VS Y : The probability of knowing the concept X : The concept ID The blue bar indicate the probability of the knowing domain The red bar indicate the probability of the knowing domain The equation which used to to calculate the probability of knowing the entire domain in the concept space VS is Bayes Version 2 115 The Accurate Probability of Knowing the Domain of Learning Object Based on the Evaluation Result of the Entire Concepts in DS Y : The probability of knowing the concept X : The concept ID The blue bar indicate the probability of the knowing
domain The red bar indicate the probability of the knowing domain The equation which used to to calculate the probability of knowing the entire domain in the concept space DS is Bayes Version 2 116 The Accurate Probability of Knowing the Domain of Learning Object Based on the Evaluation Result of the Entire Concepts in PS Y : The probability of knowing the concept X : The concept ID The blue bar indicate the probability of the knowing domain The red bar indicate the probability of the knowing domain The equation which used to to calculate the probability of knowing the entire domain in the concept space PS is Bayes version 2 117
The Size of footprint Skill Level parameter L2 L3 L4 L5 L6 Sum 14 Verified 7 4 2 1 4 18 13 Derive Potentia d l 12 11 3 2 3 31 13
11 2 2 3 31 12 12 11L3; 11 10 8 7 6 4 4 2 2 0 3 L2 L3 Verified Derived Potential 4 2 L4 3 3
1 2 L5; 2 L5 L6 118 Degreef, E., Doignon J.-P., Ducamp A., & Falmagne J.-C. (1986). Languages for the assessment of knowledge. Journal of Mathematical Psychology, 30, 243-256 Falmagne, J.-Cl. and Doignon, J.-P. (1988), A class of stochastic procedures for the assessment of knowledge. British Journal of Mathematical and Statistical Psychology, 41: 123. doi: 10.1111/j.2044-8317.1988.tb00884.x Zwillinger, D., & Kokoska, S. (2000). standard probability and Statistics tables andformulae. CRC Press. 120