BKT 2014 - International Workshop on Approaching Twenty Years of Knowledge Tracing

As we approach 20 years since the introduction of BKT, what lessons have we learned? Is there any consensus as to which modifications to BKT are most effective and how to integrate them? In this workshop, we would like to open floor to the discussion of recent advances in student modeling with BKT, take stock of the promises and failures of BKT, and develop integrative solutions. The four foci of the workshop will be:
1) Community standards and tools for evaluating BKT and student models more generally
2) Determining which modifications to BKT are most effective, and moving toward a unified model that integrates these modifications
3) Characterizing the advantages and disadvantages of BKT over successful “black box” approaches
4) The next 20 years of student modelling with BKT
Lessons Learned, Open Challenges, and Promising Developments
Community Standards and Model Comparison
In the field of Computational Linguistics, researchers have agreed to common evaluation metrics and datasets. For example, for the task of Part of Speech Tagging, the Wall Street Journal dataset has became the de facto standard that every published article uses for comparisons. Even the cross validation splits have been standardized with sections 2?21 dedicated for training, section 24 for development and section 23 is queried only to report the final results of the paper.
The situation is not so bright in the Educational Data Mining field. For student modeling, the evaluation decisions are done in an ad hoc way making comparisons between papers very difficult. Forthetaskofstudentmodelling,notonlyaretherenouniversallystudieddatasets,but researchers use vastly different methodology to evaluate their approaches. For example, some researchers evaluate their approaches on the final trials of the students used for training, others do so on the complete trial sequences of students held out from the training set, and others use only the final trials of held?out students. There is not even consensus on which performance metric to report, with the favorites including AIC, BIC, classification accuracy, AUC, and log likelihood. The fact that student modeling datasets are often unbalanced complicates the issue further.
Unified Approaches for Student Modelling
There has been a proliferation of interesting research suggesting extensions to or modifications of BKT. These variants tend to be studied in isolation and compared to off?the?shelf BKT. Little effort has been put into evaluating the relative merits on variant over another, and almost no work has been done integrating the most promising variants into a unified model.
The situation is even more complex because in parallel with BKT, a literature has evolved to explore latent variable models that infer static characteristics of students, skills, and problems. These latent variable models??including IRT, PFA, AFM, and NNMF??appear superficially incompatible with BKT, but various efforts have begun to argue that the approaches are complementary and in fact can be synthesized. In this workshop we invite submissions to discuss unified frameworks that integrate the various student modeling techniques and in so doing, exhibit their combined strengths.
Improving Performance without a Black Box
In 2010, an educational data mining competition was held in conjunction with the KDDD conference. The best performing method was not based on BKT but rather on standard machine learning techniques that used a large collection of features as input to black?box models (such as neural nets and support?vector machines), which were then combined as an ensemble to make predictions. Although this black?box approach achieved higher classification accuracy, this accuracy came at the expense of being able to interpret the model parameters. In contrast to these techniques, BKT obtains parameters that have meaning at the psychological level and can be used to assess and characterize students.
If BKT embodies mechanisms of human learning, then the inductive bias it provides should
ultimately lead to better predictions than a pure data?science approach. Yet we don’t appear to be there yet. One session in our proposed workshop will focus on the issue of how we can develop models like BKT that have structure and offer insight to the learning scientist, yet achieve levels of performance comparable to black?box approaches.
The next 20 years of student modelling
What is the future of the field? We are interested in future directions of student modelling. For example, extensions of BKT that can accommodate “big data” on the scale that is now being collected outside of academia.