Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-848d4c4894-4rdrl Total loading time: 0 Render date: 2024-06-28T07:39:56.399Z Has data issue: false hasContentIssue false

23 - Leveraging the Integrated Development Environment for Learning Analytics

from Systems Software and Technology

Published online by Cambridge University Press:  15 February 2019

By
Adam Carter ,
Christopher Hundhausen  and
Daniel Olivares
Edited by
Sally A. Fincher  and
Anthony V. Robins
Sally A. Fincher
Affiliation:
University of Kent, Canterbury
Anthony V. Robins
Affiliation:
University of Otago, New Zealand
Get access

Summary

In recent years, learning process data have become increasingly easy to collect through computer-based learning environments. This has led to increased interest in the field of learning analytics, which is concerned with leveraging learning process data in order to better understand, and ultimately to improve, teaching and learning. In computing education, a logical place to collect learning process data is through integrated programming environments (IDEs), where computing students typically spend large amounts of time working on programming assignments. The possibility of using IDEs both to collect learning process data and to strategically intervene in the learning process suggests an exciting design space for computing education research: that of IDE-based learning analytics. In order to facilitate the systematic exploration of this design space, we present an IDE-based data analytics process model with five primary activities: (1) Operationalize Behaviors, (2) Collect data, (3) Analyze data, (4) Design intervention, and (5) Deliver intervention. For each activity, we identify key design dimensions, and review relevant computing education literature. We conclude by discussing future directions of IDE-based learning analytics.
Type
Chapter
Information
The Cambridge Handbook of Computing Education Research , pp. 679 - 706
Publisher: Cambridge University Press
Print publication year: 2019

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Ahadi, A., Lister, R., Haapala, H., & Vihavainen, A. (2015). Exploring machine learning methods to automatically identify students in need of assistance. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (pp. 121–130). New York: ACM.CrossRefGoogle Scholar
Ahonen, L., Cowley, B., Torniainen, J., Ukkonen, A., Vihavainen, A., & Puolamaki, K. (2016). S1: Analysis of Electrodermal Activity Recordings in Pair Programming from 2 Dyads, PLoS One. Retrieved from http://journals.plos.org/plosone/article/asset?unique&id=info:doi/10.1371/journal.pone.0159178.s001Google Scholar
Anderson, J. R., Conrad, F. G., & Corbett, A. T. (1989). Skill acquisition and the LISP tutor. Cognitive Science, 13(1), 467–506.CrossRefGoogle Scholar
Anderson, J. R., & Skwarecki, E. (1986). The automated tutoring of introductory computer programming. Communications of the ACM, 29(9), 842–849.CrossRefGoogle Scholar
Askar, P., & Davenport, D. (2009). An investigation of factors related to self efficacy for Java programming among engineering students. Turkish Journal of Educational Technology, 8(1), 26–32.Google Scholar
Auvinen, T., Hakulinen, L., & Malmi, L. (2015). Increasing students’ awareness of their behavior in online learning environments with visualizations and achievement badges. IEEE Transactions on Learning Technologies, 8(3), 261–273.CrossRefGoogle Scholar
Baker, R. S. J., & Siemens, G. (2014). Educational data mining and learning analytics. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences, 2nd edn. (pp. 253–274). Cambridge, UK: Cambridge University Press.Google Scholar
Banovic, N., Buzali, T., Chevalier, F., Mankoff, J., & Dey, A. K. (2016). Modeling and understanding human routine behavior. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 248–260). New York: ACM.Google Scholar
Barr, A., Beard, M., & Atkinson, R. C. (1976). The computer as a tutorial laboratory: The Stanford BIP Project. International Journal of Man–Machine Studies, 8(1), 567–596.Google Scholar
Bednarik, R., & Tukiainen, M. (2004). Visual attention tracking during program debugging. In Proceedings of the Third Nordic Conference on Human-computer Interaction (pp. 331–334). New York: ACM.Google Scholar
Begel, A. (2016). Fun with software developers and biometrics: Invited talk. In Proceedings of the 1st International Workshop on Emotion Awareness in Software Engineering (pp. 1–2). New York: ACM.Google Scholar
Braught, G., & Midkiff, J. (2016). Tool design and student testing behavior in an introductory Java course. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (pp. 449–454). New York: ACM.CrossRefGoogle Scholar
Bosch, N., & D’Mello, S. K. (2013). Sequential patterns of affective states of novice programmers. Presented at The First Workshop on AI-Supported Education for Computer Science (AIEDCS 2013), Memphis, TN.Google Scholar
Brown, N. C. C., Kölling, M., McCall, , , D., & Utting, I. (2014). Blackbox: A large scale repository of novice programmers’ activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (pp. 223–228). New York: ACM.Google Scholar
Buffardi, K., & Edwards, S. H. (2013). Impacts of adaptive feedback on teaching test-driven development. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (pp. 293–298). New York: ACM.Google Scholar
Bulling, A., Blanke, U., & Schiele, B. (2014). A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys, 46(3), 33:1–33:33.Google Scholar
Busjahn, T., Schulte, C., Sharif, B., & Antropova, M. (2014). Eye tracking in computing education. In Proceedings of the Tenth Annual Conference on International Computing Education Research, Glasgow (pp. 3–10). New York: ACM.Google Scholar
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human–Computer Interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Cardell-Oliver, R. (2011). How can software metrics help novice programmers? In Proceedings of the Thirteenth Australasian Computing Education Conference – Volume 114 (pp. 55–62). Darlinghurst, Australia: Australian Computer Society, Inc.Google Scholar
Carter, A. S. (2013). OSBIDE. Retrieved from http://osbide.codeplex.comGoogle Scholar
Carter, A. S., & Hundhausen, C. D. (2015). The design of a programming environment to support greater social awareness and participation in early computing courses. Journal of Computing Sciences in Colleges, 31(1), 143–153.Google Scholar
Carter, A. S., & Hundhausen, C. D. (2016). With a little help from my friends: An empirical study of the interplay of students’ social activities, programming activities, and course success. In Proceedings of the 2016 ACM Conference on International Computing Education Research (pp. 201–209), New York: ACM.Google Scholar
Carter, A. S., & Hundhausen, C. D. (2017). Using programming process data to detect differences in students’ patterns of programming. In Proceedings of the 48th ACM Technical Symposium on Computer Science Education (pp. 105–110), New York: ACM.Google Scholar
Carter, A. S., Hundhausen, C. D., & Adesope, O. (2015). The normalized programming state model: Predicting student performance in computing courses based on programming behavior. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (pp. 141–150), New York: ACM.Google Scholar
Carter, J., & Dewan, P. (2010). Are you having difficulty? In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (pp. 211–214). New York: ACM.Google Scholar
Corbett, A. T., & Anderson, J. R. (1992). The LISP intelligent tutoring system: Research in skill acquisition. In J. Larkin, R. Chabay & C. Scheftic (Eds.), Computer Assisted Instruction and Intelligent Tutoring Systems: Establishing Communication and Collaboration (pp. 73–110), Hillsdale, NJ: Erlbaum.Google Scholar
Denny, P., Luxton-Reilly, A., & Carpenter, D. (2014). Enhancing syntax error messages appears ineffectual. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education (pp. 273–278), New York: ACM.Google Scholar
Dominguez, A. K., Yacef, K., & Curran, J. R. (2010). Data mining for individualized hints in e-learning. In Proceedings of International Conference on Educational Data Mining (pp. 91–100). International: International Educational Data Mining Society.Google Scholar
Eclipse.org (2016). Usage Data Collector User Guide. Retrieved from https://eclipse.org/org/usagedata/userguide.phpGoogle Scholar
Edwards, S. H., & Perez-Quinones, M. A. (2008). Web-CAT: Automatically grading programming assignments. In Proceedings of the 13th Annual Conference on Innovation and Technology in Computer Science Education (pp. 328–328), New York: ACM.Google Scholar
Ericsson, K. A., & Simon, H. A. (1984). Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT Press.Google Scholar
Feliciano, J., Storey, M., & Zagalsky, A. (2016). Student experiences using GitHub in software engineering courses: A case study. In Proceedings of the 38th International Conference on Software Engineering Companion (pp. 422–431), New York: ACM.Google Scholar
Guzdial, M. (1994). Software-realized scaffolding to facilitate programming for science learning. Interactive Learning Environments, 4(1), 1–44.Google Scholar
Guzdial, M., Hohmann, L., Konneman, M., Walton, C., & Soloway, E. (1998). Supporting programming and learning-to-program with an integrated CAD and scaffolding workbench. Interactive Learning Environments, 6(1–2), 143–179.Google Scholar
Haaranen, L., Ihantola, P., Hakulinen, L., & Korhonen, A. (2014). How (not) to introduce badges to online exercises. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (pp. 33–38), New York: ACM.Google Scholar
Hartmann, B., MacDougall, D., Brandt, J., & Klemmer, S. R. (2010). What would other programmers do: Suggesting solutions to error messages. In Proceedings of the 28th Conference on Human Factors in Computing Systems (pp. 1019–1028), New York: ACM.Google Scholar
Herold, B. (2014). “Landmark” student-data-privacy law enacted in California. Education Week. Retrieved from http://blogs.edweek.org/edweek/DigitalEducation/2014/09/_landmark_student-data-privacy.htmlGoogle Scholar
Hundhausen, C. D., & Adesope, O. (2017). Leveraging Programming and Social Analytics to Improve Computing Education Workshop, Tacoma, WA. Retrieved from https://icer.acm.org/icer-2017/leveraging-programming-and-social-analytics-to-improve-computing-education/Google Scholar
Hundhausen, C. D., Agrawal, A., Fairbrother, D., & Trevisan, M. (2010). Does studio-based instruction work in CS 1?: An empirical comparison with a traditional approach. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education (pp. 500–504), New York: ACM.Google Scholar
Hundhausen, C. D., Douglas, S. A., & Stasko, J. T. (2002). A meta-study of algorithm visualization effectiveness. Journal of Visual Languages and Computing, 13(3), 259–290.Google Scholar
Ihantola, P., Sorva, J., & Vihavainen, A. (2014). Automatically detectable indicators of programming assignment difficulty. In Proceedings of the 15th Annual Conference on Information Technology Education (pp. 33–38), New York: ACM.Google Scholar
Ihantola, P., Vihavainen, A., Ahadi, A., & Toll, D. (2015). Educational data mining and learning analytics in programming: literature review and case studies. In Proceedings of the 2015 ITiCSE on Working Group Reports (pp. 41–63), New York: ACM.Google Scholar
Jadud, M. C. (2006). Methods and tools for exploring novice compilation behaviour. In Proceedings of the Second International Workshop on Computing Education Research (pp. 73–84), New York: ACM.Google Scholar
Jin, W., Barnes, T., Eagle, M., Johnson, M. W., & Lehmann, L. (2012). Program representation for automatic hint generation for a data-driven novice programming tutor. In International Conference on Intelligent Tutoring Systems (pp. 304–309), Berlin, Germany: Springer Verlag.Google Scholar
Johnson, J. (2010). Designing with the Mind in Mind: Simple Guide to Understanding User Interface Design Rules. San Francisco, CA: Morgan Kaufmann Publishers, Inc.Google Scholar
Johnson, P. (2010). Hackystat – A framework for collection, analysis, visualization, interpretation, annotation, and dissemination of software development process and product data. Retrieved from https://code.google.com/p/hackystat/Google Scholar
Kevic, K., Walters, B. M., Shaffer, T. R., Sharif, B., Shepherd, D. C., & Fritz, T. (2015). Tracing software developers’ eyes and interactions for change tasks. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (pp. 202–213), New York: ACM.Google Scholar
Kim, E., Helal, S., & Cook, D. (2010). Human activity recognition and pattern discovery. Pervasive Computing, 9(1), 48–53.Google Scholar
Kölling, M., Quig, , Patterson, B., , A., & Rosenberg, J. (2003). The BlueJ system and its pedagogy. Journal of Computer Science Education, 13(4), 249–268.Google Scholar
Leinonen, J., Longi, K., Klami, A., & Vihavainen, A. (2016). Automatic inference of programming performance and experience from typing patterns. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (pp. 132–137), New York: ACM.CrossRefGoogle Scholar
Luke, J. A. (2015). Continuously Collecting Software Development Event Data As Students Program (MSc thesis). Virginia Tech, Blacksburg, VA.Google Scholar
Müller, S. C. (2015). Measuring software developers’ perceived difficulty with biometric sensors. In Proceedings of the 37th International Conference on Software Engineering – Volume 2 (pp. 887–890). New York: IEEE Press.Google Scholar
Müller, S. C., & Fritz, T. (2015). Stuck and Frustrated or in Flow and Happy: Sensing Developers’ Emotions and Progress. In Proceedings of the 37th International Conference on Software Engineering – Volume 1 (pp. 688–699). Piscataway, NJ: IEEE Press.Google Scholar
Müller, S. C., & Fritz, T. (2016). Using (bio)metrics to predict code quality online. In Proceedings of the 38th International Conference on Software Engineering (pp. 452–463), New York: ACM.Google Scholar
NetBeans.org (2016). NetBeans Usage Data Tracking. Retrieved from http://netbeans.org/about/usage-tracking.htmlGoogle Scholar
Nguyen, L. N. N., Rodriguez-Martin, D., Catala, A., Perez-Lopez, C., Sama, A., & Cavallaro, A. (2015). Basketball activity recognition using wearable inertial measurement units. In Proceedings of the XVI International Conference on Human Computer Interaction (pp. 60:1–60:6). Berlin, Germany: Springer.Google Scholar
Norman, D. A. (2013). The Design of Everyday Things: Revised and Expanded Edition. New York: Basic Books.Google Scholar
Norris, C., Barry, F., FenwickJr., J. B., Reid, K., & Rountree, J. (2008). ClockIt: Collecting quantitative data on how beginning software developers really work. SIGCSE Bulletin, 40(3), 37–41.CrossRefGoogle Scholar
Olivares, D. (2015). Exploring learning analytics for computing education. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (pp. 271–272), New York: ACM.Google Scholar
Papancea, A., Spacco, J., & Hovemeyer, D. (2013). An open platform for managing short programming exercises. In Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research (pp. 47–52), New York: ACM.Google Scholar
Parker, M. C., Rogers, K., Ericson, B. J., & Guzdial, M. (2017). Students and Teachers use an online AP CS Principles eBook differently: Teacher behavior consistent with expert learners. In Proceedings of the 2017 ACM Conference on International Computing Education Research (pp. 101–109), New York: ACM.Google Scholar
Piech, C., Sahami, M., Huang, J., & Guibas, L. (2015). Autonomously generating hints by inferring problem solving policies. In Proceedings of the Second ACM Conference on Learning @ Scale (pp. 195–204), New York: ACM.Google Scholar
Pintrich, D., Smith, D., Garcia, T., & McKeachie, W. (1991). A manual for the use of the motivated strategies for learning questionnaire (Technical report No. NCRIPTAL-91-B-004). Ann Arbor, MI: National Center for Research to Improve Postsecondary Teaching and Learning. Retrieved from http://eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/23/3c/44.pdfGoogle Scholar
Rivers, K., & Koedinger, K. R. (2013). Automatic generation of programming feedback: A data-driven approach. Presented at The First Workshop on AI-Supported Education for Computer Science (AIEDCS 2013), Memphis, TN.Google Scholar
Robinson, D. (2017). How Do Students Use Stack Overflow? Retrieved from https://stackoverflow.blog/2017/02/15/how-do-students-use-stack-overflow/Google Scholar
Rodrigo, M. M. T., & Baker, R. S. J. D. (2009). Coarse-grained detection of student frustration in an introductory programming course. In Proceedings of the Fifth International Workshop on Computing Education Research Workshop (pp. 75–80), New York: ACM.Google Scholar
Rodrigo, M. M. T., Baker, R. S., Jadud, M. C., & Tabanao, E. S. (2009). Affective and behavioral predictors of novice programmer achievement. SIGCSE Bulletin, 41(3), 156–160.Google Scholar
Rosson, M. B., Carroll, J. M., & Sinha, H. (2011). Orientation of undergraduates toward careers in the computer and information sciences: Gender, self-efficacy and social support. ACM Transactions on Computing Education (TOCE), 11(3), 1–23.CrossRefGoogle Scholar
Rovai, A. P. (2002). Development of an instrument to measure classroom community. Internet and Higher Education, 5, 197–211.CrossRefGoogle Scholar
Scacchi, W. (2002). Process models in software engineering. In Encyclopedia of Software Engineering, John Wiley & Sons, Inc. Retrieved from http://dx.doi.org/10.1002/0471028959.sof250Google Scholar
Shaw, T. (2004). The emotions of systems developers: An empirical study of affective events theory. In Proceedings of the 2004 SIGMIS Conference on Computer Personnel Research: Careers, Culture, and Ethics in a Networked Environment (pp. 124–126). New York: ACM.Google Scholar
Shi, Y., Ruiz, N., Taib, R., Choi, E., & Chen, F. (2007). Galvanic skin response (GSR) as an index of cognitive load. In CHI ‘07 Extended Abstracts on Human Factors in Computing Systems (pp. 2651–2656), New York: ACM.Google Scholar
Shneiderman, B. (1976). Exploratory experiments in programmer behavior. International Journal of Man–Machine Studies, 5(2), 123–143.Google Scholar
Siemens, G., & Baker, R. S. J. (2012). Learning analytics and educational data mining: towards communication and collaboration. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 252–254), New York: ACM.Google Scholar
Soloway, E., Bonar, J., & Ehrlich, K. (1983). Cognitive strategies and looping constructs: an empirical study. Communications of the ACM, 26, 853–860.Google Scholar
Spacco, J., Fossati, D., Stamper, J., & Rivers, K. (2013). Towards improving programming habits to create better computer science course outcomes. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education (pp. 243–248), New York: ACM.Google Scholar
Spacco, J., Hovemeyer, D., & Pugh, W. (2004). An Eclipse-based course project snapshot and submission system. In Proceedings of the 2004 OOPSLA Workshop on Eclipse Technology eXchange (pp. 52–56), New York: ACM.Google Scholar
Spacco, J., Hovemeyer, D., Pugh, W., Emad, F., Hollingsworth, J. K., & Padua-Perez, N. (2006). Experiences with Marmoset: Designing and using an advanced submission and testing system for programming courses. SIGCSE Bulletin, 38(3), 13–17.Google Scholar
Stack Overflow (2012). Stack Overflow. Retrieved from http://stackoverflow.comGoogle Scholar
Stamper, J., Eagle, M., Barnes, T., & Croy, M. (2013). Experimental evaluation of automatic hint generation for a logic tutor. International Journal of Artificial Intelligence in Education, 22(1–2), 3–17.Google Scholar
Verbert, K., & Duval, E. (2012). Learning analytics. Learning and Education, 1(8), Retrieved from http://nbn-resolving.de/urn:nbn:de:0009-5-33367Google Scholar
Verbert, K., Govaerts, S., Duval, E., & Klerkx, J. (2014). Learning dashboards: An overview and future research opportunities. Personal and Ubiquitous Computing, 18(6), 1499–1514.Google Scholar
Vihavainen, A., Vikberg, T., Luukkainen, M., & Pärtel, M. (2013). Scaffolding students’ learning using test my code. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education (pp. 117–122), New York: ACM.Google Scholar
Watson, C., Li, F. W. B., & Godwin, J. L. (2013). Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In Proceedings of the 2013 IEEE 13th International Conference on Advanced Learning Technologies (pp. 319–323). Washington, DC: IEEE Computer Society.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×