I see education-based research projects playing two parts in my scholarship of teaching:
- to evaluate the impact of a given teaching practice or intervention used in the classroom
- to support and give back to the education research community by providing results of evidence-based teaching practices
The following sections provide brief overviews of some of the research projects I have initiated or have been involved with.
Investigating the use of a programming practice tool in CS1
Motivated by previous experience with proof-of-concept programming practice tools to encourage student programming practice in an introductory programming course at UBC, I have piloted PrairieLearn as a more robust and reliable tool in CS1 at UVic.
PrairieLearn is an online platform for creating and delivering high-quality learning experiences and assessments for university students in STEM (Science, Technology, Engineering, and Mathematics). The tool provides infrastructure that allows instructors to write a single question that generates hundreds of variants of that question that will appear slightly different to a student. The benefits of this feature is two-fold:
- PrairieLearn can be used to create practice problems that students can repeatedly complete until they have mastered a concept. Students receive real-time feedback allowing them to dictate when their learning happens as they are not reliant on human feedback.
- PrairieLearn can also be used to generate exams for a class in which the questions cover the same concept yet the actual questions between students are slightly different. These easily created question variants make copying from another student futile as the answers will not be the same.
PrairieLearn facilitates automated grading of these questions that go beyond standard multiple choice and fill-in-the-blank. These automatically graded questions can involve solving of math/physics problem, reading PrairieLearn generated graphs, etc. Similar tools exist (ie. Webworks) but this tool provides a more elaborate interface for question creation as well as support for creating computer science related questions. Many computer science courses have learning outcomes surrounding “code-writing”, that is, creating software to run on a computer that solves a given problem. PrairieLearn provides instructors of these courses with all of the benefits describe above: creation of question variants and real-time automated feedback both about the correctness and quality of a student’s submitted answer.
A summary of the key features of PrairieLearn include:
- the use of auto-grading to enable expansion to larger and more diverse student populations
- mitigates academic misconduct by supporting the generation of question variants
- allows students to repeat question variants in an interactive environment and allows instructors to have frequent assessment within their courses worth a lower number of marks that leads to increased practice by students and subsequently potential improvements in learning.
- allows instructor to create complex and sophisticated questions without requiring advanced knowledge of the underlying infrastructure of PrairieLearn
In Sept 2023 we ran a term long study to assess student self-efficacy and test anxiety through surveys after each PrairieLearn supervised test.
Summary of preliminary results:
- test anxiety was higher and self-efficacy was lower for women
- correlation found between self-efficacy and grade
- no correlation found between test anxiety and grade
The results of this research have been presented at WCCCE2023, WCCCE2024 and ITiCSE 2024 with the papers PrairieLearn in CS1: An Experience Report and Test Anxiety and Self-Efficacy in a Computer-Based Test Environment and poster Test Anxiety, Self-Efficacy & Prior Experience published in the conference proceedings.
Investigating the role of abstraction in first-year programming
Working with undergraduate student Kezia Devathasan and Dr. Daniela Damian, we have begun a study to look the implications of introducing abstraction as a concrete concept in CS2 using the CSC115 summer 2021 course as a case study:
This research project will investigate various teaching techniques in increasing the students’ understanding of computer science (CS) concepts and in particular abstraction in first year programming courses. Previous research shows that abstraction is difficult to teach in introductory courses because it has no formal rules and it is seen as a soft topic. The student will design an empirical study where certain techniques are introduced and an evaluation instrument that measures students’ understanding will be designed and deployed before and after the techniques are used in CSC115 (Fundamentals of Programming II). Questions on the evaluation instrument (e.g. survey) will likely involve self-assessment regarding how well students feel they understand concepts (abstraction), as well as a few technical questions. Specific to the technical questions, we would like to take note of how well students understand the difference between a data type and its implementation. Qualitative and quantitative analyses will be conducted to understand sources of confusion amongst students, and to measure if an understanding of abstraction has a relationship to programming ability at the introductory level.
The results of this study was presented at SPLASH-E 2022 with the paper The Role of Abstraction in Introductory Programming published in the conference proceedings.
Adding points of synchronization to asynchronous online content delivery
With the shift to online instruction in the midst of a pandemic we were encouraged as instructors to provide asynchronous access to course materials so as not to disadvantage students with timezone or technology and bandwidth constraints. From an instructor’s perspective, while wanting to ensure accessibility of course materials, it is also important to ensure students are keeping up with the material through frequent assessments and feedback. These check-points are critical for courses in which the material builds on itself, becoming increasingly challenging as the term progresses.
In the design of CSC 110 online course as described the Curriculum Development page, I leveraged a flipped-classroom approach with assigned pre-lecture was leveraged to support in-class problem solving hosted online.
Access to pre-lecture content (slides, videos) was asynchronous but low-stakes, graded quizzes on this material were due before lecture to introduce a synchronization point and to encourage students to have looked at the material before lecture.
While lectures were held synchronously, they were recorded to allow for asynchronous access.
Weekly lab and assignment exercises with increased stakes in terms of marks relative to the quizzes were posted at the beginning of each week. Due dates for labs and assignments provided synchronization points and again, served to give students feedback on their progress. While students were able to attend synchronous lab time for support, this was optional.
Three synchronous online exams provided students with more summative feedback as they worked through problems on their own enforced by strict time constraints and limiting the exam to forward navigation.
The following table overviews a typical week’s schedule highlighting these points of synchronization.
| Mon | Tues/Wed | Thurs | Fri/Sat | Sun | |
| Videos | to be watched before quiz | to be watched before quiz | |||
| Quizzes | due before lecture | due before lecture | |||
| Lecture | problem solving: students given time to try/ask questions, then solution is demoed | problem solving: students given time to try/ask questions, then solution is demoed | |||
| Lab | released | optional help in registered labs | due by end of day | feedback given | |
| Assignment | released | due by end of day |
In an attempt to assess the aspects of the this course design I performed an analysis of student interaction with asynchronous course materials and participation in synchronous lab and lecture time with final grades.
With lab attendance being so low it was difficult to determine any relationship but lecture attendance had a slight correlation to final grades (.354). Student interaction with the materials (videos and slides) before coming to lecture showed a moderate correlation to final grades.
A presentation of this approach was given at UVic’s Let’s Talk About Teaching online conference and as part of UVic Engineering’s From Stress to Success Speaker Series in August 2020. The presentation overviewed challenges and rewards encountered in the design and deployment of this online offering along with an analysis of student participation data.
Investigating the use of programming practice tools in an introductory programming course
Motivation
APSC 160 is the introductory programming course for UBC engineers and has been designed to follow a flipped-classroom model. Students watch screencasts before class to introduce concepts, clicker questions at the beginning of class to gauge their understanding and fuel explanations and finally the larger portion of the class time is spent going through problems on a worksheet. Labs each week serve as low-stakes, computer-based tests that are supported with related pre-lab exercises and solutions.
In my anecdotal observations, students were doing very little programming on their computers outside of lab and lecture time. Often they were disengaging with the pre-lecture videos and coming to class unprepared for the worksheet.
In a WCCCE ’16 paper I proposed the addition of a programming practice tool into the course that could be used by students for both pre-lecture preparation in conjunction with the screencasts and for practice and review of material covered in the worksheets.
Initial Design and Deployment
Taking an existing programming practice tool, I worked with a student development team to adapt and deploy this tool at UBC over the summer of 2016. This tool was chosen for its extensive data collection capabilities that would allow us to track student interaction. The deployed tool provided students with formative feedback through code-reading and code-writing questions that were broken down by lecture topic with both pre- and post-lecture questions. The pre-lecture questions intended to test student understanding of pre-lecture assigned screencasts and post-lecture questions intended to give students more in-depth practice, augmenting in-class worksheets.
This development included:
- updating of the deprecated portions of the tool source code
- creation of questions for each of the 16 modules of the course
- 2-4 pre-lecture questions per module
- 4-11 post-lecture questions per module
Below are example screenshots showing the categories of suggested pre-lecture and post-lecture exercises for each of the course modules (links on the left menu bar). Sample questions from two different modules are shown.
No solutions were provided for the problems but the tool has an integrated hint system that provides pointers and suggestions based on common mistakes. Additionally, students were encouraged to discuss problems and solutions on the course discussion forum.
The practice tool was offered to three sections of APSC 160 2016W1 and was requested for use by APSC 160 2016W2 instructor for two more sections. The tool was provided as an optional resource to approximately 900 students across 2 terms and 5 sections where approximately 500 accounts were created and with each account making between 2 and 672 attempts to solve a problem.
In analyzing the tool interaction data we noticed the practice tool was heavily used before the midterms and final exam but not consistently across the term as intended. Given students created anonymous accounts we were unable to correlate grades with the tool interaction data. We did however observe increased activity on the forums and development of an online community of support with students both asking and answering each others questions relating to the practice tool exercises.
A poster containing background references and initial observations was presented at the UBC Science Education Open House (2017) and can be found on the Dissemination and Funding page.
Adaptations and Redeployment
The tool was updated to integrate with UBC’s Campus Wide Login (CWL) to allow us to track student interaction by student identifier. This feature was leveraged to allocate grades for completion of pre-lecture questions before the corresponding lecture deadline. Grades were awarded for correct answers to questions submitted before the deadline, allowing students unlimited attempts and assigning the highest score at the deadline as the student grade. The questions remained available, along with post-lecture questions for student practice, but grades were not awarded for this practice.
This intervention was applied to all three sections of APSC 160 2018W1 which consisted of 368 students. A student completing all pre-lecture questions correctly before the set deadline would have achieved a score of 14. As is illustrated in the graph below, a significant number of students were motivated to distribute their access of the tool across the term. The data showing over 56% of the students achieved a score of at least 13/14 and over 79% of the students achieved a score of at least 10/14.
Project Status
The current version of the tool can be found here: GitHub Repo
The project has received funding support from the CWSEI and details can be found on the Dissemination and Funding page.
Presentations related to this project were given at WCCCE (2016), the UBC-Okanagan Annual Learning Conference (2017) and the UBC Science Education Open House (2017) can be found on the Dissemination and Funding page.
With my move to the University of Victoria I intend to continue to experiment with programming practice tools to encourage students to read and write code in preparation for and in review of lecture. My current focus in on the development of content necessary to support a flipped-classroom with a longer term goal to integrate of pre and post lecture programming practice into student learning.
Investigating ways to support Self-Regulated Learning (SRL) and measuring self-efficacy
Development of student meta-cognitive skills is necessary for them to become life-long learners. I initially experimented with educating students on the impact of study strategies on success in Computer Science classes described in the Curriculum Development section in the CPSC 110 materials.
Later, I participated in a Community of Inquiry as part of the PhD research project: SRL in University courses led by Silvia Mazabel from Educational & Counselling Psychology & Special Education. My participation in this study involved completing questionnaires, providing examples where SRL would be integrated into my course, access to my course page and multiple meetings with three other participating instructors from other departments and faculties across campus. Additionally, Silvia followed up by interviewing student volunteers from the class.
Drawing from literature on Self-Regulated Learning, in the APSC 160 2018 W1 offering I introduced a self-reflection components into the course. The flipped-classroom style of this class naturally has a SRL component, as students work at their own pace through exercises and I would either use their solutions or construct a solution to clarify misconceptions in front of the class.
This project allowed me to both experiment with SRL in the classroom as well as get both ideas and feedback from the community of instructors. In the end Silvia provided each community member with results of her study that demonstrated that my perceptions aligned with those of the students.
For example, Silvia recording me as saying things like:
- “I think that’s really where all the learning happens, getting them involved in the process gets them to the point that you want them to be.”
- “It feels chaotic but at the same time I look around and people are engaged in their learning and they’re trying and they’re making mistakes and fixing them”
While student comments supported this perspective:
- “She showed you rather than told you”
- “She asked us questions and gave us hints to help solve the problems”
- “Giving people the opportunity to try and share other paths and realize if it works or not, that kind of give the incentive, you feel good about seeing your work shown as an example in the classroom”
Supporting English Language Learners
Our post-secondary institutions are seeing a growing number of English Language Learners (ELLs) in the classroom from a wide range of backgrounds. Vantage College leverages the Content and Language Integrated Learning (CLIL) approach [1] to teaching, in which the language exercises are also a means of developing discipline specific skills. For each discipline specific course that a Vantage College student takes, they are also registered in a linked course English language course. Specifically for Computer Science (CS1), students were registered in a CS1-Content and Language Enrichment course comprised of a weekly one-hour CS1 language focused class (CS1-Language Class) that was taught by an Academic English faculty member and a weekly one-hour CS1 content focused tutorial (CS1-Content Tutorial) that was taught by computer science teaching assistants.
The CS1-Language Class and CS1-Content Tutorial aim to develop students’ English language skills while enhancing their understanding of the discipline specific content. While often the role language skills play within a discipline specific course is clear, such as reading a textbook, writing lab reports and developing vocabulary but with other courses it is not as explicit. In CS1, students are introduced to an additional language, a programming language. Students must translate an ambiguous problem described in plain English to an explicit solution written in a programming language, a challenging skill for learners irrespective of language.
Working closely with the Computer Science and Academic English teams we leveraged Universal Design for Learning (UDL) principles to guide the design of curriculum for CS1-Language Class and CS1-Content Tutorial. The results of this work were published and presented at WCCCE 2017 and can be found on the Dissemination and Funding page.
We also see the potential for the exercises developed in our interdisciplinary approach to have application to a population beyond ELLs to support development of problem-solving skills and soft-skills such as group work, communication and negotiation. In a workshop hosted at UBC-Okanagan 2017 Learning Conference, we offered participants hands-on experience with a select set of English language exercises. Participants were shown how the learning outcomes of the language courses (CS1-Content and Language Enrichment) are achieved while offering students additional exposure to the discipline specific content (CS1-Content Tutorial). A reference to this workshop can be found on the Dissemination and Funding page.
[1] Coyle, D., Hood, P. and Marsh, D. 2010. Content and language integrated learning. Ernst Klett Sprachen.
Negotiation
Looking at the curriculum development that I have done for VANT 140 in the Curriculum Development page you will see that Vantage Science moved away from a tight integration of the language and content components of science courses. To support this disconnection I redeveloped the curriculum for VANT 140 – CPSC 110 2016W2 creating 11 weekly lesson plans with corresponding learning outcomes and exercises.
The curriculum development for these tutorials was based largely on literature related to the use of negotiation within second-language learning environment. Characteristics of task-based learning activities from related literature were leveraged and adapted to fit learning objectives of CPSC 110 in the development of lesson plans. That is, each lesson plan consisted of a task to be performed by the students with points of negotiation injected, either with classmates coming to a common solution or in presenting a solution or answer to the TA and classmates. This interactive approach to learning requires negotiation for meaning between learners and draws attention to knowledge and skills that are problematic for the learner allowing other students and TAs to help fill these knowledge gaps.
In anecdotal reflections by the TAs, students engaged in discussion when there were concrete trade-offs they could discuss but disengaged when asked to articulate a less concrete process for solving a problem. Motivating attendance was also said to be a challenge as students were not awarded any marks for this portion of the class.
A poster containing background references and concrete examples of lesson plans and student work followed by TA observations was presented at the UBC Science Education Open House (2017) and can be found on the Dissemination and Funding page.