In the October edition of Edubytes, our guest editors are members of the UBC LT Hub Learning Analytics team (LA team): Jeff Longland, Craig Thompson, Justin Lee, Sung Hwang, Alison Myers, and Lily Chen. They give an update on their role in supporting Learning Analytics at UBC and discuss the current state of the field relevant to their lens as practitioners who are developing institutional Learning Analytics at scale.
Putting Learning Analytics into Practice
In July 2021, Edubytes explored the role of Learning Analytics in education during the COVID-19 pandemic. The editorial, titled “Learning Analytics in the COVID Era”, highlighted important challenges to make the best use of Learning Analytics including: the rapid evolution of the educational technology ecosystems, difficulty in accurate predictive models, and the fact that data is not neutral.
One notable change since 2021 is an updated definition from the Society for Learning Analytics Research (SoLAR). SoLAR has expanded the description of “Learning Analytics” to reflect the maturation of the field in the decade since the first Learning Analytics and Knowledge (LAK) conference. SoLAR recognizes that Learning Analytics is a field of research and teaching that has intersections between learning, analytics, and human-centered design. See the full definition from SoLAR at: “What Is Learning Analytics.”
In 2022 SoLAR published the second edition of the Handbook of Learning Analytics which provides topic revisions and extensions to chapters and remains a valuable resource for an introduction to many of the areas of Learning Analytics. Chapters that may be of particular interest to practitioners include Chapter 2. A Practitioner’s Guide to Measurement in Learning Analytics: Decisions, Opportunities, and Challenges, and Chapter 17: Institutional Analytics.
In June 2024, the first-ever global, practitioner-focused Learning Analytics event, LAP24, was held. This 24-hour event displayed Learning Analytics in practice across the globe, providing UBC and other institutions with a platform to discuss their approaches and innovations, with a focus on practitioners. Members of the UBC community presented an institutional panel titled Tales from the Trenches: Perspectives on Learning Analytics at UBC, highlighting the diverse ways Learning Analytics has been, and continues to be, implemented at UBC.
The LT Hub Learning Analytics Team
The LA team was formed during the Learning Analytics Project which ran from 2017-2020. The project focused on three inter-related areas of activity:
- Engaging the UBC community through consultations, pilot projects and workshops.
A notable pilot that was supported by the project is the Teamable Analytics tool. The project is led by Dr. Bowen Hui and won the best demo award at the 2022 Learning Analytics and Knowledge Conference (LAK).
Another successful pilot project was the Student Flows dataset and dashboards. This dataset was collaboratively designed by folks across UBC who discovered they were conducting similar analyses of student enrollment pathways and wanted to share their approaches. You can learn more about this project from the 2021 ETUG presentation “A Collaborative Approach to Understanding Enrolment Patterns: the student flows project” from Craig Thompson and Annay Burke (PAIR). This work won the Best Presentation Award at the Canadian Institutional Research and Planning Association Conference in 2021.
Student hackathons have been a cornerstone of community engagement. The LA team and partners, including the Learning Analytics and Visual Analytics community and researchers/practitioners of Learning Analytics in various faculties continue to run the Learning Analytics Hackathon, having just hosted the 10th event where students were tasked with “hacking” the Canvas API, with a specific focus on discussion forum data. Learn more in the LAK 2024 paper “Hackathons for Awareness and Community Engagement in Learning Analytics” (PDF).
- Exploring the ethical issues regarding the use of learning data.
The Learning Data Committee was established and authored a Purpose and Principles document (PDF) to guide Learning Analytics efforts at UBC.
- Iteratively developing the technical and solution architecture needed at UBC.
The LA team developed infrastructure to ingest, process and store learning data, including the instrumentation of learning tools to emit events. You can learn more about the data infrastructure in the BCNet 2023 presentation “Learning Events and Learning Record Stores: A Practical Introduction” (PPT) from Jeff Longland.
At the completion of the Learning Analytics Project, the LT Hub established on-going Learning Analytics support to assist with data tool/visualization innovation, access to data, and support for Learning Analytics research.
Learning Analytics in Practice and at Scale
SoLAR, as a Learning Analytics community, takes a realistic, pragmatic, and transparent approach to the challenges in the field: issues such as avoiding one-size-fits-all solutions, and understanding the importance of context. The LAK16 Conference hosted the first LAK failathon (PDF) was held, which highlighted failures as learning opportunities. The 2015 paper “Let’s not forget: Learning analytics are about learning” (PDF) was a reminder of the goals of the field. At the LAK24 conference, the Best Paper award winner was “Have Learning Analytics Dashboards Lived Up to the Hype? A Systematic Review of Impact on Students’ Achievement, Motivation, Participation and Attitudes” (PDF), which questions the current body of evidence and concludes with a call to action for increased methodological rigour in the evaluation of Learning Analytics dashboards.
The challenges of Learning Analytics in practice and at scale are regularly front of mind for practitioners. NYU’s Learning Analytics Research Network (LEARN) shared its reflections as a conversation in “Scaling Up Learning Analytics: A Conversation.” At UBC, the LA team has a broad range of responsibilities: developing and maintaining technical infrastructure, conducting analysis, fulfilling data requests, and supporting LA tools – all of which can create challenges and competing priorities when supporting Learning Analytics at the institutional scale. For example, data infrastructure is foundational to meaningful Learning Analytics work, but it also requires on-going maintenance and tasks related to cybersecurity, data integrity, and data literacy.
The LA team takes the responsibility of understanding its data and appropriate use seriously. The ecosystem of learning tools and the data they produce not only need to be managed and stored, but they also need to be understood to make sense of the information and to give guidance on what data to use, where to use it and how to use it. A common example of a Learning Analytics request that has overlooked nuance is “time on task.” The question “how long are folks spending on page X in Canvas” seems like a simple question answered by calculating the difference between two student clicks. Alas, there is a human on the other side of those clicks and we can’t assume they were focused on a task in between those clicks. Time on task and whether it measures learning is a topic explored by many, including in the 2015 research “Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings,” which had a stated aim, in addition to its research findings, of raising awareness and initiating conversations about time-on-task estimations.
While challenges are important to recognize, so are the opportunities to learn from them. The role of a LA practitioner is to grapple with the intersection of learning, analytics, human-centered design, and data infrastructure daily. There is a broad range of Learning Analytics practitioners and researchers across UBC. While the focus here has been primarily on the LT Hub LA team, we also want to acknowledge the other Learning Analytics work that takes place across UBC. Many of these teams and practitioners existed before the establishment of the 2017 Learning Analytics project. While this does not recognize all the Learning Analytics work and individuals at UBC, you can learn more about Learning Analytics beyond the LT Hub.
Additional Resources
- For more information and how to request Learning Analytics support for members of UBC, visit the Learning Analytics support website.
- UBC continues to be an institutional member of SoLAR which allows members of the UBC community free SoLAR memberships.
- More information about “Instrumenting learning events at UBC” presented by Jeff Longland at ETUG, 2021.
- The written summary companion to the video “Scaling Up Learning Analytics: A Conversation.”
