From January 2012 to December 2012, UBC conducted a year-long, multi-phase mixed-methods research study to understand users’ experiences (students and faculty) with Connect, UBC’s new Learning Management System (LMS). Each phase involved three data collection points: (1) an initial online survey, (2) a follow-up survey, and (3) individual interviews. With the goal of engaging in evidence-based practice for UBC’s implementation of the new LMS, the study provided an opportunity to better understand users’ experiences, professional development needs, and system operations prior to the broader rollout.
Profile of Respondents
The majority of instructor respondents had several years of teaching experience as well as experience using a LMS. Of the instructors who responded to the initial online survey, 76% had six years or more of teaching experience at the university level. Further, 46% identified as innovators or early adopters of technology, 46% identified as mainstream adopters, and 8% identified as late adopters. Of the students who responded to the initial online survey, 81% were enrolled in an undergraduate degree program. 33% identified as innovators or early adopters of technology, 57% identified as mainstream adopters, and 10% identified as late adopters.
Findings
Instructor and student respondents had various overall impressions of Connect, ranging from finding the system awkward and not intuitive to being impressed by its functionality. For example, Doug1, a first year student at UBC Vancouver commented, “When I first got into the website I was a bit taken aback because looking at the entire thing, it looks like it’s going to be a very complicated system to use.” Adam, a first year student at UBC Vancouver said, “The interface of the new LMS wasn’t very intuitive. It seems like it has a lot of functionality but all of that functionality just seems jumbled together. While there is so much you can do with the Connect system, it’s not very well laid out.” On the contrary, Carole, an instructor at UBC Vancouver said, “It just seemed really user-friendly.” Susan an instructor at UBC Vancouver noted, “It’s a bit more flexible. There are things I like about it. You can personalize a little bit more.”
Overall, 48% of instructors feel that they are receiving enough support, and they are accessing support primarily through Instructional Support Staff. The majority of instructors (60%) provide some form of support to their Teaching Assistants, and many other instructors provide some support for each of the tools in Connect to either their students or Teaching Assistants. While many students (36%) do not believe that the support structures put in place have helped them to be successful with Connect, many (35%) do not seek support. The primary sources of support for students are peers (38%) and the course instructor (34%). Upon examining student responses by year of study, we found that students in their first year are two times more likely to find the new LMS easy to learn compared to those in their fourth year, provided that the level of support is held constant.
Advanced statistical analysis of the student and instructor data revealed some important findings regarding students’ and instructors’ perceptions of support and finding the system easy to learn. For example, students who felt supported half-way through the term are 9 times more likely than students who did not feel supported, to give a favourable response to the question, “Overall, the new LMS is easy to learn.” Similarly, instructors who felt supported half-way through the term are 11 times more likely than instructors who did not feel supported, to give a favourable response to the same question. These findings highlight the significance of ensuring that students and instructors feel supported in order to increase the likelihood that they will find the system easy to learn.
During interviews, instructors spoke about their experiences with communication related to Connect. For example, Bettie, an instructor at UBC Vancouver, spoke about her preference to receive communication focused on changes or new functionality. She stated that she would like to receive communication, “Just when things have changed or if there is new things or new functionality. If there is new instruction modules, I’d love to hear.” Susan, an instructor at UBC Vancouver also expressed a desire to receive communication that was focused on her specific needs as an instructor by stating, “If I had an explanation of why this is an advantage from an instructor perspective maybe that would help.” In speaking about preferences for receiving communication, many instructors spoke about their desire to receive communication from their department support staff. For example, Bettie said, “If it came from Bob [support staff] I would know that it was relative to this. I do get lots of emails from different parts of UBC so if it specifically said, information on Connect then I would be likely to read it.”
To support research-informed and evidence-based practice, CTLT is currently disseminating findings from the LMS research project to various stakeholder groups within UBC, including the LMS program steering committee, the Learning Technology Advisory Group, and staff working at the teaching and learning centres and information technology units. We are currently supporting stakeholder groups to generate recommendations from the data in order to inform the evidence-based rollout of Connect, including operating procedures, professional development, and user support. In addition, CTLT has been sharing the research with individuals beyond the UBC community in order to help inform other institutions’ LMS transitions. For more information about the LMS research project, please contact Roselynn Verwoord, Evaluation and Research Coordinator, at roselynn.verwoord@ubc.ca or Tammy Yasrobi, Communications Specialist, at tammy.yasrobi@ubc.ca.
1 All named participants have pseudonyms. The pseudonyms were chosen by the Evaluation and Research Coordinator in order to protect the identity of the participants involved in the study.
This article was published in the July 2013 CTLT Newsletter, Dialogues. Below is a list of the articles included in the issue:
- Join us for the CTLT Summer Institute!
- Flipped Lab: Examining What Works in a Flipped Classroom
- Truth and Reconciliation at UBC: Confronting Our Legacies
- Connect Research Project: Results of An Innovative Study on User Experience (currently viewing)
- Connecting Communities of Practice at UBC
- Facilitator Spotlight: Natalie Baloy
Find out more information about the CTLT Newsletter, Dialogues.