Monthly Archives: March 2022

Assessing Students’ Learning — Not Their Googling Skills! — in an Online Physiology Course

As of March 2020, when the SARS-COV-2 pandemic sent teachers and students home to figure out online instruction and learning, I had been teaching high school biology/AP biology for 27 years and anatomy & physiology at the two local community colleges for 7 years. Since I had been practicing flipped coursework for years, I knew that my biggest challenge would be how to fairly assess my students and their learning. This challenge would be compounded by an at-home virtual testing environment without any proctoring.

As I pondered the best approach to my assessment challenge, I was naturally drawn to the College Board’s 2012-13 redesign of the AP (Advanced Placement) Biology curriculum and examination. In the redesign, the AP curriculum focuses on four “Big Ideas” or broad themes covering a number of subtopics/concepts that are further broken down into learning objectives for students. The examination focuses on measuring student learning and skills using what the College Board (AP Higher Education, 2012-2013) calls an “evidence-centered-design approach that parallels the curriculum’s understanding-by-design approach.” The examination consists of a mix of multiple-choice and short-answer/free-response questions. I know from my many years of grading student AP essays/short answers that, when students turn to Google for their answers, they often fail. Students will frequently regurgitate the rubrics for grading the prompts rather than dissecting and answering the question. Subsequently, the students fail to demonstrate their own learning or understanding of the material. This is unfortunate as it is also a missed opportunity for feedback, correction and/or remediation.

In designing a new accelerated online physiology course, I really wanted the course assessments to mimic the AP Biology style of assessments. I wanted them not only to be aligned with course objectives, but to require students to think about and demonstrate the skills and concepts they were learning. I was skeptical, but hopeful I could also find an approach in which I would not have to rewrite the entire examination from scratch each term. In my search for related pedagogies, I ran across an article in the May 2020 HAPS Educator, “Testing in the Age of Active Learning: Test Question Templates Help to Align Activities and Assessments,” and recognized the name of one of the authors, Dr. Greg Crowther (Everett Community College, Everett, WA) from a previous association. I reached out to Greg and requested some more details about Test Question Templates (TQTs). What I found was a pedagogical gold mine!

The TQTs are based on somewhat general learning objectives, much like the four Big Ideas of the AP Biology exam. Students often ignore these learning objectives because they don’t know what they mean or how they will be assessed, but TQTs are formatted as input-output statements that tell the student exactly what they will be assessed on. Two examples (“Example A” and “Example B”) are provided for the students, followed by a prompt encouraging students to create their own test question following the template format.

The timing of my find was perfect for incorporating TQTs into the design of the new course. Since I am totally online, I took the time to video each TQT. On video, I present the input-output statement for each TQT and present Example A, along with approaches to answering the question or solving the problem. My TQT videos are attached to a weekly discussion board in the course management system, where students are then encouraged to work on solving Example B and creating a third example. I frequently visit the discussion board and provide feedback and guidance as needed throughout the week.

Below is an example of a TQT input-output statement and examples given to students ahead of the examination in the discussion board and used to model the examination question:

TQT 3.1. Given the chemical structure or chemical formula of an ion or molecule (chemical structure or text description), list the most likely mechanism(s) by which it crosses cell membranes.

  • Example A: See structure below left. By which process(es) is this molecule most likely to cross cell membranes? Explain your reasoning. [add chemical structure of a molecule like urea]
  • Example B: See structure below right. By which process(es) is this molecule most likely to cross cell membranes? Explain your reasoning. [add image of a peptide like insulin]
  • Example C: Make up an example (think of an ion or molecule that you’ve heard of) and ask your classmates!

In the previous unit, students had been instructed on chemical structures/formulas and bonding properties. In this unit, students are asked to extend and apply their understanding of chemical structures, bonding properties (polar, nonpolar, ionic) with their new knowledge of cell membrane structure (phospholipid) and cell transport mechanisms (passive or active).

Examinations are carefully aligned with the objectives, formative assessments and exact input-output statements given to students in the TQTs. The examination contains 10-11 short answer questions and approximately 25-30 multiple choice questions. I have added a statement on the examination for students to sign, reminding them not to use any outside resources (people, notes, internet….) along with the consequences for doing so. Students are reminded to use what they are learning in the course to answer and solve exam problems/questions. I explain to students how I will know if they don’t follow the rules.

I will admit that the new course has gotten off to a rough start. For reasons I can only guess at, more than half my students are procrastinating until the last minute to start assignments (lecture, reading, lab, formative assessments, TQTs…). This approach is not consistent with my suggestions to space out their learning, practice, or repetition of concepts that we know is so important to learning and applying the information to new situations.

Not surprisingly, students who participated during the week and spaced-out lecture segments, formative assessments and TQTs did much better on the examination than those who did not. Those who chose alternative approaches to the course material often googled their way through the examination and failed miserably. Using Google, they could identify a molecule, how it is made, and where it is found, but they couldn’t answer the questions asked.

It has taken several examinations to convince many of the students that physiology is not simply about googling or memorizing facts, but about developing critical thinking skills and a higher-order understanding of the material that will persist beyond the course. More students are now actively preparing, studying and asking more complex questions throughout the week than previously (as evidenced by the course management system analytics and student contact). Many have shown improvement not only on their overall exam scores, but in their demonstrations of reasoning on assignments and exams.

After the initial rough weeks of getting students on board, students are now reaching out via email to report progress in their learning, growth, and ability to connect the material to their work as CNAs and Medical Assistants.  For example, one young man in the course writes, “As we’ve progressed onward to future chapters I feel like my knowledge is increasing gradually and I personally feel that like I CAN do this, it has been a struggle I’m not going to lie and say it was a breeze but, I feel like I’m truly getting a ton of knowledge from these chapters, I’ve found much interest on the systems we’ve been studying especially with the TQT examples and formative questions that you help me with your feedback.” Another young lady states, “I am sorry I am not doing well. I have never been forced to study before and though the TQTs are hard I am finding that I am learning a lot and am really interested in learning more. I am glad I didn’t give up.”

In summary, both the AP Biology redesign assessment questions and the TQTs have allowed me to better assess my students’ knowledge and skills. These approaches have also given me insight into student misconceptions and helped me provide feedback, remediation, and other support as needed. I can easily write (or rewrite) questions based on the TQT input/output statements without having to rewrite entire examinations each term. Students are learning that simply googling will not let them ace the exams; instead, they are learning to more carefully read the questions and answer the questions based on their own understanding.

“ACKNOWLEDGMENTS: The author thanks Greg Crowther for help implementing TQTs and for feedback on this blog post.”

References:

  1. AP Higher Education (2012-2013). AP Course and Exam Redesign. https://aphighered.collegeboard.org/courses-exams/course-exam-redesign
  2. Crowther, G., Wiggins, B., Jenkins, L. HAPS Educator (May 2020). “Testing in the Age of Active Learning: Test Question Templates Help to Align Activities and Assessments.”
    Julie Gallagher, professor of anatomy and physiology, has been teaching at Barstow Community College (Barstow, CA) since 2014 and was a high school AP Biology teacher for 27 years at Serrano HS (Phelan, Ca).  Believing in equity and inclusion, Professor Gallagher has built state-of-the-art online anatomy and physiology courses, focused on helping all students succeed.
From a Group to a Team: Medical Education Orientation Curriculum for Building Effective Teams

I am part of a small team of Core Educators in the pre-clerkship undergraduate medical education program at the Lewis Katz School of Medicine at Temple University (LKSOM).  Last year we introduced a new curriculum to our medical students.  Part of this restructuring involved changing the format of the week-long orientation for first year students.  Operating under the new title of Transition to Medical School (TTMS), we introduced education programming amongst traditional orientation activities in which we specifically address the importance of teamwork, while providing a three-part series of 1.5- to 2-hour sessions given over three days to allow the students to get to know each other, learn about team dynamics in education and medicine, and develop their small teams; practice with patient cases to get experience with a type of active learning activities which form part of the backbone of their pre-clerkship education; reflect on the previous two sessions as part of their team’s norming process.  The focus of this blog is to describe the first session of this series, which was designed to dismantle preconceived notions of team learning, highlight the potential impact of high functioning teams, and participate in asset mapping to aid in forming of teams.

A problem which we identified as we transitioned to more case-based learning leading up to the curricular change, and that was particularly highlighted during the transition to virtual and then hybrid teaching and learning during the Covid-19 pandemic, was that medical students often struggle to learn in dysfunctional small groups if they do not first gain the skills to create and sustain high functioning, collaborative teams. Ineffective group dynamics led to limitations in students learning the material and resulted in less buy-in of the value of the case-based activities.  In addition, the downstream effects of dysfunctional team dynamics are well documented and include poor patient outcomes1. This is important as our competencies include preparing students for working in patient care teams.

We began the first education session with a word cloud activity to allow students and faculty to learn about the students’ pre-conceived ideas regarding group work.  Students were asked to submit using software (we used mentimeter.com) a word or phrase that came to mind when we said “group work”; the app then collated and displayed their responses in a figure composed of words.  Words which were submitted by multiple participants appeared larger in the word cloud (see figure for an example of a word cloud).  In our word cloud (not shown) the most frequently submitted words included “collaboration”, “communication”, “stressful”, “teamwork”, “frustrating”, and “compromise”.  Other words and phrases which appeared included “painful”, “judgment”, “overwhelming”, “open minded”, “unequal effort”, “hearing every voice”, “more work”, “understanding”, “innovative”, “constructive”, “helpful” “divide and conquer”, and “mixed bag”.  It was evident and probably not surprising that there was a range of responses from the more skeptical or negative to the more positive and enthusiastic.

Next, we shared information gathered from the literature with regards to the importance of small group, active learning in medical education.  The literature indicates that students who participate in small group learning activities demonstrate improved levels of critical thinking as compared with their peers who participate in lecture-based activities only2-4.  It has also been shown that small group work promotes communication skills5, active learning, cooperation, engagement, and retention of material6.

We then spent a few minutes reviewing the importance of diverse, effective teams in medicine.  The literature indicates that vulnerable patients with multiple chronic conditions have many doctors on their care team.  The number of people involved in a patient’s care is also increased by the nature of interprofessional roles in medicine.  Care teams include physicians (attendings, fellows, residents), medical students, nurses, physician assistants, nurse practitioners, medical assistants, pharmacists, case managers, social workers, physical and occupational therapists, technicians, pathologists, lab specialists, front desk personnel, billing specialists, and many more.  Therefore, it is imperative that students practice their communication and teamwork skills to provide their patients with the best possible care.

We also described to the students the difference between a “group” and a “team”.  A “group” can be defined as a number of people who are associated together in work or activity and has a set leader.  The group members may not work with each other but report directly to that leader, only hold themselves accountable, and rarely assess progress or celebrate successes7.  Revisiting the list above from our students’ word cloud activity, “unequal effort”, “divide and conquer”, and “more work” may be used to describe this kind of group.  In contrast, a “team” includes a small number of people with complimentary skills, who are committed to a common goal and purpose, who set performance goals and hold themselves mutually accountable.  They may share leadership and value open-ended discussion and active problem-solving7.  The terms “open minded”, “hear every voice”, “collaboration”, and “communication” from our students’ word cloud are aspects of a team.

Next, we asked the students to move into their assigned teams of 6-7 students for an asset mapping activity.  The goal of asset mapping is to create more equitable team dynamics by having students identify their own assets and share them with their team.  Each team was assigned to stay together for their first semester courses, so this experience not only allowed the students to think about their contributions to the team, but also served as an icebreaker in a classroom setting for the students before they began their first course.  We used an asset map (see figure) we adapted from George Pfeifer and Elisabeth Stoddard from Worcester Polytechnic Institute, who authored “Equitable and Effective Teams: Creating and Managing Team Dynamics for Equitable Learning Outcomes”8 and from Cliff Rouder of Temple University’s Center for the Advancement of Teaching, who authored “Asset Mapping: An Equity-Based Approach to Improving Student Team Dynamics”9.  Students were given time individually to complete their asset map, and then were instructed to share parts of their maps with their teammates.  Anecdotally, we were impressed with the depth of conversations, the degree of engagement and participation with each team, and the enthusiasm the students shared with each other.  An anonymous RedCap survey was given to the students after TTMS ended, and 87% of responding students indicated they found the asset mapping session useful (response rate was 97% of the class).

The Association of American Medical Colleges (AAMC) reports 11% of students in medical schools identify as historically underrepresented in medicine.  At LKSOM, our current M1 and M2 classes are both comprised of ~30% students who are historically underrepresented in medicine.  Our students come from a diversity of backgrounds and lived experiences, and have varying interests, skills, passions, and responsibilities.  Asset mapping provided a mechanism by which our students could initially learn about and from each other, and later led to conversations which allowed the teams to set their goals and expectations, and hopefully work towards providing a more equitable experience.  Asset mapping can be used to reassess team dynamics and for forming new teams as students progress through the curriculum.  This tool can also be used to help students optimize team dynamics for those who are struggling or underperforming.

This is an example of how sharing the literature with respect to the value of small group learning, team dynamics, and the role of asset mapping was useful in the building of teams in the first semester of medical school.  However, these tools could be adapted and used for learners at any level, or for team building within our departments.

The LKSOM Core Educator Team includes: Jill Allenbaugh MD, Bettina Buttaro PhD, Linda Console-Bram PhD, Anahita Deboo MD, Jamie Garfield MD, Lawrence Kaplan MD, David Karras MD, Karen Lin MD, Judith Litvin PhD, Bill Robinson PhD DPT, Rebecca Petre Sullivan PhD

 

References:

  1. Mitchell R, Parker V, Giles M, Boyle B. The ABC of health care team dynamics: understanding complex affective, behavioral, and cognitive dynamics in interprofessional teams. Health Care Manage Rev. 2014 Jan-Mar;39(1):1-9. doi: 10.1097/HCM.0b013e3182766504. PMID: 24304597.
  2. Tiwari, Agnes & Lai, Patrick & So, Mike & Yuen, Kwan. (2006). A Comparison of the Effects of Problem-Based Learning and Lecturing on the Development of Students’ Critical Thinking. Medical education. 40. 547-54. 10.1111/j.1365-2929.2006.02481.x.
  3. Charles Engel (2009) An Internet Guide to Key Variables for a Coherent Educational System Based on Principles of Problem-Based Learning, Teaching and Learning in Medicine, 21:1, 59-63, DOI: 10.1080/10401330802384888
  4. Kamin, Carol & O’Sullivan, Patricia & Younger, Monica & Deterding, Robin. (2001). Measuring Critical Thinking in Problem-Based Learning Discourse. Teaching and learning in medicine. 13. 27-35. 10.1207/S15328015TLM1301_6.
  5. Walton H. Small group methods in medical teaching. Med Educ. 1997 Nov;31(6):459-64. doi: 10.1046/j.1365-2923.1997.00703.x. PMID: 9463650.
  6. Van Amburgh JA, Devlin JW, Kirwin JL, Qualters DM. A tool for measuring active learning in the classroom. Am J Pharm Educ. 2007 Oct 15;71(5):85. doi: 10.5688/aj710585. PMID: 17998982; PMCID: PMC2064883.
  7. Katzenbach, JR & Smith, DK. (2005). The discipline of teams. Harvard business review. 83. 162-+.
  8. Pfeifer, Geoffrey and Elisabeth A. Stoddard (2019). “Equitable and Effective Teams: Creating and Managing Team Dynamics for Equitable Learning Outcomes” in Kristin Wobbe and Elisabeth A. Stoddard, eds. Beyond All Expectations: Project-Based Learning in the First Year.
  9. Rouder, C (2021). Asset Mapping: An Equity-Based Approach to Improving Student Team Dynamics.  Temple University Center for the Advancement of Teaching.  https://teaching.temple.edu/edvice-exchange/2021/03/asset-mapping-equity-based-approach-improving-student-team-dynamics.
Dr. Rebecca Petre Sullivan earned her Ph.D. in Physiology from the Lewis Katz School of Medicine at Temple University and completed a Post-Doctoral Fellowship in the Interdisciplinary Training Program in Muscle Biology at the University of Maryland School of Medicine.  She taught undergraduate biology courses at Ursinus College and Neumann University.  As an Associate Professor of Physiology in the Department of Biomedical Education and Data Science and the Department of Cardiovascular Sciences, and as a Core Basic Science Educator, she is currently course director in the Pre-Clerkship curriculum at LKSOM and at the Kornberg School of Dentistry; in addition to teaching medical and dental students, she also teaches physiology in Temple’s podiatry school, in the biomedical sciences graduate program, and in the physician assistant program.  She is a member of Temple University’s Provost’s Teaching Academy.  She was the recipient of the Mary DeLeo Prize for Excellence in Basic Science Teaching in 2020, the Golden Apple Award in 2017 and 2021, and the Excellence in Education Award, Year 2 in 2020 from LKSOM, and the Excellence in Undergraduate Teaching Award from Neumann University in 2012.