Evaluation and Implementation

Implementing interventions, assessing the quality of instructional products, and optimizing the products based on an efficient understanding of evaluation data.

Presentation Collaboratory Fellowship Program

University of Georgia Division of Academic Enhancement, Fall 2022 - Spring 2023

A year-long scholarship program teaching communication, public speaking, and presentation skills to underserved undergraduate students. Created for the Division of Academic Enhancement at the University of Georgia as an initial pilot, the program now serves thirty fellowship students each academic year with specific education, targeted training, and financial support for their continued education.

View Details

Researching Best Practices

As this was a brand new program, I began the design process by researching for best practices regarding the teaching and development of communication skills in post-secondary students. I found research covering programs with similar goals being used in medical schools to prepare doctors for interacting with their patients. While the context and age of these students is different from this new program I was developing, the research still provided pedagogical insights and general best practices for developing communication and public speaking skills.

From my research, I gathered some principles that would guide the design of the entire program. Firstly, these skills are developed longitudinally through regular and applied practice, rather than spontaneously improving through instructional interventions alone. Practice is essential to the success of the program. Secondly, video recording and review by students is essential to teaching them the metacognitive skills to evaluate their own performance objectively rather than relying on memory and "in the moment" perceptions. Thirdly, students need to see modeling of public speakers from professionals and from their peers. Lastly, feedback given to students must be timely, regular throughout the longitude of the program, and in a consistent format every time it is received. Many resources suggest developing a rubric to use for the duration of the program and some provide existing instruments that are ready for use.

Implementing the Program

I was hired by the University to write and deliver the program over the course of a school year. After conducting preliminary research into best practices for this kind of teaching, I began developing the necessary materials to run the program while also developing the instructional interventions. Per the recommendation of the research, I worked with a SME to create a custom rubric to assess students throughout the program. I also created a program handbook that details policies and procedures regarding the , learning outcomes, program schedule, workshops, individual consultations, attendance, student expectations, and the final capstone project. I designed and developed the instructional materials including workshop slide decks, student guides for each workshop, and practice assignments. At the same time, I created and disseminated the application for students to complete.

Once the necessary program documentation and starting materials were ready, I reviewed applications and accepted our cohort of fellowship students into the program. I opened the program with a catered introductory workshop so that students could meet myself and my colleagues, and so that we could cover all of the program policies and procedures from the handbook. Essentially, I would lead a workshop once a month which all students would attend. This workshop would involve me giving direct instruction on essential principles and skills necessary for students as they begin developing their public speaking and communication skills. After the workshop, students would have an assignment which they complete individually that involves personal reflection, practice of a specific skill, or the development of a performance task. Each student would then meet with me individually for a one-on-one consultation where we focus on that student's individual strengths, weaknesses, and desired developmental goals. This process would repeat each month throughout the program until its conclusion in April with a capstone project.

Research showed that students need longitudinal practice in authentic and applied settings to best develop their skills. As a result, I developed a final capstone project which would be developed over the course of the entire program. Students would select a research topic of their choice and would present the results of their research in a short "poster session" to small group audiences. They would also give a short elevator pitch briefly explaining their topic to the entire audience before we start the small group rotations from poster to poster. This capstone gave students experience in both short- and long-form public speaking tasks, as well as experience with speaking to larger audiences and to smaller groups.

Revising Interventions

As a brand new program, the students in the cohort were periodically surveyed and assessed to determine if revisions to the initially-planned sequence and format would be needed. The student surveys focused on their desired outcomes, satisfaction with the program to date, and their experiences in the workshops and individual coaching sessions. We specifically noticed that students did not feel that the early workshops were interactive or engaging enough, so the future workshops were revised to create more student-student interaction. We also saw that there was sometimes some confusion with the individual assignments regarding the purpose or intended outcome, so the instructions were rewritten to clarify exactly what the outcome should look like and all assignments were given an introduction that more clearly explained the purpose of the assignment to the students.

Evaluating the Outcomes

Preliminary research indicated that students would benefit best form regular, timely, and consistent feedback and assessment. Feedback also needs to be in a regular and predictable format so that students can monitor and track their progress and development over the course of the program. Further, this helps to highlight areas for improvement which might need further targeting and development during individual consultations. With this in mind, I worked alongside a communications SME to modify a rubric provided by the National Communication Association to use for the duration of the program. This modified rubric assesses student performance in two domains: preparation, and performance. Each domain is comprised of three categories. In preparation, students are assessed based on their topic, thesis, and organization. In performance, the students are assessed based on their language, delivery, and supporting materials. A copy of the rubric is supplied in the materials below. This rubric evaluated performance of students formatively during the program and in practice sessions, and summatively at the end of their capstone presentations.

The program was also evaluated at the conclusion the program to ascertain the success of the program with the student cohort, and to determine if the success warranted continuation in future years beyond this initial pilot. Notably, the final evaluation demonstrated that students in the cohort significantly increased their confidence and preparatory skills in undertaking public speaking and presentation skills. All of the responding students particularly enjoyed the individual consultations and wanted to see the program continue for future students. A final report was written and an executive summary provided for division leadership who decided to award additional funding to the program for subsequent deliveries.

Lessons Learned

While this wasn't the first time I had written and implemented a new curriculum or program for students, it was the first program I created and taught for a higher education audience. I learned just how important conducting initial research can be in guiding a program in successful directions. I wanted to take constructivist approach when designing this program because undergraduate students already spend plenty of time in lecture-based classrooms and a more "hands-on" approach would certainly be more engaging. Using existing literature to justify this design approach beyond mere intuition no doubt contributed to the success of the program and the strength of its reception by students and division leadership. The fact of the matter is that this was a pilot launch with limited funding. We wanted to secure funding for additional years but needed to demonstrate concrete improvements in our students and an evidence-based pedagogical approach. Using a well-designed evaluative instrument, paired with regular assessment of the program and surveys of the students, helped us make an even stronger case for the continued existence of the program.

Download Materials

Music History eLearning Lesson

EDIT 6190e - Design Thinking and Development Tools, Fall 2021, Dr. Choi

A 30-minute Storyline 360 lesson about Romanticism in music history, designed for use as a “sub plan” for high school music educators. This module was created using Articulate Storyline to result in fully-digital, entirely online product that makes heavy use of message design and multimedia design principles to present content and assess the learner.

View Details

Design Thinking

A focus for the course is to introduce and expose instructional designers to the design thinking process that is more common in product design circles. We were challenged to apply and engage in human-centered design and design thinking paradigms for the duration of this project.

I started with the empathize and define stages. I chose to tackle a problem that many public school teachers face: the issue of leaving "sub plans" behind when they have to be absent. To define the problem very specifically, I chose to focus on creating a solution for music teachers. Music teachers, especially at middle and high school levels, have very large class sizes and it can be difficult to find work for those students to do that is meaningful, productive, and relevant when the teacher has to be absent. I planned to create an eLearning module that a teacher could plan to use in their absence to guide the students through music content, but which doesn't require the input of an instructor. These first stages were easy to approach because this is a problem I have seen and experienced myself as a former high school band director.

Next comes ideation. I brainstormed some possible solutions to this problem, and came up with a number of potential options or avenues. You can see these early sketches in the documents below. Music history time periods were an obvious choice that I did ultimately choose to pursue, but there were also options for composer biographies and profiles, and lessons on common musical forms and structures. I ultimately chose to focus on music history time periods because I felt that the content would be most conducive to the asynchronous format, would be manageable to develop, and as a final product could be serialized easily with other music history eras (like the Baroque, Classical, 20th Century) to create a "package" to offer to teachers.

Prototyping is the next stage, where I used worked to transform my sketches into static frames in Storyline, and then planned the basic user flow and micro-interactions. Once I was satisfied with the static layouts and the user flow, I began to fully develop the project into an interactive and functional module. This constituted the bulk of the work as I had to learn about using Storyline's layers, interactions and animation programming, variable tracking, and more. User testing was the "final" linear stage of the design thinking process where I had some test users experience the completed module to provide feedback on animation, navigations, interaction, pacing, and more. This user testing helped me to further refine my design to improve the user experience of the final product.

Designing for the Digital Format

I created a lesson plan, viewable in the documents below, to outline the general user flow through the module. I loosely based this lesson on Gagne's Nine Events of Instruction because I liked the linear, self-contained approach of the framework and felt that it would serve as an excellent skeleton for an eLearning module. My intent was that this lesson should take a user 30-45 minutes to complete if they take their time with the listening examples and read the text for understanding. This did present some challenges because the amount of content was significant, especially for a digital product. Mayer's 12 Principles of Multimedia Learning provided excellent guidance for the design of the content. Taking a cognitivist approach, I worked to ensure that each slide presented manageable chunks of information at a time. I also wanted the user to have control of the pacing of information to prevent them from being overloaded. You can see examples of this throughout the module where the user can have smaller cards of content slide on and off screen to read, and where the user can select different chunks of information to read more about, before advancing to subsequent topics and activities. I even went to the trouble to create and program multimedia controls for the audio examples so that learners could play and pause the examples as needed. The redundancy principle informed the writing of the recorded dialogue to provide instructions and guidance rather than re-reading content on screen.

Establishing a Design Language

Articulate Storyline can be used quickly, and carelessly, to produce a final product that looks little more than a PowerPoint with a navigation bar on the side. It was my goal from the beginning to create a final product that justified the use of the tool in the first place. This meant I had to start from the ground up to create a design language that would give the resulting module a consistent look and feel, and which would take full advantage of the Storyline functionality. I first worked to establish the basics. Font pairings and sizes, theme colors and accents, background images, and basic UI elements first. In my ideation phase, I had planned to make use of a technique that is common in web design: card-based interactions. This layered look provides a sense of visual hierarchy to the final screen, allows for users to control the flow of information, and helps to segment material into manageable chunks. In creating my design language for the final product, I had to determine how cards would look, how text and UI elements would be presented on the cards, and how they would animate on and off screen. This was a process that was ongoing throughout the development and after user testing as I worked to refine the product toward a pleasing, usable, and efficient design.

Aural Skills Assessments

Assessment proved to be tricky for this module. The goal was to have an instructional product that could be left behind when the instructor can't be present in the classroom, so the entire assessment would have to be completed by the module itself. It was also critical to have an assessment so that the instructor could justify their use of the module as a legitimate learning tool. I took inspiration from the types of aural skills tests that music students take in their studies. In our collegiate music history classes we regularly take "listening tests" where an excerpt from a piece will be played and we will be required to either name the composer, or the title, or the time period, or otherwise pick out and explain some detail that we noticed only by listening to it. This is a higher-level assessment than just recalling the dates of a time period or listing the names of composers who were active at the time. It's a higher DOK task that requires the learner to apply their knowledge of the period's characteristics and identify them in a new context. I specifically chose to use three questions: one about piano sonatas, one about symphonies, and one about opera. This allowed me to span some of the most popular genres of music from the time period which were specifically mentioned in the module. I was restricted to using recordings that were in the public domain, so I spent a lot of time finding example pieces that would very clearly highlight the characteristics discussed in the lesson. The UI design of the assessment was also very particularly created, so as to make it clear which example a learner is listening to, giving them the ability to listen to them in any order, allowing the learner to control playback, and to make it obvious which choice they have selected before submitting.

Lessons Learned

Working in Storyline requires the instructional designer to translate their mental images and sketches into a very technical, and very architectural, format. I learned a lot about using layers and triggers to control elements and slides, setting and storing variables to control interactions, and I gained a new appreciation for the amount of detail that is required to create even very simple uesr interfaces and controls. These skills have translated directly into the designs of other projects and tasks, especially into web design areas, where the very logical and structural layout of content matters a great deal.

View ModuleView Documents