Are You Strapped In?
Author: Monica Myhill, CMP and Dr. Jack J. Phillips
November 2007
Features
ROI: The Seat Belt for Meetings and Events
The evaluation of meetings and events and the subsequent return on investment (ROI) calculation has been a hot topic now for many years in the meetings industry. Yet, the topic can be compared to the use of seat belts in automobiles. The first automobile powered by a gas engine was built by Karl Benz in 1885. Yet, seat belts were not considered standard equipment for U.S. automobiles until the 1960s—a full 75 years later. It was not until December 1984 that New York State mandated seat belt use for front seat passengers—almost 100 years after the first car. Like seat belts, the ROI evaluation process contributes to the protection and well being of the meeting planner or “driver” and the car passengers, which represent the meeting attendees and stakeholders. Through meeting evaluation, meeting planners can justify and strengthen their role as a strategic contributor to their organization’s success, thus ensuring that their position, department and budgets will not be downsized or eliminated. Through meeting evaluation, meeting attendees are protected and sheltered from a series of unproductive meetings. Other meeting stakeholders can rest assured that meetings are contributing toward the success of the organization. The timing of automobile seat belt adoption is similar to that of ROI adoption for meetings and events. The first professional association for meeting planners was founded in 1920 as the American Society of Association Executives (ASAE). Seventy-five years later, in the 1990s, the meeting industry was introduced to the concept of ROI for meetings and events through the work of Meeting Professionals International and Fusion Productions. Since then, we have witnessed the slow, but steady adoption of ROI for meetings and events. Industry-Wide Adoption of ROI
This adoption of ROI for meetings and events is due primarily to accountability demands and the rise of the meeting planner as a strategic contributor. Accountability is now expected of all functions and departments within an organization. Each area must demonstrate how their programs and projects are contributing to the success and bottom line of the organization. Meetings and events are being held to these same standards. Secondly, meeting professionals have taken on a more strategic role as meeting managers. This business management mindset has caused them to place more emphasis on economic issues within their function. Today, meeting managers are more aware of bottom-line issues in their organizations, are linking their meetings to the organization’s objectives and are taking decisive steps toward improving their meetings and events. Naturally, these strategic meeting managers have embraced comprehensive meeting evaluations and ROI calculations for their meetings.Benefits of Meeting Evaluation and ROI Calculations
Strategic meeting managers are finding numerous benefits to conducting comprehensive meeting evaluations and ROI studies. Some of these benefits include: • Determine Whether Meeting Objectives Were Achieved. Measurable meeting objectives are at the heart and core of meeting evaluation. Through the evaluation process, one can determine whether objectives were met and to what extent. • Compare Meeting Costs to the Benefits. With today’s business focus on the bottom-line, determining a meeting’s cost-effectiveness is crucial. Through the ROI process, the fully-loaded costs of the meeting are compared with the meeting benefits that have been isolated to the particular meeting and converted to a monetary value. • Improve Meetings and Events. Probably the most common purpose of evaluation is to determine the effectiveness of the meeting design, elements, messages, content, delivery methods and presenters. Through this feedback, adjustments and improvements tofuture meetings and events can be identified and made. • Marketing of Future Meetings. Evaluation results can provide valuable marketing material for future meetings. The best results to feature are how attendees utilized the meeting content back on their jobs and the subsequent impact to their business measures.
Phillips ROI Methodology
Dr. Jack J. Phillips can be considered the designer of the first seat belt for meetings and events. The Phillips ROI Methodology was developed by Dr. Phillips in the 1970s, refined through application and use in the 1980s and implemented globally during the 1990s. Currently, more than 2,000 organizations are using this methodology in 20 different fields. It is estimated that at least 5,000 impact studies are conducted annually worldwide. To date, 20 books have been published on the methodology and its application in the business environment. Those specifically geared to the meetings industry are: Proving the Value of Meetings & Events: How and Why to Measure ROI and the upcoming Return on Investment in Meetings and Events: Tools and Techniques to Measure the Success of all Types of Meetings and Events.
Levels of Evaluation
The Phillips ROI Methodology utilizes six levels of evaluation, which are vital in evaluating a meeting and determining its ROI. This framework is used to create measurable meeting objectives as well as a plan to evaluate the success of the meeting. Level 0 – Statistics, Scope, and Volume collects data on meeting statistics as well as the scope and volume of meeting attendance, exhibitors, marketing efforts, press coverage, budgetary measures, website traffic, etc. Much of this information can be found in an organization’s records, upon review of an organization’s balance sheet or through a scan of media coverage. Level 1 – Reaction, Satisfaction & Planned Action evaluation determines attendee satisfaction levels with the meeting as well as the planned actions of the attendees following the meeting. This level of measurement gathers reactions to the meeting elements, presenters, meeting facility and meeting content. In addition, attendees are asked what they plan to do with the meeting content, professional contacts or sales leads following the meeting. Level 2 – Learning evaluation focuses on determining the extent to which knowledge, skills, opinions and professional contacts have been acquired during the meeting. This data can be collected through the use of tests, skill practices, case studies, job simulations, participant feedback and facilitator observations.
Level 3 – Application measures the extent to which skills, knowledge, and professional contacts learned or acquired at the meeting were utilized or applied. This level of data is typically collected two to four months following the meeting in order for the attendee to have had an opportunity to apply the meeting content. The barriers or enablers that support or hinder the attendee from applying the meeting content are also investigated at this level in the evaluation process. While this feedback may point to issues within the organization that the meeting planner has little influence over, the meeting planner should be knowledgeable of them when planning future meetings.
Level 4 – Business Impact evaluation determines whether the application of meeting content resulted in an improvement to business measures. The business measures most impacted by meetings and events include increased sales, cost savings, time savings and greater productivity. This data is most often obtained from performance monitoring of organizational records and participant questionnaires.
Level 5 – ROI evaluation addresses the meeting’s return on investment. The return on investment is determined by dividing the net meeting benefits by the fully-loaded meeting costs. The net benefits are the meeting benefits minus the costs. In formula form, the ROI becomes: This is the same basic formula used in evaluating other investments where the ROI is traditionally reported as earnings divided by investment.
Methodology Steps
As shown in Figure 1: Evaluation and ROI Methodology Model, the Phillips ROI Methodology includes four major steps: evaluation planning, data collection, data analysis and reporting. Within the first step, evaluation planning, measurable meeting objectives and an evaluation plan are created. Then, data are collected during the meeting and following the meeting. During the data analysis step, improvements in business measures are isolated to the meeting, converted to a monetary value and then compared to the fully-loaded costs of the meeting. The last step is the reporting of results to key stakeholders.
Step 1: Evaluation Planning Not only are measurable meeting objectives a necessary step in developing a successful meeting, they are also a key component in the meeting evaluation process and should correspond to the Phillips levels of evaluation. Without them, it is impossible to determine the success of the meeting. Table 2 shows meeting objectives tied to the levels of evaluation from a recent pharmaceutical sales meeting.
Step 2: Data Collection Meeting objectives also guide decisions on the data collection method, when data should be collected and who will provide the data. For example, to determine whether the Level 0 objective “95% or more of the North American sales force will attend the meeting” was met, the meeting’s registration numbers must be compared with the sales force roster once the meeting is completed. To determine the Level 4 objective of an increase in sales, gathering sales data before and three months after the meeting would be necessary. Questionnaires, the most popular data collection tool for meetings and events, can be used to capture Level 0, 1, 2, 3 and 4 data. Through questionnaires, attendees are asked to provide responses to a variety of scaled multiple choice and open-ended questions. These questionnaires can be distributed in paper form to attendees on-site; delivered to attendees through a session’s audience-response equipment or sent as a weblink through e-mail following the meeting. Online questionnaires are very popular because they can be administered quickly, easily and cheaply through online survey tools such as Survey Monkey or Zoomerang. Through them, a questionnaire can be created, distributed and the results analyzed.
Step 3: Data Analysis A summary of the questionnaire results is created during the data analysis stage. With online survey tools, this summary can be provided automatically to respondents and key stakeholders in “real-time.” When an ROI calculation is desired, business measures must also be isolated to the meeting, credibly converted to a monetary value and then compared with the fully-loaded meeting costs. • Isolating the Effects of the Meeting This portion of the ROI and evaluation methodology answers the question “How do you know it was your meeting that improved the business measures?” It is necessary to isolate the effects of the meeting on these measures to determine what percentage of the improvement, if any, can be attributed to the meeting. Control groups, trend lines and participant estimates are the most-used techniques to isolate this impact. • Converting Data to Monetary Value If the meeting’s ROI is wanted, benefits or business measures isolated to the meeting must be converted to a monetary value. An organization’s standard values, historical costs, expert input or participant estimates can be utilized to convert benefits to a monetary value. • Tabulating Fully-Loaded Costs Most meeting managers are very efficient at controlling and tabulating meeting costs. Yet, a fully-loaded meeting cost is a bit different, as it includes not only the direct meeting and travel costs but also program development costs, value of participant and staff time, and overhead administrative costs. • Calculating the ROI Once the monetary meeting benefits and fully-loaded meeting costs are tabulated, determining the ROI is easy. Return on investment (ROI) is the financial equation that compares net benefits (earnings) to the meeting costs (investment) and is easily recognized by executives, managers and administrators with fiduciary responsibility within organizations.
Step 4: Communicating Results It is important to communicate the results throughout the evaluation process—not just after the study is complete. Since meetings and events are often planned many months out, it may be necessary to share results and future recommendations from the post-meeting results before the full study is complete. Results can be shared in regularly scheduled department meetings, memos, newsletters, company intranet, e-mail blasts or articles. The meeting results should be customized to the interests and needs of your various stakeholders. For example, prepare a one-page executive summary for senior executives that covers the top-line data and recommendations only—not whether attendees liked meal functions. Your hotel liaison, on the other hand, will appreciate feedback on how attendees felt about their facility, service levels and meals.
Evaluation Targets
Not all meetings are candidates for higher level evaluations or ROI measurements. As shown in Figure 2: Evaluation Targets for Meetings and Events, only 30% of an organization’s meetings should be taken to a Level 3: Application evaluation study. Only 10–20% should be taken to a Level 4: Business Impact evaluation. ROI studies should only be conducted on 5–10% of an organization’s meetings and events. The recommended percentages decrease as one moves up the pyramid due to the time and effort involved in the evaluation process.Best Meetings and Events for ROI or Higher-Level Evaluation
Meetings and events best suited for Level 3: Application, Level 4: Business Impact or Level 5: ROI Evaluation Studies have several or all of the following factors:• Long meeting life cycle • Meeting is linked to organization’s top goals, objectives and issues • High meeting costs • Significant visibility of the meeting • Considered a big meeting with a large number of attendees • Meeting requires a significant investment of time on the part of the meeting organizer and/or the meeting attendee • Needs assessment conducted prior to design of meeting • Senior executives are interested in a higher level evaluation or ROI measurement
Challenges to the Use of the Phillips ROI Methodology
Meeting professionals have experienced some challenges when utilizing the Phillips ROI Methodology—none of which are insurmountable. • Lack of Measurable Meeting Objectives: Once meeting managers make the decision to pursue a higher level meeting evaluation, they often realize that their meeting objectives are vague and not measurable. This, however, presents an opportunity for them to facilitate a strategic discussion with key stakeholders about the meeting. The process of creating measurable meeting objectives can be the most difficult, but most rewarding step in the evaluation process.
• Meeting Not Linked to Organization Goals and Key Performance Indicators (KPIs): Some meetings are not directly linked to the organization’s strategic goals or Key Performance Indicators (also known as KPIs). These linkages must be made in order to set measurable objectives at Level 3: Application and Level 4: Business Impact. Yet, this linkage brings to light the importance of the meeting to the organization’s success and to its bottom line. • Perceived Effort Involved in Process: Granted the first higher level evaluation study will take more time and effort than the previously conducted reaction/satisfaction “smile sheets.” However, there are on-line evaluation tools that can cut down the time and effort involved in creating questionnaires, analyzing the data and reporting the results. For example, GuideStar Research offers MeetingMetrics, a comprehensive on-line meeting measurement and reporting system specifically for meetings and events.
Are you ready to move your meetings and career forward? Then, strap in!
LEVEL Meeting Objectives 0 Statistics, Scope, and Volume Measures meeting statistics such as the scope and volume of meeting attendance, press coverage, budgetary measures, etc. and documents attendee demographic information. 1 Reaction & Planned Action Measures attendee reaction to and satisfaction with the meeting and captures planned actions. 2 Learning Measures changes in knowledge, skills, opinions and professional contacts acquired at the meeting. 3 Application Measures changes in attendee behavior following the meeting. These changes involve use of knowledge, skills and professional contacts. 4 Business Impact Determines changes in business measures such as sales, time, costs and quality. 5 Return on Investment Compares meeting benefits to the costs.
LEVEL Meeting Objectives 0 Statistics, Scope, and Volume • 95% or more of the North American sales force will attend the meeting. 1 Reaction & Planned Action • Attendees will rate the meeting elements a 4.0 or better on a 5.0 scale of poor to excellent.• On a scale of 0% (not at all) to 100% (extremely critical), attendees will indicate that applying the meeting content to their job is 70% to 100% critical to their success. 2 Learning • Attendees will rate the following statement 4.0 or better on a 5.0 scale of strongly disagree to strongly agree: I can effectively verbalize key points from the new clinical study that support the product’s core selling messages.• Attendees will score 75 or better on the product dosing chart. 3 Application • Within 3 months of the meeting, 70% of attendees will indicate "some change," "significant change" or "very significant change" in their use of skills and knowledge acquired at the meeting. • Attendees will identify barriers and enablers to applying learned skills and knowledge in the workplace. 4 Business Impact • Increase sales of new product by 5% within 3 months of meeting. 5 Return on Investment • Achieve a 15% return on investment using first-year meeting benefits.
ROI PERSPECTIVES
ROI calculations can be made from the perspective of multiple stakeholder groups. Associations, nonprofit organizations and government entities should consider the perspective of the meeting attendee, exhibitor and sponsor. The best meetings and events for nonprofits to take to ROI studies and higher level evaluation studies (those studies that go beyond Level 2: Learning) are annual conferences, trade shows and specialty or in-depth member training or certification programs. Corporations may want to consider not only conducting studies from their own perspective, but also that of the meeting attendee. For example, a franchisor company can calculate the ROI for franchise owners participating in meetings and events. Some of the corporate meetings, events and programs that are suitable for higher level evaluation and ROI studies include user/client conferences, marketing events, training meetings, incentive programs, annual sales kick-off meetings and events tied to corporate sponsorship.
Monica Myhill, CMP, President of Meeting Returns can be reached by e-mail at monica@meetingreturns.com or by phone at 303-220-1920
Dr. Jack J. Phillips of the ROI Institute provides consulting services for Fortune 500 companies and workshops for major conference providers. www.roiinstitute.net
