Final Report of the Customer Satisfaction Work Group of the Workforce Information Council Section I – Executive Summary The Customer Satisfaction Work Group (Work Group) of the Workforce Information Council (WIC) was appointed to explore the issue of customer satisfaction as it relates to labor market information (LMI) providers and users of the information The Work Group was composed of eleven members, two support personnel and a consultant They met formally three times between October 2002 and June 2003, and gave quarterly updates to the WIC The Work Group utilized existing documents and gathered other reports to guide its work Many of the concepts employed are private sector marketing concepts The group verbalized the process of developing labor market information products and services as being a process of data collection, data analysis and distribution of information products and services The process defined in this manner seems void of customer satisfaction measurement and needs assessment Therefore, the Work Group concentrated on incorporating the needs and satisfaction of customers into the LMI creation process Two models were constructed to demonstrate how customer satisfaction and needs assessment can guide the creation of meaningful labor market information and move toward a demand-driven workforce information system One model charts a state scenario and the other extends the model to federal products The result is a dynamic, customer-driven labor market and workforce information system which is responsive to customer needs and wants in a changing environment The process embraces not only the concept of assessing customer satisfaction, but also of working (through evaluation teams and the WIC) toward continuous improvement of the system This is the objective of the requirement to “consult” with customers as set forth in Section 309 of the Workforce Investment Act of 1998 As part of infusing customer satisfaction into the labor market information development process, some critical components have been identified They include: • • • The establishment of evaluation teams in each state, charged with turning customer satisfaction assessment results into improved products and services, and with sharing customer satisfaction processes and improvements with other states and federal agencies; Nine measurable attributes which the Work Group believes will lead to more meaningful customer assessment activities and provide for easier information sharing; A methodology that state and federal agencies can employ to “think through” the customer satisfaction process The methodology is displayed in a handbook developed by the Work Group which will be made available to assist federal and • state agencies as they work to develop protocols and choose methods for assessment; Strategies for documenting demand for labor market information products which can provide guidance to funding agencies, in addition to encouraging broader use of available information This document has five sections including this executive summary Section II outlines the Work Group’s approach to work Section III discusses and models the relationships between customer satisfaction and labor market information Section IV reviews pilot studies, and Section V of this document responds specifically to the charter given to the Work Group by the WIC The response in Section V notes findings by the group and then provides seven actionable recommendations These recommendations include establishing or identifying a customer satisfaction focal point for sustaining the effort and providing on-going customer satisfaction technical assistance, offering training to agency representatives on implementing customer satisfaction programs, exploring customer satisfaction as a deliverable in the BLS cooperative agreement and continuing it as a deliverable in the ETA core products grant, and expanding the efforts of the Work Group into a demanddriven marketing effort for workforce information There are also attachments under various tabs Tab III is a customer satisfaction handbook Tab IV reviews pilot study results Tab V is the draft document “Workforce Information Customer Satisfaction Assessment: A Primer for State and Local Planning” developed by Mediacall, Inc and the Heldrich Center Tab VI documents BLS customer satisfaction activities Tab VII is the project work plan which set the course for the Work Group Tab VIII is feedback information from federal agencies on their customer feedback programs Tab IX is a tally of state customer satisfaction activities, and Tab X is the first progress report of the Work Group Section II – Approach to Work Introduction The Customer Satisfaction Work Group (Work Group) of the Workforce Information Council (WIC) was appointed to explore the issue of customer satisfaction as it relates to labor market information providers and users of the information The WIC defined the scope of the Work Group by providing a charter with five specific outcomes: What are the broad parameters for a customer satisfaction measurement system? What customers and products/services should the system address? How should the activities related to customer satisfaction measurement be communicated to state LMI offices and state employment statistics agencies? How can their support for the effort be encouraged? What technical assistance should be provided to states in implementing customer satisfaction measurement within their states? How should customer satisfaction measurement be incorporated into federal funding agreements for LMI activities? The Work Group was composed of eleven members, two support personnel and a consultant, including: • • • • • • • • • • • Olaf Bjorklund, Employment and Training Administration, Federal Co-chair James McFadden, Chief, Office of Occupational and Labor Market Information, Delaware Department of Labor, State Co-chair Garry Breedlove, Florida Agency for Workforce Innovation Tom Gallagher, Manager, Research and Planning, Wyoming Department of Employment Rod Fortran, New York Department of Labor, Division of Research and Statistics Yolanda Harris, Marketing Communication Specialist, Illinois Department of Employment Security Betty McGrath, Economist, North Carolina Employment Security Commission Peter Phelan, Center For Workforce Information and Analysis, Pennsylvania Department of Labor and Industry Mary Ann Regan, Director, Center for Workforce Information and Analysis, Pennsylvania Department of Labor and Industry Vivian Shapiro, Assistant Commissioner, New Jersey Department of Labor Stanley Stephenson, San Francisco Regional Commissioner, Bureau of Labor Statistics Support Staff • • Dixie Sommers, Center on Education and Training for Employment, Ohio State University Nancy Pyon, Center on Education and Training for Employment, Ohio State University Consultant • Don Norton, Vice President, Mediacall Marketing and Advertising, Inc The Work Group began meeting quarterly with the consultant in October 2002 Other meetings were held in January and May 2003 The WIC was apprised of status of the project at meetings in December 2002, March and June 2003 A presentation was made on the Work Group efforts at the BLS LMI conference in May 2003 Several documents were used to guide the effort Some of the documents were developed by staff specifically for the project and others were collected to build in previous efforts These documents include: A draft “Customer Satisfaction Primer,” funded by the ETA and written by Mediacall and the Heldrich Center for Workforce Development at Rutgers This document also notes what actions states are implementing, as well as private sector customer satisfaction practices; A report from the Bureau of Labor Statistics (BLS) on the type of customer satisfaction activities it has undertaken; A “Project Work Plan” developed by the Work Group; Examples of “Customer Feedback Information” collected from federal agencies; and Tally of Customer Satisfaction Activities currently being undertaken by states Copies of these documents are attached as appendices to this document Similarly, the group developed a report at the mid-point of the process to document resources It is also attached as “Report 1.” Approach to Work The Work Group began to approach customer satisfaction with concepts set forth in the Primer, noted above The group set forth working definitions These include: o Customer satisfaction: meeting or exceeding the expectations of the customer It is recognized that there may be many attributes to satisfying the customers o Customer expectations: attributes that are in the mind of the customers, and consist of all parameters of what they, the customers, need, want and think they need and want o Customer feedback: information gathered from the customers to assess their level of satisfaction There were many discussions of “marketing” as it relates to labor market information The group embraced the concept of marketing as a process In the private sector, the process involves conducting research (listening to the customer to ascertain needs and wants), developing products to meet those needs, promoting products, and then re-evaluating product effectiveness Products are changed through innovation or discontinued, as noted below The group verbalized the process of developing labor market information products and services as being a process of data collection, data analysis and distribution of information products and services The process has not historically recognized the customer as part of the development process, as noted below LMI Product Development Model Section 309 of the Workforce Investment Act of 1998 clearly mandates that customer satisfaction data be gathered in an effort to continuously improve the LMI system The Work Group has been keenly aware of this mandate, and worked toward developing a model which could incorporate private sector concepts into the labor market information creation process In that regard, the group was less concerned with a process that developed a single number for measuring satisfaction Rather, the group focused on ways to collect information regarding how labor market information products can be improved to better meet the needs of users The resultant model could be seen as follows: Integrating Customer Satisfaction into LMI Product Development Based on this fundamental model of incorporating the needs and satisfaction of customers into the LMI creation process, the work group then set about establishing models which could allow this process to thrive in the complex state and federal workforce information development system In addition to seeking and responding to customer satisfaction input, the Work Group identified the lack of documentation regarding the usefulness of and demand for workforce and labor market information Therefore, the Work Group’s effort focused on developing a process for assessing customer satisfaction with and documenting demand for labor market information Section III – Customer Satisfaction and Labor Market Information Modeling the Process In addition to framing the approach to work with the models noted in Section II, the Work Group established two models to further define the process that states and federal agencies should use in assessing customer satisfaction The first model is for states and is outlined below: State Product Customer Needs and Satisfaction The following is a verbal description of this process: The process notes the three fundamentals of creating labor market and workforce information: data collection, data analysis and information distribution Information distribution encompasses both products and services These products and services are the items that are evaluated by the customers Results of the assessment of customer satisfaction are given to an evaluation team at the state level Work Group research determined that these teams already exist in many states as the entity which develops and distributes current information products The evaluation team then makes changes at any one of the three levels of creating labor market information, as it is able depending on funding and other considerations Furthermore, the evaluation team shares information with a customer satisfaction focal point – a central collecting point yet to be determined Training on customer satisfaction best practices and techniques, as well as lessons learned on document preparation, then should be offered to states, perhaps through the Labor Market Information Training Institute Finally, the state evaluation shares information with representatives at the federal level to facilitate improvement or changes to the workforce system This same process should be used for developing new programs, products and services The second model is noted below The right side of the model below is the same as the state model above, but the model is completed by adding customer satisfaction measurement at the national level and the relationship between federal and state entities System-wide Customer Satisfaction for Workforce Information The following is a verbal description of this process: Federal activities are in blue, state activities in green, and those entities which are state and federal cooperative have both colors The process notes the three fundamentals of creating labor market and workforce information as the central activity of both federal and state entities: data collection, data analysis and information distribution The process described above is mirrored at the federal level for national or federal product customers Results are reported to the WIC, which may choose to work through an on-going Customer Satisfaction Work Group For products which have consortia or task forces, satisfaction results will be sent to that entity for change recommendations The entity for making change may either be the Bureau of Labor Statistics or the Employment and Training Administration, which, like the state model, will effect change at any of the three levels of data collection, data analysis or information distribution Federal entities then communicate the customer satisfaction results to the state focal point to facilitate possible improvements in state products and assessment practices The end result of this entire process is a dynamic, customer-driven labor market and workforce information system that is responsive to customer needs and wants in a changing environment The process embraces not only the concept of assessing customer satisfaction, but also of working (through evaluation teams, federal agencies and the WIC) toward continuous improvement of the system Components of Assessment Part of the charter from the WIC was to determine how to encourage support for customer satisfaction assessment from the states Work Group members concluded that the process of assessment would have to be broken into easy-to-follow components to be embraced by states In addition, the group assumed that there would not be one mandatory survey imposed upon the states Therefore, one particular survey instrument or assessment mechanism would not be recommended, but rather several assessment options would be offered To meet the objective of encouraging states to participate, the Work Group is providing a handbook entitled “Customer Satisfaction Made Easy for Labor Market Information Professionals.” The information discussed in this section of the report has been discussed by the group and formulated into the development of the handbook Who to Assess: The Work Group consulted Section 309 of the Workforce Investment Act of 1998 which clearly indicates the customer groups need to be assessed They include: • Businesses make decisions about product and financial markets, business location, and employee recruitment, compensation, and training • Individuals, including young people and adults, make choices about careers, education, training, and job search • Elected Officials and Policy Makers, including Workforce Investment Boards, make decisions about law, policy, budgets, and regulations • Program Planners determine what workforce and economic development services to provide, and evaluate program performance • Education and Training Providers, including teachers and curriculum specialists, design, deliver, and evaluate programs that develop students’ knowledge and skills • Intermediaries, such as parents, counselors, teachers, mentors, placement workers, and case workers, assist others in choosing education and training opportunities, and in finding employment, and • Researchers and the Media study how the labor market works, and conduct policy research The Work Group recommends that agencies place the customer into one of these groups This will facilitate information sharing between states and federal organizations in terms of how customer groups utilize core products In addition, the Work Group believes it is important that products be assessed at a customer-familiar level As an illustration, a candy customer sees a chocolate bar The chocolate bar manufacturer sees that same bar as sugar, cocoa, milk and other ingredients In LMI terms, the customer sees a brochure that lists projected job opportunities The LMI shop sees that same brochure as projections data, skills sets and wage records State and federal agencies should ask the customer about the satisfaction level in terms familiar to the customer, but report a level of satisfaction in terms of each core data product State Evaluation Team: The models for customer satisfaction noted above feature a state evaluation team This team has three purposes: o To oversee the customer satisfaction effort at the state level, and incorporate results of that effort into continuous improvement of state products; o To share information gathered in the customer satisfaction effort, both in terms of process and conclusions, with other states; o To share information gathered in the customer satisfaction effort with representatives and organizations at the federal level to encourage systemic change States may want to consider having a publications specialist, an LMI projections professional and an information analyst on the team to represent some of the individuals involved in product production It should be noted that the composition of these teams will vary by state During research developed by the group it was discovered that many states already have such teams that also function as the teams that develop LMI products Attributes: A review of “customer satisfaction” surveys at the state and federal levels found that most surveys actually are an attempt to measure customer service The Work Group believes that customer service is one aspect of the total customer satisfaction picture These service-oriented questions include “Was our staff helpful?” “Could you find the 10 information easily?” To provide further guidance here, the Work Group identified nine (9) attributes which states should consider measuring when assessing customer satisfaction They include: Accuracy: The information is accurate enough for the customer’s use Relevancy: The information is relevant to solving the customer’s problem Accessibility: The customer could easily access the information Understandability: The customer could understand what the content means Comparability: The customer is able to compare the information with other information they use Geographic Detail: The information is in sufficient geographic detail Timely: The information is timely enough for the customer’s needs Completeness: The customer could solve the problem with this information only Importance: The information is important in the customer’s overall problem solving work Not all of these apply to every information piece all the time But these attributes can give information providers guidance on how to make products more meaningful, and provide evidence to support new product development Methodologies: It is important to emphasize that customer satisfaction assessment is not just a survey or a number As noted earlier, the Work Group believes that customer assessment is a process and there is a risk of being number-oriented If a state or federal agency scores a 77% on a satisfaction rating, what does this tell them about how to alter a product to make it more meaningful? Quite possibly, the agency knows nothing more about how to improve the product This number-orientation could further be complicated by the very nature of labor market information research As an industry, LMI professionals are numbers people Methodologies that profess to gauge customer satisfaction through anything less than a stratified random sample with significant return rate could be rejected outright Customer satisfaction measurement through other methods could represent a paradigm shift within the LMI industry Furthermore, the Employment and Training Administration has discouraged the development of one preferred methodology and has, in fact, encouraged utilizing a variety of approaches to customer satisfaction assessment The Work Group also embraces that approach However, customers can and should (as the sample survey in the handbook points out) be asked a relative level of satisfaction with a document on a Likert scale of to with being highest To provide a framework for this descriptive research, the Work Group endorses a process of Problem Definition and Problem Solution, which has been refined to a simple “Assessment Statement.” This protocol helps enable agencies to choose a methodology which best meets the needs identified in the problem definition, as noted below 11 This process provides a valid framework for a state to “think through” the customer satisfaction assessment process Based on this framework, the Work Group recommends that a variety of assessment mechanisms be used to assess customer satisfaction, including but not limited to phone, mail and Internet surveys, focus groups and interviews In fact, many states have used even more assessment methods than those listed which provide actionable customer satisfaction feedback Two of the most common methods being utilized are surveys and focus groups To facilitate this effort, the Work Group developed a survey which can be adapted for states to administer, and a document on how to conduct a focus group Again, details of these processes are available in the handbook Section IV of this document discusses how the methodologies in this document have been piloted Needs Assessment: Customer satisfaction implies that there is a product with which a customer can express satisfaction or dissatisfaction A truly proactive approach is to identify a target customer group and assess their needs for products that may or may not exist The next step would be to create a product to meet those needs and then assess satisfaction with that product One of the piloting efforts taking place in Pennsylvania is conducting this proactive technique with local workforce board professionals utilizing a similar research protocol as outlined above 12 Documenting Demand The Work Group believes that there is not a clear picture of the utility of or demand for labor market information Statistics regarding demand and usage are important in that they can provide guidance to funding agencies, in addition to encouraging broader use of available information With this in mind, the group suggests that states document items such as: • • • • • • Statistics of product usage; Web site visits, using Web Metrix and other programs; Developing customer lists by product; Other Formal Satisfaction Feedback, such as special surveys, focus groups and directed interviews; Anecdotal information, such as testimonials from customers regarding how information is used to solve problems; Proof of usage, through letters or other measurements where people have noted “This is how I use it and its value to me.” Marketing Implications Part of the issue of documenting demand concerns the fact that labor market information products have never been aggressively marketed except at the state level in a few states The Work Group believes that while it is important to satisfy existing customers, there are tens of thousands of potential customers who are in need of information products but not know they exist As noted earlier, customer satisfaction is a marketing concept The marketing cycle is not complete unless products are adequately promoted Only when LMI products are effectively marketed can their demand and usage be fully document This effort has not yet been undertaken, and possibly to the detriment of economic development and other efforts in communities throughout the nation Section IV – Pilot Studies The Work Group believes it is important to provide states and federal agencies with customer satisfaction assessment tools which can be understood and then implemented Toward that end, several of the methodologies outlined in the previous section are being piloted The purposes of the pilot studies include: • • • • To test the usability of the nine attributes in assessing customer satisfaction measurement; To test a proposed survey instrument; To provide examples of focus group questions and processes; To refine protocols that will be outlined in the handbook 13 Three states volunteered to be included in the study The states and the projects are as follows: Delaware: A survey followed by directed interviews on a monthly labor market information brochure with a defined audience; A focus group with a finite group of state leaders to determine the use of existing products and need for new products; Illinois: Focus groups with two similar but geographically diverse customer groups regarding a set of career counseling documents; Pennsylvania: Open-ended survey and directed interviews/focus groups with local workforce board representatives to determine products and services that will be needed during the coming year (needs assessment.) An attachment to this document provides detailed protocol, comments and recommendations which have resulted from these pilots Section V – Response to Charter The Customer Satisfaction Work Group is providing the following direct responses to the charter from the Workforce Information Council The responses take the form of findings and actionable recommendations What are the broad parameters for a customer satisfaction measurement system? Response The Customer Satisfaction Work Group believes that customer satisfaction should be incorporated into the process of developing labor market information products Following the model set forth in this report, customer satisfaction should be measured at the local, state and national levels There are nine actionable attributes which should be measured, in addition to assessing unmet information needs State and federal agencies should use a variety of assessment mechanisms including but not limited to phone, mail and Internet surveys, focus groups and interviews States and federal agencies should recognize the importance of mechanisms such as the WIC and state evaluation teams (many already exist) for the implementation of customer recommendations for product improvement What customers and products/services should the system address? Response We believe in assessing customers as outlined in WIA section 309 with emphasis on the business community While we recognize that states and federal agencies not have the resources to assess customer satisfaction on every product at all times, over time, we believe agencies should work toward assessing all products and services that are offered It is important that assessment be made of the products and services as our customers 14 know them (pamphlets, brochures, websites, documents, other information), and not as core products or statistical data products (CES, OES) as they are known in LMI cooperative agreements However, for some intensive data users, the product may indeed be the statistical data product How should the activities related to customer satisfaction measurement be communicated to state LMI offices and state employment statistics agencies? How can their support for the effort be encouraged? Response The Work Group has overseen the development and publication of a handbook, which is a logical outgrowth of the Primer We also believe that narrative portions of state one-stop grant annual reports to the ETA should be summarized and published And the group believes it is important to include other stakeholder groups in our customer satisfaction work group activities Common LMI websites should be used to post results of activities and success stories, and seminars on customer satisfaction assessment should be presented at national and regional LMI and workforce development meetings What technical assistance should be provided to states in implementing customer satisfaction measurement within their states? Response In addition to providing the workbook, the Work Group has piloted studies of methodologies noted in this report We encourage on-going independent validation of customer satisfaction methodologies recommended by the work group, and encourage making available the handbook, a tutorial, and collateral materials to implement LMIspecific methodologies for conducting customer satisfaction assessments and forming customer service evaluation teams in states How should customer satisfaction measurement be incorporated into federal funding agreements for LMI activities? Response The Work Group is putting forward as an actionable recommendation the continuation of the ETA core products grant requirement to assess and report customer satisfaction, and exploring customer satisfaction measurement as a funded deliverable in the BLS cooperative agreement Recommendation for Action We recommend that states, ETA and BLS implement the process outlined in this report We recommend that a work group or organization be established, appointed or identified for sustaining the customer service effort This group or organization should be charged with ensuring that on-going customer satisfaction technical assistance is provided and that customer satisfaction activities are documented, synthesized and made available to the WIC and states We recommend the WIC use the documented state customer satisfaction results to guide recommendations regarding the workforce information system We recommend training sessions be offered through existing training mechanisms (such as the LMI Institute) on implementing customer satisfaction programs and 15 the importance of the customer service activity to a more stable funding environment We recommend that capacity be built to sustain a customer satisfaction assessment process for Labor Market Information • We recommend that customer satisfaction measurement be explored as a deliverable in the BLS cooperative agreement, and that it be continued as a deliverable in the ETA core products grant We recommend that customer satisfaction assessment be expanded into a demand-driven marketing effort for workforce information Section VI – Conclusion The Customer Satisfaction Work Group of the Workforce Information Council was appointed to explore the issue of customer satisfaction as it relates to labor market information providers and users of the information The group believes that listening to the voice of the customer is a critical component in the process of developing more meaningful labor market information products and services The group has been able to document a process which, while not mandated to be followed, provides a practical and valid approach to infusing the needs and wants of customers into product development A handbook is being provided as a part of this effort that can serve as a guide to agencies seeking to utilize this process Work Group members believe that the success of building a demand driven workforce and labor market information system is dependent upon having states and federal agencies conduct customer satisfaction assessment, share the results of both the processes and information gathered, document demand for information products and services and begin a process of fully marketing and promoting the wealth of information that is available to benefit the public and private sectors in the nation 16 ... Approach to Work Introduction The Customer Satisfaction Work Group (Work Group) of the Workforce Information Council (WIC) was appointed to explore the issue of customer satisfaction as it relates... the customers o Customer expectations: attributes that are in the mind of the customers, and consist of all parameters of what they, the customers, need, want and think they need and want o Customer. .. responding to customer satisfaction input, the Work Group identified the lack of documentation regarding the usefulness of and demand for workforce and labor market information Therefore, the Work Group? ??s