Co-authored with Noga Chipman
Why is this important?
"Would you please tell me which way I ought to go from here?" "That depends a good deal on where you want to get to," said the Cat. "I don't much care where—" said Alice. "Then it doesn't matter which way you go," said the Cat. "—so long as I get somewhere," Alice explained. "Oh, you're sure to do that," said the Cat, "if you only walk long enough..." (Alice in Wonderland, Lewis Carroll)
Every organization wants to implement solutions and programs with the highest possible chances of success: programs with minimum risks and visible return on investment. Programs that are easily accepted by staff are feasible to implement and achieve desired results. To determine whether the chosen program meets these criteria, it is customary to accompany it with a control program that includes various elements of evaluation and measurement. The first step in building such a program is defining objectives - carefully planned goals that allow monitoring of their achievement.
The quote that opens the article contains several important insights about goal-setting and emphasizes the importance of the process: First, we learn that to reach the desired destination, we must take action. We cannot rely on luck or fate to bring us the desired results, and we must take active steps to reach those business goals required for organizational advancement. Additionally, Lewis Carroll teaches us that the critical time to define high-level goals is at the beginning of the journey. This enables continued structured work according to a work plan, reaching the defined goal quickly while monitoring process progress. Therefore, a lack of careful planning and failure to set intermediate goals that complement the high-level objectives may lead the organization to an undesirable place: the organization might be drawn into implementing projects without a clear business purpose. This action will lead the organization to unwanted and uncontrolled expenditure of resources: time, money, and human resources.
Some Definitions: What Exactly are Evaluation and Measurement?
Evaluation and measurement are established processes for tracking goal achievement. From a practical standpoint, this tracking allows us to ensure compliance with laws, standards, and regulations, enables us to receive feedback from program participants, helps us identify trends and changes, assists in planning follow-up activities and modifying intermediate goals, and, most importantly - promotes opportunity for improvement. Measurement is finding the size of a specific characteristic and expressing this size in appropriate units of measure. For example, measuring 200 monthly portal entries. However, the value obtained doesn't provide much information when standing alone. Are 200 entries few or many? Measurement alone doesn't provide an answer to this question. Evaluation is a complementary process during which the value of a characteristic is determined relative to a scale and according to predetermined norms and values. For instance - few/many. For example, 200 monthly entries in a community serving a population of 25 users in secondary work processes might be evaluated as "many." Conversely, 200 monthly entries in a portal serving a global employee population in their core work processes would be evaluated as "few."
Theoretical Model for Evaluation and Measurement
One popular approach to evaluation and measurement standards is Kirkpatrick's approach (1977). This theoretical model was originally designed to evaluate the effectiveness of training or educational programs. Training, like knowledge management, is a soft field that presents challenges in numerical quantification. Therefore, the model can be implemented, with some modifications to its definitions, beyond the training niche and applied to evaluating knowledge management solutions.
Kirkpatrick's original evaluation model includes four steps:
Reactivity: This stage measures whether the solution "reaches" the target audience - do they know about it? Do they use it? Are they interested in it? This step is quite common in the evaluation world and is implemented through surveys and questionnaires.
Learning: This stage measures whether learning occurred. Do organization members know more? Has the organization's intellectual capital grown? Learning is measured by evaluating the extent to which training program participants understand the principles, facts, and techniques conveyed in the program. This step is also common and typically measured through paper-and-pencil achievement tests.
Behavior Change: This stage measures whether the solution has been integrated into work processes and has created actual additions or changes. In other words, it assesses the extent to which training led to behavioral change at work.
Results: This stage measures whether there is an impact on the organization. Has the organization progressed toward its goals or objectives? This is the results measurement stage. Possible results include cost reduction, changes in financial turnover, changes in absence and complaint levels, and increases in quantity and quality of performance.
Organizations have been found less frequently use the last two steps of the model when evaluating training programs - possibly due to difficulty in defining appropriate measurement tools. Moreover, most training evaluations never progress beyond the first step! This is done under the mistaken assumption that if trainees know the training, benefits will also exist in the three additional criteria. However, contrary to this assumption, researchers Alliger & Janak (1989) found that:
Learning (step 2 in the model) was related to both results (step 4) and behavior (step 3)
Behavior (step 3) was related to results (step 4)
Trainee awareness (1) was not related to the other three evaluation criteria!
Nevertheless, the importance of the first measurement step should not be underestimated, as unfamiliarity with the program necessarily prevents its implementation. This fact reinforces the need for careful measurement and evaluation of all four stages.
Evaluation of Knowledge Management Solutions - From Theory to Practice
When evaluating a knowledge management solution, it's important to remember that it is not a goal or objective but rather a supporting tool for achieving the organization's strategic goals. The objective is output that adds value to the organization, such as accident prevention, work method optimization, economic value, and more. Defining the goal we aim to reach at the end of the journey requires defining sub-goals and stages along the way and monitoring their achievement. Therefore, Kirkpatrick's evaluation model, which is based on intermediate stages and gradual progression and additionally dedicates significant attention to evaluating organizational results, naturally meets this need.
However, several adaptations to the model's four stages are needed to adapt it to the world of knowledge management:
Awareness: In the renewed model, we try to evaluate how aware users are of the existence of knowledge management solutions. Do they know about the content areas within it? Are they interested in it? This awareness is a critical stage toward implementing the knowledge management solution - since we necessarily won't experiment with a tool we're unaware of. As measurement tools for this stage, we can use responses arising in focus groups, level of cooperation in training, corridor conversations, and responses to teasers and posters.
Activity: In the renewed model, we measure the level of solution usage at this stage. How much are the various contents accessed? How much new content is being added in any given period? As measurement tools for this stage, we can use usability reports or statistical reports, such as breakdowns by items and breakdowns by user populations, checking the number of entries and amount of discussion component usage.
Implementation in Work Processes: In the renewed model, we measure at this stage the level of solution implementation and integration into work processes and knowledge retrieval during common organizational work processes. As measurement tools for this stage, we can use surveys, observations, work process questioning, usability tests, and more.
Evaluated Results: In the renewed model, we evaluate the content area's contribution to achieving the organization's goals and objectives at this stage. As tools for this stage, we use manager and employee reports and the meaning attributed to the solution in company conversations, in-depth interviews, conferences, and generally - in organizational discourse.
Getting Up Tomorrow Morning...
What should we do now? How can we implement the renewed evaluation model described in the article on organizational knowledge management work? For this purpose, we can use the following points:
To achieve a goal, we need to define a controllable objective and measurable intermediate goals.
It's important to distinguish between the goal and the tool - the knowledge management solution is a tool for achieving output and business-added value for the organization.
Establishing a work plan and planned dates for measuring and intermediate goals is essential. This establishes evaluation as part of planned work processes and ensures execution, feedback, and improvement.
Care must be taken to evaluate the four stages: awareness, activity, implementation, and evaluated results. Each stage has a distinct importance that is crucial to project success!
As measurement tools, it's possible and recommended to use qualitative and quantitative tools (2Know-KM Magazine Issue 97): statistics, surveys, focus groups, questionnaires, and interviews. Combining these tools in each stage will increase evaluation accuracy, promote knowledge management solution implementation, and enable achieving organizational goals.
Good luck!
Yorumlar