Frequently Asked Questions on Evaluation (FAQs)

In my training programs, talks, lectures, and consulting assignments. I am asked a lot of questions. I thought to write about the same as FAQs.

Many people start to ask a question by saying “I am asking something silly” or “This is a stupid question I am asking”. I wonder what makes them say that. For me, a question is a question. There is nothing silly or stupid about it. A person asks because he wants to know something.

Q. Is it worth the investing in evaluation?  

Ans. That is a call a company needs to take. For eg. If you are spending $ 80000 for a training program, isn’t it worth spending $ 10000 on the evaluation? At least here you would know if you want to make investments in future.

Q. What are the benefits of evaluation?  

Ans

• Evaluation has plenty of benefits such as
• Reduce costs, increases revenue, saves time
• Makes everyone in the company accountable
• Training becomes a serious business
• All the programs, policies and processes are aligned with business measures
• The probability of things going wrong reduces
• Makes the vendors accountable
• SOPs are streamlined
• Evaluation earns a person seat in the board room
• We no more do work on feel-good factor. Our work is based on data that is tried and tested
• Budgets are easier to get sanctioned

 Q. Do we need to know statistics to conduct an evaluation study?

Ans. We do not need to know advanced statistics, but good to know basic statistics that include measures of central tendency.

Q. Do we need to wait long (three months to one year) to complete the evaluation? We do not have that much time.

Ans. Evaluation is a process. Any process to implement and see the value takes a minimum of three months. We suggest a time period of at least six months.

Q. Is a pilot test essential?  

Ans. Experience shows that pilot testing a program is always beneficial. Especially for high-value programs. Eg. If you want to invest $50000, it is better to check the pilot and forecast if it is worth investing in the program.

Q. Evaluation is time-consuming  

Ans. How and where one wants to invest time is subjective. It is better to invest time in evaluation rather than repent on the amount, efforts and time spent on programs that give no ROI.

Q. Are companies investing in evaluation on programs?

Ans. Yes, The major industry that does evaluation is the Non-Government Organizations (NGOs) and departments under corporate social responsibility (CSR). Other companies investing in the evaluation are those which are data-driven companies. Where accountability and measurement is the culture and the HR are willing to take risks, show accountability and have a say in the decision making. However, the trend shows that now HR and L&D departments are getting serious and want to show value for money.

Q. Can soft skills, behavior be measured?  

Ans. Majority of the things can be measured provided we intend to measure and we know how to measure.

Three things are vital

•          Operational definition of the concept to be measured  
•          Measures that are linked to the business needs
•          Data availability
•      Various tools, techniques, and softwares on measurements are available.

Q. Is it really possible to measure anything?   

Ans. Yes, our entire world is governed by measurements. Some measurements are available, some are derived.

Eg.  Crossing the road: We take the distance and make certain assumptions about the speed of the car before we cross the road. Our brain is complex and is capable of processing information at a speed with a multiplier effect. The brain processes the information so fast that everything is governed at the subconscious level and we act consciously. All the actions that we do are based on measurements whether it is the force, speed or time.

Q. Is learning EXCEL enough to conduct evaluation study?   

Ans. Yes, Excel is enough for fewer data. For larger data, knowledge of ‘SPSS’ or ‘R’ is needed. 

For any questions do write to me (wagle.sonali@gmail.com)

With Gratitude

Sonali Wagle

Can empathy be developed in 2 days?

We have been getting a lot of enquiries to evaluate leadership programs on empathy. Empathy, it is an irony that this is the most talked subject and yet forgotten when it is required the most. Companies conduct programs on empathy from two days to a week. I always wonder can empathy be taught during training programs or retreats. Microsoft CEO Satya Nadella talks about empathy in his book ‘Hit Fresh’. One of the few CEOs, to talk about implementing empathy in the company and pass on to its customers through products and services. The first and the foremost he asserts to bring in empathy is a culture change. He also mentions that how personal setbacks help to bring in empathy.

Does a personal setback definitely bring in empathy or empathy can be buildup irrespective of the personal setbacks? Empathy again is a subjective term. The terms compassion, loving kindness are all synonyms.

Many organizations do employee engagement activities now under the concept of CSR and they feel that is enough. Is this sufficient to drive empathy in organizations or something more is required? Can an organization renew itself and set in the culture of empathy and how?

With Gratitude

Sonali Wagle

Sampling in Evaluation Studies

When one wants to know the exact number of defective pieces a box may contain, we need to test and count them all. However, when in comes to large quantities, sampling works the best.

Many times HR professionals question how do we do sampling for our training programs. 

Case in Point

Name of the training: Leadership training

Number of people to be trained: 90

Departments: All

Estimated cost: Rs. 75 lakhs

Rs. 75 Lakhs is a major investment. For all our ROI studies and before investing so much money we suggest our clients, to do a pilot test and take a sample before deciding if the training would give any ROI. Following are the steps that we take1.    

1. Decision to do a pilot test.
2. Identify a maximum of 10 people for the pilot.
3. Out of the 10 people, one person must represent each department. So we have a heterogeneous group.
4.  Identify at least two impact measures per participant.
5. Conduct the training with action plan documents to be submitted post one month.
6. Measure reaction and learning.
7. Make the necessary changes in the content based on the reaction and learning feedback.
8. Wait for a month.
9. Conduct focus group/ interview the participants involved.
10. Check if impact measures match the action plan documents.
11. Forecast training impact.
12. If the forecast is in the positive direction, launch the training for the remaining 80 people.

Please note one need not take the sample of 30 people that is considered a minimum required in statistics. When it comes to training, 10 people are a good representation to check the impact of the training.

For all our impact studies, we emphasize the need to do a pilot and check a sample. Unfortunately, this is overlooked on account of the following reasons

1. The urgent need to conduct training, so there is no time to wait and do a pilot.
2. There is a mandate from the top management to launch training immediately.
3. HR people are co-coordinating for various things and hence need for sample test is not felt.
4. Post-pilot, what if we realize that the training is not giving the impact then how do we deal with it.
5. We have to use the sanctioned budget.
6. We have developed the content so 10 people undergo the training or 60 people, in any case, we have invested in the content.
7. We are short-staffed to keep doing pilot testing.

There is a need to go beyond our traditional way we conduct training and not just experiment with the training format (gaming, activities, and simulation) but also the way we execute the end-to-end training process.

Happy Sampling

Sonali

Do we believe in measurement?

Dear Friends

In one of the conferences, one speaker said, you cannot measure everything. I have spent a decade in measurements, evaluation, ROI and HR analytics, I differed with him. Of course, measurements often face with obstacles.

We live in a life of uncertainty and to measure this uncertainty is a challenge. I come from the field of Mathematics and Statistics, know that everything can be measured provided you intend to measure and know how to measure. Many models and tools exist which we can use to measure. Dr. Jac Fitz-enz, Dr. Jack Phillips, and Douglas Hubbard have written much on this subject.

When it comes to Business management especially with respect to HR and training, that a question of measurement arises. When I interview HR heads, many often respond that soft skills, policies, HR processes cannot be measured and that there is no use measuring it. Why are HR professionals not confident to measure their own work?

Few thoughts

1.    If HR professionals cannot measure what they do then how will they bring value to their role?

2.    If HR professionals cannot quantify and measure what they do then on what basis do the management decide their salary increment and incentives, which many feel it is their right.

3.    If HR professionals still believe that HR cannot be measured and training does not give ROI, then do we need HR and training departments?

Case in Point: In one of my consulting assignments, I presented our final measurement report on one of their training programs. One person who was a financial auditor said, “all that is mentioned in the report is based on probabilities, ranges, assumptions, and estimations. How do we buy-in this?”

It is ironic that the same person when goes to the medical doctor accepts the medical report full of ranges and assumptions. This same person finalizes the company balance sheet that is full of estimations (as mentioned in the balance sheet - this figure is based on management estimations). When I happen to answer this, he became defensive. I also mentioned to him that this is the way you evaluate risks, uncertainty and do valuation. I failed to understand what stops such people from accepting that measurement is possible. And what is acceptable in finance can be accepted in HR too.

Another example was the people attending the presentation, many were from finance, who had no clue or rather ignored that HR processes, policies can be measured. How many of you have you ever seen HR professionals attending a balance sheet presentation by the auditors?

After all decision sciences, actuarial science and risk analytics is all based on measurements and we can use the same models in business management too. Nobody ever thought that gravity could be measured until Newton came up with the formula. How can we forget Einstein’s theory of relativity?  

Peter Drucker said, “What gets measured gets done.” Of course, we cannot ignore and eliminate the human element and experience involved in the measurements. Though it is only the human element that believes that measurement is possible.

With Gratitude

Sonali Wagle

wagle.sonali@gmail.com

www.exponentialadvisory.co.in

Is Your Organization ROI Ready?

Greetings!

Completed an impact study for a client for their leadership program.  The client approached me stating that training has taken place and the CEO needs to know the monetary returns of the training. Can we calculate the ROI on the leadership program?

Indeed, I said yes. What followed next was a series of interviews with the participants, supervisors, training managers to check how the training had benefitted them on their job. The interviews were taken two months post training. Few of the responses are listed below

1. I understood what it means by trust in leadership.
2. Better understanding of team
3. Now I can lead by example
4. This training had been a great initiative
5. The program helped me to enhance my knowledge and skill
6. This training was required to better understand organizational goals and processes

One thing I realized was participants were not much aware as to what is expected of the training. This was my first question to the VP learning if participants were aware as to what was expected of them and how were they going to document it?

After much discussion, we came to a conclusion that Business Alignment was missing. How can ROI be calculated of the business alignment itself was missing. My next question to the VP was

"Is your company ROI ready?"

What do you think about your organization? Is your organization ROI ready? Do write to me on sonali@exponentialadvisory.co.in

With Gratitude
Sonali Wagle

V-Model for Business Alignment

Dear Colleagues                                                               (time to read the blog - 3 Minutes)

Another ROI Impact study completed and it dawns on me that in spite of so much investment in the training domain, we still miss the business alignment.

Reasons for business alignment gap

Ø  The need to do business alignment – The HR do not feel the need to do business alignment. They drive the training and hope and assume that this training would be successful. HR is happy with a good score on the reaction questionnaire. So what next?

Ø  Lack of awareness – Many are not aware that training itself can be business aligned and that it can give business results. Though things are changing with constant scrutiny on the HR department, data and results are yet to be seen.

Ø  Last minute demand – Many companies set their yearly calendar, yet there are last minute training demands. So where does business alignment fit in? What follows is

• Finding vendor/ trainer
• Venue
• Nominating participants

And again what we have is training with WOW effect and no ROI. This reminds me of fast food- burgers and pizzas that gives us the wow effect that we enjoy and bear the cost with no health benefits.

One such tool used to develop business alignment is the V-Model  as given in the Phillips ROI Methodology. The major advantage of the V-model is to set-up and design the training to align to the measures that need improvement. Business alignment is the process to ensure that any training drive business results. To achieve this, the training can be positioned for success, and then evaluated accordingly.

The three phases of the V-Model are

•            Needs assessment
•            Development of training objectives
•       Training evaluation

The V-model describes the levels of evaluation. V-model is a vital step for need analysis as this gives critical linkages between the needs, objectives and the evaluation. The V-model details the needs at the five levels, beginning with pay off needs and progressing to preference needs. The objectives can be derived directly from the needs identified. For every specific need, multiple objectives can be developed. Below is a suggested example of V-Model

Please feel free to share training and business alignment for the programs you do.

With Gratitude

Dr. Sonali Wagle