From time-to-time I invite guest bloggers to publish related articles. As you’ll see, Mike Nering is an expert in quantifying engagement and here he discusses approaches to measurement.
The measurement of engagement in many ways is the like the measurement of any psychological trait. Indeed, some latent psychological traits are easier to measure than others, but generally as long as you ask good questions, and the right number of questions, you’ll be able to quantify just about any psychological trait that interests you. The problem is that there is no easy process for developing a psychological measurement instrument. This has to do with the fact that there are many factors that make the development of an instrument difficult, such as:
- Purpose of the assessment
- Context of the assessment
- Consequences of the assessment (e.g., decisions based on performance)
- Definition of the audience
- Articulation of the construct of interest
Generally, when we want to measure a psychological trait of an individual we think of an assessment. An assessment is simply a bunch of questions, and these questions can take on any form we can think of. Of course we all know of multiple-choice, true-false, or survey questions, and I’m sure we can image computer simulations or games that might also measure different psychological things. I like to think of these as tasks, tasks that are specifically developed to measure a psychological construct.
The reason why the measurement of psychological traits in not an easy process is because it is not as easy as you might think to develop a bunch of assessment tasks to perfectly measure a particular psychological trait. For example, some tasks may simply do a better job of capturing the psychological trait than other tasks. That is, some tasks might perform better, and be a more efficient way of measuring something. As a result of this measurement error, we need to develop and administer tasks to a bunch of people. In the end we can think of this measurement problem as meeting at the cross roads of tasks and people. Just as items might not align to a psychological construct, people vary in the degree to which they possess a particular psychological trait (e.g., think of extroversion or introversion).
With the development of any assessment there is a rather involved process that typically starts with construct definition, proceeds through task development and data collection, and concludes with analysis and the reporting of results. Here I’m going to focus on the process of quantifying the interaction that people have with tasks. That is, I’m going to discuss why you need a psychometric model during the analysis, so that you are able to make the best interpretation about the task-person interaction.
Let’s suppose that are tasks are simple true-false type questions for now. When presented with a task a person is asked to respond either true or false. You are an engaged at work (true or false). You would think that all we need to do is ask a bunch of questions like this, and add up the number of true and false responses. The problem with a simple sum score like this is that it treats all tasks the same. As we discussed earlier it is reasonable to think that some tasks tap into the construct differently than other tasks. And this is where psychometrics can be used to help. Instead of ignoring the characteristics of the tasks – using a psychometric model explicitly factors task characteristics into person level scores. An example of a psychometric scale for a task can be seen in Figure 1.
What we see in the Figure 1 diagram is what we call a trace line for an item. This is mathematical approach that can be used to describe the person-task interaction. This diagram represents a single item, and the probability of responding true to that item, given a person’s standing on the psychometric scale. In this example, the higher a person is on the psychometric scale, the higher the probability of a true response. The change is probability of a correct response is gradual (i.e., it is not a step function) as we move across our psychometric scale. The psychometric scale is an arbitrary numerical scale (in this case going from -4 to +4), and it is our ruler that we use to place both our people our tasks.
Tasks can vary in terms of how they relate to the psychometric scale. For example, in the Figure 2 diagram we see two tasks where our new task is located further up the psychometric scale compared to the task in the Figure 2 diagram.
For the right most trace line, for a given point along on psychometric scale we can see that there is a lower probability of a true response compared to the left most trace line. We can think of the right trace line as requiring a higher degree of the psychometric scale for us to see a true response. In our third diagram below we see a collection of tasks that vary in many different ways. By allowing our trace lines for each task to vary in complex ways, like in Figure 2, we are recognizing that the construct itself is complex and requires a variety of tasks in order to have some degree of confidence in our measurement process.
Just as we locate tasks along our psychometric scale we can also locate people. By having people response to a variety of tasks, like those depicted in Figure 3, we can use various statistical procedures to obtain a statistical likelihood function. This function persists on our psychometric scale, and is used to locate the most likely position on the scale for a given individual. The process for achieving this likelihood function is not really all that important. The point in presenting this is to show how we use responses to tasks as a way of locating an individual on our scale.
The center point of the curve is the ‘maximum’ of our likelihood function and is used to represent an individuals standing on the psychometric scale. Because different people respond to tasks in different ways, each person will have their own likelihood function like the one above, and the location of each persons’ likelihood function depends on how they responded to the various tasks. By using methods like this we can provide a degree of precision in the measurement process that is completely lost in the sum score approach.
This quantification process is rather complex in that we need to understand the statistical characteristics of our tasks, and then administer the tasks to the people that we are interested in measuring. In the end we are given a person’s location along a scale. This numerical representation of a psychological constructed is rather limited by itself. The power of psychometrics really comes into play once we can group people, and further explore our construct of interest. For example, if we are measuring engagement, we may find a group of disengaged people (i.e., a group of people located along a similar point of our engagement psychometric scale), and we may find other data to support why these individual might be disengaged.
Psychometric methods are best used when in conjunction with other data, or with specific purposes in mind. Knowing the nature and level of disengagement is really a starting point for, and helps to give an understanding of the nature of disengagement and the magnitude of this construct of interest. In fact, psychometric methods should be seen as merely one step in a larger measurement process.
Related Article: Customer Engagement Roadmapping
Thanks for sharing your thoughts about psychometrics.
Regards