Module Components

• Learning Activities

Topics

• Sampling Overview
• Sampling for Quantitative
• Sampling for Qualitative
• Measurement
• Levels of Measurement
• Reliability and Validity of Measures
• Estimating Reliability and Validity
• Modue 3 Self-Review

NRSG 790 - Methods for Research and Evidence-Based Practice

Module 3: Research Methods

Reliability and Validity of Measures

In order for assessments to be sound, they must be free of bias and distortion. Reliability and validity are two concepts that are important for defining and measuring bias and distortion.

Reliability refers to the extent to which assessments are consistent. Another way to think of reliability is to imagine a kitchen scale. If you weigh five pounds of potatoes in the morning, and the scale is reliable, the same scale should register five pounds for the potatoes an hour later. Likewise, instruments such as classroom tests and national standardized exams should be reliable – it should not make any difference whether a student takes the assessment in the morning or afternoon; one day or the next.

Validity refers to the accuracy of an assessment -- whether or not it measures what it is supposed to measure. Even if a test is reliable, it may not provide a valid measure. Let’s imagine a bathroom scale that consistently tells you that you weigh 130 pounds. The reliability (consistency) of this scale is very good, but it is not accurate (valid) because you actually weigh 145 pounds. Also, if a test is valid, it is almost always reliable.

The above situation shows four possible outcomes:

1. In the first one, you are hitting the target consistently, but you are missing the center. You are consistently measuring the wrong value – reliable but not valid.
2. In the second you are spreading you hits across the target randomly and getting some right for the group in the center but not all – valid but not reliable.
3. In the third you are you are spreading your hits in a wide area but missing the center – neither reliable or valid.
4. Last, you consistently hit the center – reliable and valid

FACTORS AFFECTING RELIABILITY & VALIDITY

Manner in which the measure is scored

Example: Having an essay test in English scored by a non English speaking person rather than an English speaking scorer.

Characteristics of the measure itself

Example: Measuring blood pressure by asking the subject whether or not they think their blood pressure is normal rather than using a recently calibrated blood pressure cuff.

Physical and/or emotional state of the individual at the time of measurement

Example: Asking a mother who is late picking up her toddler from nursery school to complete a 10 page satisfaction questionnaire before leaving the exam room as compared to a mother who has no pressing need to leave in a hurry.

Properties of the situation in which the measure is administered

Example: Interviewing a subject while standing in the check out line in a grocery store as compared to interviewing a subject in a private room with comfortable seating and a quiet atmosphere.
If the attribute being measured does not change , a reliable tool or method should produce stable observations and scores

Example: Responses from an environmental survey mailed to residents living in Baltimore City that is forwarded to a respondent who has relocated to a rural area of a different state as compared to respondents who are still living in Baltimore City.

Since one can never directly compare an observed score with a true score, the true score is estimated by obtaining and comparing multiple observations about the attribute of interest

Example: If the interest is in the activities of daily living skills of a geriatric patient living in an assisted living facility observing the patient over time performing a variety of skills as compared to observing the patient on only one occasion.

Reliability and validity of a tool or method should be assessed each time it is used to ascertain that it is performing as planned.