J. Dairy Sci. 2008. 91:119-126. doi:10.3168/jds.2007-0496
© 2008 American Dairy Science Association ®
Evaluation of a Lameness Scoring System for Dairy Cows
P. T. Thomsen1,
L. Munksgaard and
F. A. Tøgersen
University of Aarhus, Faculty of Agricultural Sciences, Research Centre Foulum, Tjele, Denmark
1 Corresponding author: PeterT.Thomsen{at}agrsci.dk
 |
ABSTRACT
|
|---|
Lameness is a major problem in dairy production both in terms of reduced production and compromised animal welfare. A 5-point lameness scoring system was developed based on previously published systems, but optimized for use under field conditions. The scoring system included the words "in most cases" in the descriptions of the clinical signs evaluated. This was done to avoid a situation in which cows might not fit into any of the categories. Additionally, a number of clinical signs used in other lameness scoring systems, considered of less importance in relation to lameness, were not included. Only clinical signs were included that could easily be assessed within a few seconds from a distance. The scoring system was evaluated with intra-and interobserver agreement using kappa statistics. The evaluation was done before and after training 5 observers. Weighted kappa values ranged from 0.38 to 0.78 for intraobserver agreement, with mean kappa values across all observers of 0.60 and 0.53 before and after training, respectively. Weighted kappa values ranged from 0.24 to 0.68 for interobserver agreement, with mean kappa values across all pairs of observers of 0.48 and 0.52 before and after training, respectively. Training had only a limited positive effect on intra- and interobserver agreement. Additionally, how the different lameness categories are distributed along a theoretical scale representing the full spectrum of lameness from "absolutely normal gait" to "as lame as a cow can possibly be" was evaluated. This evaluation was done using the polychoric correlation coefficient. The estimated within-observer polychoric correlation coefficient ranged from 0.76 to 0.96, and there were no significant differences between the thresholds used to classify cows into different lameness categories by different observers before or after training. In conclusion, the results suggest that the lameness categories were not equidistant and the scoring system has reasonable reliability in terms of intra- and interobserver agreement.
Key Words: lameness scoring system • interobserver agreement • intraobserver agreement • dairy cow
|
INTRODUCTION
|
|---|
Lameness is a frequent and serious problem among dairy cows. Lameness negatively affects production and animal welfare (Green et al., 2002; Whay, 2002; Ettema and Østergaard, 2006). For many purposes it is important to be able to evaluate the lameness status of a large number of cows using a simple, quick, and reliable method. These purposes include the evaluation of risk factors affecting the incidence or prevalence of lameness in a dairy herd, evaluation of the impact of lameness on animal welfare and production, and as a tool in day-to-day health management in dairy herds. A large number of scoring systems for the evaluation of dairy cow lameness exist (see review by Whay, 2002). Some have been used extensively (Sprecher et al., 1997) and others in just one or a few studies (Winckler and Willen, 2001). But relatively few of the existing lameness scoring systems have been evaluated scientifically regarding intra- and interobserver agreement (Winckler and Willen, 2001; Flower and Weary, 2006; Thomsen and Baadsgaard, 2006), even though such an evaluation is very important for proper interpretation of the results both within and between studies.
Many of the lameness scoring systems use a 5-point ordinal scale (Sprecher et al., 1997; Winckler and Willen, 2001; Flower and Weary, 2006) and many of the scoring systems are modifications of previously published scoring systems (Winckler and Willen, 2001; Haskell et al., 2006; Rajkondawar et al., 2006). The available number of lameness scoring systems is relatively large and many of the systems are modifications of each other and therefore very much alike. To our knowledge, only 1 study has evaluated whether the different categories of an ordinal lameness scoring system are equidistant (Engel et al., 2003). This means that how the different lameness categories are distributed along a theoretical scale representing the full spectrum of lameness from "absolutely normal gait" to "as lame as a cow can possibly be" has rarely been evaluated. Intuitively, many people would believe that 5 ordinal lameness categories would cover the full spectrum of lameness with the distances between categories being the same for all categories (equidistant), but this might not be the case. An alternative situation might be that the distances between lameness categories are not the same. One might imagine a situation where lameness categories 1 and 2 could be used more or less interchangeably to describe nonlame cows; lameness category 3 used to describe moderately lame cows; and lameness categories 4 and 5 used more or less interchangeably to describe cows that are obviously lame. Information about the distances between categories would be valuable for the correct interpretation of results, because no gold standard exists.
The objectives were to describe a lameness scoring system for dairy cows based on direct observations of animals and evaluate this scoring system regarding intra- and interobserver agreement. Additionally, the effect of training was evaluated of the observers on intra- and interobserver agreement. Finally, the distances were evaluated among the 5 lameness categories of the scoring system to assess equidistance. In contrast to Engel et al. (2003), this evaluation was based on a method that does not depend on a second subjective scoring of the gait of the cows (visual analog scale).
|
MATERIALS AND METHODS
|
|---|
Scoring System
A modified scoring system (Table 1
) was developed based on locomotion scoring systems of Sprecher et al. (1997), Flower and Weary (2006), Haskell et al. (2006), and Rajkondawar et al. (2006). The presence or absence of certain clinical signs for each lameness category is presented in a schematic way in Table 2, which may be used as a guideline to differentiate between different lameness categories. During the development of the modified scoring system, emphasis was put on 3 major points in which the scoring system differs from other systems: 1) the scoring system should be relatively simple; 2) the system should avoid a situation in which cows do not fit into any of the categories by the descriptions; and 3) the system should include no observations relating to the behavior of the cow getting up or lying down. Discussion of the background and reasons for this is given in the Results and Discussion section.
View this table:
[in this window]
[in a new window]
|
Table 2. The presence or absence of certain clinical signs for each lameness category in a lameness scoring system for dairy cows
|
|
Housing and Cows
All observations were conducted at the Danish Cattle Research Centre (Foulum, Denmark) in a free-stall barn with free-stalls and slatted floor. The slatted floor in the free-stall barn had been renovated a few months before the study to improve the nonskid properties of the floor. A special machine cut grooves (approximately 4 mm wide and 3 mm deep) in the slats (perpendicular to the openings between the slats). This was done 3 mo before the experiment and the cows therefore had ample opportunity to adjust to these changes. The cows observed were selected randomly among all 135 lactating cows in the herd and included Danish Holstein (n = 56), Danish Red dairy breed (n = 41), and Danish Jersey cows (n = 38). The mean milk yield was 8,959 kg of ECM/yr.
Observers
The observers had some to much experience regarding lameness scoring of cows. Nevertheless, none of the observers had used the new lameness scoring system before this experiment. Observers 1 and 2 developed the scoring system. Observer 1 was a veterinarian with 6 yr of experience as a bovine practitioner and 5 yr of experience where he assigned lameness scores to >15,000 cows using the lameness scoring system described by Sprecher et al. (1997). Observer 2 was a researcher with 23 yr of experience in animal behavior and stress physiology. Observer 3 was an agricultural technician with 6 yr of experience, who has been involved in a number of research projects in which lameness scores and a number of other clinical signs were recorded. Observers 4 and 5 were agricultural technicians with 10 and 5 yr of experience, respectively. They work at the Danish Cattle Research Centre and their duties include routine lameness scoring of all cows at the center every second week.
Intra- and Interobserver Agreement and Effect of Training
Each individual cow was observed simultaneously by all observers as she walked a minimum of 15 m on a relatively dry (newly scraped) slatted floor in an alley (3 m wide) between 2 rows of free-stalls. The cow was encouraged to walk by a person walking 3 to 5 m behind her, but was not forced to walk quickly. The observers were placed in the free-stalls and were able to observe the cow from the side, from the front, and from behind. If 1 or more observers did not have adequate time for their observations, the cow was walked past the observers again. The examination of each cow lasted approximately 1 min. The observers assigned lameness scores to each cow individually and were not allowed to discuss or compare their scorings during or after the observations.
The study included 2 separate experiments:
Experiment 1.
Intra- and interobserver agreement was evaluated without training of the observers. The observers were allowed to read the descriptions of the 5 lameness categories (Table 1
) 5 min before the experiment. Thereafter, the observers assigned scores to the cows independently. A total of 50 individual cows were scored twice in the same day by all 5 observers. The time between the 2 scorings was approximately 2 h. Cows were left undisturbed and were allowed to lie down between the 2 scorings. The observers did not discuss their observations or the general use of the scoring system in the period between experiments 1 and 2.
Experiment 2.
This experiment took place 1 wk later and was identical to experiment 1 with the exception that the observers had a short training session before experiment 2. The training session lasted approximately 30 min and included the joint observation and scoring of 8 cows. Training cows included 2 cows each with lameness scores 1, 2, 3, and 4. Each individual cow was first scored independently by all observers, and afterward the scores were compared and discussed. Additionally, observers 1 and 2, who had developed the scoring system, gave a short introduction to clinical signs that were considered typical for each lameness score. Immediately after the training session the observers scored 50 individual cows. The cows from the training session were not scored again and therefore not included in the results. Each individual cow was observed twice with an interval of approximately 2 to 3 h and each observer scored the cows independently as described for experiment 1.
Statistical Analyses
Kappa and weighted kappa coefficients were calculated for the intra- and interobserver agreement with and without training (experiments 1 and 2, respectively). The PROC FREQ procedure of SAS (Version 9.1, SAS Institute Inc., Cary, NC) with the Cicchetti-Allison weight type for the weighted kappa was used. In PROC FREQ, an alternative Fleiss-Cohen weight type can be applied. This weight type generally yields higher kappa values, but the Cicchetti-Allison weights were more biologically relevant, because they put less weight on observations that were relatively far apart (e.g., lameness scores 2 and 4 given to an individual cow). The weighted kappa values found may be considered as conservative.
To investigate whether the distances between the 5 lameness categories of the scoring system were equidistant, the thresholds were assessed among the different lameness categories using the polychoric correlation coefficient. It is appropriate to use this coefficient when the scoring scale was assumed to reflect an underlying continuous variable (also called a latent variable), which was observed only on discrete levels. This idea is illustrated in Figure 1 for a variable in which the number of scorings in each of 4 categories is shown as a bar chart. The underlying normal distribution is overlaid on the bar chart and 
1, 2, and 3 indicate the thresholds separating the category levels, assuming0 = –
and 4 =. In this example there are 4 categories, but this can be generalized to any number of categories. Scorings can be obtained from 2 observers looking at the same cows at the same time or from 1 observer looking at the same cows twice with a short time interval in between, assuming that the 2 observations are correlated. From the assumptions of underlying normal distributions, this may be modeled as a bivariate normal distribution. The polychoric correlation coefficient is an estimate of the correlation between 2 latent trait variables. If there is good agreement between the 2 observations, then the polychoric correlation coefficient is large and the estimated thresholds should be concordant. If a bivariate standard normal density is categorized according to I and J thresholds, the one-stage approach described by Olsson (1979) estimated the I + J + 1 parameters of this model from an observed (I – 1) x (J – 1) contingency table. The R statistical programming environment (R Development Core Team, 2006) and the "polycor" package (Fox, 2006) were used to estimate the various parameters of the latent variables.
|
RESULTS AND DISCUSSION
|
|---|
During the development of the scoring system much emphasis was placed on providing a system that should be relatively simple. Therefore, there was a need to avoid the inclusion of clinical signs that could not easily be assessed within a few seconds from a distance (e.g., whether joints move freely or not). Additionally, the descriptions of the 5 levels of the scoring system often include the words "in most cases". Certain clinical signs in most cases are present in cows being classified as belonging to a certain level, but sometimes this may not be the case. Most lame or severely lame cows (lameness scores 4 and 5) would have an arched back when standing and walking. However, there were often observed cows that were obviously lame (no or only limited weight bearing on 1 leg), but that did not show signs of an arched back when standing (and sometimes when walking). Such a lame cow naturally should be classified as lame even though an arched back was not always evident. Additionally, a nonlame cow could have an arched back due to abdominal pain. Therefore, the words "in most cases" were added to the description of the arched back. This is a major strength and weakness of the scoring system. The phrase "in most cases" was needed to avoid cows being misclassified (e.g., an obviously lame cow that does not have an arched back when standing should naturally not be classified as "not lame"). Conversely, the phrase "in most cases" makes the descriptions less rigid and clearer. No observations relating to the behavior of the cow lying down or getting up were included in the lameness score. These movements are clearly not part of the gait of the cow, but belong to another pattern of physical activity.
The distribution of lameness scores for the 2 repeated scorings for each individual observer before and after training, respectively, is presented in Figure 2. Values of kappa and weighted kappa for intra- and interobserver agreement before and after training are presented in Tables 3 and 4.
View larger version (19K):
[in this window]
[in a new window]
|
Figure 2. Distribution of lameness scores for 2 repeated scorings for individual observers before and after training, respectively.
|
|
View this table:
[in this window]
[in a new window]
|
Table 3. Intraobserver agreement in an evaluation of a lameness scoring system before and after training of the observers, respectively
|
|
View this table:
[in this window]
[in a new window]
|
Table 4. Interobserver agreement in an evaluation of a lameness scoring system before and after training of the observers
|
|
The estimated within-observer polychoric correlation for observers 1 to 5 was 0.96, 0.76, 0.79, 0.84, and 0.89, respectively. A plot of each observer’s thresholds between lameness categories is shown in Figures 3 and 4 (before and after training, respectively). These thresholds with the associated estimates of standard errors are also presented in Table 5.
View larger version (15K):
[in this window]
[in a new window]
|
Figure 3. Plots of thresholds between lameness categories for each individual observer before training. Each pair of panels show the thresholds for an observer estimated from the 2 repeated scorings in experiment 1.
|
|
View larger version (14K):
[in this window]
[in a new window]
|
Figure 4. Plots of thresholds between lameness categories for each individual observer after training. Each pair of panels show the thresholds for an observer estimated from the 2 repeated scorings in experiment 2.
|
|
View this table:
[in this window]
[in a new window]
|
Table 5. Thresholds between lameness categories and estimated SE (in parentheses) for individual observers before and after training
|
|
The agreement within and between observers when using the simple 5-point scale for estimating lameness by direct observation showed a moderate to good agreement with greater agreement within observers than between observers. Furthermore, the results suggested that the thresholds between scores did not differ between observers. Comparing the estimated thresholds between scores for lameness for each observer using a confidence interval with a width of 2 standard errors suggested that the estimated thresholds do not differ between observers. All observers seemed to use approximately the same thresholds to categorize cows with different scores for lameness, and the estimated polychoric correlations were relatively high for all observers.
The assumption of an underlying continuous variable representing the exact degree of lameness seems reasonable. Lameness per se is not categorized into a number of distinct classifications, but constitutes a continuum from "absolutely normal gait" to "‘as lame as a cow can possibly be". Still, illustrating lameness using a number of discrete categories may be an appropriate and more practically applicable approach. The use of a visual analog scale to evaluate lameness (Welsh et al., 1993; Engel et al., 2003) is an example of describing lameness on a (pseudo) continuous scale. Handling of such data is more difficult and Engel et al. (2003) found a larger variation between observers when using a visual analog scale compared with a scale with 9 ordinal categories.
Our results suggest that the 5 lameness categories were not equidistant. The upper and lower limits of the lameness score 3 were relatively close to each other compared with the limits for score 2. From Table 2 it can be seen that one of the differences between scores 3 and 4 was that the observer could identify which leg was affected in cows having a score 4, whereas score 3 indicated that the cow was lame, but the affected leg was not evident. Because experienced observers were used, it might be that when a cow showed abnormal gait it would most often be categorized as belonging to score 4, because the observers were able to identify the affected leg because of their experience. The number of cows receiving lameness score 5 was limited. A low number of severely lame cows were found in other studies (Cook et al., 2004; Espejo et al., 2006). Thus, no reliable estimate for the threshold between scores 4 and 5 is available.
The weighted kappa values ranged from 0.38 to 0.78 for intraobserver agreement, with the mean kappa values of 0.60 and 0.53 before and after training, respectively. Weighted kappa values ranged from 0.24 to 0.68 for interobserver agreement, with the mean kappa values of 0.48 and 0.52 before and after training, respectively. These levels generally were considered as good agreement (Fleiss, 1981) and moderate agreement (Dohoo et al., 2003), respectively. Higher kappa values were found for intraobserver agreement than for interobserver agreement, which agrees with Martin et al. (1987), as they stated that kappa values for intraobserver agreement were likely higher than those for interobserver agreement.
The intraobserver agreement decreased slightly after training, whereas the interobserver agreement increased slightly. Training of the observers seemed to have had only minor effects. The amount of training used may have been insufficient in relation to having a positive impact on intraobserver agreement and had only a small positive impact on interobserver agreement. On the other hand, given the relatively high agreement even before training, the fact that the improvement in kappa values before and after training was limited might indicate that the description of the scoring system (Table 1) was clear, adequate, and unequivocal. Additionally, the observers were experienced. The increase in the simple kappa values for interobserver agreement was larger than the increase in the weighted kappa values, indicating that more scores had become absolutely identical between pairs of observers after the training session. Studies have evaluated the effect of training on intra- and interobserver agreement in a large number of settings in veterinary and human medicine. Still, the effect of training on intra- and interobserver agreement differs markedly from study to study. Some studies demonstrated a positive effect of training (Polderman et al., 2001; Kristensen et al., 2006), whereas other studies have not demonstrated any effect of training (Ford et al., 2000; van Tubergen et al., 2003; Leone et al., 2006).
The short time between the observations decreased the possibility that the true lameness status of the cow could change from the first to the second scoring within each experiment. On the other hand, the short time between the scorings might increase the possibility that one or more of the observers could remember a specific cow and her score from the first scoring. We find this unlikely due to the high number of cows observed. Additionally, the observers were instructed not to try to remember any scores from the first scoring.
|
CONCLUSIONS
|
|---|
Our results suggest that training of experienced observers had only a limited effect on intra- and interobserver agreement. Distances between lameness categories were not equidistant. We have shown that 5 experienced observers have acceptable levels of intra- and interobserver agreement and use approximately the same thresholds to classify dairy cows when using the simple system for scoring lameness. Therefore, this system may be a useful tool for scoring lameness in dairy cows under practical conditions. Nevertheless, the system needs further validation regarding the relationship between lameness categories and lesions leading to lameness.
|
ACKNOWLEDGEMENTS
|
|---|
We greatly acknowledge the help from technicians Helle Møller Nielsen, Mette Eriksen, and Carsten Kjærulff Christensen, who participated in the lameness scoring of the cows. We wish to thank Tine Rousing Nielsen, University of Aarhus, who gave us valuable comments on the manuscript.
Received for publication July 3, 2007.
Accepted for publication September 24, 2007.
|
REFERENCES
|
|---|
Cook, N. B., T. B. Bennett, and K. V. Nordlund. 2004. Effect of free stall surface on daily activity patterns in dairy cows with relevance to lameness prevalence. J. Dairy Sci. 87:2912–2922.[Abstract/Free Full Text]
Dohoo, I., W. Martin, and H. Stryhn. 2003. Veterinary epidemiologic research. AVC Inc., Charlottetown, Prince Edward Island, Canada.
Engel, B., G. Bruin, G. Andre, and W. Buist. 2003. Assessment of observer performance in a subjective scoring system: Visual classification of the gait of cows. J. Agric. Sci. 140:317–333.[CrossRef]
Espejo, L. A., M. I. Endres, and J. A. Salfer. 2006. Prevalence of lameness in high-producing Holstein cows housed in freestall barns in Minnesota. J. Dairy Sci. 89:3052–3058.[Abstract/Free Full Text]
Ettema, J. F., and S. Østergaard. 2006. Economic decision making on prevention and control of clinical lameness in Danish dairy herds. Livest. Sci. 102:92–106.[CrossRef]
Fleiss, J. L. 1981. Statistical methods for rates and proportions, 2nd ed. John Wiley, New York, NY.
Flower, F. C., and D. M. Weary. 2006. Effect of hoof pathologies on subjective assessments of dairy cow gait. J. Dairy Sci. 89:139–146.[Abstract/Free Full Text]
Ford, J. C., K. O’Rourke, J. P. Veinot, and V. M. Walley. 2000. Histologic estimation of coronary artery stenoses: Reproducibility and the effect of training. Cardiovasc. Pathol. 9:251–255.[CrossRef][Medline]
Fox, J. 2006. Polycor: Polychoric and polyserial correlations. R package version 0.7–2. http://cran.r-project.org/src/contrib/Descriptions/polycor.html
Green, L. E., V. J. Hedges, Y. H. Schukken, R. W. Blowey, and A. J. Packington. 2002. The impact of clinical lameness on the milk yield of dairy cows. J. Dairy Sci. 85:2250–2256.[Abstract/Free Full Text]
Haskell, M. J., L. J. Rennie, V. A. Bowell, M. J. Bell, and A. B. Lawrence. 2006. Housing system, milk production, and zero-grazing effects on lameness and leg injury in dairy cows. J. Dairy Sci. 89:4259–4266.[Abstract/Free Full Text]
Kristensen, E., L. Dueholm, D. Vink, J. E. Andersen, E. B. Jakobsen, S. Illum-Nielsen, F. A. Petersen, and C. Enevoldsen. 2006. Within-and across-person uniformity of body condition scoring in Danish Holstein cattle. J. Dairy Sci. 89:3721–3728.[Abstract/Free Full Text]
Leone, M. A., P. Gaviani, and G. Ciccone. 2006. Inter-coder agreement for IDC-9-CM coding of stroke. Neurol. Sci. 27:445–448.[CrossRef][Medline]
Martin, S. W., A. H. Meek, and P. Willeberg. 1987. Veterinary epidemiology. Principles and methods. Iowa State University Press, Ames.
Olsson, U. 1979. Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika 44:443–460.[CrossRef]
Polderman, K. H., E. M. F. Jorna, and A. R. J. Girbes. 2001. Interobserver variability in APACHE II scoring: Effect of strict guidelines and training. Intensive Care Med. 27:1365–1369.[CrossRef][Medline]
R Development Core Team. 2006. R. A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-Project.org.
Rajkondawar, P. G., M. Lui, R. M. Dyer, N. K. Neerchal, U. Tasch, A. M. Lefcourt, B. Erez, and M. A. Varner. 2006. Comparison of models to identify lame cows based on gait and lesion scores, and limb movement variables. J. Dairy Sci. 89:4267–4275.[Abstract/Free Full Text]
Sprecher, D. J., D. E. Hoesteler, and J. B. Kaneene. 1997. A lameness scoring system that uses posture and gait to predict dairy cattle reproductive performance. Theriogenology 47:1179–1187.[CrossRef][Medline]
Thomsen, P. T., and N. P. Baadsgaard. 2006. Intra- and interobserver agreement of a protocol for clinical examination of dairy cows. Prev. Vet. Med. 75:133–139.[CrossRef][Medline]
van Tubergen, A., L. Heuft-Dorenbosch, G. Schulpen, R. Landewe, R. Wijers, D. van der Heijde, J. van Engelshoven, and S. J. van der Linden. 2003. Radiographic assessment of sacroiliitis by radiologists and rheumatologists: Does training improve quality? Ann. Rheum. Dis. 62:519–525.[CrossRef]
Welsh, E. M., G. Gettinby, and A. M. Nolan. 1993. Comparison of a visual analog scale and a numerical rating scale for assessment of lameness, using sheep as a model. Am. J. Vet. Res. 54:976–983.[Medline]
Whay, H. 2002. Locomotion scoring and lameness detection in dairy cattle. In Pract. 24:444–449.[Abstract/Free Full Text]
Winckler, C., and S. Willen. 2001. The reliability and repeatability of a lameness scoring system for use as an indicator of welfare in dairy cattle. Acta Agri. Scand. A (Suppl. 30):103–107.
This article has been cited by other articles:
|
|
|
|
 
R. F. Fitzgerald, K. J. Stalder, P. M. Dixon, A. K. Johnson, L. A. Karriker, and G. F. Jones
The Accuracy and Repeatability of Sow Body Condition Scoring
Professional Animal Scientist,
August 1, 2009;
25(4):
415 - 425.
[Abstract]
[PDF]
|
|
|
|
|
|
|
P. T. Thomsen
Rapid screening method for lameness in dairy cows
Vet Rec.,
May 30, 2009;
164(22):
689 - 690.
[Full Text]
[PDF]
|
|
|
TOP
ABSTRACT
INTRODUCTION
