HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) __Meta-Analysis Reference List Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Lean, I. J. Articles by Dohoo, I. R. Search for Related Content PubMed PubMed Citation Articles by Lean, I. J. Articles by Dohoo, I. R.

Invited review: Use of meta-analysis in animal health and reproduction: Methods and applications

I. J. Lean^*,,1, A. R. Rabiee^*,, T. F. Duffield and I. R. Dohoo

^* SBScibus, PO 660, Camden, New South Wales, Australia
University of Sydney, Camden, New South Wales, Australia
Department of Population Medicine, University of Guelph, Guelph, Ontario, N1G 2W1 Canada
Centre for Veterinary Epidemiological Research, Atlantic Veterinary College, 550 University Avenue, Charlottetown, Prince Edward Island, Canada

¹ Corresponding author: ianl{at}dairydocs.com.au

	ABSTRACT

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

 
The objectives of this paper are to provide an introduction to meta-analysis and systematic review and to discuss the rationale for this type of research and other general considerations. We highlight methods used to produce a rigorous meta-analysis and discuss some aspects of interpretation of meta-analysis drawing on examples from the animal and veterinary science literature. Meta-analysis is a rapidly expanding area of research that has been relatively underutilized in animal and veterinary science. It is a quantitative, formal, epidemiological study design used to systematically assess previous research studies to derive conclusions about that body of research. Outcomes from a meta-analysis may include a more precise estimate of the effect of treatment or risk factor for disease, or other outcomes, than any individual study contributing to the pooled analysis. The examination of variability or heterogeneity in study results is also a critical outcome. The benefits of meta-analysis include a consolidated and quantitative review of a large, and often complex, sometimes apparently conflicting, body of literature. Meta-analytic methods place less emphasis on dichotomous outcomes from null hypothesis significance testing and greater emphasis on determining the magnitude and the precision of an effect of interest. A substantial benefit of meta-analysis is the potential to investigate new hypotheses using existing data, both through the development of a priori hypotheses and by examination of the heterogeneity in study responses. The specification of the outcome and hypotheses that are tested is critical to the conduct of meta-analyses, as is a sensitive literature search. A failure to identify the majority of existing studies can lead to erroneous conclusions; however, there are methods of examining data to identify the potential for studies to be missing; for example, by the use of funnel plots. Many of the statistical methods to conduct meta-analysis are widely used. Bayesian methods are well suited to meta-analysis. The post-hoc methods used to evaluate heterogeneity and publication bias, which include the I ² statistic, L’Abbé plots, Galbraith plots, Rosenthal’s N, and influential study analysis are exclusively used in meta-analysis. Examples where meta-analyses have been repeated in animal science or veterinary medicine show good consistency in estimates of effect. Findings of studies to date have provided new understandings of rumen modifiers, milk fever, parasite control, mastitis, somatotropin, and reproductive manipulations. Rigorously conducted meta-analyses are useful tools to improve animal well-being and productivity. The need to integrate findings from many studies ensures that meta-analytic research is desirable and the large body of research now generated makes the conduct of this research feasible.

Key Words: meta-analysis • dairy cow • sample size • case definition

	INTRODUCTION

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

 
The scientific investigation of animal health and reproduction seeks to provide findings that can be confidently applied to improve health, production, and reproduction. The classical cycle of observation providing a hypothesis, testing of the hypothesis, and refinement of the hypothesis by further testing can now be extended to include the investigation of hypotheses through production of estimates of effect and confidence in these effects using a series of existing studies. The objectives of this paper are to introduce meta-analysis and systematic review, and to discuss the rationale for this type of research and other general considerations. We highlight the methods used to produce a rigorous meta-analysis and discuss some aspects of interpretation of meta-analysis drawing on examples from the animal and veterinary science literature.

Meta-Analysis and Systematic Review
Glass (1976) defined meta-analysis as "The statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings." Meta-analysis is a quantitative, formal, epidemiological study design used to systematically assess the results of previous research to derive conclusions about that body of research. Typically, but not necessarily, the study is based on randomized, controlled clinical trials. Outcomes from a meta-analysis may include a more precise estimate of the effect of treatment or risk factor for disease, or other outcomes, than any individual study contributing to the pooled analysis. Identifying sources of variation in responses; that is, examining heterogeneity of a group of studies, and generalizability of responses can lead to more effective treatments or modifications of management. Examination of heterogeneity is perhaps the most important task in meta-analysis. The Cochrane collaboration (Cochrane Collaboration, 2008) has been a long-standing, rigorous, and innovative leader in developing methods in the field. Major contributions include the development of protocols that provide structure for literature search methods, and new and extended analytic and diagnostic methods for evaluating the output of meta-analyses. Use of the methods outlined in the handbook should provide a consistent approach to the conduct of meta-analysis.

Meta-analyses are a subset of systematic review. A systematic review attempts to collate empirical evidence that fits prespecified eligibility criteria to answer a specific research question (Sargeant et al., 2006). The key characteristics of a systematic review are a clearly stated set of objectives with predefined eligibility criteria for studies; an explicit, reproducible methodology; a systematic search that attempts to identify all studies that meet the eligibility criteria; an assessment of the validity of the findings of the included studies (e.g., through the assessment of risk of bias); and a systematic presentation and synthesis of the attributes and findings from the studies used. Systematic methods are used to minimize bias, thus providing more reliable findings from which conclusions can be drawn and decisions made than traditional review methods (Antman et al., 1992; Oxman and Guyatt, 1993). Systematic reviews need not contain a meta-analysis—there are times when it is not appropriate or possible; however, many systematic reviews contain meta-analyses. Westwood et al. (1998a,b) provided a quantitative, systematic review of protein nutrition in cattle and included several pooled effect estimates and meta-analyses. These provided evidence of the strength of correlation between nitrogen concentrations in the rumen, blood, and milk and the effects on first-service conception rate of increasing CP in the diet and effects of increasing protein protection with isonitrogenous diets.

The benefits of meta-analysis include a consolidated and quantitative review of a large, and often complex, sometimes apparently conflicting, body of literature; for example, the effects of recombinant somatotropin (rbST) on milk production and health of cattle (Dohoo et al., 2003a,b). Meta-analysis can be used by regulatory authorities to provide a quantitative evaluation of the efficacy of a product available for registration. The hypotheses that can be raised in a meta-analysis may extend well beyond those examined in the original studies; for example, meta-analysis was used to evaluate the multivariable effects of macromineral concentrations in the diet on the risk of milk fever (Oetzel, 1991), whereas most of the original studies were univariable in design.

The value of meta-analysis to address the marked increase in literature available to scientists and practitioners in most fields of scientific endeavor has been recognized. These methods have extended beyond the social sciences and human medicine, where they were first developed and adopted, and span disciplines from astronomy to zoology (Petticrew, 2001). From a few early publications, the number of published meta-analyses in medical science had increased to 400 per year in the year 2000 (Lee et al., 2001). While researching this paper, we identified more than 1,000 papers published using quantitative or systematic review in medical science in the year 2008. Statistical techniques to combine results from separate randomized controlled trials, economic evaluations, and epidemiological studies have become more common in animal and veterinary science, possibly because of the increasing size of the research literature, and partly because of the demand for rigorous efficacy and safety standards. Despite a wealth of suitable studies, however, there has not been growth in meta-analysis in animal and veterinary science comparable to that in medicine. Our literature search using PubMed, CAB (Commonwealth Agricultural Bureaux), and Scirus identified only 150 formal meta-analyses and quantitative reviews in cattle. Early examples of meta-analysis in animal and veterinary science include Oetzel (1991), Enevoldsen (1993) investigating factors influencing milk fever, Morgan and Lean (1993) examining effects of GnRH on the probability of pregnancy, and Gross et al. (1999), who examined effects of anthelmintic treatment on milk production. Some areas of veterinary and animal science have benefited from well-executed quantitative review including treatment of parasites (Gross et al., 1999; Sanchez et al., 2004), disease conditions (Fourichon et al., 2000), rumen modification (Lean and Wade, 1997; Duffield et al., 2008a,b,c), rbST (Dohoo et al., 2003a,b), and prediction of milk fatty acids (Moate et al., 2008).

The inclusion of observational medical studies in meta-analyses led to considerable debate over the validity of meta-analytical approaches, as there was necessarily a concern that the observational studies were likely to be subject to unidentified sources of confounding and risk modification (Greenland 1994a,b; Olkin, 1994). Pooling such findings may not lead to more certain outcomes. Animal health and production has a distinct advantage over medical research in being more able to make use of randomized controlled clinical trials.

Meta-analyses are conducted to assess the strength of evidence present on a disease, treatment, or aspects of production or reproduction. One aim is to determine whether an effect exists; another aim is to determine whether the effect is positive or negative and, ideally, to obtain a single summary estimate of the effect. The results of a meta-analysis can improve precision of estimates of effect, answer questions not posed by the individual studies, settle controversies arising from apparently conflicting studies, and generate new hypotheses. In particular, the examination of heterogeneity is vital to the development of new hypotheses. Although dichotomous outcomes are inherent to null hypothesis testing, Nakagawa and Cuthill (2007) note that a key component of meta-analytic thinking is to move away from desiring dichotomous outcomes from null hypothesis significance testing and to place greater emphasis on determining the magnitude of an effect of interest and the precision of estimate of that magnitude of effect. The principles of meta-analysis mean that the cases in one study are not directly compared with those in another, and each study is analyzed separately, unless dealing with pooled analyses using individual data from original studies.

A large number of meta-analyses are undertaken with the broad aim of summarizing existing evidence on a subject; for example, Gross et al. (1999) to evaluate the effect of anthelmintic treatment on milk production in cattle and Robert et al. (2006) to evaluate mastitis and the effects of antibiotic treatment of dry cows. Others are undertaken to inform specific decisions (Sanchez et al., 2004) to evaluate the effect of different classes of anthelmintics on milk production of cattle and may be incorporated in economic models in a decision analysis framework. A well-conducted meta-analysis can provide unbiased overviews of the available evidence on product claims. There is considerable interest in the use of meta-analysis to evaluate the use of new technologies for regulatory evaluation purposes; for example, to assess efficacy and safety of new therapeutics in animal health, production, and reproduction.

	GENERAL CONSIDERATIONS IN META-ANALYSIS

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

 
Individual or Aggregated Data
The majority of meta-analyses are based on a series of studies to produce a point estimate of an effect and measures of the precision of that estimate. However, methods have been developed (Stewart and Clarke, 1995; Simmonds et al., 2005) for the meta-analyses to be conducted on data obtained from original trials. This approach may be considered the "gold standard" in meta-analysis because it offers advantages over analyses using aggregated data, including a greater ability to validate the quality of data and to conduct appropriate statistical analysis. Further, it is easier to explore differences in effect across subgroups within the study population than with aggregated data. This approach relies on cooperation between researchers who conducted the relevant studies. We used this approach to examine the effect of monensin on time to conception (Duffield et al., 2008c) using raw data obtained from those involved in the original studies. Researchers who are aware of the potential to contribute or conduct these studies will provide and obtain additional benefits by careful maintenance of original databases and making these available for future studies.

Study Power
A major rationale for use of meta-analysis is to increase the statistical power of hypotheses. Most disease and many reproductive outcomes are dichotomous and, consequently, require large numbers of animals to determine statistically significant differences between treatments. Figure 1 provides estimates of numbers of animals required to determine differences between 2 groups of subjects with statistical power of 0.9 and = 0.05 for some common diseases at anticipated incidences. Similarly, to detect a 5% difference in conception rates to a single service with an = 0.05 and statistical power of 0.9 requires 460 cows per group.

View larger version (12K): [in this window] [in a new window]   Figure 1. Sample size per group for an intervention targeted at a 50% reduction in incidence, based on estimated incidence of disease in the control group. For example, milk fever, dystocia and abomasal disorders (white bar), ketosis, retained placenta (gray bar), and metritis, mastitis and lameness (black bar). From Lean et al. (2008).

View larger version (44K): [in this window] [in a new window]   Figure 2. Risk ratios (RR) [and their 95% confidence interval (CI) and weights for individual trials] determined from the results of 42 trials comparing risk of pregnancy (PR) conception at 1st and 2nd insemination and overall risk of pregnancy in dairy cows injected with GnRH or analog at the time of insemination in cows at first and second insemination and repeat breeder cows (data obtained from Morgan and Lean, 1993). Box sizes are proportional to the inverse variance of the estimates. Summary estimates of treatment effects are shown using: 1) a fixed effects approach (Mantel-Haenszel = M-H subtotal), 2) a random effects approach (DerSimonian and Laird = D+L subtotal), and 3) the estimated predicted interval of a future trial; this prediction can provide further evidence of the effectiveness of treatment in future studies. A full list of the studies referenced here is available online (http://jds.fass.org/content/vol92/issue8/).

In both cases, an increased precision of estimate of effect over any single, even very large, study and increased external validity of the meta-analysis that used data from a large number of herds and production systems, provided valuable information on the effects of treatment.

Evidence in Animal Studies
There are many examples of observational epidemiological study designs that provide useful evidence of disease or reproduction that include case-control studies (Davis et al., 1980), and cohort studies (Curtis et al.,1984; Degaris et al., 2008). We recommend, in general, that the evidential base used in a meta-analysis consist of randomized controlled clinical studies or prospective cohort studies as these should provide the most rigorous data. Studies on the effects of disease on reproductive performance of dairy cattle (Fourichon et al., 2000) and effects of milk production on disease and reproduction in dairy cattle (Ingvartsen et al., 2003) are examples, however, of the appropriate application of observational studies in quantitative review. Figure 3 shows that there is a range of different study types that can be used to examine health and reproduction and that these provide different insights into these processes. Figure 3 also shows that the studies vary in their validity, with larger multi-site, multicenter studies having great external validity and smaller, more tightly controlled, laboratory or experimental station studies having greater internal validity.

View larger version (17K): [in this window] [in a new window]   Figure 3. Study designs that are used in animal and veterinary science, indicating strength of evidence of causality and whether the study design is of greater internal or external validity (modified from Dohoo et al., 2003).

Case Definition and Implications
The definition of the outcome (i.e., health disorder or production measure) is critical to the conduct of successful meta-analyses. A failure to adequately define the condition or to identify a lack of consistency in the definition in studies can lead to problems of interpretation in meta-analysis.

Definitions of disease often reflect historical understandings of disease (e.g., "milk fever") or terms that reflect aspects of the perceived etiology (e.g., "grass tetany"). Classification of disease should be accurate and reflect standardized and repeatable criteria. Attempts have been made to standardize reproductive terminology; for example, Hubbert et al. (1972). These efforts have provided some clarity to what was an unclear literature. However, there have been few consolidated attempts to define disease processes. Kelton et al. (1998) used an approach of providing standardized definitions of disease. Further, in the case of disease, there needs to be a detailed examination of Evans’ postulates (Evans, 1976) to support the definition. In the case of metabolic disease, we propose the following postulates consistent with those of Evans: 1) the definition of disease should be consistent with current understandings of the biochemical basis of the disorder or reflect modifications thereof; 2) the proportion of individuals with the disease should be significantly higher in those exposed to the supposed cause than in those who are not; 3) exposure to a supposed cause should be present more commonly in those with than in those without the disease, when all other risk factors are held constant; 4) the number of new cases of disease should be significantly higher in those exposed to a supposed cause than in those not so exposed, as shown by prospective studies; 5) a spectrum of host responses from mild to severe should follow exposure to a supposed cause along a logical biological gradient; 6) a measurable change in metabolism should appear regularly following exposure to a supposed cause in those lacking this response before exposure, or should increase in magnitude if present before exposure; this pattern should not occur in individuals not so exposed; 7) experimental reproduction of the disease should occur with greater frequency in animals or man appropriately exposed to a supposed cause than in those not so exposed; 8) elimination (e.g., removal of a specific risk factor) or modification (e.g., alteration of a deficient diet) of the supposed cause should decrease the frequency of occurrence of the disease; and 9) all relationships and associations should be biologically and epidemiologically credible.

Approaches consistent with these postulates have been used to define the metabolic diseases ketosis (Lean et al., 1994; Duffield et al., 2009) and acidosis (Bramley et al., 2008). Curtis (1997) and Sheldon et al. (2006) also used careful definitions of uterine infection and examined these definitions using risk factors for and outcomes of uterine infection. Importantly, all these researchers demonstrated that the definitions developed either predicted loss of production or increased risk of other disease.

Furthermore, it is not always simple to make decisions about the appropriateness of the grouping of a series of studies. Rabiee et al. (2005) decided to include the Pre-synch and Co-synch methods, and minor modifications of these for synchronizing estrus in category called "modified Ovsynch." Advantages of that decision included the opportunity to examine the most recent developments in the field; hence, those methods of ovulation synchrony with the least data at the time of investigation. It was also considered likely that these treatments provided a consistent physiological approach to the synchrony of ovulation. A critical validation of the determination to use these studies as a single category was the post hoc finding of strong homogeneity of the group of studies, supporting the hypothesis that the modified Ovsynch group of treatments was consistent in physiological action. It is vital to stress that meta-analysis, or for that matter, any other study design, needs to consider, in detail and with rigor, the biological basis of the study.

The definition of disease is critical to the process of investigation. If there is a nondifferential error in the definition of disease (i.e., errors of classification are not related to potential risk factors investigated), the effect is to bias the result toward the null. If there are differential errors in definition, the results are unpredictable and can drive the findings toward the null or toward a positive but potentially spurious finding (Kelsey et al., 1996).

	METHODS

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

 
Literature Search
A sound meta-analysis is characterized by a thorough and disciplined literature search. A clear definition of hypotheses to be investigated provides the framework for such an investigation. It is important to obtain all relevant studies, because loss of studies can lead to bias in the study. Typically, published papers and abstracts are identified by a computerized literature search of electronic databases that can include Commonwealth Agricultural Bureau (CAB), Biological Abstracts, Life Sciences, ScienceDirect (www.sciencedirect.com), Scirus (www.scirus.com/srsapp), PubMed (www.ncbi.nlm.nih.gov./entrez/query.fcgi), Agricola (agricola.nal.usda.gov), ISI Web of Knowledge (http://www.isiwebofknowledge.com), and Google Scholar (http://scholar.google.com). Database searches should be augmented with hand searches of library resources for relevant papers, books, abstracts, and conference proceedings. Cross-checking of references, citations in review papers, and communication with scientists who have been working in the relevant field are important methods used to provide a comprehensive search. Communication with pharmaceutical companies manufacturing and distributing test products can be appropriate for studies examining the use of pharmaceutical interventions.

It is not feasible to find absolutely every relevant study on a subject. Some or even many studies may not be published, and those that are might not be indexed in computer-searchable databases. The reviews should attempt to be sensitive; that is, find as many studies as possible, to minimize bias and be efficient. It may be appropriate to frame a hypothesis that considers the time over which a study is conducted or to target a particular subpopulation. The decision whether to include unpublished studies is difficult. Although language of publication can provide a difficulty, it is important to overcome this difficulty, provided that the populations studied are relevant to the hypothesis being tested. Sanchez et al. (2004), for example, sought to include papers in English, Spanish, French, Portuguese, and Italian, and their search included peer-reviewed journals, abstracts, conference proceedings, and theses.

Inclusion or Exclusion Criteria and Potential for Bias
Studies are chosen for meta-analysis based on inclusion criteria. If there is more than one hypothesis to be tested, separate selection criteria should be defined for each hypothesis. Inclusion criteria are ideally defined at the stage of initial development of the study protocol. The rationale for the criteria for study selection used should be clearly stated.

One important potential source of bias in meta-analysis is the loss of trials and subjects. Ideally, all randomized subjects in all studies satisfy all of the trial selection criteria, comply with all the trial procedures, and provide complete data. Under these conditions, an "intention-to-treat" analysis is straightforward to implement; that is, statistical analysis is conducted on all animals that are enrolled in a study rather than those that complete all stages of study considered desirable. However, not all animals provide complete data in large field studies. We strongly recommend that published studies provide full details of subject loss and recommend that, whenever possible, papers are analyzed on an intention-to-treat basis.

Further, not all studies are completed, because of protocol failure, treatment failure, or other factors. Nonetheless, missing subjects and studies can provide important evidence. It is desirable to obtain data from all relevant randomized trials, so that the most appropriate analysis can be undertaken. Chan et al. (2004) and Dwan et al. (2008) have discussed the significance of missing trials to the interpretation of intervention studies in medicine. Journal editors and reviewers need to be aware of the existing bias toward publishing positive findings and ensure that papers that publish negative or even failed trials be published, as long as these meet the quality guidelines for publication.

There are occasions when authors of the selected papers have chosen different outcome criteria for their main analysis. In practice, it may be necessary to revise the inclusion criteria for a meta-analysis after reviewing all of the studies found through the search strategy. Variation in studies reflects the type of study design used, type and application of experimental and control therapies, whether or not the study was published, and, if published, subjected to peer review, and the definition used for the outcome of interest. There are no standardized criteria for inclusion of studies in meta-analysis. Universal criteria are not appropriate, however, because meta-analysis can be applied to a broad spectrum of topics. Published data in journal papers should also be cross-checked with conference papers to avoid repetition in presented data.

Clearly, unpublished studies are not found by searching the literature. It is possible that published studies are systemically different from unpublished studies; for example, positive trial findings may be more likely to be published. Therefore, a meta-analysis based on literature search results alone may lead to publication bias. Efforts to minimize this potential bias include working from the references in published studies, searching computerized databases of unpublished material, and investigating other sources of information including conference proceedings and graduate dissertations.

Quality scores have been used to include or exclude studies from a meta-analysis (Jüni et al., 2001), and ranking methods were developed primarily for studies that sought to include data provided from observational, as opposed to experimental studies, and can be subjective. Before assessing study quality, a quality assessment protocol and data forms should be developed. The goal of this process is to reduce the risk of bias in the estimate of effect. To reduce bias toward a particular view, reviewers should read only the methods and results of the published reports, and all identifying information, such as details providing the journal of publication, authors, institution in which the authors work, are removed from copies of reports given to assessors. The study design, including details of the method of randomization of subjects to treatment groups, criteria for eligibility in the study, blinding, method of assessing the outcome, and handling of protocol deviations are important features defining study quality. When studies are excluded from a meta-analysis, reasons for exclusion should be provided for each excluded study. Usually, more than one assessor decides independently which studies to include or exclude, together with a well-defined checklist and a procedure that is followed when the assessors disagree. Two people familiar with the study topic perform the quality assessment for each study. This is followed by a consensus meeting to discuss the studies excluded or included. Practically, the blinding of reviewers from details of a study such as authorship and journal source is difficult.

We do not generally recommend the use of quality scores to exclude studies. We have examined the effect of including or excluding studies based on "quality" criteria in several data sets and found little difference in outcomes of the pooled assessments. We consider that the post hoc evaluation of the studies objectively ranked on quality attributes, however, can be of value in understanding sources of heterogeneity in a group of studies.

Statistical Analysis
The most common measures of effect used for dichotomous data are the risk ratio (also called relative risk) and the odds ratio. The dominant method used for continuous data are standardized mean difference (SMD) estimation. Methods used in meta-analysis for post hoc analysis of findings are relatively specific to meta-analysis and include heterogeneity analysis, sensitivity analysis, and evaluation of publication bias.

All methods used should allow for the weighting of studies. The concept of weighting reflects the value of the evidence of any particular study. For dichotomous outcomes, the weighting for each study is a function of the number of animals enrolled in the study and in the proportion with the outcome of interest. For an example, see equations 1 to 3. In effect size estimates, the weighting reflects the number of animals used and the variance of the study. Usually, studies are weighted according to the inverse of their variance (Egger et al., 2001). It is important to recognize that smaller studies, therefore, usually contribute less to the estimates of overall effect. This concept is demonstrated visually in Figure 2, in which the size of the box in the forest plot indicates the contribution of the study to the overall effect. However, well-conducted studies with tight control of measurement variation and sources of confounding contribute more to estimates of overall effect than a study of identical size less well conducted.

The following texts, Web site, and papers should be consulted for greater detail on the statistical methods used (Cochrane Collaboration, 2008; http://www.cochrane-handbook.org/; Petiti, 1994; Stangl and Berry, 2000; Egger et al., 2001; Whitehead, 2002). The computer program Stata (Intercooled Stata V.10.2, StataCorp., College Station, TX) provides a comprehensive suite of programs that can be used in meta-analysis. Some detail on the common statistical methods used is presented and examples of when these have been used in studies using cattle are provided.

It has been argued that Bayesian models are more appropriate to decision making in medical policy than frequentist approaches (Spiegelhalter et al., 2000; Congdon, 2001). Furthermore, some workers have suggested that meta-analyses are a "natural" fit for Bayesian statistical approaches (Lindley, 1972). Examples of Bayesian analysis are rare in the animal and veterinary science literature. Rabiee et al. (2004, 2005) chose to use a random-effects, Bayesian model to examine the use of controlled release intravaginal devices that release progesterone and Ovsynch programs on reproductive performance of dairy cows and outlined the statistical methods used. The advantage perceived for this analysis was a benefit of Bayesian approaches in which data, particularly the number of studies, were sparse and the outcomes are more robust than frequentist approaches (Stangl and Berry, 2000). Other advantages of the Bayesian approaches include the production of credible (confidence) intervals rather than P-values, that all parameters are random rather than fixed effects, and the potential to incorporate existing knowledge or even subjective estimates of effect in "informed priors" (Stangl and Berry, 2000; Congdon, 2001; Whitehead, 2002).

One of the foremost decisions to be made when conducting a meta-analysis is whether to use a fixed-effects or a random-effects model. A fixed-effects model is based on the assumption that the sole source of variation in observed outcomes is that occurring within the study; that is, the effect expected from each study is the same. Consequently, it is assumed that the models are homogeneous; there are no differences in the underlying study population, no differences in subject selection criteria, and treatments are applied the same way (Stangl and Berry, 2000). Random-effects models have an underlying assumption that a distribution of effects exists, resulting in heterogeneity among study results. Consequently, as software has improved, random-effects models and Bayesian approaches that require greater computing power have become more frequently conducted. This is desirable because the strong assumption that the effect of interest is the same in all studies is frequently untenable. Whitehead (2002) recommends comparing the fixed effects and random effect models developed as this process can yield insights to the data.

Multilevel modeling is now an accepted statistical analysis method for hierarchical data (Goldstein, 1995). Meta-analysis can also be viewed as a special case of multilevel analysis (Hox and de Leeuw, 2002). When we have a hierarchical data set, with subject within studies at the first level and studies at the second level, the multilevel approach is appropriate. If the original data are available, a standard multilevel analysis can be carried out, predicting the outcome variable using the available individual and study-level explanatory variables. Access to the original raw data is unusual; more frequently, the published results are provided in the form of P-values, means, standard deviations, or correlation coefficients. Raudenbush and Bryk (1985), Bryk and Raudenbush (1992), Kalaian and Raudenbush (1996), and van den Noortgate and Onghena (2003) showed how the multilevel approach could be applied in meta-analysis. Kalaian and Raudenbush (1996) also demonstrated the use of a mixed model for meta-analysis as a special case of the multilevel regression model. The analysis is performed on available statistics instead of raw data, and as a result, some specific restrictions must be imposed on the model (Hox and de Leeuw, 2002). The major advantage of using multilevel analysis instead of classical meta-analysis methods is flexibility. In multilevel meta-analysis, it is simple to include study characteristics as explanatory variables in the model. If there is a hypothesis that the study characteristics can influence the outcomes, these will be included on a priori grounds in the analysis. Alternatively, when it is found that the study outcomes are heterogeneous, the available study variables can be explored to explain the heterogeneity.

Hierarchical models or multilevel models for meta-analysis require the application of some specialized approaches (Thompson et al., 2001). This is because the information in a meta-analysis is usually derived from 2 levels: studies at the higher level and participants within studies at the lower level. Sometimes additional levels of data may be relevant; for example, centers in a multicenter trial or clusters in a cluster-randomized trial. A hierarchical framework is appropriate whether meta-analysis is conducted on summary statistic information or individual patient data (Turner et al., 2000). Such a framework is particularly relevant when random effects are used to represent unexplained variation in effect estimates among studies.

Hierarchical models are useful in several contexts and they can be used to 1) allow for the imprecision of the variance estimates of treatment effects within studies; 2) allow for the imprecision in the estimated between-study variance estimate, 3) provide methods that explicitly model binary outcome data, rather than use summary statistics; 4) investigate the relationship between underlying risk and intervention benefit; and 5) extend methods to incorporate either study-level characteristics or individual-level characteristics. Hierarchical models can be more relevant when individual data on both outcomes and covariates are available (Higgins, et al., 2001). However, even when using such methods, care still needs to be exercised to ensure that within- and between-study relationships are not confused. Hierarchical modeling requires appropriate software, either using a classical statistical approach (e.g., SAS Proc Mixed, SAS Institute Inc., Cary, NC) or a Bayesian approach (e.g., WinBUGS).

Dichotomous Data.
Given that many of the earlier meta-analyses related to cattle were on the effect of reproductive manipulations on conception or pregnancy, a risk ratio or relative risk (RR) analysis was used. This method used data extracted from individual studies to calculate the RR for individual studies and subsequently the Mantel-Haenszel (MH) test, a fixed-effect model, was used to calculate pooled effects.

The MH method weights studies by sample size and allows stratification by study. Separate 2 x 2 tables are constructed and a pooled estimate of relative risk calculated. A chi-square statistic with 1 df is used to assess the significance of the summary measure of effect, and the standard error of the estimate is used to compute a confidence interval around the relative risk. The following formulae and methods are used for the calculation of MH relative risk (Rothman, 1986):

Mantel-Haenszel relative risk (RR_MH) is calculated with the following formula:

where a, b, N₀, N₁, and T_i are calculated from 2 x 2 tables constructed for each trial (Table 1).

where a, b, N_o, N₁, M₁, and T are calculated from 2 x 2 tables constructed for each trial and ln is the natural log. Confidence intervals for the RR_MH can then be calculated using this variance:

Other fixed-effect methods used for dichotomous data include the Peto method for determining odds ratios that is suitable for observational studies (Petiti, 1994). A random-effects method was described by DerSimonian and Laird (1986) and is suitable for conducting a random-effects analysis on dichotomous outcomes from studies.

Other approaches to dichotomous data include the fixed effects logistic regression methods used by Oetzel (1991) and Enevoldsen (1993) in response to the original meta-analysis of Oetzel (1991). Later, Lean et al. (2006) used a random-effects logistic regression to examine an expansion of the original data set of Oetzel (1991). Logistic regression models provide the considerable advantage of being able to control for effect modifying or confounding variables that may influence the interpretation of the effects identified. Charbonneau et al. (2006) used a mixed-models analysis with an arc-sine transformation to evaluate effects of dietary cation anion difference in precalving diets on milk fever.

Continuous Data.
Continuous data are analyzed using SMD, which is also called effect size (ES; Petiti, 1994), in which the difference between treatment and control groups means is standardized using the standard deviations of control and treatment groups.

Effect size can be described in several ways: 1) standardized normal deviate, 2) correlation coefficient, and 3) SMD. The SMD is defined as the difference between 2 population means divided by the standard deviation of one or both populations or by a common population standard deviation. It is often used when an experimental population is compared with a control population, to investigate the effect of an intervention. The SMD was first used by Cohen (1969) to perform power calculation for t-tests. Subsequently, this measure of ES has been frequently used to summarize the results of group comparison studies (Glass, 1976; Glass et al., 1981). Another approach was developed by Hedges and Olkin (1985) who proposed making the weights independent of the observed ES by using the overall ES estimate to estimate the weight.

There are several statistical methods for estimating ES: 1) Cohen (1969), 2) Hedges and Olkin (1985), and 3) Glass et al. (1981). When the pooled standard deviations of the 2 groups are not available, mean differences are standardized by using the overall treatment and control group standard deviation. An ES so derived is a "Z" or normal probability score, which reflects the probability that the treated group mean arose from the control group population (Hedges and Olkin, 1985). Effect size estimates are pooled using the method of Hedges (1982). Differences between treated and control group means are weighted by sample size and averaged to provide an indication on the extent of the pooled treatment effect.

ES Calculation.
In the fixed-effects case, all the ES estimate the single population effect size. If each ES could be measured with an infinite sample, all the ES in the meta-analysis would be the same.

The ES for each study is calculated using the following formula

where is the experimental group mean, is the control group mean, and S_p is the pooled standard deviation. The pooled standard deviation of the control and treatment groups is used as the estimate of S_p as described by Mullen and Miller (1991). Standard deviation (S) is extracted from the papers reporting data as follows:

Where an overall standard error of the mean (SEM_p) is reported for the treatment and control groups, S_p is calculated thus:

where n_p is the total number of cows in the treatment and control groups (Huntsberger and Billingsley, 1987).

Where the SEM is reported for the treatment and control groups separately, the standard deviation is calculated with the formula above for each sample group and the standard deviations are then pooled.

Where papers reported standard deviations for the treatment and control groups, these are also pooled with the formula:

where n_e and n_c are the number of cows in the experimental and control groups, respectively, and s_e and s_c are the standard deviations for the experimental and control groups (Hedges and Olkin,1985).

The overall ES for all trials (ES_o) is calculated weighted for sample size of each trial:

where n_i is the number of cows in each trial and k is the number of studies being assimilated (Hedges and Olkin, 1985).

The variance of the effect size for each trial is calculated (Hedges and Olkin, 1985):

and 95% confidence intervals for the ES of each trial and for ES_o:

Weights are calculated

where W_i is the weight assigned to each study, and var_i is the variance.

The methods used in meta-analysis of continuous data, particularly in reference to nutrition studies in animals, have been explored by St-Pierre (2001) and Sauvant et al. (2008). Sauvant et al. (2008) explore the use of mixed models and regression analysis in meta-analysis, and the methods used in these papers should be examined along with those used by Duffield et al. (2008a) for continuous data. The use of analytic methods that allow an assessment of heterogeneity is a substantial consideration. Duffield et al. (2008b) reported significant effects of monensin on decreasing DMI and milk fat percentage in lactating dairy cows. The analyses also illustrated the extremely consistent effect of monensin on DMI, yet a very variable effect on milk fat percentage, which led to further exploration of sources of that heterogeneity, including dietary factors. Both St-Pierre (2001) and Sauvant et al. (2008) examine the methods and effects of weighting studies. An important consideration for meta-analysis of nutrition studies is the use of crossover or Latin square designs. These studies pose particular challenges in animal health or production. Concerns with these studies include the potential for carry-over of effects from one period to another, suitability of the condition to study concerning follow-up periods or stage of lactation, a potential for exclusion bias, and effects of stage of lactation. The carry-over effect in a Latin square design is illustrated in a study of feeding fish oil and monensin to lactating dairy cattle (Cant et al., 1997). In this investigation, a 3-wk period for washout/adaptation was used. However, there was clearly carry-over from the fish oil treatment to the monensin group because 2 of 4 cows fed the monensin only treatment had detectable docosahexaenoic acid in the milk fatty acid profile. After checking the treatment order, we confirmed that the 2 cows had previously been fed fish oil before receiving the monensin only treatment. We [C. J. Sniffen (Fencrest LLC, Holderness, NH), M. B. de Ondarza (Paradox Nutrition LLC, West Chazy, NY), and I. J. Lean; unpublished data] recently identified very substantial differences in weighting of Latin square and randomized controlled trials examining the effects of diet on milk protein production of dairy cattle. This difference reflected the much smaller variance in milk protein production of cows in Latin square studies. A more conservative weighting based on the square root of the number of animals in studies may be the most appropriate way to weight studies that include Latin square designs. However, serious consideration should be given to the appropriateness of including Latin square designs in pooled analyses.

	HETEROGENEITY

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

 
Arguably, the greatest benefit of conducting meta-analysis is to examine sources of heterogeneity, if present, among studies. If heterogeneity is present, the summary measure must be interpreted with caution (Greenland, 1987). When heterogeneity is present, one should question whether and how to generalize the results. Understanding sources of heterogeneity will lead to more effective targeting of prevention and treatment strategies and will result in new research topics being identified. Part of the strategy in conducting a meta-analysis is to identify factors that may be significant determinants of subpopulation analysis or covariates that may be appropriate to explore in all studies.

To understand the nature of variability in studies, it is important to distinguish between different sources of heterogeneity. Variability in the participants, interventions, and outcomes studied has been described as clinical diversity (Cochrane Collaboration, 2008), and variability in study design and risk of bias has been described as methodological diversity (Cochrane Collaboration, 2008). Variability in the intervention effects being evaluated among the different studies is known as statistical heterogeneity and is a consequence of clinical or methodological diversity, or both, among the studies. Statistical heterogeneity manifests itself in the observed intervention effects varying by more than the differences expected among studies that would be attributable to random error alone. Usually, in the literature, statistical heterogeneity is simply referred to as heterogeneity.

Clinical variation will cause heterogeneity if the intervention effect is modified by the factors that vary across studies; most obviously, the specific interventions or participant (animal) characteristics that are often reflected in different levels of risk in the control group when the outcome is dichotomous. In other words, the true intervention effect will differ for different studies. Differences between studies in terms of methods used, such as use of blinding or differences between studies in the definition or measurement of outcomes, may lead to differences in observed effects. Significant statistical heterogeneity arising from differences in methods used or differences in outcome assessments suggests that the studies are not all estimating the same effect, but does not necessarily suggest that the true intervention effect varies. In particular, heterogeneity associated solely with methodological diversity indicates that studies suffer from different degrees of bias. Empirical evidence suggests that some aspects of design can affect the result of clinical trials, although this may not always be the case.

The scope of a meta-analysis will largely determine the extent to which studies included in a review are diverse. Meta-analysis should be conducted when a group of studies is sufficiently homogeneous in terms of animals involved, interventions, and outcomes to provide a meaningful summary. However, it is often appropriate to take a broader perspective in a meta-analysis than in a single clinical trial. Combining studies that differ substantially in design and other factors can yield a meaningless summary result, but the evaluation of reasons for the heterogeneity among studies can be very insightful. It may be argued that these studies are of intrinsic interest on their own, even though it is not appropriate to produce a single summary estimate of effect.

Statistical Assessment of Heterogeneity
Variation among the trial level ES or risk ratios are usually assessed using Cochran’s Q statistic, a chi-squared (²) test of heterogeneity. The null hypothesis is that the effect of treatment was the same across k trials, and the null hypothesis is rejected if the heterogeneity test statistic was greater than a critical value that separated the upper 10% of a ² distribution with (k – 1) df. This test has relatively poor power to detect heterogeneity among small numbers of trials (Egger and Smith, 2003); consequently, an level of 0.10 is used to test hypotheses.

Heterogeneity of results among trials is better quantified using the I² statistic (Higgins et al., 2003), which describes the percentage of total variation across studies that is due to heterogeneity rather than chance. Where Q is the ² heterogeneity statistic and k is the number of trials, I² is calculated as

Uncertainty intervals for I² (dependent on Q and k) are calculated using the method described by Higgins and Thompson (2002). Negative values of I² are put equal to zero, consequently I² lies between 0 and 100%. A value >50% may be considered substantial heterogeneity (Higgins and Thompson, 2002). This statistic is less influenced by the number of trials compared with other methods used to estimate the heterogeneity and provides a logical and readily interpretable metric.

Approaches to Heterogeneity
Given that there are several potential sources of heterogeneity in the data, several steps should be considered in the investigation of the causes. Although random-effects models, either frequentist or Bayesian, are appropriate, we consider that it is still very desirable to examine the data to identify sources of heterogeneity and to take steps to produce models that have a lower level of heterogeneity, if appropriate. Further, if the studies examined are highly heterogeneous, we consider that it is not appropriate to present an overall summary estimate, even when random effects models are used. As Petiti (1994) notes, statistical analysis alone will not make contradictory studies agree; critically, however, one should use common sense in decision-making. Despite heterogeneity in responses, if all studies had a positive point direction and the pooled confidence interval did not include zero, it would not be logical to conclude that there was not a positive effect, provided that sufficient studies and subject numbers were present. The appropriateness of the point estimate of the effect is much more in question.

L’Abbé plots (L’Abbé et al., 1987) are used to examine responses in dichotomous outcomes. The L’Abbé plots are used to show the variation in observed results by plotting the event probability in the treatment group on the vertical axis and that in the control group on the horizontal axis. The visual presentation is easy to assimilate, because the source of variability in response can be from deviation from the expected results, as represented by the null result straight line, by the control group, the treated group, or both. The information provided by these deviations can be used to identify sources of heterogeneity. For instance, if a L’Abbé plot reveals that there is a great variability in outcome from control groups but a similar outcome in intervention groups, heterogeneity in the treatment effect across individual studies may be at least partially explained by differences in the control populations. In this case, it may be more fruitful to investigate differences in control groups.

Galbraith plots (Galbraith, 1988) provide a graphical display to obtain a visual impression of the amount of heterogeneity from a meta-analysis. For each trial, the z statistic is plotted against the reciprocal standard error. The unweighted regression line constrained through the origin, with its 95% confidence interval, has a slope equal to the overall log risk ratio, or log odds ratio, or log hazard ratio in a fixed effects meta-analysis. The position of each trial on the horizontal axis gives an indication of the weight allocated to it in a meta-analysis. The position on the vertical axis gives the contribution of each trial to the Q statistic for heterogeneity. In the absence of heterogeneity, we could expect all the points to lie within the confidence bounds (positioned 2 units over and below the regression line). Figure 4 from Rabiee et al. (2005) shows the heterogeneity in Ovsynch and prostaglandin F₂ data using a Galbraith plot (I² = 73.3%, P < 0.0001). Clearly, many of these studies were heterogeneous. Identifying characteristics of these studies is critical to the investigation of sources of heterogeneity.

View larger version (9K): [in this window] [in a new window]   Figure 4. Galbraith plot for assessing heterogeneity in Ovsynch and prostaglandin F2 comparison data in lactating dairy cows (developed using the data from Rabiee et al., 2005).

Stratified analyses can also be used to reduce heterogeneity. Morgan and Lean (1993) used this approach to reduce, but not eliminate, heterogeneity in the GnRH treatment at insemination data. In this case, the dose of GnRH and insemination number influenced treatment responses. Burton and Lean (1995) found that the weighted average reduction in days open between treated and control cows was 2.6 d for trials with abnormal cows and 3.3 d for trials including normal and abnormal cows for studies examining the use injections of prostaglandin F₂ after calving.

Sensitivity analyses have also been used to examine the effects of studies identified as being aberrant concerning conduct or result, or being highly influential in the analysis. Rabiee et al. (2004, 2005) used this method in exploring sources of heterogeneity in the controlled internal drug-releasing insert (CIDR) and Ovsynch data. Baker and Jackson (2008) have proposed a method that reduces the weight of studies that are outliers in meta-analyses. All of these methods for examining heterogeneity have merit, and the variety of methods available reflects the importance of this activity.

	PUBLICATION BIAS

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

 
Although the intent of a meta-analysis is to find and assess all studies meeting the inclusion criteria, it is not always possible to obtain these. A critical concern is the papers that may have been missed. There is good reason to be concerned about this potential loss because studies with significant, positive results are more likely to be published and, in the case of interventions with a commercial value, to be promoted, than studies with nonsignificant or "negative" results. Studies that produce a positive result, especially large studies, are more likely to have been published and, conversely, there has been a reluctance to publish small studies that have nonsignificant results. Further, publication bias is not solely the responsibility of editorial policy as there is reluctance among researchers to publish results that were either uninteresting or are not randomized (Dickersin and Berlin, 1992). There are, however, problems with simply including all studies that have failed to meet peer-review standards. All methods of retrospectively dealing with bias in studies are imperfect.

Managing Publication Bias
It is important to examine the results of each meta-analysis for evidence of publication bias. An estimation of likely size of the publication bias in the review and an approach to dealing with the bias are inherent to the conduct of many meta-analyses. Several methods have been developed to provide an assessment of publication bias; the most commonly used is the funnel plot. The funnel plot was developed by Light and Pillemer (1984) and provides a graphical evaluation of the potential for bias. If publication bias is not present, the funnel plot is expected to be symmetrical, as shown in Figure 5. In this example from Duffield et al. (2008a), there is a symmetrical pattern of distribution around the point estimate, indicated by the point of the funnel on the y-axis. In a study in which there is little publication bias, larger studies (as indicated by the size of the circles in the plot) tend to cluster closely to the point estimate. As studies become less precise (i.e., have a higher standard error), the results of the studies can be expected to be more variable and are scattered to both sides of the more precise larger studies. Figure 5 shows that the smaller, less precise studies are, indeed, scattered to both sides of the point estimate of effect and that these seem to be symmetrical showing little evidence of publication bias.

View larger version (11K): [in this window] [in a new window]   Figure 5. Funnel plot of blood glucose data in lactating dairy cows supplemented with monensin. x-axis is the standard error (SE) of standardized mean difference (SMD), and y-axis is the standardized mean difference (SMD) (data from Duffield et al., 2008a).

View larger version (11K): [in this window] [in a new window]   Figure 6. Funnel plot of blood β-hydroxybutyrate data in lactating dairy cows supplemented with monensin. x-axis is the standard error (SE) of standardized mean difference (SMD), and y-axis is the standardized mean difference (SMD) (data from Duffield et al., 2008a).

The adjusted rank correlation (Begg’s method) is used to assess the correlation between estimates and their variances. The deviation of Spearman’s rho () values from zero provides an estimate of funnel plot asymmetry. Positive values indicate a trend toward higher effect estimates in studies with smaller sample size (Begg and Mazumdar, 1994). The Egger’s test is used to determine whether there is an association between study results and their precision. A regression analysis is conducted using the following equation:

where SND is the standard normal deviate defined as diagnostic d divided by its standard error SE(d), a is the intercept, and b is the slope. The intercept value (a) provides an estimate of asymmetry of funnel plot, with positive values (a > 0) indicating a trend toward higher levels of test accuracy in studies with smaller sample size (Egger et al., 1997).

Contour-Enhanced Funnel Plots.
Contour-enhanced funnel plots have been proposed by Peters et al. (2008) to include contour lines corresponding to statistical significance (P = 0.01, 0.05, 0.1). This approach allows the statistical significance of study estimates and areas in which studies are perceived to be missing to be considered. The contour-enhanced funnel plots may help to differentiate asymmetry caused by publication bias from that due to other factors. For example, if studies appear to be missing in areas of statistical nonsignificance, then this adds credence to the possibility that the asymmetry is caused by publication bias. Conversely, if the supposed missing studies are in areas of higher statistical significance, this suggests that the observed asymmetry may be more likely to be due to factors other than publication bias; for example, variable study quality or a failure to publish findings that were not statistically significant (Figure 7). In Figure 7, the 2 studies that show very positive effects that lie to the right of the 0.01 line represent strong positive responses that are unlikely to be matched by negative studies. If there are no statistically significant studies, then publication bias may not be a plausible explanation for funnel plot asymmetry (Ioannidis and Trikalinos, 2007).

View larger version (20K): [in this window] [in a new window]   Figure 7. Contour-enhanced funnel plot of pregnancy rate in cycling and repeat breeder dairy cows injected with GnRH at the time of insemination (data from Morgan and Lean, 1993).

Trim and Fill.
A rank-based data augmentation technique, the "trim and fill" method can be used to estimate the number of missing studies and to produce an adjusted estimate of test accuracy by imputing suspected missing studies (Duval and Tweedie, 2000). Smaller studies are omitted until the funnel plot is symmetrical (trimming). The trimmed funnel plot is used to estimate the true center of the funnel, and then the omitted studies and their missing counterparts around the center are replaced (filling). This provides an estimate of the number of missing studies and adjusted treatment effect, including the filled studies. Sanchez et al. (2004) used a trim and fill method to explore the potential for missing data to influence the effects of anthelmintic treatments on milk production.

	INFLUENTIAL STUDY ANALYSIS

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

 
Influential studies are of interest in statistical analysis. Large, well-conducted studies are important to identify, as are studies with a profoundly influential effect on the point estimate. In the case of a large study that has been well conducted but differs in estimate of effect to smaller studies, the key concern is whether the large study provides a stronger evidential base than the other studies conducted. In the case of the small study with strong effect, it is often illuminating to examine the study for differences in study design or analysis that may have influenced the outcome (Tobias, 1999). The following influential study analyses from Duffield et al. (2008a,c) show a series of studies in which very few studies were influential and one in which there is clear evidence of a study with a strong effect (Figures 8 and 9).

View larger version (49K): [in this window] [in a new window]   Figure 8. Influence analysis for monensin effect (SMD) on 3-hydroxybutyrate (Duffield et al., 2008a). A full list of the studies referenced here is available online (http://jds.fass.org/content/vol92/issue8/).

View larger version (24K): [in this window] [in a new window]   Figure 9. Unpublished figure illustrating the influence of a single study on the impact of monensin on the relative risk (RR) of dystocia (Duffield et al., 2008c). The influential study is Melendez et al. (2006). The overall RR of dystocia with the study included is 1.39, whereas it is 1.07 with the study removed. A full list of the studies referenced here is available online (http://jds.fass.org/content/vol92/issue8/).

The studies of Gross et al. (1999) and Sanchez et al. (2004), who studied the effects of anthelmintic treatments on milk production, agreed in both significance and direction. Gross et al. (1999) estimated a 0.63 kg of milk per cow per day response to anthelmintic treatment, whereas Sanchez et al. (2004) found an adjusted estimate of 0.35 kg of milk per cow per day response to treatment.

Dietary and other risk factors for milk fever have been examined in 4 meta-analyses providing multivariable models and one model providing a univariable estimate (Charbonneau et al., 2006). Two of the studies used identical data, and Oetzel (1991), Enevoldsen (1993), and Lean et al. (2006) developed 2 models of similar statistical fit using an expanded milk fever data set including the original data from Oetzel (1991). Oetzel (1991) found a significant univariable relationship between DCAD and risk of milk fever, as did Charbonneau et al. (2006), albeit with a slightly different definition for DCAD. The multivariable model of Lean et al. (2006) supported that finding. The consistency of estimate of effect for variables retained within the multivariable models in Table 2 is reasonably high, despite a finding that both magnesium and exposure acted to confound the coefficients for calcium in the models developed by Lean et al. (2006). Estimates of increased risk of milk fever with CP and lactation number from the Oetzel (1991) data are consistent with unpublished models that included these factors investigated by Lean et al. (2006), but are not consistent with the effect estimated by Enevoldsen (1993). Despite the latter finding, there is strong evidence for repeatable and consistent findings from these studies. Adherence to protocols such as those provided by the Cochrane Collaboration should increase the repeatability of results (Cochrane Collaboration, 2008).

	CONCLUSIONS

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

Results of studies to date have demonstrated the merit of this approach. Traditional review methods recommended that calcium concentrations be maintained at 1.0 to 1.2% of dietary intake before calving (Oetzel, 2000). These concentrations appear to increase milk fever risk, whereas meta-analyses (Oetzel, 1991; Lean et al., 2006) found a quadratic risk with low (<0.6% of DMI) and possibly high concentrations (>1.5% of DMI) reducing risk of milk fever. The use of GnRH at insemination was considered ineffective (Wright and Malmo, 1992), whereas meta-analysis found significant positive effects (Morgan and Lean, 1993). The effect of treatment of cattle with anthelmintics on milk production was considered in need of clarification (Vercruysse and Claerebout, 2001); the very consistent meta-analyses of Gross et al. (1999) and Sanchez et al. (2004) provide evidence of a positive response to these treatments and factors that influence the magnitude of the response. The meta-analysis of the health effects of rbST in dairy cattle was influential in the Canadian decision to deny registration of the product Dohoo et al. (2003b). Duffield et al. (2008a,b,c) identified several new understandings regarding use of monensin to modify rumen function.

No single study, whether meta-analytic or not, will provide the definitive understanding of responses to treatment, diagnostic tests, or risk factors influencing disease. Despite this limitation, meta-analytic approaches have demonstrable benefits in addressing the limitations of study size, can include diverse populations, provide the opportunity to evaluate new hypotheses, and are more valuable than any single study contributing to the analysis. The conduct of the studies is critical to the value of a meta-analysis and the methods used need to be as rigorous as any other study conducted. Methods and insights provided in this paper should assist those seeking to evaluate or conduct meta-analyses.

Received for publication February 17, 2009. Accepted for publication April 19, 2009.

	REFERENCES

TOP ABSTRACT INTRODUCTION GENERAL CONSIDERATIONS IN META... METHODS HETEROGENEITY PUBLICATION BIAS INFLUENTIAL STUDY ANALYSIS CONCLUSIONS REFERENCES

 


Antman, E. M., Lau, J., Kupelnick, B., Mosteller, F. and Chalmers, T. C.. 1992. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: Treatments for myocardial infarction. JAMA 268:240–248.[Abstract/Free Full Text]

Baker, R. and Jackson, D.. 2008. A new approach to outliers in meta-analysis. Health Care Manage. Sci. 11:121–131.[CrossRef]

Becker, B. J. 2005. Failsafe N or file-drawer number. Pages 111–125 in Publication Bias in Meta-Analysis. H. R. Rothstein, A. J. Sutton, and M. Borenstein, ed. John Wiley & Sons, Chichester, UK.

Begg, C. B. and Mazumdar, M.. 1994. Operating characteristics of a rank correlation test for publication bias. Biometrics 50:1088–1101.[CrossRef][Medline]

Bramley, E., Lean, I. J., Fulkerson, W. J., Stevenson, M. A., Rabiee, A. R. and Costa, N. D.. 2008. The definition of acidosis in dairy herds predominantly fed on pasture and concentrates. J. Dairy Sci. 91:308–321.[Abstract/Free Full Text]

Bryk, A. S., and S. W. Raudenbush. 1992. Hierarchical linear models. Sage Publications, Newbury Park, CA.

Burton, N. R. and Lean, I. J.. 1995. Investigation by meta-analysis of the effect of prostaglandin F2 administered post partum on the reproductive performance of dairy cattle. Vet. Rec. 136:90–94.[Abstract]

Cant, J. P., Fredeen, A. H., MacIntyre, T., Gunn, J. and Crowe, N.. 1997. Effect of fish oil and monensin on milk composition in dairy cows. Can. J. Anim. Sci. 77:125–131.

Chalmers, T. C., Sacks, H. S., Levin, H., Reitman, D. and Nagalingam, R.. 1987. Meta-analysis of clinical trials as a scientific discipline: II replicate variability and comparison of studies that agree and disagree. Stat. Med. 6:733–744.[Medline]

Chan, A. W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C. and Altman, D. G.. 2004. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 291:2457–2465.[Abstract/Free Full Text]

Charbonneau, E., Pellerin, D. and Oetzel, G. R.. 2006. Impact of lowering dietary cation-anion difference in nonlactating dairy cows: A meta-analysis. J. Dairy Sci. 89:537–548.[Abstract/Free Full Text]

Cochrane Collaboration. 2008. Cochrane Handbook for Systematic Reviews of Interventions. http://www.cochrane-handbook.org/

Cohen, J. 1969. Statistical Power Analysis for the Behavioral Sciences. Academic Press, New York, NY.

Congdon, P. 2001. Bayesian Statistical Modelling. John Wiley & Sons, London, UK.

Curtis, C. R., Erb, H. N., Sniffen, C. J. and Smith, R. D.. 1984. Epidemiology of parturient paresis: Predisposing factors with emphasis on dry cow feeding and management. J. Dairy Sci. 67:817–825.[Abstract/Free Full Text]

Curtis, M. A. 1997. Uterine infections in dairy cows. PhD Thesis. University of Sydney, Australia.

Davis, J. P., Chesney, P. J., Wand, P. J., La, M. and Venture, M.. 1980. Toxic-shock syndrome: Epidemiologic features, recurrence, risk factors, and prevention. N. Engl. J. Med. 303:1429–1435.[Abstract]

Degaris, P. J., Lean, I. J., Rabiee, A. R. and Heuer, C.. 2008. Effects of increasing days of exposure to prepartum transition diets on milk production and milk composition in dairy cows. Aust. Vet. J. 86:341–351.[CrossRef][Medline]

DerSimonian, R. and Laird, N.. 1986. Meta-analysis in clinical trials. Control. Clin. Trials 7:177–188.[CrossRef][Medline]

Dickersin, K. and Berlin, J. A.. 1992. Meta-analysis: State of the science. Epidemiol. Rev. 14:154–176.[Free Full Text]

Dohoo, I. R., DesCôteaux, L., Leslie, K., Fredeen, A., Shewfelt, W., Preston, A. and Dowling, P.. 2003b. A meta-analysis review of the effects of recombinant bovine somatotropin. 2. Effects on animal health, reproductive performance, and culling. Can. J. Vet. Res. 67:252–264.[Medline]

Dohoo, I. R., Leslie, K., DesCôteaux, L., Fredeen, A., Dowling, P., Preston, A. and Shewfelt, W.. 2003a. A meta-analysis review of the effects of recombinant bovine somatotropin. 1. Methodology and effects on production. Can. J. Vet. Res. 67:241–251.[Medline]

Dohoo, I. R., W. Martin, and H. Stryhn. 2003. Introduction to observational studies. Pages 139–150 in Veterinary Epidemiologic Research. AVC Inc., Charlottetown, Prince Edward Island, Canada.

Duffield, T. F., Lissemore, K. D., McBride, B. W. and Leslie, K. E.. 2009. Impact of hyperketonemia in early lactation dairy cows on health and production. J. Dairy Sci. 92:571–580.[Abstract/Free Full Text]

Duffield, T. F., Rabiee, A. R. and Lean, I. J.. 2008a. A meta-analysis of the impact of monensin in lactating dairy cattle. Part 1. Metabolic effects. J. Dairy Sci. 91:1334–1346.[Abstract/Free Full Text]

Duffield, T. F., Rabiee, A. R. and Lean, I. J.. 2008b. A meta-analysis of the impact of monensin in lactating dairy cattle. Part 2. Production effects. J. Dairy Sci. 91:1347–1360.[Abstract/Free Full Text]

Duffield, T. F., Rabiee, A. R. and Lean, I. J.. 2008c. A meta-analysis of the impact of monensin in lactating dairy cattle. Part 3. Health and reproduction. J. Dairy Sci. 91:2328–2341.[Abstract/Free Full Text]

Duval, S. and Tweedie, R.. 2000. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56:455–463.[CrossRef][Medline]

Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J. P. A., Simes, J. and Williamson, P. R.. 2008. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 3:e3081.[CrossRef][Medline]

Egger, M., and G. D. Smith. 2001. Principles of and procedures for systematic reviews. Pages 23–42 in Systematic Reviews in Health Care: Meta-Analysis in Context. M. Egger, G. Davey Smith, and D. G. Altman, ed. British Medical Journal Books, London, UK.

Egger, M., Davey Smith, G., Schneider, M. and Minder, C.. 1997. Bias in meta-analysis detected by a simple, graphical test. BMJ 315:629–634.[Abstract/Free Full Text]

Enevoldsen, C. 1993. Nutritional risk factors for milk fever in dairy cattle: Meta-analysis revisited. Acta Vet. Scand. 89(Suppl.):131–134.

Evans, A. S. 1976. Causation and disease. The Henle-Koch postulates revisited. Yale J. Biol. Med. 49:175–195.[Medline]

Fourichon, C., Seegers, H. and Malher, X.. 2000. Effect of disease on reproduction in the dairy cow: A meta-analysis. Theriogenology 53:1729–1759.[CrossRef][Medline]

Galbraith, R. F. 1988. A note on graphical presentation of estimated odds ratios from several clinical trials. Stat. Med. 7:889–894.[Medline]

Glass, G. V. 1976. Primary, secondary and meta-analysis of research. Educ. Res. 5:3–8.

Glass, G. V., B. McGaw, and M. L. Smith. 1981. Meta-analysis in social research. Sage Publications, Newbury Park, CA.

Goldstein, H. 1995. Multilevel statistical analysis. Edward Arnold, London, UK.

Greenland, S. 1987. Quantitative methods in the review of epidemiologic literature. Epidemiol. Rev. 9:1–30.[Medline]

Greenland, S. 1994a. Invited commentary: A critical look at some popular meta-analytic methods. Am. J. Epidemiol. 140:290–296.[Abstract/Free Full Text]

Greenland, S. 1994b. Invited commentary: A critical look at some popular meta-analytic methods. Am. J. Epidemiol. 140:300–301.[Free Full Text]

Gross, S. J., Ryan, W. G. and Ploeger, H. W.. 1999. Anthelmintic treatment of dairy cows and its effect on milk production. Vet. Rec. 144:581–587.[Abstract/Free Full Text]

Hedges, L. V. 1982. Estimation of effect size from a series of independent experiments. Psychol. Bull. 92:490–499.[CrossRef]

Hedges, L. V., and I. Olkin. 1985. Statistical Methods for Meta Analysis. 1st ed. Academic Press Inc., Orlando, FL.

Higgins, J. P. T., Thompsom, S. G., Deeks, J. J. and Altman, D. G.. 2003. Measuring inconsistency in meta-analysis. BMJ 327:557–560.[Free Full Text]

Higgins, J. P. T. and Thompson, S. G.. 2002. Quantifying heterogeneity in a meta-analysis. Stat. Med. 21:1539–1558.[CrossRef][Medline]

Higgins, J. T., Whitehead, A., Turner, R. M., Omar, R. Z. and Thompson, S. G.. 2001. Meta-analysis of continuous outcome data from individual patients. Stat. Med. 20:2219–2241.[CrossRef][Medline]

Hox, J. J., and E. D. de Leeuw. 2002. Multilevel models for meta-analysis. Pages 139–155 in Multilevel Analysis: Techniques and Applications. J. Hox, ed. Lawrence Erlbaum Associates Inc., Publishers, Mahwah, NJ.

Hubbert, W. T., Dennis, S. M., Adams, W. M., Bierschwal, C. J., Biggers, J. D., Carroll, E. J., Dunne, H. W., Hutton, N. E., Kendrick, J. W., Kenney, R. M., Kirkham, W. W., Leipold, H. W., Lingard, D. R., McDonald, L. E., Miner, M. L., Mossman, H. W., Murdick, P. W., Oberst, F. H., Priester, W. A., Jr., Ramage, J. C., Simon, J., Smith, R. E., Swift, B. L., Wagner, W. C., Williams, E. I. and Zemjanis, R.. 1972. Recommendations for standardizing bovine reproductive terms. Cornell Vet. 62:216–237.[Medline]

Huntsberger, D. V., and P. Billingsley. 1987. Elements of statistical inference. Allyn and Bacon, Boston, MA.

Ingvartsen, K., Dewhurst, R. and Friggens, N.. 2003. On the relationship between lactational performance and health: Is it yield or metabolic imbalance that cause production diseases in dairy cattle? Livest. Prod. Sci. 83:277–308.[CrossRef]

Ioannidis, J. P. and Trikalinos, T. A.. 2007. The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey. Can. Med. Assoc. J. 176:1091–1096.[Abstract/Free Full Text]

Iyengar, S. and Greenhouse, J. B.. 1988. Selection problems and the file drawer problem. Stat. Sci. 3:109–135.[CrossRef]

Jüni, P., Altman, D. G. and Egger, M.. 2001. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 323:42–46.[Free Full Text]

Kalaian, H. A. and Raudenbush, S. W.. 1996. A multivariate mixed linear model for meta-analysis. Psychol. Methods 1:227–235.[CrossRef]

Kelsey, J. L., A. S. Whittemore, A. S. Evans, and W. D. Thompson. 1996. Methods in Observational Epidemiology. Oxford University Press, New York, NY.

Kelton, D. F., Lissemore, K. D. and Martin, R. E.. 1998. Recommendations for recording and calculating the incidence of selected clinical diseases of dairy cattle. J. Dairy Sci. 81:2502–2509.[Abstract]

L’Abbé, K. A., Detsky, A. S. and O'Rourke, K.. 1987. Meta-analysis in clinical research. Ann. Intern. Med. 107:224–233.[Abstract/Free Full Text]

Lean, I. J., Bruss, M. L., Troutt, H. F., Galland, J. C., Farver, T. B., Rostami, J., Holmberg, C. A. and Weaver, L. D.. 1994. Bovine ketosis and somatotropin: Risk factors for ketosis and effects of ketosis on health and production. Res. Vet. Sci. 57:200–209.[Medline]

Lean, I. J., DeGaris, P. J., McNeil, D. M. and Block, E.. 2006. Hypocalcemia in dairy cows: Meta-analysis and dietary cation anion difference theory revisited. J. Dairy Sci. 89:669–684.[Abstract/Free Full Text]

Lean, I. J., and L. K. Wade. 1997. Effects of monensin on metabolism, production, and health of dairy cattle. Pages 50–70 in Usefulness of Ionophores in Lactating Dairy Cattle. K. L. Leslie, ed. Univ. Guelph, Ontario, Canada.

Lean, I. J., Westwood, C. T. and Playford, M. C.. 2008. Livestock disease threats associated with intensification of pastoral dairy farming. N. Z. Vet. J. 56:261–269.[Medline]

Lee, W. L., Bausell, R. B. and Berman, B. M.. 2001. The growth of health-related meta-analyses published from 1980 to 2000. Eval. Health Prof. 24:327–335.[Abstract/Free Full Text]

Light, R. J., and D. B. Pillemer. 1984. Summing Up: The Science of Reviewing Research. Harvard University Press, Cambridge, MA.

Lindley, D. V. 1972. Bayesian Statistics, A Review. SIAM, Philadelphia, PA.

Melendez, P., Goff, J. P., Risco, C. A., Archbald, L. F., Littell, R. C. and Donovan, G. A.. 2006a. Effect of administration of a controlled release monensin capsule on incidence of calving-related disorders, fertility, and milk yield in dairy cows. Am. J. Vet. Res. 673:537–543.

Moate, P. J., Chalupa, W., Boston, R. C. and Lean, I. J.. 2008. Milk fatty acids II: Prediction of the production of individual fatty acids in bovine milk. J. Dairy Sci. 91:1175–1188.[Abstract/Free Full Text]

Morgan, W. F. and Lean, I. J.. 1993. Effects of gonadotropin-releasing hormone treatment in cattle: A meta-analysis of the effects on conception at the time of insemination. Aust. Vet. J. 70:205–209.[CrossRef][Medline]

Mullen, B., and N. Miller. 1991. Meta-Analysis. In Research Methods in Social Relations. 6th ed. C. M. Judd, E. R. Smith, and L.H. Kidder, ed. Harcourt Brace Jovanovich College Publishers, New York, NY.

Nakagawa, S. and Cuthill, I. C.. 2007. Effect size, confidence interval and statistical significance: A practical guide for biologists. Biol. Rev. Camb. Philos. Soc. 82:591–605.[Medline]

Oetzel, G. R. 1991. Meta-analysis of nutritional risk factors for milk fever in dairy cattle. J. Dairy Sci. 74:3900–3912.[Abstract]

Oetzel, G. R. 2000. Management of dry cows for the prevention of milk fever and other mineral disorders. Vet. Clin. North Am. Food Anim. Pract. 16:369–386.[Medline]

Olkin, I. 1994. Invited commentary: Re: "A critical look at some popular meta-analytic methods. Am. J. Epidemiol. 140:297–299.[Free Full Text]

Orwin, R. G. 1983. A fail-safe N for effect size in meta-analysis. J. Educ. Stat. 8:157–159.[CrossRef]

Oxman, A. D. and Guyatt, G. H.. 1993. The science of reviewing research. Ann. N. Y. Acad. Sci. 703:125–133.[Medline]

Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R. and Rushton, L.. 2008. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J. Clin. Epidemiol. 61:991–996.[CrossRef][Medline]

Petiti, D. B. 1994. Meta-analysis, decision analysis and cost-effectiveness analysis. Page 115 in Methods for Quantitative Synthesis in Medicine. Oxford University Press, New York, NY.

Petticrew, M. 2001. Systematic reviews from astronomy to zoology: Myths and misconceptions. BMJ 322:98–101.[Free Full Text]

Rabiee, A. R., Lean, I. J., Dorton, K. L., Engstrom, M. E. and Sanchez, W. K.. 2008. Effect of feeding Diamond V Yeast Culture^TM on milk production and dry matter intake in lactating dairy cows: A meta-analysis. J. Anim Sci. 86(E-Suppl. 2):589. (Abstr.)

Rabiee, A. R., Lean, I. J. and Stevenson, M. A.. 2004. A Bayesian meta-analysis of the effects of administering an intra-vaginal (CIDR) device in combination with other hormones on the reproductive performance of cycling, anoestrous and inseminated cows. N. Z. Vet. J. 52:384–393.[Medline]

Rabiee, A. R., Lean, I. J. and Stevenson, M. A.. 2005. Efficacy of ovsynch program on reproductive performance in dairy cattle: A meta-analysis. J. Dairy Sci. 88:2754–2770.[Abstract/Free Full Text]

Raudenbush, S. W. and Bryk, A. S.. 1985. Empirical Bayes meta-analysis. J. Educ. Stat. 10:75–98.[CrossRef]

Robert, A., Seegers, H. and Bareille, N.. 2006. Incidence of intramammary infections during the dry period without or with antibiotic treatment in dairy cows—A quantitative analysis of published data. Vet. Res. 73:25–48.

Rosenthal, R. 1979. The "file drawer problem" and tolerance for null results. Psychol. Bull. 86:638–641.[CrossRef]

Rothman, K. J. 1986. Modern Epidemiology. Little, Brown and Company, Boston, MA.

Sanchez, J., Dohoo, I. R., Carrier, J. and DesCoteaux, L.. 2004. A meta-analysis of the milk-production response after anthelmintic treatment in naturally infected adult dairy cows. Prev. Vet. Med. 63:237–256.[CrossRef][Medline]

Sargeant, J. M., Rajic, A., Read, S. and Ohlsson, A.. 2006. The process of systematic review and its application in agri-food public-health. Prev. Vet. Med. 75:141–151.[Medline]

Sauvant, D., Schmidely, P., Daudin, J. J. and St-Pierre, N. R.. 2008. Meta-analyses of experimental data in animal nutrition. Animal 2:1203–1214.

Sheldon, I. M., Lewis, G. S., LeBlanc, S. and Gilbert, R. O.. 2006. Defining postpartum uterine disease in cattle. Theriogenology 65:1516–1530.[CrossRef][Medline]

Simmonds, M. C., Higgins, J. P. T., Stewart, L. A., Tierney, J. F., Clarke, M. J. and Thompson, S. G.. 2005. Meta-analysis of individual patient data from randomized trials: A review of methods used in practice. Clin. Trials 2:209–217.[Abstract/Free Full Text]

Sinclair, L. C., and M. B. Bracken. 1992. Effective Care of the Newborn Infant. Oxford University Press, Oxford, UK.

Spiegelhalter, D. J., Myles, J. P., Jones, D. R. and Abrams, K. R.. 2000. Bayesian methods in health technology assessment: a review. Health Technol. Assess. 4:1–130.[Medline]

St-Pierre, N. R. 2001. Integrating quantitative findings from multiple studies using mixed model methodology. J. Dairy Sci. 84:741–755.[Abstract]

Stangl, D. K., and D. A. Berry. 2000. Meta-analysis in medicine and health policy. Stata Tech. Bull. 8:108–110. Marcel Dekker, New York, NY.

Stewart, L. A. and Clarke, M. J.. 1995. Practical methodology of meta-analyses (overviews) using updated individual patient data. Stat. Med. 14:2057–2079.[Medline]

Thompson, S. G., Turner, R. M. and Warn, D. E.. 2001. Multilevel models for meta-analysis, and their application to absolute risk differences. Stat. Med. Res. 10:375–392.[CrossRef]

Tobias, A. 1999. Assessing the influence of a single study in meta-analysis. Stata Help. StataCorp., College Station, TX.

Turner, R. M., Omar, R. Z., Yang, M., Goldstein, H. and Thompson, S. G.. 2000. A multilevel model framework for meta-analysis of clinical trials with binary outcomes. Stat. Med. 19:3417–3432.[CrossRef][Medline]

van den Noortgate, W. and Onghena, P.. 2003. Multilevel meta-analysis: A comparison with traditional meta-analytical procedures. Educ. Psychol. Measure. 63:765–790.

Vercruysse, J. and Claerebout, E.. 2001. Treatment vs. non-treatment of helminth infections in cattle: Defining the threshold. Vet. Parasitol. 98:195–214.[CrossRef][Medline]

Westwood, C. T., Lean, I. J. and Kellaway, R. C.. 1998a. Indications and implications for testing of milk urea in dairy cattle: A quantitative review. (1) Dietary protein sources and metabolism. N. Z. Vet. J. 46:87–96.[Medline]

Westwood, C. T., Lean, I. J. and Kellaway, R. C.. 1998b. Indications and implications for testing of milk urea in dairy cattle: A quantitative review. (2) Effect of dietary protein on reproductive performance. N. Z. Vet. J. 46:123–130.[Medline]

Whitehead, A. 2002. Meta-analysis of controlled clinical trials. John Wiley & Sons Ltd., Chichester, UK.

Wright, P. J. and Malmo, J.. 1992. Pharmacologic manipulation of fertility. Vet. Clin. North Am. Food Anim. Pract. 8:57–89.[Medline]

This Article Abstract Full Text (PDF) __Meta-Analysis Reference List Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Lean, I. J. Articles by Dohoo, I. R. Search for Related Content PubMed PubMed Citation Articles by Lean, I. J. Articles by Dohoo, I. R.