SOWK 322: Social Welfare Policy & Program I : Evaluating Sources

Has this data been repackaged? If so, ask:

• Is there proper documentation to lead you to the primary source?
• Would it be useful to get more information from the primary source?
• Could there be anything missing from the secondary version?

Is this data current enough for my purposes? 

• When were the data collected?
  There is often a time lag between collection and reporting because of the time required to analyze the data.
• Are these the newest figures? 
  Sometimes the newest available figures are a few years old. Make sure you can verify that there isn't something newer.

To fully understand your data, what it can tell you, and how much it will strengthen your argument, look in the following places:

1. Data Documentation

   • Find the website of the institution that creates, disseminates, or hosts the data you are using

   • Look for documentation: User Guides, Codebook, Questionnaire or "survey instrument," Statistical overview

   • Look for Purpose / overview / background, Methodology, and Data quality

2. The Research Literature

   • Search for research articles that use the data you're researching

   • Citation chain - look at their references and find who cited them

3. Specialized Bibliographies

   • With large surveys, it is often the case that the web site where they are hosted will also contain a bibliography of research that is about or uses that data. Look for these on the project web sites for the data.

   • ICPSR maintains a Bibliography of Data Related Literature, which you can search from their web site.

   • Often, topical entries in subject encyclopedias will have sections on data. For example the entry for Recidivism in the Encyclopedia of Crime and Justice has a discussion of data commonly used to measure recidivism.

Note: some data will be much easier to research than others, and not all three suggestions above will apply to all datasets.

Content on this page was adapted from "Data, Datasets, and Statistical Resources" by Gould Library licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Source: Who or what is the source?

• Who is presenting the statistics?
  The data source and the reporter/citer/publisher are not always the same. When feasible, it is best to go to the original source (or at least know and evaluate the source).
• Can you trace the data or statistics back to the original source? If the source is unclear, be skeptical about the data/statistics.
• Who collected the data?
• What are the credentials of the original source?

Motive: What’s in it for them?  

• What was the purpose of the collection or study?  Why was the information collected? Who sponsored the production of these data?
• Who was the intended audience for or users of the data/statistics?
• Was it collected as part of the mission of an organization, for advocacy, or for business purposes?  
• Do the statistics/data from the reporter/citer/publisher match the statistics/data of the data source?
• Does the reporter/presenter/data source have a known bias?  Even a reputable source and collection method can introduce bias, and biased sources can be accurate, but you need to check them carefully.

 Authority: Who produced the data/statistics?

• What are the author's/data producer's credentials?
• How widely known or cited is the producer?
• Is the measure or producer contested?
• Who else uses the data?
• Is the producer an expert on the subject?  What are their credentials?
• If an individual, what organizations are they associated with?  Could that association affect the work?

 Review: Go over the collection methods & completeness carefully.  

• What methodology was used?  Is the methodology documented?   
• How are the data collected - count, measurement, or estimation? 
• What was the total sample size?  How does that compare to the size of the population it is supposed to represent?
• What methods were used to select the sample population?  How was the population sampled?  (Was it self-selection, random sample, etc)
• What populations were included and/or excluded?
• If a survey:
  • What was the response rate?
  • Can you read the survey questions used?
• Are there any notable errors in facts or conclusions?  
• Is the information current enough?  
  Remember that there is often a delay between data collection and data reporting because analysis and publication take time.
• Does it sound reasonable/plausible?

Content on this page was adapted from "Savvy Info Consumers: Data & Statistics" by University of Washington Libraries.