Systematic assessment is a fundamental across all of the sciences and accounts for enormous advances. Consider some familiar and unfamiliar examples in natural, biological, and social sciences.

This chapter focuses on practical issues about how to evaluate and present one’s results. This information can be used to complement tests of statistical significance and decision making in evaluating and presenting the data. The chapter begins with data evaluation. With all its objections, null hypothesis significance testing (NHST) still dominates and as such the researcher (and reader) ought to be skilled in the approach, mindful of its liabilities, and have an overflowing quiver of options to improve the yield from one’s research. In this chapter, we will discuss practical issues but not specific statistical tests and options and what and when to do them.

We previously discussed several considerations that are used to guide evaluation and selection of measures, as well as the use of multiple measures to assess constructs. There are several considerations that are used to guide evaluation and selection of measures, as well as multiple measures to assess constructs. The reason was the inherent limitation of any measure capturing all facets of a given construct. Also, there is a “method factor,” which means that the findings obtained by measuring a construct in a particular way (e.g., self-report) can be due to the content or construct of the measure as well as this method or way in which that construct was assessed. In most circumstances, we would like to know that the results are not restricted to one way of assessing the construct of interest.

We have now covered a variety of concepts, including threats to validity and various sources of bias that guide thinking when designing, executing, and evaluating research. All that will be critical to keep in mind as we move forward to elaborate research design issues. Yet we begin here with the first step of a research, namely, what will be studied? How and where does one get an idea for an actual study?

In many ways, methodology is all about interpretation of findings in a study. As scientists we engage in special methodological practices, so the results can be interpreted in one way rather than another. Thus, we want to interpret the findings by explaining how a particular variable of interest to us, rather than some other influence, artifact, or bias (e.g., pre-existing group differences, “chance”) is the basis for the results. Also, when the data are collected and analyzed, we want to explain the results in ways that are consistent with what we actually found. Often an investigator makes a little leap moving from the data analysis to the interpretation of what was found. The study then is revealed to be poorly designed. That is, in reading a report of the study, we say, “If this is what the investigator wished to conclude then this was not quite the right way to design the study.” Thus, data interpretation issues are squarely within the realm of methodology. Indeed, it is helpful before designing a study to know exactly what you would like to conclude if your theory or hypothesis is supported. The design is built around making it so that you can reach that conclusion.

We have discussed internal and external validity, which are fundamental to research. Two other types of validity, referred to as construct validity and data-evaluation validity, also must be addressed to draw valid inferences. All four types relate to the conclusions that can be reached about a study. Construct and data-evaluation validity are slightly more nuanced than are internal and external valid-ity. They are more likely to be neglected in the design of research in part because they are not handled by commonly used practices. For example, random assignment of subjects to various experimental and control conditions nicely handles a handful of internal validity threats (e.g., history, maturation, testing, and selection biases), and we are all generally aware of this.

By far the most common research designs within psychology compare groups of subjects who are exposed to different conditions that are controlled by the investigator. The general strategy can entail a variety of different arrangements depending on the groups included in the design, how assessment is planned, and when and to whom the experimental condition is presented. This chapter considers fundamentals of group designs and various options when the investigator manipulates or systematically varies conditions and controls the assignment of subjects to different conditions. We begin with discussing individuals who are to participate in the study, their assignment to groups, and specific arrangements that constitute the experimental manipulation.

When we discuss or consider empirical research, there is a specific methodological paradigm we have in mind. That paradigm or approach is within the positivist tradition and includes the whole package of concepts and practices (e.g., theory, hypothesis testing, operational definitions, careful control of the subject matter, isolation of the variables of interest, quantification of constructs, and statistical analyses).

We have distinguished ethical issues as the responsibilities of researches in relation to participants from scientific integrity as those responsibilities related to the standards and obligations in conducting and reporting research. These can be distinguished at the margins (e.g., ethical issues related to ensuring that informed consent is voluntary for each subject vs. integrity issues such as plagiarizing or fabricating data). Yet ethical issues and integrity overlap in core values (e.g., transparency, honesty) and in how one represents one’s work (e.g., as we describe the project to the participants and as we describe our work to other professionals and the scientific community).

The important concept of plausible rival hypothesis addresses those competing interpretations that might be posed to explain the findings of a particular study. Methodology helps rule out or at least make implausible competing interpretations. An experiment does not necessarily rule out all possible explanations. The extent to which it is successful in ruling out alternative explanations is a matter of degree. From a methodological standpoint, the better the design of an investigation, the more implausible it makes competing explanations of the results. There are a number of specific concepts that reflect many of the interpretations that can interfere with and explain the results of a study. The concepts are critical too as they serve as a methodological checklist so to speak. When planning a study or evaluating the results of a completed study, it is extremely useful to know the many concepts we cover and how they will be or were handled in the design of the study.