Experimental design is one aspect of a scientific method. A well-designed, properly conducted experiment aims to control variables in order to isolate and manipulate causal effects and thereby maximize internal validity, support causal inferences, and guarantee reliable results. Traditionally employed in the natural sciences, experimental design has become an important part of research in the social and behavioral sciences. Experimental methods are also endorsed as the most reliable guides to policy effectiveness. Through a discussion of (...) some of the central concepts associated with experimental design, including controlled variation and randomization, this chapter will provide a summary of key ethical issues that tend to arise in experimental contexts. In addition, by exploring assumptions about the nature of causation and by analyzing features of causal relationships, systems, and inferences in social contexts, this chapter will summarize the ways in which experimental design can undermine the integrity of not only social and behavioral research but policies implemented on the basis of such research. (shrink)

Some useful statistical tools and concepts; Plot or pen technique; The completely randomized design; Randomized complete block design; The latin square design; The choice of treatments and the factorial experiment; Other factorial experiments; Confounding in factorial experiments; Factorial experiments with main effects confounded- split plot and aplit block designs with vatiations; Incomplete block designs: general considerations and the one-restrictional lattices with treatments in complete replicates; Lattice designs with more one restriction on the allocation of treatments in the (...) complete block; Other incomplete block designs; Balanced designs; Some additional designs; Covariance. (shrink)

This, the third edition of Fundamentals of Experimental Design, has five added chapters - those on regression (Chapters 12, 14, and 15), multivariate analysis (Chapter 18), and the matrix algebra appropriate to the level of presentation of this material (Chapter 13). I have noted in the preface other additions in this third edition. The added material should enhance the value of the book as a textbook and a reference. Given these additions, however, alternative approaches in using the current (...) edition as a textbook may merit consideration. It may help to note that Chapters 16 and 17 (analysis of covariance, trend analysis) do not depend on the material in Chapters 12 through 15, although the student should know something about simple linear regression to be able to understand fully the material in Chapters 16 and 17. In any event, the instructor who wants to teach only the material in the first two editions can do so by dropping the added chapters - 12 through 15, and 18 - from the syllabus. (shrink)

I review three answers to the question: How can phenomenology contribute to the experimental cognitive neurosciences? The first approach, neurophenomenology, employs phenomenological method and training, and uses first-person reports not just as more data for analysis, but to generate descriptive categories that are intersubjectively and scientifically validated, and are then used to interpret results that correlate with objective measurements of behaviour and brain activity. A second approach, indirect phenomenology, is shown to be problematic in a number of ways. Indirect (...) phenomenology is generally put to work after the experiment, in critical or creative interpretations of the scientific evidence. Ultimately, however, proposals for the indirect use of phenomenology lead back to methodological questions about the direct use of phenomenology in experimental design. The third approach, 'front-loaded' phenomenology, suggests that the results of phenomenological investigations can be used in the design of empirical ones. Concepts or clarifications that have been worked out phenomenologically may operate as a partial framework for experimentation. (shrink)

Peter Spirtes, Clark Glymour, Richard Scheines, Christopher Meek, S. Fineberg, E. Slate. Prediction and Experimental Design with Graphical Causal Models.

Sections include: experiments and generalised causal inference; statistical conclusion validity and internal validity; construct validity and external validity; quasi-experimental designs that either lack a control group or lack pretest observations on the outcome; quasi-experimental designs that use both control groups and pretests; quasi-experiments: interrupted time-series designs; regresssion discontinuity designs; randomised experiments: rationale, designs, and conditions conducive to doing them; practical problems 1: ethics, participation recruitment and random assignment; practical problems 2: treatment implementation and attrition; generalised causal inference: a (...) grounded theory; generalised causal inference: methods for single studies; generalised causal inference: methods for multiple studies; a critical assessment of our assumptions. (shrink)

Computational modeling plays an important role in prediction and optimization of real systems and processes. Models usually have some parameters which should be set up to the proper value. Therefore, parameter estimation is known as an important part of the modeling and system identification. It usually refers to the process of using sampled data to estimate the optimum values of parameters. The accuracy of model can be increased by adjusting its parameters to the optimum value which need a richer dataset. (...) One simple solution for having a richer dataset is increasing the amount of data, but that can be costly and time consuming. When using data from animals or people, it is especially important to have a proper plan. There are several available methods for parameter estimation in dynamical systems; however there are some basic differences in chaotic systems due to their sensitivity to initial condition. Accordingly, in this paper, a new cost function which is proper for chaotic systems is applied to the chaotic one-dimensional map. Then the efficiency of a newly introduced intelligent method experimental design in extracting proper data is investigated. The results show the success of the proposed method. (shrink)

We applaud the authors' basic message. We note that the negative research emphasis is not special solely to social psychology and judgment and decision-making. We argue that the proposed integration of null hypothesis significance testing (NHST) and Bayesian analysis is promising but will ultimately succeed only if more attention is paid to proper experimental design and implementation.

Standard practices in experimental economics arise for different reasons. The “no deception” rule comes from a cost-benefit tradeoff; other practices have to do with the uses to which economists put experiments. Because experiments are part of scientific conversations that mostly go on within disciplines, differences in standard practices between disciplines are likely to persist.

This paper considers ways that experimental design can affect judgments about informally presented context shifting experiments. Reasons are given to think that judgments about informal context shifting experiments are affected by an exclusive reliance on binary truth value judgments and by experimenter bias. Exclusive reliance on binary truth value judgments may produce experimental artifacts by obscuring important differences of degree between the phenomena being investigated. Experimenter bias is an effect generated when, for example, experimenters disclose (even unconsciously) (...) their own beliefs about the outcome of an experiment. Eliminating experimenter bias from context shifting experiments makes it far less obvious what the “intuitive” responses to those experiments are. After it is shown how those different kinds of bias can affect judgments about informal context shifting experiments, those experiments are revised to control for those forms of bias. The upshot of these investigations is that participants in the contextualist debate who employ informal experiments should pay just as much attention to the design of their experiments as those who employ more formal experimental techniques if they want to avoid obscuring the phenomena they aim to uncover. (shrink)

To what extent does the design of statistical experiments, in particular sequential trials, affect their interpretation? Should postexperimental decisions depend on the observed data alone, or should they account for the used stopping rule? Bayesians and frequentists are apparently deadlocked in their controversy over these questions. To resolve the deadlock, I suggest a three‐part strategy that combines conceptual, methodological, and decision‐theoretic arguments. This approach maintains the pre‐experimental relevance of experimental design and stopping rules but vindicates their (...) evidential, postexperimental irrelevance. †To contact the author, please write to: Tilburg Center for Logic and Philosophy of Science, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands; e‐mail: [email protected]. (shrink)

In order to maximize human performance, defence forces continue to explore, develop, and apply human performance enhancement (HPE) methods, ranging from pharmaceuticals to (bio)technological enhancement. This raises ethical, legal, and societal concerns and requires organizing a careful reflection and deliberation process, with relevant stakeholders. We discuss a range of ethical, legal, and societal aspects (ELSA), which people involved in the development and deployment of HPE can use for such reflection and deliberation. A realistic military scenario with proposed HPE application can (...) serve as a starting point for such an iterative and participatory process. Stakeholders can discuss this application, modify its features, and design appropriate processes around it – for instance, procedures for informed consent. We propose that organizing aspects into these three categories – ethical, legal, and societal – can help involve appropriate interlocutors at different moments: legal aspects with people in strategy or management roles, from the start of a project; ethical aspects with people in operations and medical roles, during development; and societal aspects with people in communication and personnel roles, during deployment. Notably, we developed and discussed this framework and the three aspects in close collaboration with personnel from the military. (shrink)

Recently, there has been an increased interest in developing statistical methodologies for analyzing single case experimental design data to supplement visual analysis. Some of these are simulation-driven such as Bayesian methods because Bayesian methods can compensate for small sample sizes, which is a main challenge of SCEDs. Two simulation-driven approaches: Bayesian unknown change-point model and simulation modeling analysis were compared in the present study for three real datasets that exhibit “clear” immediacy, “unclear” immediacy, and delayed effects. Although SMA (...) estimates can be used to answer some aspects of functional relationship between the independent and the outcome variables, they cannot address immediacy or provide an effect size estimate that considers autocorrelation as required by the What Works Clearinghouse Standards. BUCP overcomes these drawbacks of SMA. In final analysis, it is recommended that both visual and statistical analyses be conducted for a thorough analysis of SCEDs. (shrink)

This argument map presents Paul Loewi’s crucial experiment in which he showed that neural transmissions of signals are chemical in nature, not electrical, in form of an argumentation. The map can be used in science education to show how the formulation of hypotheses should be related to a corresponding determination of experimental designs.

The Hooks et al. review of microbiota-gut-brain literature provides a constructive criticism of the general approaches encompassing MGB research. This commentary extends their review by: highlighting capabilities of advanced systems-biology “-omics” techniques for microbiome research and recommending that combining these high-resolution techniques with intervention-based experimental design may be the path forward for future MGB research.

Almaatouq et al. propose an “integrative approach” to increase the generalisability and commensurability of experiments. Yet their metascientific approach has one glaring omission (and misinterpretation of) – the role of sample diversity in generalisability. In this commentary, we challenge false notions of subsumed duality between contexts, population, and diversity, and propose modifications to their design space to accommodate sample diversity.

During the twentieth century statistical methods have transformed research in the experimental and social sciences. Qualitative evidence has largely been replaced by quantitative results and the tools of statistical inference have helped foster a new ideal of objectivity in scientific knowledge. The paper will investigate this transformation by considering the genesis of analysis of variance and experimental design, statistical methods nowadays taught in every elementary course of statistics for the experimental and social sciences. These methods were (...) developed by the mathematician and geneticist R. A. Fisher during the 1920s, while he was working at Rothamsted Experimental Station, where agricultural research was in turn reshaped by Fisher’s methods. Analysis of variance and experimental design required new practices and instruments in field and laboratory research, and imposed a redistribution of expertise among statisticians, experimental scientists and the farm staff. On the other hand the use of statistical methods in agricultural science called for a systematization of information management and made computing an activity integral to the experimental research done at Rothamsted, permanently integrating the statisticians’ tools and expertise into the station research programme. Fisher’s statistical methods did not remain confined within agricultural research and by the end of the 1950s they had come to stay in psychology, sociology, education, chemistry, medicine, engineering, economics, quality control, just to mention a few of the disciplines which adopted them. (shrink)

A “stopping rule” in a sequential experiment is a rule or procedure for deciding when that experiment should end. Accordingly, the “stopping rule principle” states that, in a sequential experiment, the evidential relationship between the final data and an hypothesis under consideration does not depend on the experiment’s stopping rule: the same data should yield the same evidence, regardless of which stopping rule was used. In this essay, I reconstruct and rebut five independent arguments for the SRP. Reminding oneself that (...) the stopping rule is a part of an experiment’s design and is no more mysterious than many other design aspects helps elucidate why some of these arguments for the SRP are unsound. (shrink)

Since modern medicine is based substantially in clinical medical research, the flaws and ethical problems that arise in this research as it is conceived and practiced in the United States are likely to be reflected to some extent in current medicine and its practice. This paper explores some of the ways in which clinical research has suffered from an androcentric focus in its choice and definition of problems studied, approaches and methods used in design and interpretation of experiments, and (...) theories and conclusions drawn from the research. Some examples of re-visioned research hint at solutions to the ethical dilemmas created by this biased focus; an increased number of feminists involved in clinical research may provide avenues for additional changes that would lead to improved health care for all. (shrink)

In this first of two companion papers to a logico-mathematic, structural methodology , a meta-level analysis of the non metric structure is presented in relation to critiques based on standard experimental, statistical, and computational methods of contemporary psychology and cognitive science. The concept of a non metric methodology is examined as it relates to the epistemological and scientific goals of experimental, statistical, and computational methods. While sharing in these goals, differences and similarities between the two methodological approaches are (...) outlined. It is argued that typical experimental methods are not sufficient to extract and validate semantic information in verbal narratives. It is further suggested that a logico-mathematic, structural methodology can yield invariant law-like cognitive processes by careful methodological control of the specific case — instead of those found by current methods that produce “laws” based on statistical frequency. Lastly, the issue of experimental manipulation in relation to the logico-mathematic, structural methodology is examined. (shrink)

In many areas of the social and behavioral sciences, the nature of the experiments and theories that best capture the underlying constructs are themselves areas of active inquiry. Integrative experiment design risks being prematurely exploitative, hindering exploration of experimental paradigms and of diverse theoretical accounts for target phenomena.

Can you think of a meme that made you laugh or a political post that made you angry? These examples illustrate how social media use (SMU) impacts how people feel. Similarly, how people feel when they initiate SMU may impact the emotional effects of SMU. Someone feeling happy may feel more positively during SMU, whereas someone feeling sad may feel more negatively. Using an experimental design, we examined whether following SMU, those in a happy mood would experience increases (...) in positive affect (PA) and those in a sad mood would experience increases in negative affect (NA). A large sample of college students (N = 703) were randomly assigned to a happy, sad, or neutral mood induction before SMU. PA and NA were assessed at baseline, post-mood induction, and after SMU. Contrary to hypotheses, after SMU, people in happy moods experienced decreases in PA, and those in sad moods experienced decreases in NA, reflecting SMU having a dampening effect on emotions. PA and NA were significantly lower after SMU compared to baseline and did not vary by condition. How young adults feel when they log onto SMU matters in understanding how SMU impacts PA and NA, but on average, emotional experiences are dampened. (shrink)

As etiologic concepts of diseases gradually changed from humoral to microbial in the 19th century, syphilis presented particularly great challenges to doctors and scientists. Was “syphilis” merely a synonym for “venereal disease,” and did all manifestations attributed to it have the same cause? The discovery in 1879 of the gonococcus by Albert L. Neisser , and of the cause of chancroid, or soft chancre, in 1890 by Augusto Ducrey , established that venereal disease and syphilis were not synonymous, but syphilis (...) continued to mystify investigators. Symptoms could disappear for years and then—in most but not all persons—reappear with entirely different “tertiary” manifestations. .. (shrink)

Almaatouq et al.'s prescription for more integrative experimental designs is welcome but does not address an equally important problem: Lack of adequate theories. We highlight two features theories ought to satisfy: “Well-specified” and “grounded.” We discuss the importance of these features, some positive exemplars, and the complementarity between the target article's prescriptions and improved theorizing.

Stroke continues to be a leading cause of disability. Basic neurorehabilitation research is necessary to inform the neuropathophysiology of impaired motor control, and to develop targeted interventions with potential to remediate disability post-stroke. Despite knowledge gained from basic research studies, the effectiveness of research-based interventions for reducing motor impairment has been no greater than standard of practice interventions. In this perspective, we offer suggestions for overcoming translational barriers integral to experimental design, to augment traditional protocols, and re-route the (...) rehabilitation trajectory toward recovery and away from compensation. First, we suggest that researchers consider modifying task practice schedules to focus on key aspects of movement quality, while minimizing the appearance of compensatory behaviors. Second, we suggest that researchers supplement primary outcome measures with secondary measures that capture emerging maladaptive compensations at other segments or joints. Third, we offer suggestions about how to maximize participant engagement, self-direction, and motivation, by embedding the task into a meaningful context, a strategy more likely to enable goal-action coupling, associated with improved neuro-motor control and learning. Finally, we remind the reader that motor impairment post-stroke is a multidimensional problem that involves central and peripheral sensorimotor systems, likely influenced by chronicity of stroke. Thus, stroke chronicity should be given special consideration for both participant recruitment and subsequent data analyses. We hope that future research endeavors will consider these suggestions in the design of the next generation of intervention studies in neurorehabilitation, to improve translation of research advances to improved participation and quality of life for stroke survivors. (shrink)

Context: The integration of data measured in first- and third-person frameworks is a challenge that becomes more prominent as we attempt to refine the ties between the dimensions we assume to be objective and our experience itself. As a result, cognitive science has been a target for criticism from the epistemological and methodological point of view, which has resulted in the emergence of new approaches. Neurophenomenology has been proposed as a means to address these limitations. The methodological application of this (...) discipline, even in its mildest form, enriches the methodology typically used in cognitive sciences. Problem: Nowadays psychological studies are difficult to replicate. As a way to achieve replication of results published in a previous study in order to develop a methodological adaptation suitable for electroencephalographic measurements in a subsequent experiment, first-person accounts from the participants in our pilot study were included in the experiment construction. This study’s objective is to show the benefit of including a mild-neurophenomenology-inspired approach in the adaptation from an original paradigm, which requires, foremost, the ability to replicate the original results. Method: Interviews with open and semi-structured questions were carried out at the end of an Approach-Avoidance Task (AAT. The first-person reports, together with the behavioral outcomes of each pilot, were taken into account for the development of the next piloting phase until replication of the original results was achieved, and the final experimental design was elaborated. Results: A sequence of four pilots, where the integration of third- and first-person information derived from subjects’ behavior and reported experiences while carrying them out rendered the behavioral replication we sought to achieve, providing support for a first-person enriched cognitive science paradigm. Implications: Including first-person accounts systematically during the development and performance of classic cognitive paradigms ensures that those paradigms are measuring what they claim to measure. This is the next logical step to improve replication rates, to refine the explanation of the results and avoid confounding third-person data interpretation. Constructivist content: Including first-person experiences and acknowledging the active role that participants’ experiences regarding the paradigm had in the modeling of its final version is in concordance with a constructivist standing. (shrink)

Recently, there has been increasing interest in the use of heavily restricted randomization designs which enforce balance on observed covariates in randomized controlled trials. However, when restrictions are strict, there is a risk that the treatment effect estimator will have a very high mean squared error. In this article, we formalize this risk and propose a novel combinatoric-based approach to describe and address this issue. First, we validate our new approach by re-proving some known properties of complete randomization and restricted (...) randomization. Second, we propose a novel diagnostic measure for restricted designs that only use the information embedded in the combinatorics of the design. Third, we show that the variance of the MSE of the difference-in-means estimator in a randomized experiment is a linear function of this diagnostic measure. Finally, we identify situations in which restricted designs can lead to an increased risk of getting a high MSE and discuss how our diagnostic measure can be used to detect such designs. Our results have implications for any restricted randomization design and can be used to evaluate the trade-off between enforcing balance on observed covariates and avoiding too restrictive designs. (shrink)

Methodological practices differ between economics and psychology because economists use game theory as the basis for the design and interpretation of experiments, while psychologists do not. This methodological choice explains the “four key variables” stressed by Hert-wig and Ortmann. Game theory is currently the most rigorous basis for modeling strategic choice.

PurposeThe purpose of this study was to introduce an orthogonal experimental design to improve the efficiency of building and optimizing models for freezing of gait prediction.MethodsA random forest model was developed to predict FOG by using acceleration signals and angular velocity signals to recognize possible precursor signs of FOG. An OED was introduced to optimize the feature extraction parameters.ResultsThe main effects and interaction among the feature extraction hyperparameters were analyzed. The false-positive rate, hit rate, and mean prediction time (...) were 27%, 68%, and 2.99 s, respectively.ConclusionThe OED was an effective method for analyzing the main effects and interactions among the feature extraction parameters. It was also beneficial for optimizing the feature extraction parameters of the FOG prediction model. (shrink)