Factorial design: A powerful research tool

In this article, we’ll discuss factorial design experiments, a powerful methodology any researcher must know. Like any statistical research technique, it may sound a bit heavy. Therefore, I tried to simplify the contents in order to accurately explain what is factorial design (or factorial experiments) and how to use it.

What is factorial design?

Factorial design is a methodology from statistics sciences that we use extensively in the field of Cognitive Psychology and Behavioral Psychology. It’s also used in educational, forensic, health, ABA and other branches of psychology. However, Behaviorism and Cognitivism are paramount in UX research, which is the subject we’re going to discuss.

First thing first: you may find factorial design and factorial experiments as interchangeable terms. They’re not exactly the same, but this interchangeability is quite extended. So, if you see any of these terms, they’ll probably refer to the same thing.

As for the definition, we can say that…

In statistics, a factorial experiment is an experiment whose design consists of two or more factors, each with discrete possible values or “levels”, and whose experimental units take on all possible combinations of these levels across all such factors.

https://en.wikipedia.org/wiki/Factorial_experiment

How do we use factorial experiments in user research?

In research on the causative or controlling factors of human behavior, we recognize the fact that such factors are never affected by a single variable. Instead, they occur in conjunction with multiple factors, interacting in a complex way. For those working in user experience research, this is basic user psychology.

It is known that in specific experiments that contrast causal relationships between variables, the way in which the various factors are controlled or manipulated is not indistinct.

If we identify a factor as the only independent variable (“IV”, factor hypothesized as “causal”) and control for all other possible sources of variation on the dependent variable (“AV”, affected variable, or behavior of interest), we run the risk that the results obtained are only applicable to said configuration of variables.

This is the most common occurrence in those cases that behavior is affected in different ways by one variable depending on how the other variables are present.

An interaction between factors or IV’s occurs whenever the effect of a factor on AV depends in turn on the presence / absence or degree to which another factor is present.

Remember that in an experiment, the factor must be manipulated. That is, the experimenter must be able to make it adopt different values. For example, “presence-absence”, “much-little-nothing”, etc., in order to compare how the AV performs under each of these values.

For example, if the IV were the color of a button and the AV the selection rate of the same, the manipulated values ​​of the factor could be different colors, while the selection rate of the button should be registered under each of those values, thus establishing which of these colors produces a greater response.

Suppose that the place on the page where the button is positioned is also a factor that alters the selection behavior.

In this case, if we were to control this variable by leaving it constant (for example, on the upper right of the screen), the results obtained on the effect of color on the selection rate could be biased at that particular level of the “position” factor. Even without being applicable to other possible positions. You can see more information about this in the article about A / B and Multivariate testing

Because of this, most research on human behavior uses so-called “multivariate” (also known as MVT) or “factorial” designs. Thus, instead of controlling alternative factors to the one of interest, they are incorporated into the design as independent variables, being manipulated like the main factor. This way, these statistical designs have two or more independent variables, whose values ​​are combined with each other, in order to be able to empirically study both the isolated and joint effects of the various variables on behavior.

In the example mentioned above, a possible factorial design involving the factors “color” and “position” could be that the first adopts three values ​​(green, blue and red) and the second four values ​​of the position on the screen (upper -right, upper-left, lower-right, lower-left). Thus, the design would involve recording the button’s selection rate in the following combinations:

Why is factorial design so important in UX?

Factorial designs allow us to study user psychology by definig the following information :

• the effect of each factor separately on AV
• the presence or absence of interaction between them
• and finally, if there is an interaction, how we can describe said interaction.

Factorial designs can contain two, three or more manipulated factors, making the analysis and interpretation of the results more complex as the design grows.

They are usually formally represented by a formula that specifies the number of factors manipulated and the number of values ​​adopted by each one. In the case of the example mentioned, it is a 3 x 4 factorial design (two factors, one with 3 and the other with 4 values).

These designs can, according to the objectives and resources of the investigation, be independent measures, repeated measures, or repeated measures in some of their factors.

Independent measures is the design where as many groups of subjects as combinations of factor values ​​must be assembled. In the case of our example, 12 groups should be created, each performing a task only under a specific combination.

In the case of repeated measures, it would be a single group of subjects who should perform the task under all combinations of values.

Finally, in repeated measures design in any of its factors, all subjects perform the task under all levels of one of the factors, but are divided into groups to receive only one of the values ​​of the other factor.

In the example, it could be three different groups, one that does the task with a green button, another with a light blue button, and another with a red button. But the subjects of the three groups perform the task with the button presented in all four possible positions.

Finally, the data obtained in this type of design is usually treated through statistical tests called “Analysis of variance” (also known as ANOVA). These tests are specific for more than one factor, called two-way, three-way, etc., according to the amount of manipulated factors. We’ll cover this topic in a future article.

More Factorial Design Resources

If you want to know more about Factorial Design and how can you use it in your projects, we made a curated selection of resources and bibliography about Factorial Design.

The following are academic research papers, so you can be sure you’ll learn a lot about the subject:

A Modern Theory of Factorial Designs – R. Mukerjee, C.F. J. Wu

Factorial Design Considerations – S. Green, P. Liu, J. O’Sullivan

More Academic Papers by Mariana Arismendi, PhD

Mariana Arismendi, PhD

Mariana is a Doctor in Psychology, specialized in research methodologies.

She is a

• Professor of Methods of Scientific Research at the University of Buenos Aires
• Professor of Fundamental Processes at the Universidad del Salvador.
• Investigator at the National Commission for Science, Education and Technology (CONICET)
• Published Author
• Former researcher for government projects