Factorial vignette surveys are a powerful way of combining traditional survey methods with experimental methods. However, the vignettes can be somewhat time-consuming to put together, particularly if you are studying multiple factors or if your factors have many dimensions.

I use the Qualtrics platform to run factorial surveys from time to time. Here is a general set of instructions, based on the guides put together by Luzia Helfer, Leiden University & Marjolijn De Wilde, at the University of Antwerp for how to put together a factorial vignette survey in Qualtrics using some time-saving Python code I’ve put together.

1) Download my Python script from GitHub. This script will generate a Qualtrics basic TXT-formatted survey file.

2) Alter the python code to fit your particular survey needs and story details. The code is setup by default to generate all 360 possible vignettes for a 5-factor survey (a [3 * 4 * 2 * 5 * 3] arrangement). You can quite easily change the number of factors or the number of dimensions within factors by altering the code. Suggestions on how to do so are in the comments within the code.

3) Generate the TXT file by typing ‘python questiongen.py > outputfile.txt’ in command prompt. Or just ‘python questiongen.py’ to see what your output file is going to look like first.

4) Head back over to your web-browser, and create a new survey in Qualtrics using the Quick Survey builder.

5) Once you are in “Edit Survey” mode, click on “Advanced Options” and choose “Import Survey.” Import the outputfile.txt you generated using the python script. If you have a large universe of questions, it may take Qualtrics some time to parse the file. Once Qualtrics has finished importing the file, you’ll see the entire universe of vignettes has been imported into its own question “block.”

6) Go to survey flow.

7) Click “Add a New Element Here” at the very bottom of the survey flow screen and choose the “Randomizer”. Using the “Move” button, drag the randomizer to the very beginning of your survey. Now comes the tedious part (depending on your universe size). Drag and drop each vignette block under the randomizer.

8) Set the Randomizer to “Randomly Present [X] of the following elements.” [X] is how many vignettes you want your respondent to see from the universe. Also, you can make your life easier by making sure “Evenly Present Elements” is selected.

9) Click “Save Flow.”

10) Congrats, you’ve created a factorial vignette survey in Qualtrics.