Lighthouse Studio gives users a number of ways to look at how long respondents spent in your survey. As a user, you can take advantage of these timers to identify questionable respondents, improve your survey experience, and get better data.
What time data do we store?
When a respondent first enters a survey, we record their start time (“sys_StartTime”) and end time (“sys_EndTime”). The end time is updated each time they submit an answer so it always records the last time they accessed the survey. An additional variable, “sys_ElapsedTime,” computes the total time between the start time and end time.
We also record the time spent on each page of the survey (sys_PageTime_X). This records the time elapsed between when the server sends a page to a respondent and when it gets a response back for that page. If a respondent visits a page more than once (due to a backward skip or using the “back” button), the sys_PageTime attempts to track the cumulative time. An additional variable, “sys_SumPageTimes,” computes the total time across all pages of the survey.
Tip: In Lighthouse Studio, you can change the time zone for “sys_StartTime” and “sys_EndTime”. Before uploading your survey, go to Compose – Survey Settings… and click on the General Settings tab. In the lower right quadrant, you’ll find a “Time Zone” setting. Since this can be set independently for each survey, you can set it to your time zone or your client’s time zone.
How can I use this data?
- Many people use the sys_ElapsedTime variable as an indicator of how much attention a respondent was paying to a survey. One common practice is to eliminate, say, the fastest 5%-10% of respondents. I’ll often look at how long it takes me to skim the instructions and then click through randomly. Any respondent that gets through faster than I do when I’m knowingly not paying attention gets the boot. Be careful to factor in survey branching, though. If there’s legitimate reasons for some respondents to see considerably shorter surveys than others, you won’t want to accidentally remove them for speeding simply because they were faster.
- Use sys_PageTime to identify questions that are difficult for respondents to answer. Look for correlations between page times and dropout rates. Perhaps a question that isn’t really important to you is causing your survey to run longer than anticipated. This is not only bad for the respondent experience, it could actually cost you money, since many panels base pricing partly on the time it takes to complete your survey.
- Add up page times across questions to see how long a particular section took. I might want to remove respondents that sped through the conjoint section, even if their overall time was acceptable.
- Did a respondent take 10 minutes to complete because they were actually considering each question, or did they just pause on the last question so they wouldn’t get disqualified for speeding? We recently used page times to prove that a CAPI field interviewer was conducting much shorter interviews than their contract allowed. Using time stamps, we could show that they were pausing on the last page for 45 minutes before submitting, turning a 15-minute interview into what looked like an “hour-long” interview.
While any single measure can help identify questionable respondents, we usually look at more than one indicator before eliminating respondents. In addition to timers, you might find it useful to look for straight lining, nonsensical open end responses, low fit statistics (RLH in CBC, for instance), mismatches in responses (“CEO” of a company with “10,000 or more employees” who earns “$20,000 or less” working “part time”???), and other indicators.
Are there any issues to be aware of?
Some of the variables and features may not be available or may function differently in older or newer versions of Lighthouse Studio/SSI Web. This article references features and variables available in Lighthouse Studio v9.5.3.
Fun Fact: You may have noticed that the start and end times are initially recorded on the server as large numbers such as “1486408690” (“sys_StartTimeStamp” and “sys_EndTimeStamp”). This “UNIX Time” is the number of seconds that have elapsed since 00:00:00 January 1, 1970 UTC. Factoring in leap years (but not leap seconds) “1486408690” works out to be 47 years, 36 days, 19 hours, 18 minutes and 10 seconds since “Unix Time” began, or 2:18PM EST, February 6, 2017. Since this is also saved in a more readable format, you probably won’t ever need to use the UNIX time. But if you do any advanced scripting, it may be easier to work with the UNIX time.