Design Studies

Summary

• Overview
• Design Spaces
• Teaching and Learning
• Experiments
• Report
  • Context
  • Research Questions
  • Subject
  • Experimental Conditions
  • Variables
  • Method
  • Data
  • Discussion
  • Conclusions

Overview

Design is about making decisions

Decisions can come from multiple places

• Customer
• End user
• Technology specification
• Competitive products

Design Study := rigorous and systematic evaluation of the factors that influence a design and consists of a comparison of various possible approaches

Begin with relevant criteria, metrics, and threshold

Design Spaces

Explore a space of possibilities

Factors to examine:

• aesthetic
• objective

Teaching and Learning

Design can’t be taught, only learned by doing

Experiments

Overall goal of design study is repeatability

Report

Objective

Dispassionate

Multiple sections

1. Context
2. Research questions
3. Subjects
4. Experimental conditions
5. Variables
6. Method
7. Results
8. Discussion
9. Conclusions

Context

Background or motivation for study

Necessary information to understand the study

Research Questions

Examines tradeoffs between various non-functional requirements

• Tradeoffs framed as questions
  • “How is the memory footprint affected by changing X”

Questions should be neutral and numbered

Subject

Subjects are the thing being studied

Studies typically include multiple subjects

• Each subject should be briefly described and differentiated

Experimental Conditions

A Software design study normally means running several versions of a program, making measurements and evaluating the results

Configuration information should be explicitly documented

Variables

Independent and dependent variables must be identified and appropriate metrics specified

3 column table

Research Question	Independent	Dependent

Method

Number of trials

Measurement devices and tools

Randomization technique

Significant digits

Explicit statement of which subjects will be run and with what arguments

Briefly describe any statistical techniques you will use

Data

The point of a study is to produce data

This section presents the data collected and their statistical analysis

Discussion

Opportunity to interpret the data you collected and provided a discussion of its implications

Reflect on the experiment itself

Conclusions

Summarize your results and draw conclusions

Provide explicit answers to each of the research questions