Refined metrics for measuring ideation effectiveness

Idea generation is an important step in the engineering design process, and as a result significant research efforts have focused on developing methods to aid designers in exploring design possibilities. Metrics to evaluate design exploration are thus necessary to make conclusions and comparisons among idea generation methods. Metrics have previously been proposed, identifying novelty, variety, quantity, and quality to characterize sets of designs and the degree to which they describe design space exploration. This article describes flaws in the variety metric and proposes a new metric to eliminate the flaws. Additionally, a single metric is proposed to evaluate the quality of design space exploration during concept generation, enabling application of a single metric to compare idea generation processes and methodologies.