Coherence and correspondence in engineering design: informing the conversation and connecting with judgment and decision-making research

Text-only Preview

Judgment and Decision Making, Vol. 4, No. 2, March 2009, pp. 147–153
Coherence and correspondence in engineering design: informing
the conversation and connecting with judgment and
decision-making research
Konstantinos V. Katsikopoulos?
Max Planck Institute for Human Development &
Massachusetts Institute of Technology
Abstract
I show how the coherence/correspondence distinction can inform the conversation about decision methods for engi-
neering design. Some engineers argue for the application of multi-attribute utility theory while others argue for what
they call heuristics. To clarify the differences among methods, I ?rst ask whether each method aims at achieving coher-
ence or correspondence. By analyzing statements in the design literature, I argue that utility theory aims at achieving
coherence and heuristics aim at achieving correspondence. Second, I ask if achieving coherence always implies achiev-
ing correspondence. It is important to provide an answer because while in design the objective is correspondence, it is
dif?cult to assess it, and coherence that is easier to assess is used as a surrogate. I argue that coherence does not always
imply correspondence in design and that this is also the case in problems studied in judgment and decision-making
research. Uncovering the conditions under which coherence implies, or does not imply, correspondence is a topic where
engineering design and judgment and decision-making research might connect.
Keywords: coherence, correspondence, design. engineering, Pugh process, multi-attribute utility theory, heuristics.
1 Introduction
is increasingly recognized that engineering design [is a]
decision-intensive process” (p. 342). The NSF has, since
Kenneth Hammond (1996, 2007) has pointed out that a
1996, sponsored numerous workshops on decision-based
method can be evaluated both according to its internal
design. The Accreditation Board for Engineering and
consistency, or coherence, and its external performance,
Technology also de?nes engineering design as a decision-
or correspondence. It is important to keep this distinc-
making process.
tion in mind when comparing decision methods because
What decisions do design engineers make? Design
one method could be achieving coherence while another
engineers choose among alternative concepts. A design
method could be achieving correspondence. For exam-
concept is a technical speci?cation of an artifact that is
ple, the take-the-best heuristic (Gigerenzer & Goldstein,
detailed enough so that the engineer can predict, reason-
1996) violates a criterion of coherence (transitivity) that
ably accurately, how the artifact will function. For exam-
is satis?ed by linear regression, while, under some condi-
ple, a concept of a chair would specify the material used
tions, take-the-best outperforms regression in a criterion
to build each chair part and the geometrical relationships
of correspondence (predictive accuracy). In this article, I
among the parts. There are typically many attributes on
show how the coherence/correspondence distinction can
which design concepts can be evaluated. Examples of
inform the conversation about decision methods within a
attributes for a chair concept are durability, comfort, or
?eld that has had minimal overlap with JDM (judgment
production cost. Hereafter, I refer to design concepts as
and decision making), the ?eld of engineering design.
simply designs.
In 1999, National Science Foundation (NSF) engineer-
ing design program director George Hazelrigg wrote: “It
Some engineers (Thurston, 1991; 2001) argue for the
application of multi-attribute utility theory for choosing
?This work was supported by a German Science Foundation (DFG)
Fellowship for Young Researchers KA 2286/4–1. I thank Jon Baron,
among designs, while others argue for what they call
Don Clausing, Phil Dunwoody, Jonathan Evans, Dan Frey, Robin Hoga-
heuristics, such as Stuart Pugh’s convergence process
rth, Chris Magee, two anonymous reviewers, and the participants of the
(Pugh, 1981; 1990). This debate has by and large ig-
2007 Brunswik Society Meeting for their comments. Address: Kon-
nored the coherence/correspondence distinction. I ask
stantinos V. Katsikopoulos, Center for Adaptive Behavior and Cogni-
tion, Max Planck Institute for Human Development, Lentzeallee 94,
two questions that use the distinction and inform the de-
14195 Berlin, Germany. Email: [email protected]
bate.
147

Judgment and Decision Making, Vol. 4, No. 2, March 2009
Coherence and correspondence in engineering design
148
The ?rst question is whether each method aims at
In the rating/weighting method (Scott & Antonsson,
achieving coherence or correspondence (or both). By an-
1999), the worth of a design is calculated by adding
alyzing published statements in the design literature, I ar-
the attribute levels of the designs, multiplied by the
gue that multi-attribute utility theory aims at achieving
weight of each attribute. This method is the analogue
coherence while the Pugh convergence process aims at
of the weighted linear model in JDM research. The rat-
achieving correspondence.
ing/weighting method conforms to (1) and (2). But it vi-
The second question is if achieving coherence always
olates (3) because the weights of the attributes must be
implies achieving correspondence. It is important to an-
stated explicitly by the design engineers. Saaty’s (1980)
swer because while in design the objective of decision-
Analytic Hierarchy Process (AHP) conforms to (1) but
making is correspondence, it is dif?cult to assess it, and
dispenses with (2) and (3): AHP calculates a measure of
coherence that is easier to assess is used as a surrogate.
worth for each design but this measure takes into account
For the surrogate (coherence) to be useful for inferring
the other designs, using the explicit statements of the en-
the objective (correspondence), the relationship between
gineer.
the two must be known. I argue that coherence does
There is a method that dispenses with all three prin-
not always imply correspondence in design, and that this
ciples of multi-attribute utility theory. In Pugh’s (1981,
is also the case in decision problems studied in JDM. I
1990) convergence process, a group of experts is asked
conclude that the study of conditions under which coher-
to compare each design with a benchmark. For each at-
ence implies, or does not imply, correspondence is a topic
tribute, they judge whether the design is equally good (0),
where design and JDM research might connect.
better (+1), or worse (–1) than the benchmark. For exam-
Before asking and answering the two questions, I re-
ple, the durability of a new chair may be compared to the
view two methods for making decisions that have a
durability of a benchmark swivel chair. Even though it
prominent place in engineering design theory and prac-
can sometimes be determined experimentally which one
tice.
of two chairs is more durable, the judgments are the opin-
ions of experts. These judgments are not weighted or
summed up. Pugh (1990, p. 77, emphasis in the original)
2 Decision methods in engineer- writes: “The scores or numbers...must not be summed
ing design: Multi-attribute utility algebraically.” That is, there is no design worth, violating
(1), (2), and (3).
theory and the Pugh convergence
The numbers that appear in the Pugh process are there
process
to probe. The idea is that if a design is worse on a partic-
ular attribute, this is a stimulus to think whether it can be
Deborah Thurston (1991) proposed that Keeney and
improved, and to generate new designs. Designers are en-
Raiffa’s (1976; 1993) multi-attribute utility theory, be
couraged to change existing designs and create new ones.
applied to design. This theory satis?es the following
On each iteration, all designs, including the ones created
three principles (for more details see Katsikopoulos &
throughout the process, are compared to the benchmark.
Gigerenzer, 2008).
Through conversation among designers, the benchmark
(1) Worth of Designs. For the decision-maker, each
of the next iteration emerges. When it becomes clear that
design has a worth associated with it, measured by a nu-
a design is best, the process is terminated.
merical value (i.e., utility).
In the JDM literature, there are many opinions on what
(2) Absolute Evaluation of Designs. The worth of a
is and is not a heuristic, and on what it means to fully
design to the decision-maker is determined in an absolute
specify a heuristic. It seems that, in the design litera-
way (i.e., without considering the other designs), by us-
ture, the Pugh process is considered to be a heuristic. For
ing a function that maps the attributes of the design to its
example, Franssen (2005, p. 55) writes: “. . . Pugh . . .
utility.
method . . . is only a ‘heuristic tool.’ ” Frey et al. (2009)
(3) Assessment of Utilities. The utility function is as-
explicitly label the Pugh process as a heuristic.
sessed by questioning the decision-maker about her or his
Connections can be drawn between the decision meth-
preference structure over the design space (i.e., the utility
ods that designers use and fast and frugal heuristics
function is not stated explicitly but revealed through the
(Magee & Frey, 2006).1 While it may be a stretch to call it
answers).
1Magee and Frey (2006) discuss a design exercise where undergrad-
Thurston writes: “Utility analysis cannot be the only
uate students had to develop a paper airplane that would ?y a given
analytic tool employed in design” (Thurston, 2001, p.
distance consistently. It was observed that students seemed to use “one-
182). Other decision methods used in design can be de-
reason” decision heuristics when testing and creating airplane designs
(even though, as the authors acknowledge, this research did not employ
scribed in terms of whether they satisfy the three princi-
controlled studies). The authors review work on fast and frugal heuris-
ples of multi-attribute utility theory or not.
tics, compare it to the thinking and reasoning that goes on in design,

Judgment and Decision Making, Vol. 4, No. 2, March 2009
Coherence and correspondence in engineering design
149
a fast and frugal heuristic, the Pugh process shares some
3.2 Achieving correspondence in design
practices with methods such as take-the-best (Gigeren-
decision-making
zer & Goldstein, 1996). For example, both use pairwise
comparisons and ignore some pieces of information (e.g.,
I use Hammond’s de?nition of correspondence.
He
attribute weights). Interestingly, these practices are of-
writes: “There are two general ways we evaluate an-
ten avoided in design decision-making (Saari & Sieberg,
other person’s judgments [coherence and correspon-
2004).
dence]. One is to ask if they are empirically correct:
I reviewed two methods for making decisions in de-
When someone judges this tree to be ten feet tall, will
sign: multi-attribute utility theory and the Pugh conver-
a yardstick prove that she is right?” (Hammond, 2007, p.
gence process. In sum, in utility theory the decision-
xvi).
maker deliberates and may secure accountability, while
A difference between coherence and correspondence is
in the Pugh process the decision makers rely on intuition
that in coherence the criterion is internal (logical consis-
and aim at boosting creativity. In the next section, I sam-
tency) while in correspondence the criterion is external
ple and analyze statements, in the design literature, about
(success in the real world). While criteria of logic are es-
the two methods. The goal is to examine if utility theory
sentially domain-independent, criteria of correspondence
and the Pugh process aim at achieving coherence, corre-
depend on the decision problem.
spondence, or both.
In engineering, correspondence is typically achieved
by a design that “works.”2 The user determines what re-
quirements the design must satisfy so that it can be said to
3 Multi-attribute
utility
theory work. In engineering jargon, these are called functional
requirements. After the user articulates how she wants the
aims at achieving coherence; the artifact to function, the engineer expresses the functional
Pugh process aims at achieving requirements in technical terms, often as mathematical
constraints involving attributes.
correspondence
For example, consider a user who says that she wants
an of?ce chair that “will last for some time.” A technical
I ?rst de?ne what it means to achieve coherence and cor-
description of the functional requirement is that “the time
respondence in design decision-making.
to failure exceeds 10,000 hours of use by a female with
physical characteristics that are within the middle 99% of
the normal adult range.”
3.1 Achieving coherence in design decision-
making
3.3 The Pugh process aims at correspon-
Hammond (2007, p. xvi) de?nes coherence as “the con-
dence
sistency of the elements of the person’s judgment.” I use
the same de?nition for coherence in engineering design.
For more than ?fty years, economists have been dis-
For example, suppose that an engineer chooses chair
cussing decision theories in terms of axioms (Allais,
design A. To evaluate the coherence of this decision, it
1952/1979; Starmer, 2000). Decision theories can be
needs to be checked whether the statements the engineer
descriptive (what does a person do?), normative (what
made in order to decide for A were internally consistent
should an ideal person do?), and prescriptive (what
or not. If the engineer maintained that design A would
should a real person do?), and it has been argued that the
be chosen over design B if C were considered as an an-
common ground of all three is a set of axioms (Luce &
other possible design, but B would be chosen over design
von Winterfeldt, 1994).
A if C were not considered, she has violated a coherence
Engineers have also discussed decision methods in
requirement called the Independence of Irrelevant Alter-
terms of axioms. For example, Thurston (2006, p. 19)
natives. Another instance of failure of coherence is In-
labels axioms as “rules for clear thinking.” A common
transitivity where A is chosen over B, B is chosen over
C, and C is chosen over A. Some coherence requirements
2Correspondence can also be measured by human-factors criteria
are related to the three principles of multi-attribute util-
such as transparency or usability, and by “broader” criteria such as crit-
ity theory. The worth-of-designs principle implies tran-
ical acclaim or success in the market. For more examples of coherence
sitivity and the absolute-evaluation-of-designs principle
and correspondence criteria in engineering design and on some com-
ments on their relation, see Evans, Foster, and Katsikopoulos (in press).
implies independence of irrelevant alternatives.
Computer science and engineering also provide examples of coherence
and correspondence criteria. For example, computer code has to sat-
and conclude that “Our current belief is that engineering designers use
isfy syntax requirements (coherence) and produce the “desirable” out-
a toolbox of fast and frugal heuristics” (Magee & Frey, 2006, p. 486).
put (correspondence).

Judgment and Decision Making, Vol. 4, No. 2, March 2009
Coherence and correspondence in engineering design
150
argument against heuristic methods is that, under some
and C has the values of 3,000, 3,2000, and 3,400 N/mm,
conditions, they violate some axioms.
respectively, it cannot be that a designer is free to rank
Franssen (2005, p. 55) presents an existence proof
them as C > A > B (Scott & Antonsson, 1999, pp. 223–
for the claim that, depending on the initial benchmark,
224). That is, they say that engineering reality constraints
the design chosen by the Pugh process may vary. This
engineering judgment. This is a correspondence argu-
is a violation of independence of irrelevant alternatives.
ment. Franssen (2005, pp. 48–49) replies with a textbook
Franssen labels such violations as “dif?culties.”
case of a coherence argument: “. . . it is of paramount
Hazelrigg (1996) has criticized the group decision-
importance to realize that preference is a mental concept
making aspects of the Pugh process because they can lead
and is neither logically nor causally determined by the
to violations of independence of irrelevant alternatives as
physical characteristics of a design option.”
well. To argue this, Hazelrigg uses Arrow’s (1950) im-
possibility theorem. Informally, the theorem says that
there is no method that translates the single preference
orders of more than three designers to a “group” prefer-
3.4 Utility theory aims at coherence
ence order, so that ?ve axioms are always jointly satis?ed.
One of these axioms is the independence of irrelevant al-
Utility theory is a mathematical theory that reigns in
ternatives.
economics (Starmer, 2000), operations research (Keeney
Hazelrigg (1996, p. 161) uses Arrow’s theorem to con-
& Raiffa, 1976; 1993), and decision analysis (Howard,
clude that “the methods of . . . Quality Function De-
1968). Mathematical theories proceed from axioms to
ployment (QFD)3 can lead to highly erroneous results.”
theorems. As such, utility theory aims at achieving co-
Franssen (2005, p. 55) writes: “[the Pugh process] does
herence.
not meet Arrow’s requirement” and, “Presumably, be-
Does utility theory aim at achieving correspondence as
cause he is well aware of dif?culties like these, Pugh is-
well? There is a sense in which the answer seems to be
sues a warning that his method . . . is only a ‘heuristic
yes. Keeney and Raiffa (1976; 1993) advocate its use
tool’.”4
because they believe that utility theory will lead to suc-
These authors are correct in pointing out that the Pugh
cess in real-world decision problems. Why would this be
process does not aim at achieving coherence. Nowhere
true? The answer implicit in the design literature — at
in the writings of Pugh is there a concern with adhering
least among proponents of multi-attribute utility theory
to the axioms of decision theories.5 On the other hand,
— is that achieving coherence always implies achieving
aiming at correspondence, in the title of his paper, Pugh
correspondence. For example, Thurston (2001, p. 176)
(1981, emphasis added) introduced process convergence
writes: “Unaided human decision-making often exhibits
as “a method that works.” Similarly, Frey et al. (2009)
inconsistencies, irrationality, and suboptimal choices . . . .
and Clausing and Katsikopoulos (2008) argued in favor
To remedy these problems, decision theory is built on a
of the Pugh process by saying that it can lead to success
set of ‘axioms of rational behavior.’ ” Here, lack of coher-
in real design problems.
ence (inconsistencies) is mentioned in the same sentence
Failing to acknowledge that the Pugh process aims
with lack of correspondence (suboptimal choices) as if to
at achieving correspondence but not coherence, has led
express that the two are conceptually very close to each
to conversations where design researchers talk past each
other. In fact, the second sentence directly suggests that
other. Consider the unrestricted-domain axiom (that fea-
lack of correspondence will be remedied by coherence.
tures in Arrow’s theorem): “Each member of the design
My opinion is that, even if the proponents of multi-
group is free to rank designs in any way.” Scott and An-
attribute utility theory believe that it aims at achieving
tonsson (1999) argue that it is not obvious that this axiom
correspondence, the only thing we know for sure is that
should be considered true in design. For example, they
utility theory aims at achieving coherence. Given the
say that when the bending stiffness of three designs, A, B,
premise that coherence always implies correspondence,
we would conclude that utility theory aims at achieving
3One of these methods is the Pugh process; see Hauser and Clausing
correspondence, but the truth of the premise is an open
(1988) for details.
4
empirical question.
It is worthy to note that even though the dif?culties implied by
Arrow’s theorem are theoretically possible, they are rarely realized in
In the next and ?nal section of the paper, I discuss
practice (Regenwetter et al., 2006).
what we know, from both engineering design and JDM
5This does not mean that the Pugh process aims at, or would even
accept, incoherence in the sense of not adhering to basic mathematical
research, about whether achieving coherence always im-
truths such as “1 + 1 = 2.”
plies achieving correspondence.

Judgment and Decision Making, Vol. 4, No. 2, March 2009
Coherence and correspondence in engineering design
151
4 Does achieving coherence always are many possibilities. I propose the following:
imply achieving correspondence?
Achieving Coherence Implies Achieving Correspon-
dence. “For a decision problem, achieving coherence im-
Two counterexamples from en- plies achieving correspondence if whenever there are two
gineering design and JDM re- methods A and B such that A satis?es a criterion of coher-
ence and B violates this criterion, it holds that A scores
search.
higher in all criteria of correspondence than B.”
I argue that there exist counterexamples to the claim
I ?rst argue that, in design, it is important to know if co-
that achieving coherence implies achieving correspon-
herence implies correspondence. The reason is that while
dence for all decision problems. Don Clausing and I
in design the objective of decision-making is correspon-
made this point for engineering design.
dence, it is dif?cult to assess it, and coherence that is
After the golden post-World-War-II era, the Ameri-
easier to assess is used as a surrogate. For the surrogate
can manufacturing industry began to lose ground in the
(coherence) to be useful for inferring the objective (cor-
1970s, in particular compared to Japan. By the 1980s,
respondence), the relationship between the two must be
the crisis was so obvious that investigations were under-
known.
taken. The report of the MIT Commission On Industrial
The previous argument rests on two claims: First, cor-
Productivity (1989) summarizes some results. Based on
respondence is the objective of design decision-making.
the report, Clausing and Katsikopoulos (2008) argue that
Second, correspondence is dif?cult to assess in design
methods such as the Pugh process lead to higher quality
decision-making. I argue for these two claims.
designs that are produced with less cost and are delivered
First, it is in a sense obvious that the only thing that
more quickly than the designs chosen by methods such
ultimately matters in engineering is “how well the de-
as weighting-and-rating and utility theory. Recall that
sign works.” It is not acceptable to argue coherently but
the Pugh process violates coherence criteria such as the
choose a design that is not functional. The objective is to
independence of irrelevant alternatives while the weight-
achieve correspondence.6
ing/rating method and multi-attribute utility theory sat-
To argue for the second claim that correspondence is
isfy it.
dif?cult to assess in design, I start with a frequent ob-
An explanation for this counterexample is that the
servation in the JDM literature. While coherence can be
Pugh process fosters creativity while other methods sti?e
assessed during the process of making a judgment or a
it. Classical utility-based decision analysis (von Winter-
decision, correspondence can be assessed only after the
feldt & Edwards, 1986; Edwards & Fasolo, 2001) may
outcome of the judgment or decision has been observed
sti?e creativity when it focuses on the analysis of the
(Connoly, Arkes, & Hammond, 2000). With respect to
designs provided before the decision process starts, and
coherence, this observation seems to hold in engineering
neglects the generation of novel designs. This explana-
design as well. The situation is somewhat different with
tion is consistent with the results of a study by Frey et
respect to correspondence.
al. (2009) where a computer simulation was used to as-
Whether a design satis?es a functional requirement or
sess the pro?tability achieved by different decision meth-
not may be assessed both after and before a design is cho-
ods. It was found that the Pugh process outperformed the
sen. Crucially, assessing whether a functional require-
rating/weighting method when creativity was modeled as
ment is satis?ed is often dif?cult or even impossible. Ex-
part of the design decision process, while the two meth-
tensive experimentation is needed in order to establish
ods were equally pro?table when there was no creativity.
that a requirement, such as durability, is satis?ed (Frey
It is noteworthy that practitioners of classical deci-
& Dym, 2006). It is even harder to measure criteria of
sion analysis have drawn attention to creativity as well
correspondence such as transparency or critical acclaim.
(Philips, 1982). In a recent paper, Ralph Keeney (2004a,
Some correspondence criteria, such as market share, may
p. 193), re?ects on this: “[In decision analysis] more
be easier to measure but it is hard to attribute success or
emphasis must be placed on structuring decisions worth
failure to decision-making alone. For example, marketing
thinking about, and less emphasis must be based on
may greatly in?uence the sales of an artifact.
analyzing structured decisions.”
He has also written
What is the precise meaning of the statement “achiev-
speci?cally on how to create design alternatives (Keeney,
ing coherence implies achieving correspondence”? There
2004b).
The ?nding that coherence does not always imply cor-
6Note that my argument is that correspondence should have priority
respondence should not be a surprise to JDM researchers.
speci?cally in engineering design. I would not make this argument for
Some of the results of the “fast-and-frugal-heuristics”
all ?elds. For example, coherence should be the top priority in mathe-
matics. In public policy, it has been argued that a mix of coherence and
program can be interpreted as showing this as well.
correspondence may be most appropriate (Hammond, 2007).
Consider the paired comparison problem where it has

Judgment and Decision Making, Vol. 4, No. 2, March 2009
Coherence and correspondence in engineering design
152
to be judged which one of two objects has the higher
coherence implies achieving correspondence is a topic
value on a numerical criterion. This judgment is made
where engineering design and JDM research might con-
based on the values of the objects on cues (that correlate,
nect. Interestingly, these conditions may involve mathe-
albeit imperfectly, with the criterion). For example, the
matical properties as well as psychological constructs.
two objects may be companies, the criterion may be a
company’s net worth, and a cue may be whether a com-
pany is in the stock exchange.
References
Take one method to be multiple linear regression. Take
another method to be the lexicographic heuristic take-
Allais, M. (1979). Foundations of a positive theory of
the-best (Gigerenzer & Goldstein, 1996), where cues are
choice involving risk, and a criticism of the postulates
looked up one at a time and a decision is made based
and axioms of the American School. In M. Allais &
on the ?rst cue with different values on the two objects.
O. Hagen (Eds.), Expected utility hypothesis and the
It is easy to see that regression satis?es transitivity and
Allais’ Paradox, pp. 25–145. Dordrecht: D. Reidel.
take-the-best does not. Across twenty datasets (from
(Original work published 1952)
economics, biology, psychology; see Gigerenzer et al.,
Arrow, K. J. (1950). A dif?culty in the concept of social
1999), linear regression achieved a predictive accuracy
welfare. Journal of Political Economy, 58, 328–346.
of 68% and take-the-best achieved 71%.7
Baucells, M., Carasco, J. A., & Hogarth, R. M. (2008).
As in design, this counterexample contradicts the
Cumulative dominance and heuristic performance in
rhetoric of classical decision analysis. Keeney and Raiffa
binary multi-attribute choice.. Operations Research,
(1993, p. 78) have written that lexicographic heuristics
56, 1289–1304.
are “naively simple” and “will rarely pass a test of reason-
Clausing, D. P., & Katsikopoulos, K. V. (2008). Ratio-
ableness.” As Gigerenzer and Goldstein (1996, p. 663)
nality in systems engineering: Beyond calculation or
point out, “despite empirical evidence. . . lexicographic
political action. Systems Engineering, 11, 309–328.
algorithms have often been dismissed at face value be-
Connolly, T., Arkes, H. R., & Hammond, K. R. (2000).
cause they violate the tenets of classical rationality.” Lex-
Judgment and decision making: An interdisciplinary
icographic heuristics were dismissed because it was as-
reader. Cambridge, UK: Cambridge University Press.
sumed that coherence always implied correspondence.8
The question of under which conditions should heuris-
Edwards, W., & Fasolo, B. (2001). Decision technology.
tics be used for making decisions has been addressed in
Annual Review of Psychology, 52, 581–606.
the JDM literature. There is agreement that “. . . in gen-
Evans, J., R., Foster, C., & Katsikopoulos, K. V. (in
eral, . . . heuristics are quite useful, but sometimes they
press). Coherence and correspondence in engineering
lead to severe and systematic errors” (Tversky & Kahne-
design evaluations. Proceedings of the Annual Meeting
man, 1974, p. 1124). But what does “sometimes” mean?
of the American Society for Engineering Education.
In the beginning, answers were typically cast as a list of
Franssen, M. (2005). Arrow’s theorem, multi-criteria de-
experimental manipulations that increase or decrease the
cision problems and multi-attribute preferences in en-
accuracy of a heuristic. Further progress has been made
gineering design. Research in Engineering Design, 16,
recently by specifying precise models of heuristics and
42–56.
using them to analyze the performance of heuristics.
Frey, D. D., & Dym, C. L. (2006). Validation of design
There exist now mathematical analyses of the accu-
methods: Lessons from medicine. Research in Engi-
racy of lexicographic heuristics such as take-the-best,
neering Design, 17, 45–57.
and of more sophisticated methods (Martignon & Hof-
Frey, D. D., Herder, P. M., Wijnia, Y., Subramanian, E.,
frage, 2002; Hogarth & Karelaia, 2007; Baucells et al,
Katsikopoulos, K. V., & Clausing, D. P. (2009). An
2008). For example, a necessary and suf?cient condi-
evaluation of the Pugh controlled convergence method.
tion has been derived under which a lexicographic heuris-
Research in Engineering Design, 20, 41–58.
tic achieves maximum accuracy (Katsikopoulos & Mar-
Gigerenzer, G., & Goldstein, D. G. (1996). Reasoning
tignon, 2006).
the fast and frugal way: Models of bounded rationality.
Uncovering general conditions under which achieving
Psychological Review, 103, 4, 650–669.
Gigerenzer G, Todd P. M., the ABC research group
7Lages, Hoffrage, and Gigerenzer (2000) provide some evidence
(1999) Simple heuristics that make us smart. New
that decision methods that produce a higher number of intransitive
triples also have higher predictive accuracy if there is not a lot of miss-
York: Oxford University Press.
ing information. Note, however, that it is not clear that the number of
Hammond, K. R. (1996). Human judgment and social
intransitive triples is related monotonically to the “degree” of coherence
policy: Irreducible uncertainty, inevitable error, un-
(Regenwetter et al., 2006).
8For comments on the role that lexicographic heuristics can play in
avoidable injustice. Oxford: Oxford University Press.
decision analysis, see Katsikopoulos and Fasolo (2006).
Hammond, K. R. (2007). Beyond rationality: The search

Judgment and Decision Making, Vol. 4, No. 2, March 2009
Coherence and correspondence in engineering design
153
for wisdom in a troubled time. Oxford: Oxford Uni-
ogy. International Journal of Engineering Education,
versity Press.
22, 478–488.
Hauser, J. R., & Clausing, D. P. (1988). The house of
Martignon, L., & Hoffrage, U. (2002). Fast, frugal, and
quality. Harvard Business Review, 3 (May-June), 663–
?t: Simple heuristics for paired comparison. Theory
673.
and Decision, 52, 29–71.
Hazelrigg, G. A. (1996). The implication of Arrow’s im-
MIT Commission on Industrial Productivity (1989).
possibility theorem on approaches to optimal engineer-
Made in America — Regaining the competitive edge.
ing design. ASME Journal of Mechanical Design, 118,
Cambridge: MIT Press.
161–164.
Phillips, L. D. (1982). Requisite decision modeling: A
Hazelrigg, G. A. (1999). An axiomatic framework for
case study. Journal of Operational Research Society,
engineering design. ASME Journal of Mechanical De-
33, 303–311.
sign, 121, 342–347.
Pugh, S. (1981). Concept selection: A method that
Hogarth, R. M., & Karelaia, N. (2007). Heuristic and
works. Proceedings of the International Conference
linear models of judgment: Matching rules and envi-
on Engineering Design, Rome: ASME Press.
ronments. Psychological Review, 114, 733–758.
Pugh, S. (1990). Total design: integrated methods for
Howard, R. A. (1968). The foundations of decision anal-
successful product engineering. Workingham, UK:
ysis. IEEE Transactions on Systems Science and Cy-
Addison-Wesley.
bernetics, 4, 211–219.
Regenwetter, M., Grofman, B., Marley, A., & Tsetlin, I.
Katsikopoulos, K. V., & Gigerenzer, G. (2008). One-
(2006). Behavioral social choice. Cambridge: Cam-
reason decision-making: Modeling violations of ex-
bridge University Press.
pected utility theory. Journal of Risk and Uncertainty,
Saari, D. G., & Sieberg, K. K. (2004). Are partwise com-
37, 35–56.
parisons reliable? Research in Engineering Design, 15,
Katsikopoulos, K. V., & Fasolo, B. (2006). New tools
62–71.
for decision analysts. IEEE Transactions on Systems,
Saaty, T. (1980). Analytic Hierarchy Process, New York:
Man, and Cybernetics: Systems and Humans, 36, 960–
Mc-Graw Hill.
967.
Scott, M. J., & Antonsson, E. K. (1999). Arrow’s theorem
Katsikopoulos, K. V., & Martignon, L. (2006). Naive
and engineering design decision making. Research in
heuristics for paired comparisons: Some results on
Engineering Design, 11, 218–228.
their relative accuracy. Journal of Mathematical Psy-
Starmer, C. (2000). Developments in non-expected utility
chology, 50, 488–494.
theory: The hunt for a descriptive theory of choice un-
Keeney, R. L. (2004a). Making better decision makers.
der risk.” Journal of Economic Literature 38(2), 332–
Decision Analysis, 1, 193–204.
382.
Keeney, R. L. (2004b). Stimulating creative design alter-
Thurston, D. L. (1991). A formal method for subjective
natives using customer values. IEEE Transactions on
design evaluation with multiple objectives. Research
Systems, Man, and Cybernetics: Systems and Humans,
in Engineering Design, 3, 105–122.
34, 450–459.
Thurston, D. L. (2001). Real and misconceived limi-
Keeney, R. L., & Raiffa, H. (1976). Decisions with mul-
tations to decision based design with utility analysis.
tiple objectives: Preferences and value tradeoffs. New
ASME Journal of Mechanical Design, 123, 176–182.
York: Wiley.
Thurston, D. L. (2006). Utility function fundamentals, In
Keeney, R. L., & Raiffa, H. (1993). Decisions with multi-
K. E. Lewis, W. Chen, & L. C. Schmidt (Eds.) Deci-
ple objectives: Preferences and value Tradeoffs, Cam-
sion Making in Engineering Design (pp. 15–19). New
bridge: Cambridge University Press.
York: ASME Press.
Lages, M., Hoffrage, U., & Gigerenzer, G. (2000). How
Tversky, A., & Kahneman, D. (1974). Heuristics and
heuristics produce intransitivities and how intransitiv-
biases: Judgement under uncertainty. Science, 185,
ities can discriminate between heuristics. Manuscript,
1124–1130.
Max Planck Institute for Human Development, Berlin,
von Winterfeldt, D., & Edwards, W. (1986). Decision
Germany.
analysis and behavioral research. New York: Cam-
Luce, R. D., & von Winterfeldt, D. (1994). What com-
bridge University Press.
mon ground exists for descriptive, prescriptive, and
normative utility theories? Management Science. 40,
263–279.
Magee, C., & Frey, D. D. (2006). Experimentation and
its role in engineering design: Linking a student de-
sign exercise with new results from cognitive psychol-