Review status
This summary is based on the full article text.
The DOI and source URL have been verified through the official INFORMS article page.
The article is an empirical study published in Organization Science. It uses archival data from R&D project selection decisions inside a large multinational professional service firm, combined with qualitative interviews and observation of one selection meeting. The empirical design exploits quasi-random ordering of proposals in panel deliberations to estimate whether the sequence of prior funding decisions affects the funding awarded to the next R&D project.
Research question
Does the temporal sequence of R&D project evaluations affect panel funding decisions?
More specifically, the article asks whether a project that appears immediately after a funded project is less likely to receive funding or receives a lower share of its requested funding, even when project characteristics and applicant characteristics are controlled for.
The article also asks whether this sequence effect becomes stronger later in panel meetings, when evaluators may experience greater mental and emotional fatigue.
Hypotheses
The article tests two hypotheses.
Hypothesis 1 proposes that prior positive funding decisions negatively affect the panel’s funding of the subsequent project.
Hypothesis 2 proposes that the negative impact of a positive prior funding decision on the focal project’s funding decision is stronger later in panel deliberations.
Method
The study examines R&D project selection inside a large multinational professional service firm. The focal selection panel was responsible for allocating organizational resources to innovation projects and reported directly to the chief executive officer. The panel had five members: the R&D director, the director of the knowledge management department, and three directors responsible for the firm’s main practice areas.
The empirical setting is useful because the order of proposal evaluation was quasi-random. Projects were first grouped by practice area, but within practice areas the order varied: sometimes by applicant surname, sometimes by application identification number, and sometimes in ascending, descending, or fully random order. The ordering principle was unknown to both applicants and panel members. The authors test the randomness of ordering using nonparametric run tests and find support for the assumption that projects were not systematically ordered by quality-related characteristics.
The sample covers R&D project applications submitted between 2006 and 2008. During this period, the selection panel made 604 funding decisions. Because some projects were re-evaluated when information was insufficient, the full process involved 763 deliberations. The authors drop the first decision in each of the 16 meetings because there is no prior decision in that meeting. The final regression sample contains 588 funding decisions.
The dependent variable is the share of requested funding awarded. It ranges from 0 to 1: 0 means the application was rejected, and 1 means it was fully funded. In the sample, 28.7% of proposals were rejected and 37.8% were fully funded.
The main independent variable is whether the prior proposal received funding. The authors also test an alternative measure based on the amount of funding awarded to the prior proposal.
The moderator for Hypothesis 2 is order of assessment, which captures how late the project appeared in the panel meeting. Because the number of proposals varied by meeting, the authors use the log-transformed order measure.
The study uses Tobit regression because the dependent variable is bounded between 0 and 1. The authors also use logit models as robustness checks with an alternative binary dependent variable indicating whether a proposal received any funding. Standard errors are clustered by panel and introducer. The models include year fixed effects and a leave-out mean to account for the panel’s general tendency to fund projects.
The control variables cover four levels:
- R&D portfolio controls, such as funding left before the meeting, similarity to previously funded projects, whether the domain had preallocated funding, and the number of proposals in the same domain area;
- panel-level controls, such as the number of postponed proposals and whether the first proposal in the meeting was large;
- proposal-level controls, such as alignment with R&D strategy, investment proposal status, resubmission, endorsement, opposition, proposal length, project duration, proposal size, and identified delivery risk;
- applicant-level controls, such as applicant tenure and prior research funding experience.
The quantitative analysis is complemented by qualitative evidence. One researcher observed a selection meeting in which the panel evaluated 22 applications. The authors also conducted 29 semistructured interviews: 21 with successful and unsuccessful applicants, four with members of the focal panel, and four with members of other panels.
Results / key findings
The study finds a robust sequence effect in R&D project selection.
The main Tobit model shows that a prior positive funding decision has a negative effect on the share of requested funding awarded to the next proposal. The coefficient for prior proposal funded is -0.303 and statistically significant at the 5% level. The authors estimate that if a proposal is assessed after a funded proposal, the predictive share of requested funding awarded for supported applications decreases by 23%.
The result is robust to alternative specifications. When the authors use a binary dependent variable indicating whether a proposal receives any funding, the prior positive funding decision remains negative. When they use the log-transformed amount of funding awarded to the prior proposal instead of the binary prior-funding measure, the coefficient is also negative and significant (-0.033, p < 0.05).
The placebo test supports the causal interpretation. The authors create 10,000 simulated datasets by randomly reshuffling projects within meetings and re-estimating the model. The coefficient from the real sequence lies inside the fifth percentile of the simulated coefficient distribution. This indicates that the observed sequence effect is unlikely to be an artifact of random reshuffling.
The timing result supports Hypothesis 2. The interaction between prior proposal funded and order of assessment is negative and significant (-0.330, p < 0.05). This means the sequence effect becomes stronger later in the meeting. The authors estimate that when a proposal follows a funded proposal and is evaluated toward the end of a meeting, the share of requested funding awarded decreases by 55%.
The timing result is also robust. Using the alternative binary dependent variable, the interaction remains negative (-0.552, p < 0.10). Using the alternative prior-funding-awarded measure, the interaction is also negative and significant (-0.034, p < 0.05).
The article then tests alternative explanations.
The first explanation is the law of small numbers, also connected to gambler’s fallacy logic. The authors find evidence consistent with this mechanism. When both the immediately prior proposal and the second-lagged proposal were funded, the coefficient is negative and significant (-0.436, p < 0.01). This suggests that the previous two decisions matter, not only the immediately prior decision.
The second explanation is the contrast effect. The evidence does not support a simple contrast-effect explanation. If contrast effects drove the result, a high-quality previous project should make the next project look worse. Instead, the estimated quality of the previous proposal has a positive effect on the focal proposal’s funding share (0.652, p < 0.05). This points more toward an assimilation pattern than a contrast pattern.
The third explanation is a quota model. The authors test whether the result is driven by budget constraints or implicit limits on how many projects the panel wants to fund. The sequence effect remains negative after controlling for funding left before each decision. The effect also decays sharply: the immediately prior funded proposal matters most, the second-lagged funded proposal has a smaller effect, and the third-lagged funded proposal is not significant. This decay pattern is inconsistent with a simple quota explanation, because a quota model would predict that earlier funded projects should continue to matter regardless of exact sequence position.
The fourth explanation is learning. The authors control for the difference between the focal project’s predicted quality and the average quality of prior projects. This quality-difference measure is positive and significant (2.999, p < 0.01), as a learning argument would predict. However, the sequence effect remains negative and significant (-0.254, p < 0.05), meaning learning alone does not explain the result.
Several control variables are also substantively important. Alignment with R&D strategy is positively associated with funding share. In the main hypothesis model, its coefficient is 0.145 and statistically significant at the 1% level. Resubmitted proposals receive more funding, with a coefficient of 0.451 (p < 0.01). Endorsement is positive and significant (0.028, p < 0.05). Proposal size is strongly negative (-0.607, p < 0.01), meaning larger requests receive a lower share of requested funding. The number of proposals in the same domain area is also negative (-0.051, p < 0.01), indicating that proposals face more difficulty when many same-domain proposals are considered in the same meeting.
The article also explores how an individual-level sequence effect passes to the panel level. The panel made decisions through discussion rather than simple voting or averaging, so the introducer’s role was important.
First, the sequence effect is stronger when the introducer has a heavy workload. The interaction between prior proposal funded and introducer workload is negative and significant (-0.269, p < 0.05). The authors estimate that the share of funding awarded to a proposal following a funded proposal decreases by 29% if the proposal is introduced by a panel member with a heavy workload.
Second, the sequence effect is stronger when the introducer has greater expertise in the proposal’s engineering domain than the rest of the panel. The interaction is negative and marginally significant (-0.037, p < 0.10). The authors estimate that the share of requested funding awarded decreases by 36% when the introducer has greater domain expertise and the project follows a positive funding decision.
Overall, the findings show that sequence effects can influence group decisions even in expert panels, even when proposal order is intended to be fair, and even when panels use informed assessments of complex R&D projects.
Practical implications
For managers, the article shows that R&D funding outcomes can be shaped by process details that look neutral but are not.
The central practical lesson is simple: where a proposal appears in a meeting can affect how much funding it receives. A project evaluated immediately after a funded project may receive less funding, even if its own characteristics are strong. The estimated effect is economically meaningful: proposals following a funded proposal receive an estimated 23% lower share of requested funding when supported.
The timing result is especially important for managers. Sequence effects become stronger later in panel deliberations. This means long meetings, fatigue, and pressure to keep moving through the agenda can make panels more vulnerable to decision bias. The estimated decrease reaches 55% when the proposal follows a funded project and is evaluated later in the meeting.
The article suggests that random ordering alone is not enough. Randomization may prevent strategic ordering and protect procedural fairness, but it does not remove the psychological effect of a shared sequence. If all panel members evaluate proposals in the same order, the whole panel may still be influenced by the immediately prior decision.
Managers can reduce the problem through relatively low-cost process changes.
First, panels can use decision checklists that explicitly remind members about sequence effects and other biases before deliberation. The article compares this logic to checklist use among doctors, pilots, and other professionals.
Second, panel members can evaluate proposals independently before discussion. If each evaluator writes down an assessment before the meeting, the panel is less dependent on the flow of discussion and the framing of the introducer.
Third, different evaluators can be given different random proposal orders before the meeting. This would preserve fairness while reducing the chance that all panel members share the same sequence-based bias.
Fourth, meetings can include deliberate breaks or reset points. The article argues that panels often try to “press on through” long agendas, but this may increase vulnerability to sequence effects. Resetting attention may help evaluators treat each proposal more independently.
Fifth, managers should monitor introducer workload. When one panel member has to introduce many projects, the sequence effect becomes stronger. Spreading introducer responsibilities more evenly may reduce bias.
Sixth, panels should be careful when one introducer has much greater domain expertise than the other panel members. Expertise is valuable, but it can also make the panel more dependent on that person’s framing. This may allow sequence effects affecting the introducer to pass more strongly to the panel.
For practitioners, useful diagnostic questions include:
- Are proposals evaluated in long uninterrupted sequences?
- Do panel members know whether sequence effects can bias their judgments?
- Are written independent evaluations collected before group discussion?
- Does every panel member see proposals in the same order?
- Are breaks or reset points built into long selection meetings?
- Are some introducers carrying a much heavier workload than others?
- Are panel decisions overly dependent on one domain expert’s framing?
- Does the organization track whether proposal outcomes vary by meeting order?
Theoretical implications
The article contributes to innovation management by showing that R&D project selection is not only about project quality, applicant characteristics, portfolio fit, or strategic alignment. The temporal order of evaluation also matters.
The article challenges the assumption that R&D project decisions are temporally independent. Many innovation and portfolio-selection models treat each proposal as if it is evaluated on its own merits. This study shows that the prior decision can affect the next decision, even when sequence order is quasi-random.
The article contributes to research on search and selection in organizations. It suggests that organizational search should be understood as a sequence of events rather than a set of independent choices. A proposal’s fate may depend partly on what the panel just decided before it.
The article also contributes to group decision-making research. Organizations often use expert panels because groups are expected to reduce individual bias and improve deliberation. The study shows that groups can still fall prey to sequence effects. In some cases, shared decision sequences and group deliberation may transmit or reinforce biases rather than eliminate them.
The study also adds nuance to explanations of sequence effects. The findings are most consistent with a law-of-small-numbers logic, but the authors also find evidence of assimilation and learning. The mechanisms are not mutually exclusive. This matters because organizational decision-making biases can arise from several overlapping cognitive and social processes.
The post hoc analysis contributes to understanding how biases travel from individuals to panels. Introducers matter because they frame proposals for the group. When introducers face heavy workload or have stronger domain expertise than the rest of the panel, the sequence effect becomes stronger. This suggests that group-level outcomes can depend heavily on how information enters the discussion.
Limitations
The study examines one anonymized large multinational professional service firm. The findings may not generalize fully to other industries, smaller firms, public organizations, venture capital settings, hiring panels, or academic review panels.
The article studies R&D funding decisions, but the organization did not have detailed outcome data for funded and unfunded projects. Therefore, the authors cannot estimate the financial or innovation losses caused by the sequence effect.
The organization did not use formal R&D portfolio analysis or quantitative decision-support tools. The sequence effect may differ in organizations that use structured scoring models, portfolio optimization tools, or more formal stage-gate systems.
The study can identify the existence of a sequence effect, but it is difficult to fully separate the underlying mechanisms. The law of small numbers, contrast effects, quota thinking, and learning can be empirically hard to distinguish in real decision settings.
The study controls for several applicant characteristics, but it does not examine how sequence effects interact with gender, race, nationality, or other disadvantage-related characteristics.
The study focuses on a panel that used consensus-based group discussion. Results may differ in panels using anonymous voting, scoring systems, independent reviews, or purely algorithmic ranking.
Future research
Future research could test sequence effects in other high-uncertainty organizational decisions, such as venture capital funding, hiring, alliance formation, acquisition screening, promotion committees, and scientific grant review.
Researchers could examine whether structured scoring tools, portfolio dashboards, independent pre-meeting evaluations, or randomized evaluator-specific orderings reduce sequence effects.
Future studies could investigate whether sequence effects create measurable performance losses by linking selection decisions to downstream innovation outcomes.
Another useful direction would be to study how sequence effects interact with social evaluation biases. For example, future research could examine whether applicants from disadvantaged groups are more strongly affected by unlucky sequence positions.
Researchers could also examine whether artificial intelligence decision-support tools reduce or amplify sequence effects in panel decisions.
Finally, future research could compare different group decision structures, such as consensus panels, majority voting, averaged ratings, expert chair decisions, and independent scoring systems, to identify which structures are least vulnerable to sequence-based bias.