When eight of ten Delphi panelists have finished

Eight of ten panelists have finished the first round. The other two have started, but their conversations are incomplete. The host can see the difference. Should everyone wait, or should the panel move on with the contributions it has?

Imagine a fictional panel advising a city on when to open extra cooling centres during a heatwave. Its members bring expertise in public health, service operations and access for vulnerable residents. One wants forecast temperatures to trigger action; another thinks overnight heat and access to transport deserve more weight. We will use this example throughout. It is an illustration of the process, not a report from a customer panel.

For Harmonica, that eight-of-ten case requires decisions about who controls progression, whose contributions enter the feedback, and what happens to unfinished conversations. A prompt asking an AI to facilitate Delphi cannot make those decisions enforceable. The update had to follow the method through the sessions, context controls and handoffs that make a panel work.

Delphi grew out of work at RAND on eliciting expert judgment. Dalkey and Helmer’s 1963 paper describes repeated questionnaires with controlled feedback. Dalkey’s 1969 report identifies anonymity, controlled feedback between rounds and a statistical group response as central features. The design sought to reduce the influence of dominant individuals and pressure to conform. Participants could reconsider earlier answers after receiving the previous round’s results; they did not need a live discussion with the rest of the panel.

Harmonica uses a conversational, qualitative adaptation with three rounds: independent elicitation, reconsideration against aggregate feedback, and examination of convergence and remaining disagreement. Three rounds and the 80% readiness threshold are our choices for this adaptation. The qualitative feedback also differs from the numerical aggregation used in RAND’s experiments. Those differences need to remain visible when we describe what we have built.

In the cooling-centre example, the first round asks each panelist for their own judgment and its basis. Harmonica gives them a private conversation in which the facilitator can ask for a concrete indicator, explore uncertainty or clarify a proposed condition. The conversation should not introduce somebody else’s proposed trigger while the panelist is still developing their own answer.

We found that this independence needed a boundary in the software. Harmonica can normally bring relevant contributions from other conversations into a session, a feature we call cross-pollination. For recognised Delphi chains, the server now withholds that ambient peer context in every round. The host’s explicit materials remain available, and later rounds receive their controlled feedback through a separate route. A neutral instruction in the prompt would not have been enough if another part of the system could quietly supply current-round peer contributions.

Finishing the individual conversation also needed attention. A panelist who says they are done should receive a recap and an accuracy check, with a chance to correct it before confirming completion. Harmonica’s Delphi closing policy now calls for that recap immediately, rather than another attempt to fill a missing output. If the panelist has not supplied a threshold, does not know it or declines to answer, the recap must leave that state explicit. An AI-generated completion should not turn an unanswered question into an expert judgment.

Once eight eligible panelists have completed that process, our fictional round reaches its 80% threshold. It becomes ready for the host to advance. The host can also wait: the two unfinished responses may contain expertise the panel needs. Reaching the threshold does not automatically make that judgment for them. If all ten finish, advancement can happen automatically, subject to the required feedback being ready and the other progression checks passing. Completion of ten conversations says nothing about whether ten people agree.

Suppose the host chooses to move on at eight. The next round’s aggregate feedback uses completed, eligible responses only. Its instructions require an anonymised account of positions and reasons, including persistent tensions, rather than attributed contributions. The two partial conversations do not enter that feedback. The system records the source cohort and a fingerprint of the content used, then checks that the summary still matches its source basis before handing it on. If another panelist finishes while feedback is being prepared, an earlier summary cannot silently stand in for the changed set of completed responses.

After the handoff, that round’s feedback cohort stays fixed. Unfinished conversations become read-only, with drafts retained and a route to the current round. Their later activity must not quietly rewrite the feedback other panelists have already received. The same concern governs withdrawals: a participant’s request to leave does not immediately shrink the panel size used for quorum. The host reviews the change and its progression consequence. Otherwise eight completed responses could become eight of eight and trigger automatic advancement without an intentional host decision.

The next conversation needs continuity as well as a boundary. In Round 2, the cooling-centre panelist should be able to consider the aggregate alongside their own earlier reasoning about overnight heat. Harmonica has separate paths for prior-round summaries and that participant’s own earlier answers. The original host question and background also need to survive the move into a new session. If personal history is unavailable, the conversation cannot be treated as evidence that this continuity worked. Supplying the aggregate alone would leave the facilitator asking for reconsideration without the earlier position it is meant to revisit.

In the final round, the useful question may be whether an apparent difference has narrowed or merely changed wording. Our fictional panel might broadly support earlier opening while disagreeing about which warning signal should trigger it. The facilitation instructions ask for that remaining disagreement to be recorded. Agreement cannot establish accuracy by itself: Dalkey’s analysis explicitly distinguishes convergence from improvement in estimates. Our qualitative adaptation likewise needs evidence about the usefulness of its results, beyond whether panelists move closer together.

We also wanted to make the feedback inspectable after the conversation. Each chain run keeps a snapshot of its template. Template and platform defaults for round-summary instructions are pinned for the run; explicit host edits remain possible and are recorded with the effective instructions used. The saved summary carries an instruction receipt, while the completed-response handoff records its source basis. These records let a reviewer ask which contributions and instructions produced a particular piece of feedback. They do not establish that the feedback represents every contribution fairly.

The Open Facilitation Library’s Conformance Model gives this review a portable structure. It separates method essentials from an implementer’s conversational choices, the runtime mechanisms and participant outcomes. In this case, controlled feedback is a method concern; the immediate conversational recap is an interaction choice; source receipts and progression controls are Harmonica mechanisms. Whether people felt accurately represented, could reconsider freely and produced a useful result remains an outcome question.

Four parts of the workflow illustrate how a requirement connects to a mechanism and its evidence:

  • Independent elicitation and controlled feedback: separate conversations withhold ambient peer context and supply prior-round aggregates separately. Source and context checks cover specified boundaries; they do not establish anonymity or faithful facilitation in every panel.
  • An intentional asynchronous cutoff: the host chooses from 80%, while 100% permits automatic progression subject to prerequisites, with completed-only feedback. Local handoff rehearsals cover host and automatic transitions. They do not determine whether a host chose a good cutoff.
  • Continuity and honest completion: own-history context and original brief carryover support reconsideration; recap and confirmation support accurate closing. Implementation and bounded checks cover particular paths; the participant’s experience needs separate observation.
  • Inspectable feedback: frozen run configuration, pinned defaults, audited host edits and source/instruction receipts identify the recorded basis. A receipt cannot certify the quality of a summary or the panel’s conclusions.

OFL and Harmonica have different jobs in this work. OFL can describe requirements and adaptations in a form another tool could use or challenge; Harmonica implements a particular workflow and produces evidence about it. The public OFL Delphi method specification, worked conformance profile and Harmonica adapter are still drafts. The specification declares that all panelists must submit, so Harmonica’s host-controlled 80% route is an explicit adaptation, not a literal implementation of that completion rule. The profile also records an absence of source and practitioner review for its criteria, and the adapter’s historical evidence is not a current certificate. This update does not turn those artifacts into a reviewed standard or a claim of full Delphi conformance. The same attention to process also shaped our recent SWOT update, with a different session structure and handoff.

The changes have a substantial engineering trail: nine explicitly Delphi-focused pull requests covered isolation, progression, participation, closing behaviour, bounded release checks, rehearsal and activation. On 2 November 2026, we activated the reviewed template for new runs and verified its seven intended field changes. The summary settings, unrelated configuration and all 12 existing run snapshots were preserved. Existing runs therefore retained their original configuration.

The 2 November activation verified deployment and configuration. Local and CI checks, plus bounded model checks recorded in the delivery trail, support particular implementation claims; the activation operation itself created no participant session or generated feedback. That release evidence does not establish that the activated workflow produces better panel judgments.

For the cooling-centre host, moving on with eight responses still carries a cost: the missing two may see something the others have missed. Waiting carries a cost too. The updated workflow leaves that choice with the host, makes the completed-response cutoff explicit and preserves a basis for reviewing what the next round received. A useful panel result would still have to earn its value through the quality of the contributions, the feedback and the decisions people make with them.

Book a 30-minute call

Loading calendar…