KK Research / Effect Reports / Report 02
Honest Confidence 置信诚实性74 real conversations, 35 models, and a 40% fallback rate
74 场真实对话、35 份模型，以及 40% 的降级率
Every one of the 385 trait slots was filled — yet 40% of the models returned nothing except an explicit "insufficient conversation data" flag, and only 17% reached an overall confidence of 0.6. Trait coverage is not trait quality.
385 个特质槽位全部被填充——然而 40% 的模型除了显式的「对话数据不足」标记外什么都没给出，仅 17% 达到 0.6 的整体置信度。特质覆盖率不等于特质质量。
74
Real sessions真实会话
35
Models produced产出的模型
100%
Trait slot coverage特质槽位覆盖率
0.400
Median confidence置信度中位数
40.0%
Explicit fallback rate显式降级率
The Challenge
挑战
An engine that always returns a full profile looks more impressive than one that sometimes says "I don't know". It is also more dangerous. Real conversations are wildly unequal in length: in our data the median session is 3 turns, and 47.3% are a single turn — a greeting, or a lost visitor. An engine optimised to always produce output will happily generate eleven confident-looking numbers from one sentence.
一个永远返回完整画像的引擎，看起来比一个有时会说「我不知道」的引擎更体面。但它也更危险。真实对话的长度极不均衡：在我们的数据里，会话轮数中位数是 3 轮，47.3% 只有 1 轮——一句问候，或一个走失的访客。一个以「永远产出」为目标的引擎，会乐于从一句话里生成十一个看起来很有把握的数字。
The question this report answers is therefore not "how good are the scores" but "does the engine say so when it does not have enough to go on?" That is the load-bearing claim behind not fabricating user profiles.
因此本报告要回答的不是「分数有多好」，而是「在信息不足时，引擎会不会主动说出来？」 这正是「不编造用户画像」这一承诺的承重墙。
The Approach
做法
- Take the full real corpus. Every session in the live store is included — 74 of them, across text, voice and video modes. No record is filtered out for being too short, which is exactly the subset that would flatter the numbers.取全量真实语料。 线上存储中的每场会话都纳入——共 74 场，覆盖文本、语音、视频三种模式。不因「太短」剔除任何记录——而恰恰是这部分记录会对指标最有利。
- Read the generated models, not the events. Confidence figures are taken from the model records themselves, because the model store is authoritative while the client event stream is best-effort instrumentation.读生成的模型，而非事件。 置信度数字取自模型记录本身——模型存储是权威来源，而客户端事件流只是尽力而为的埋点。
- Count slots, not sessions. Each model has 11 dimensions, giving 35 × 11 = 385 slots. Coverage and confidence are reported at slot level so that a single missing dimension cannot hide inside a "model-level" average.数槽位，而不是数会话。 每份模型 11 维，共 35 × 11 = 385 个槽位。覆盖率与置信度按槽位统计，这样单个缺失维度无法藏在「模型级」平均值里。
- Separate "filled" from "meaningful". A slot counts as covered if it holds any value; it counts as earned only if the engine produced real evidence rather than its fallback text. The gap between the two is the headline of this report.把「填了」与「有内容」分开。 槽位只要有任何取值就算已覆盖；只有引擎给出真实证据、而非兜底文本时，才算已赢得。两者之间的缺口，就是本报告的核心结论。
The Result
结果
Coverage is perfect; usefulness is not. All 385 slots across all 35 models were filled — a 100% coverage rate. But the distribution of confidence tells a very different story.
覆盖率满分；可用性不是。 35 份模型的全部 385 个槽位都被填充——覆盖率 100%。但置信度的分布讲了另一个故事。
| Reading读数 | Value数值 | Share占比 |
|---|---|---|
| Models whose every dimension says only "insufficient conversation data"所有维度都只返回「对话数据不足」的模型 | 14 / 35 | 40.0% |
| Models whose every dimension sits at the neutral 50所有维度都停在中性值 50 的模型 | 15 / 35 | 42.9% |
| Models at overall confidence ≥ 0.6整体置信度 ≥ 0.6 的模型 | 6 / 35 | 17.1% |
| Models at overall confidence ≤ 0.35整体置信度 ≤ 0.35 的模型 | 11 / 35 | 31.4% |
| Trait slots sitting at exactly 0.40 confidence — the fallback bucket置信度恰好为 0.40 的特质槽位——兜底档 | 153 / 385 | 39.7% |
| Sessions completed (of 74)完成的会话（共 74 场） | 34 / 74 | 45.9% |
Overall confidence across the 35 models: median 0.400, mean 0.397, 90th percentile 0.650, maximum 0.800.
35 份模型的整体置信度：中位数 0.400，均值 0.397，90 分位 0.650，最高 0.800。
Confidence is not uniform across dimensions
置信度在各维度间并不均匀
Where the engine does commit, it commits unevenly. Observable, text-carried dimensions earn roughly 50% more confidence than the ones that depend on inference about a person's inner state.
当引擎确实给出判断时，它的把握也是不均匀的。可观察的、由文字直接承载的维度，比依赖内在状态推断的维度所赢得的置信度高出约五成。
| Dimension维度 | Mean conf.平均置信 | σσ | Relative相对水平 |
|---|---|---|---|
| communication_style | 0.460 | 0.225 | |
| directness | 0.457 | 0.232 | |
| openness | 0.450 | 0.194 | |
| engagement | 0.439 | 0.199 | |
| ai_interaction_style | 0.437 | 0.199 | |
| trust_disclosure_pattern | 0.410 | 0.171 | |
| emotional_expression | 0.393 | 0.155 | |
| reflection_style | 0.376 | 0.196 | |
| decision_style | 0.347 | 0.231 | |
| conflict_style | 0.339 | 0.243 | |
| uncertainty_response | 0.334 | 0.173 |
The pattern is coherent rather than random:
communication_style and directness — things a reader can actually see in a transcript — lead at 0.46, while uncertainty_response and conflict_style — which require seeing a person under pressure, something a three-turn chat rarely supplies — trail at 0.33.这个分布是有内在逻辑的，而非随机：
communication_style 与 directness——在文稿中真能看得见的东西——以 0.46 领先；而 uncertainty_response 与 conflict_style——需要观察一个人处在压力下的表现，而三轮回合很少提供这种机会——以 0.33 垫底。Context: how sessions actually reach the engine
背景：会话如何抵达引擎
74 landing views落地页浏览
→
73 starts开始
→
36 consent granted授权通过
→
34 mirror started进入对话
→
34 sessions completed会话完成
This funnel is taken from the client event stream, which is best-effort instrumentation: session-completion events (35) and completed-session records (34) differ by one, and consent events (36) exceed mirror-start events (34). We publish the raw stream rather than silently reconciling it, and we do not use these counts as headline figures.
该漏斗取自客户端事件流，属尽力而为的埋点：会话完成事件（35）与已完成会话记录（34）相差 1，授权事件（36）多于进入对话事件（34）。我们公布原始事件流，而不是悄悄抹平差异，也不把这些计数当作主结论。
What It Means
含义
The 40% fallback rate is the most important number we have published, and it is not a good one. Here is what we take from it.
40% 的降级率是我们公布过的最重要的数字，而且它并不好看。我们的解读如下。
- The engine prefers silence to invention. Given one-turn sessions, it returns a flagged neutral profile instead of generating eleven plausible-sounding traits. That is the designed behaviour working — and it is the reason a 100% coverage figure is not advertised as a quality claim.
- 引擎选择沉默而非编造。 面对只有 1 轮的会话，它返回一个带标记的中性画像，而不是生成十一个听着合理的特质。这正是设计预期在生效——也正是我们把 100% 覆盖率当作质量结论来宣传的原因所在（我们不这么做）。
- The real bottleneck is conversation length, not modelling. 47.3% of sessions are one turn and only 20.3% reach four turns. Improving the model cannot fix a corpus that never gave it anything to model.
- 真正的瓶颈是对话长度，不是建模能力。 47.3% 的会话只有 1 轮，仅 20.3% 达到 4 轮。语料本身没给模型任何可建模的东西，改进模型解决不了这个问题。
- Confidence now sorts the product surface. Dimensions at the bottom of the confidence table are shown as open questions rather than statements, and the UI makes the difference visible instead of flattening it.
- 置信度现在决定了产品展示方式。 置信度表末端的维度以「待确认问题」而非「结论」的形式呈现，界面让这一差别可见，而不是把它抹平。
- Four turns is the threshold we now design around. The 12 models built from sessions of four turns or more average 0.468 overall confidence, against 0.403 for the 16 models built from two turns or fewer. The dialogue engine's nudges exist to push sessions across that line.
- 「4 轮」已成为我们的设计阈值。 来自 4 轮及以上会话的 12 份模型，整体置信度均值 0.468；而来自 2 轮及以下会话的 16 份模型为 0.403。对话引擎的引导语正是为了把会话推过这条线。
The honest framing: if we reported only the 100% coverage figure, this engine would look finished. Reporting the 40% fallback rate instead shows a measurement system that is calibrated to admit its own ignorance — which is the only state in which its confident readings deserve to be believed.
诚实的表述： 如果我们只公布 100% 覆盖率，这台引擎会显得已经完工。改为公布 40% 降级率，展现的是一套被标定为「肯承认自己无知」的测量系统——而只有在这种状态下，它那些有把握的读数才值得被相信。
Reproduce it
复现方式
Pure aggregation over the JSON stores — no probe required. Count models in
data/mirror_models.json, sum their traits[].confidence values, and count sessions in data/mirror_sessions.json. The fallback rate is the share of models whose every trait carries evidence text beginning "Insufficient".
对 JSON 存储的纯聚合，无需探针。统计 data/mirror_models.json 中的模型数，汇总其 traits[].confidence；统计 data/mirror_sessions.json 中的会话数。降级率 = 所有特质的证据文本均以 "Insufficient" 开头的模型占比。
Limitations
研究局限
- Small corpus. 35 models is a calibration reading. A single burst of genuine long sessions would move every percentage on this page substantially.
- 语料量小。 35 份模型只是一个标定读数。一批真实的长会话就会让本页的每个百分比显著变化。
- The corpus is not a random sample of users. Early traffic skews to testers and internal evaluation, which likely inflates both the fallback rate (many deliberate one-turn probes) and the low-confidence tail.
- 语料不是用户的随机样本。 早期流量偏向测试者与内部评测，这很可能同时抬高了降级率（大量刻意为之的单轮探针）与低置信长尾。
- Confidence is self-reported by the engine. This report shows that the engine's confidence is internally consistent and predictably distributed. It does not show that confidence is well-calibrated — that needs an external criterion (see Report 01's limitations and the accuracy disclaimer on the reports index).
- 置信度由引擎自报。 本报告表明引擎的置信度是内部自洽且分布可预期的，但并未表明它是良好标定的——那需要一个外部效标（参见报告 01 的局限与报告中心页的准确率免责说明）。
- Event counters are best-effort. As shown in the funnel, the instrumentation stream does not perfectly reconcile with authoritative stores, so event-derived ratios should be read as approximate.
- 事件计数为尽力而为。 如漏斗所示，埋点流与权威存储并不完全对齐，因此由事件推导的比率应视为近似值。
What would prove this report wrong Falsifiable
什么能推翻本报告 可证伪
- If a manual review finds that fallback-flagged models actually contain distinct, input-specific trait values, then the flag is cosmetic and the 40% figure overstates how often the engine abstains.
- 若人工复核发现被标记为降级的模型实际上含有因输入而异的具体特质取值，则该标记只是形式，40% 这个数字夸大了引擎弃权的频率。
- If overall confidence ≥ 0.6 rises above 60% once sessions of eight turns or more dominate the corpus, the low confidence was a data-supply problem and not a modelling ceiling — which would confirm the bottleneck claim but retire the "uncertain engine" framing.
- 若当 8 轮以上会话成为语料主体后，整体置信度 ≥ 0.6 的比例升至 60% 以上，则说明低置信是数据供给问题而非建模天花板——这会印证「瓶颈在语料」的判断，同时使「引擎不确定」的表述失效。
- If the confidence ranking between dimensions reverses when a different cohort is used, then the apparent ordering is a property of these 35 models and must not be presented as a general finding.
- 若换一批样本后维度间的置信度排序发生反转，则该排序只是这 35 份模型的属性，不得作为普遍结论呈现。
Related: Report 01 — Repeatability · Report 03 — Signal coverage · KH-001
相关： 报告 01 — 重复性 · 报告 03 — 通道到位率 · KH-001