KK Research / Effect Reports / Report 01

Repeatability 重复性

Does the same conversation produce the same model twice?

同一段对话，重复建模会得到同一个模型吗？

One fixed 20-turn conversation, re-modelled three times independently through the full production endpoint. Stability score 0.7333, mean per-dimension standard deviation 13.33, worst-case range 55 points.

同一段 20 轮对话，经完整生产端点独立重复建模 3 次。稳定性分 0.7333，逐维标准差均值 13.33，最坏极差 55 分。

KKMatch Human Intelligence Research Team KKMatch 人类智能研究团队 · Published: 2026-09-12发布于：2026-09-12 · 6 min read阅读约 6 分钟

0.7333

Stability score稳定性分

13.33 /50

Mean per-dim σ逐维σ均值

Worst range (pts)最坏极差（分）

Independent runs独立重复次数

The Challenge

挑战

"Explainable" is a promise that collapses if the explanation is not reproducible. If the same words, submitted to the same engine, return a materially different profile each time, then no trait score can be presented to a user as a finding about them — it would be a random draw dressed up as insight. This is the first question any measurement instrument has to answer, and the one most AI personality products never publish: how much of the output is signal, and how much is run-to-run noise?

「可解释」这个承诺，在解释不可复现时就会崩塌。如果同样的话、送进同一个引擎，每次返回的画像都实质不同，那么任何特质分数都无法作为「关于这个人的一个发现」呈现给用户——那只是把随机抽样包装成了洞察。这是任何测量仪器都必须先回答的问题，也是大多数 AI 人格产品从不公布的问题：输出里有多少是信号，多少是运行间的噪声？

Why it matters for the product: an unstable dimension cannot support a confident sentence in a user-facing report. Knowing which dimensions are noisy is what lets us decide where to write a finding, where to write a range, and where to say nothing. 为什么这对产品重要：不稳定的维度支撑不起一句有把握的用户侧结论。知道哪些维度噪声大，才能决定哪里该下判断、哪里该给区间、哪里该保持沉默。

The Approach

做法

We built a probe that removes every source of variation except the engine itself.

我们写了一个探针，把除引擎本身之外的所有变化来源全部剔除。

Freeze the input. A single real transcript of 20 turns (desensitized, hash prefix 050bdc34) is selected from the training-sample store and parsed into a fixed turn list.冻结输入。从训练样本库中选出一段真实的 20 轮对话（已脱敏，哈希前缀 050bdc34），解析成固定的轮次列表。
Replay it N times. Each run goes through the ordinary production path — session start, full modelling, session complete — with all consent flags set. Nothing is mocked and no prompt is injected.重复送入 N 次。每次运行都走常规生产路径——会话开启、完整建模、会话结束——并置全部授权标记。不做任何 mock，不注入任何提示。
Read the traits back. For each of the 11 dimensions we collect the score from all N runs and compute mean, standard deviation and range.回读特质。对 11 个维度中的每一个，收集全部 N 次运行的分数，计算均值、标准差与极差。
Collapse to one number. A single stability score so that later model changes can be compared against this baseline at a glance.收敛为一个数。得出单一稳定性分，便于后续模型改动与本次基线一眼可比。

stability = 1 − mean(σ_dimension) / 50 over all 11 dimensions覆盖全部 11 个维度

The divisor 50 is the half-width of the 0–100 score scale, so the score reads directly as "what fraction of the scale is not explained by run-to-run noise". A perfectly repeatable engine scores 1.000; an engine whose dimensions swing by half the scale on average scores 0.000. Parameters for this run: --n 3, fixed transcript, 20 input turns.

除数 50 是 0–100 分制的半幅，因此该分数可直接读作「量表中未被运行间噪声解释掉的比例」。完全可复现的引擎得 1.000；各维度平均摆动半个量程的引擎得 0.000。本次运行参数：--n 3，固定文稿，20 轮输入。

The Result

结果

Three independent runs produced three different model records, as expected — a new model is written each time. Across the 11 dimensions the mean standard deviation was 13.33 points and the single worst range was 55 points. The spread is not uniform: it is concentrated in a few dimensions.

三次独立运行产生了三份不同的模型记录（符合预期——每次都写入一份新模型）。11 个维度的标准差均值为 13.33 分，最大单项极差 55 分。离散度并不均匀，而是集中在少数几个维度上。

Dimension维度	Mean均值	σσ	Range极差
communication_style	65.0	24.83	55
ai_interaction_style	68.3	20.14	45
engagement	68.3	20.14	45
reflection_style	35.0	17.80	40
uncertainty_response	58.3	15.46	35
openness	53.3	12.47	30
emotional_expression	66.7	11.79	25
decision_style	36.7	9.43	20
conflict_style	30.0	8.16	20
trust_disclosure_pattern	15.0	4.08	10
directness	11.7	2.36	5

What is stable. directness (σ 2.36, range 5) and trust_disclosure_pattern (σ 4.08, range 10) barely move at all. If a user is told something about their directness, that sentence will survive a re-run.

稳定的部分。 directness（σ 2.36，极差 5）与 trust_disclosure_pattern（σ 4.08，极差 10）几乎不动。如果告诉用户某个关于其「直接性」的结论，这句话在重跑后依然成立。

What is not. communication_style ranged from 30 to 85 across three runs of the same words — a swing wider than most people span. ai_interaction_style and engagement moved identically (σ 20.14, range 45), which is itself informative: they are being driven by the same feature and should not be treated as independent evidence.

不稳定的部分。 communication_style 在同一段话的三次运行中从 30 摆到 85——摆动幅度比大多数人的真实跨度还大。ai_interaction_style 与 engagement 完全同步变动（σ 20.14，极差 45），这本身也是信息：它们由同一个特征驱动，不应被当作两份独立证据。

What It Means

含义

A single model output is a sample, not a verdict. Three consequences follow, and we adopted all three:

单次模型输出是一次抽样，不是一个判决。由此得出三个后果，我们全部采纳：

Confidence is now a first-class field. Every trait carries an explicit confidence value, and every model carries an overall_confidence. Low-repeatability dimensions must earn confidence from evidence, not from the model being asked once. (Report 02 measures the result of this.)
置信度已成为一等字段。 每个特质都带显式置信度，每份模型都带 overall_confidence。可重复性低的维度必须靠证据赢得置信度，而不是靠「只跑一次」。报告 02 量化了这一改动的结果。
No user-facing sentence is written from one run. Dimensions with σ above roughly 15 (communication_style, ai_interaction_style, engagement in this reading) are reported as ranges or withheld, not as point findings.
不基于单次运行写任何用户侧结论。 σ 高于约 15 的维度（本次读数中的 communication_style、ai_interaction_style、engagement）以区间形式呈现或不予呈现，不给点结论。
The baseline is now a regression gate. This reading is stored as the first baseline entry, so a future model change that drags the stability score down is caught by comparison rather than by opinion.
基线已成为回归闸门。 本次读数存为首条基线记录，未来任何把稳定性分拉低的模型改动都会被对比捕获，而不是靠主观判断。

The honest framing: 0.7333 is a mediocre number, and we publish it as such. It says the engine is right far more often than it is wrong, but that for some dimensions a single reading is closer to an opinion than a measurement. That is the state of the instrument today, and this report exists so it can be tracked. 诚实的表述： 0.7333 是一个中等偏下的数字，我们照实公布。它说明引擎在多数情况下是对的，但对某些维度而言，单次读数更接近一个看法而非一次测量。这就是仪器当下的状态，这篇报告的存在就是为了让它可被追踪。

Reproduce it

复现方式

python _stability_probe.py --n 3 --tag repeatability-2026-09-12
Reads the fixed transcript from data/mirror_training_samples.jsonl; writes data/stability_baseline.json and appends to its history array. Add --cleanup to remove the probe's model records afterwards. The probe uses the reserved user id _stability_probe so its records are always separable from real users. 从 data/mirror_training_samples.jsonl 读取固定文稿；写出 data/stability_baseline.json 并追加到其 history 数组。加 --cleanup 可在事后删除探针产生的模型记录。探针使用保留用户 id _stability_probe，其记录始终可与真实用户区分。

Limitations

研究局限

N=3 is small. With three runs a standard deviation is an indication, not an estimate. A wider reading (N≥10) would narrow the uncertainty around each σ; the current cost is that each run performs a full model generation.
N=3 样本量小。 三次运行得出的标准差只是迹象，不是估计。更大的读数（N≥10）会收窄每个 σ 的不确定度；当前代价是每次运行都要执行一次完整模型生成。
One transcript, one speaker. A 20-turn sample from one person cannot tell us whether communication_style is unstable for everyone or unstable for this style of speech. Per-dimension noise may be input-dependent.
单一文稿、单一说话人。 来自一个人的 20 轮样本无法说明 communication_style 对所有人都不稳定，还是只对这类说话方式不稳定。逐维噪声可能是输入相关的。
High σ is not automatically an error. A dimension may legitimately have several defensible readings for ambiguous speech. This report measures agreement between runs; it does not decide which run was right.
高 σ 不一定等于错误。 面对含糊的表述，一个维度可能确实存在多个都站得住的读数。本报告衡量的是运行间的一致性，不判定哪一次是对的。

What would prove this report wrong Falsifiable

什么能推翻本报告可证伪

If a re-run with N≥10 returns a mean σ well below 5 for all dimensions, the poor stability reading was an artifact of small N and should be retracted.
若以 N≥10 复跑后，所有维度的 σ 均值显著低于 5，则本次「稳定性差」的结论属小样本假象，应撤回。
If communication_style's 55-point range cannot be reproduced on a second transcript, the instability is specific to this input rather than a property of the engine.
若 communication_style 的 55 分极差在第二段文稿上无法复现，则该不稳定性属于此输入的特例，而非引擎属性。
If fixing the random seed makes σ collapse to near zero while the score values stay the same, the noise was sampling, not modelling — the fix is trivial and the report is superseded.
若固定随机种子后 σ 塌缩至近零而分数取值不变，则噪声来自采样而非建模——修复很简单，本报告即被取代。

Back to Effect Reports返回效果实测报告