Vector similarity and semantic grouping errors

First define the relation to be found

For a question map, let the positive label mean that two phrasings require the same information under the same material conditions, with answers interchangeable in both directions. This requirement is narrower than sharing a topic.

‘What is T500's nominal volume?’ and ‘How many millilitres is T500's nominal capacity?’ may represent one task. ‘How much liquid was used in the test?’ concerns a different quantity. In the illustrative dossier this is 450 ml, compared with a nominal 500 ml. The same product and unit of measurement do not make the answers interchangeable.

Document retrieval can use a broader relation: a passage merits inspection as possible evidence. For navigation, different questions can share a topic such as ‘Temperature retention’. Combining questions for coverage measurement requires another criterion. These decisions are best retained separately.

Defining the relation also determines how the evaluation set is annotated. Labels such as ‘similar’ and ‘dissimilar’, without a rubric, leave each editor to solve a different task. Threshold tuning would then fit a number to inconsistent decisions.

What lies between a string and a score

A model converts text into a numerical representation — a vector. A chosen function then compares two representations. Sentence Transformers provides cosine similarity, dot product and distance-based scores, among other options. Their numerical scales must not be mixed.

The result belongs to a specific configuration: model, language, text unit and input preparation. Context matters too. A short ‘how long does it last?’ without a named object does not specify the same task as a full question about testing T500 at a particular temperature.

If a model uses special instructions or different preparation methods for queries and documents, these must also be recorded. Otherwise, repeating the comparison changes the task conditions. Truncating a long passage may remove a negation or qualification before similarity is even calculated.

Domain records from linguistic analysis help retain parameters and conditions, but do not automatically prove equivalence. They can support an additional check: which number denotes nominal capacity, which action a restriction applies to, and which entity is named.

How to interpret cosine similarity

For two non-zero vectors, cosine similarity is defined as follows:

cos(u, v) = (u · v) / (‖u‖ × ‖v‖).

The numerator is the dot product; the denominator is the product of the vector lengths. The value depends on the angle between their directions. If both vectors are normalized to unit length, the denominator equals one and cosine similarity equals the dot product.

The mathematical range is −1 to 1. Zero means perpendicular directions in the chosen space. A negative value should not automatically be interpreted as a contradiction between texts, nor 0.9 as a 90% probability that they ask the same question. That interpretation would require a separate model and validation.

Take the unit vectors A = (1; 0), B = (√3/2; 1/2), C = (1/2; √3/2). Their angles to the horizontal are 0°, 30° and 60°. Therefore cos(A, B) = cos(B, C) = √3/2 ≈ 0.866, while cos(A, C) = 1/2. This is pure geometry; no texts have been assigned to these points.

A pairwise threshold and a grouping rule are separate settings

In this example, we can connect all pairs with cosine similarity above 0.8 and take the graph's connected components. A, B and C then belong together, although A–C fails the threshold. Passing a pairwise threshold is not transitive. Membership of the same connected component is transitive, but does not mean that every pair within it passes the threshold.

A chain can be useful for a thematic overview: intermediate topics create a connected section. For interchangeable questions, however, it can hide differences in the required answer. The resulting group must be checked against the chosen requirement, rather than merely counting its internal edges.

Different algorithms impose different rules. In the DBSCAN documentation the parameter eps limits the neighbourhood, not the maximum distance between all points in a cluster. The result also depends on density and min_samples. Our three points illustrate a connected component; no claim is made about their outcome under an arbitrary DBSCAN configuration.

Options include checking every pair in a small group, comparing with a representative, or applying domain constraints. Each has its own failure modes. An overly general representative may hide incompatibility between the most divergent questions. Full pairwise checking costs more and also depends on evaluator quality. The chosen method should be tested on control groups, including chains that transition between tasks.

Separate candidate retrieval from final acceptance

For a large collection, it is practical first to retrieve a limited candidate list for each string, then examine pairs in detail. The first stage aims to find the necessary comparisons; the second concerns the quality of the decision on each pair. Candidate count is chosen for the task and checked for losses of required cases.

In the Retrieve & Re-Rank approach, potentially relevant documents are retrieved first; a Cross-Encoder then jointly evaluates the query and each candidate. Adapting this separation to question grouping is our own methodological choice. Document relevance and question duplication remain different relations.

If a required string never enters the list, reranking that list cannot recover it. Candidate-retrieval recall must therefore be checked separately on labelled cases. Good second-stage performance on the pairs it receives does not describe losses at the first stage.

At the acceptance stage, domain roles and conditions can be compared, or a trained pairwise model can be used. The Cross-Encoder catalogue distinguishes STS, relevance and logical-relation tasks. Choosing a model for interface convenience without checking its task does not justify treating questions as equivalent.

Logical entailment can also be directional. Six tubes entail the presence of tubes, but the presence of tubes does not entail their number. Symmetric cosine similarity does not distinguish that direction. Even correct entailment relative to the source text does not establish the truth of the original product information.

Select a threshold for a labelled task

Threshold selection starts with a rubric and examples. Editors decide which pairs ask the same question, record material conditions and resolve disagreements. The set includes similar questions with different answers and different phrasings of one task. Pairs involving frost versus room temperature, or nominal capacity versus test fill volume, are needed alongside ordinary synonyms.

Split the material into tuning and final evaluation sets. Closely related paraphrases of one original task should stay in the same split; otherwise, a familiar case appears on both sides. If transfer to another category or language is required, evaluate that transfer specifically.

BinaryClassificationEvaluator uses labelled pairs and 0/1 labels. The author's task determines what a label means. Here, the positive class means ‘the same answer task’. In article 09, the positive class was the presence of an error; identical metric formulas therefore have different domain meanings here.

For merging, precision is the proportion of correct matches among accepted pairs; recall is the proportion of accepted positive pairs among all reference-positive pairs in the chosen evaluation set. A false merge creates an extra positive pair; a missed match yields a negative decision on a required pair. A ready-made fraction must not conceal zero denominators or disputed reference labels. If only pairs supplied by candidate retrieval are checked, the result applies to that conditional set. Evaluating the full process must also account for positive pairs lost before pairwise checking.

Choose the threshold on the tuning split using a defined criterion. For example, first limit the acceptable number of false merges, then seek more correct matches among the eligible configurations. The criterion must reflect the consequences: losing a compatibility distinction may matter more than an extra check of similar questions.

With scores fixed, raising the threshold reduces the set of accepted pairs. A required match may disappear along with a false merge. The change in precision depends on the case mix; moving the threshold alone does not guarantee an improvement.

After selection, freeze the configuration and evaluate it on the held-out split. Examine errors across categories and the conditions under which scores are unstable. A successful threshold on one set is not universal, and tuning a threshold does not turn the original score into a probability of correctness.

What grouping should leave behind

A verifiable result retains the source strings, grouping criterion, main question, material conditions, disputed candidates and evidence for decisions. Within-group similarity can help navigate these materials, but cannot replace them.

Validation covers several levels: whether the necessary candidates were found, whether pairs were accepted correctly, and whether the grouping rule hid differences in the final group. Without these checks, group count says little about quality: shortening the list can mean removing duplicates or losing important questions.

If a chain leads from thinning fresh paint to restoring dried paint, it is useful to retain the shared topic while separating the answer tasks. A system need not produce a flat partition for every purpose. Choose the map's level according to what will be measured or written next.

The phrasings found do not ensure that a page will be visible in search either. Search channels obtain data, select results and display answers in different ways. Page accessibility, the structure of product information and the limits of conclusions drawn from impressions or citations therefore need separate examination.