Large language models - Lexical support - Concept access

Who Gets a Token,
and What Does It Carry?

Unequal name support and concept access in large language models

Mir Tafseer Nayeem Davood Rafiei

Department of Computing Science, University of Alberta

Paper Code Dataset
NameTrace overview: names enter language models with unequal lexical access; the framework measures task-axis concept accessibility, within-stratum persistence, transfer to unseen names, and downstream leverage.
Study overview NameTrace follows name-surface support from tokenizer access to internal task concepts, transfer across names, and later constrained choices.
Overview

Matched names may still be different model inputs

Names are often used as controlled social cues in evaluations of language models. But a tokenizer may encode one name as a single token and another as several subword pieces. NameTrace asks whether this unequal lexical support is merely a vocabulary difference or remains visible in task-relevant model computation.

The study introduces NAMETRACE, a model-native framework that measures concept accessibility from a model's own probabilities over task-specific adjective axes. It looks before open-ended generation, uses continuous weights to distinguish stronger from weaker concepts, and does not require reference answers or an external judge.

Method

From token access to downstream choices

01

Map lexical access

Measure exact single-token access for 497,583 first-name surfaces across 12 LLM-associated tokenizers.

02

Match name pairs

Compare atomic and short-fragmented names within eight race/ethnicity- and gender-associated strata, matching on observed name properties.

03

Read task concepts

Score model probabilities over weighted adjective axes for fellowship, hiring, clinical concern, and lending.

04

Probe transfer and leverage

Test whether development-name patterns predict unseen-name gaps and whether task-direction edits shift later constrained choices.

Findings

Lexical support predicts task-relevant accessibility

RQ1: Who gets direct lexical access?

Direct access is selective and model dependent: 23,095 names are atomic in at least one tokenizer, while only 4,052 are atomic in all 12. The paper's tokenizer comparison shows how sharply access counts vary by model, alongside aggregate gender- and race/ethnicity-associated counts.

Three-panel tokenizer comparison. Among the metadata-annotated names shown, atomic counts range from 4,271 for DeepSeek V3.2 to 18,170 for Aya Expanse 32B. The other panels break counts down by gender- and race/ethnicity-associated name metadata.
RQ1 - Tokenizer access Atomic first-name counts differ by tokenizer; the companion panels show the aggregate metadata composition of those vocabularies.

In a higher-frequency, high-confidence subset of 7,469 names, atomic access was 49.8% for male-associated versus 25.7% for female-associated names. Across race/ethnicity-associated metadata, access ranged from 17.6% for NH Black-associated names to 47.2% for NH White-associated names (46.4% for Asian/PI-associated names).

RQ2: Does support appear in internal task concepts?

On held-out names, NameTrace finds positive pooled atomic-minus-fragmented accessibility gaps on all four task axes:

Held-out atomic-minus-short-fragmented concept accessibility gaps
Task axisWeighted gap95% confidence interval
Fellowship / promise0.131[0.096, 0.168]
Hiring / competence0.072[0.055, 0.091]
Clinical assessment / concern0.059[0.048, 0.072]
Lending / trustworthiness0.051[0.041, 0.061]

A positive gap means greater task-aligned concept accessibility for the atomic name. These are weighted scores, not probabilities.

Results chart: pooled weighted gaps are positive for fellowship, hiring, clinical concern, and lending; the heatmap shows positive gaps across all eight name-metadata strata.
Across tasks and strata The support-linked gap is positive across the four pooled task axes and all eight name-metadata strata.

RQ3: Does support transfer and influence later choices?

Yes: patterns learned from development names predict gaps for unseen names, and interventions along measured task directions shift later constrained choices.

Transfers to unseen names

Development-name support priors account for 72.9% to 96.6% of the pooled held-out gap, depending on task.

Has downstream leverage

Interventions along measured task directions shift later constrained choices in all three primary model families.

Varies by model and task

Effects differ in magnitude and sometimes direction across architectures, layers, and training stages; they are not uniform model behavior.

Two-panel chart showing that a support prior learned on development names predicts held-out accessibility gaps, and task-direction edits shift later choices in Qwen, Llama, and Ministral models.
Transfer and intervention The learned support prior predicts unseen-name gaps; task-direction edits shift later choices in three evaluated model families.
Interpretation

Lexical comparability is part of evaluation design

The results show that demographic matching alone does not guarantee comparable model inputs. They support treating lexical access as a measurable source of variation when interpreting name-based evaluations.

Resources

Code and dataset

Explore the implementation and the NameTrace dataset.

Code on GitHub Dataset on Hugging Face

A public paper link will be added when available.