Skip to content

Context-Aware Language Detection

Overview

contextual-langdetect uses document-level context to improve language detection accuracy in mixed-language corpora. It's particularly useful when you have a strong prior belief about the number and distribution of languages in your corpus.

How It Works

  1. First Pass: Each sentence is analyzed independently using fast-langdetect
  2. Context Building:
  3. Identifies primary languages (those appearing in >10% of sentences)
  4. Builds confidence scores for each detected language
  5. Ambiguity Resolution:
  6. Uses document context to resolve ambiguous cases
  7. Applies special case handling for known detection challenges

Example: Chinese Text Processing

Consider a corpus of primarily Chinese text. Individual sentences might be misidentified:

sentences = [
    "中国是一个伟大的国家",  # Clearly Mandarin
    "你好",                 # Could be detected as zh/ja/wuu
    "谢谢大家",             # Could be ambiguous
    "我很高兴见到你"         # Could be detected as Japanese without kana
]

The library will: 1. Detect that Mandarin is the primary language 2. Notice that some sentences are detected as Japanese but lack kana characters 3. Resolve these ambiguous cases in favor of Mandarin

Special Cases

Japanese vs Chinese

  • Japanese text typically contains kana (hiragana/katakana) characters
  • If a sentence is detected as Japanese but contains no kana, in a Chinese context it's likely Chinese

Wu Chinese (wuu)

  • Wu Chinese shares many characters with Mandarin
  • In a primarily Mandarin context, sentences detected as Wu are likely Mandarin

Configuration

The library uses configurable thresholds: - Primary language threshold: 10% of sentences - Confidence threshold: 0.70 for ambiguity detection - Alternative language probability threshold: 0.30 for considering alternative languages

Best Practices

  1. Document Structure
  2. Group related sentences together
  3. Process complete documents rather than isolated sentences

  4. Keep context boundaries (e.g., paragraphs, sections) intact

  5. Language Distribution

  6. Works best when there's a clear primary language
  7. Handles 1-2 primary languages well

  8. May need tuning for documents with many equally represented languages

  9. Performance

  10. Process documents in batches rather than individual sentences
  11. Use contextual_detect() instead of multiple detect_language() calls
  12. Consider sentence length when setting confidence thresholds

Limitations of Per-Sentence Detection

Traditional language detection analyzes each sentence in isolation, which has limitations: - Short phrases may have insufficient linguistic features for reliable detection - Some phrases look similar across related languages (e.g., Chinese, Japanese) - No benefit from the context of surrounding text

Two-Pass Context-Aware Approach

contextual-langdetect addresses these limitations with a context-aware approach:

  1. First Pass - Independent Analysis:
  2. Process each sentence independently with fast-langdetect
  3. Classify each detection as "confident" or "ambiguous" based on confidence score
  4. For confident detections (above threshold), trust the detected language

  5. For ambiguous detections (below threshold), mark for further processing

  6. Document-Level Language Analysis:

  7. Identify primary languages from confident detections
  8. Find languages that appear with significant frequency (>10% of sentences)
  9. Create a statistical model of the document's language distribution

  10. Apply a prior assumption that documents typically contain 1-2 primary languages

  11. Second Pass - Context Resolution:

  12. For ambiguous sentences, use the document context to make better decisions
  13. Look at probability distribution across languages
  14. Select the most likely primary language based on both sentence-level probabilities and document-level statistics

This mimics how humans use context to understand language - we naturally use surrounding text to help decode ambiguous phrases.

Statistical Language Model

The statistical model used for context-aware detection is based on Bayesian principles:

  1. Prior assumption: Documents typically contain 1-2 primary languages
  2. This is implemented by focusing on languages that appear in >10% of confident detections

  3. Languages with only occasional appearances are considered less likely to be the document's primary language

  4. Language distribution analysis:

  5. Track frequency of each confidently detected language
  6. Calculate language prevalence as percentage of total document
  7. Languages that appear frequently (above the 10% threshold) are identified as "primary languages"

  8. In most cases, this results in 1-2 languages being identified as primary

  9. Bayesian-inspired disambiguation:

  10. For ambiguous sentences, examine their individual language probability distributions
  11. Combine these with the document's language distribution (prior)
  12. Select the language that maximizes P(language | sentence) × P(language | document)
  13. This balances individual sentence evidence with document-level context

This approach is particularly effective for documents with consistent language patterns, such as: - Single-language documents with occasional ambiguous phrases - Alternating language patterns (e.g., bilingual conversations) - Documents with one primary language and occasional phrases in another language

Special Case Handling

The library includes special case handling for common language detection challenges:

  1. Wu Chinese (wuu) correction
  2. Wu Chinese is often detected as a separate language from Mandarin Chinese

  3. When Wu is detected with low confidence and Mandarin is a primary language, treat as Mandarin

  4. Chinese/Japanese disambiguation

  5. Some Chinese sentences are misdetected as Japanese due to shared characters
  6. When a sentence is detected as Japanese without any kana characters, and Chinese is a primary language, treat it as Chinese

API Usage

Process a Document with Context Awareness

from contextual_langdetect import contextual_detect

# List of sentences to analyze
sentences = [
    "Hello world",
    "This is English",
    "你好世界",
    "这是中文",
    "Short text"  # Ambiguous due to length
]

# Process with context awareness
languages = contextual_detect(sentences)

# Print results
for sentence, lang in zip(sentences, languages):
    print(f"{sentence}: {lang}")

Access Lower-Level APIs

from contextual_langdetect import detect_language, get_language_probabilities

# Get detailed detection result
result = detect_language("Hello world")
print(f"Language: {result.language}")
print(f"Confidence: {result.confidence}")
print(f"Is ambiguous: {result.is_ambiguous}")

# Get probability distribution
probs = get_language_probabilities("Hello 你好")
print(probs)  # {'en': 0.6, 'zh': 0.4}

Benefits of Context-Aware Detection

  • Improved accuracy for short phrases and sentences
  • Reduced need for explicit language specification
  • More consistent results for mixed-language documents
  • Mimics human understanding of language in context
  • Graceful degradation when detection is challenging

Limitations

  • Language detection is probabilistic and may not be perfect
  • Very short sentences (1-2 words) remain challenging
  • Closely related languages can be difficult to distinguish
  • Depends on fast-langdetect's capabilities and limitations