Skip to Content
DocumentationProvider Philosophy

Provider Philosophy: No Data > Bad Data

Core Principle

In language learning applications, inaccurate data is worse than no data.

This principle guides all GLOST plugin packages that require external data sources.

Why No Fallbacks?

Heuristic-based fallbacks are harmful because they mislead learners:

Example: Word Length = Frequency

  • Heuristic assumes shorter words are more common
  • Fails for: “FAQ” (uncommon), “qi” (rare), “go” (very common)
  • Impact: Learners prioritize wrong vocabulary

Example: Capitalization = Proper Noun

  • Heuristic assumes capitalized words are proper nouns
  • Fails for: “I” (pronoun), “March” (could be verb or noun)
  • Impact: Incorrect grammar instruction

The Alternative: Graceful Degradation

GLOST plugins return undefined when they don’t have data:

// ✅ Good: Return undefined when unsure async getFrequency(word, language) { const data = await corpus.lookup(word); return data ? data.frequency : undefined; } // ❌ Bad: Guess based on heuristics async getFrequency(word, language) { const data = await corpus.lookup(word); if (!data) return word.length <= 3 ? "common" : "uncommon"; return data.frequency; }

When a provider returns undefined, the UI should either hide the data, show “Unknown”, or highlight as “needs review”. This is better than showing incorrect data.

Real Data Sources

Use validated sources instead of fallbacks:

Frequency:

  • English: British National Corpus (BNC), COCA
  • Thai: Thai National Corpus
  • Japanese: BCCWJ
  • Chinese: HSK word lists

POS Tagging:

  • Universal Dependencies
  • Language-specific corpora (ORCHID for Thai, MeCab for Japanese)

Difficulty:

  • CEFR word lists (A1-C2)
  • JLPT (N5-N1), HSK (1-6), Age of Acquisition databases

Implementation Guidelines

For Plugin Developers:

  • Never guess; return undefined when you don’t have data
  • Document data requirements explicitly
  • Provide validation helpers

For Language Package Maintainers:

  • Use validated sources only (established corpora and word lists)
  • Document data provenance (credit and link to sources)
  • Quality over coverage (better 80% accurate than 100% poor)

For Application Developers:

  • Handle undefined gracefully in UI
  • Don’t fill gaps with guesses
  • Allow manual annotation for missing data
  • Track coverage metrics

Benefits

  1. Trust: Users can trust displayed data is accurate
  2. Explicit Gaps: Missing data is visible and can be filled properly
  3. Quality Signal: Applications can show data confidence to users
  4. Focused Improvement: Developers know exactly where to improve

Exceptions

If you must use heuristics, make them explicit with confidence scores and source flags. However, it’s generally better to show nothing than “probably wrong” data, as users tend to trust any data shown.

Last updated on