Provider Philosophy: No Data > Bad Data
Core Principle
In language learning applications, inaccurate data is worse than no data.
This principle guides all GLOST plugin packages that require external data sources.
Why No Fallbacks?
Heuristic-based fallbacks are harmful because they mislead learners:
Example: Word Length = Frequency
- Heuristic assumes shorter words are more common
- Fails for: “FAQ” (uncommon), “qi” (rare), “go” (very common)
- Impact: Learners prioritize wrong vocabulary
Example: Capitalization = Proper Noun
- Heuristic assumes capitalized words are proper nouns
- Fails for: “I” (pronoun), “March” (could be verb or noun)
- Impact: Incorrect grammar instruction
The Alternative: Graceful Degradation
GLOST plugins return undefined when they don’t have data:
// ✅ Good: Return undefined when unsure
async getFrequency(word, language) {
const data = await corpus.lookup(word);
return data ? data.frequency : undefined;
}
// ❌ Bad: Guess based on heuristics
async getFrequency(word, language) {
const data = await corpus.lookup(word);
if (!data) return word.length <= 3 ? "common" : "uncommon";
return data.frequency;
}When a provider returns undefined, the UI should either hide the data, show “Unknown”, or highlight as “needs review”. This is better than showing incorrect data.
Real Data Sources
Use validated sources instead of fallbacks:
Frequency:
- English: British National Corpus (BNC), COCA
- Thai: Thai National Corpus
- Japanese: BCCWJ
- Chinese: HSK word lists
POS Tagging:
- Universal Dependencies
- Language-specific corpora (ORCHID for Thai, MeCab for Japanese)
Difficulty:
- CEFR word lists (A1-C2)
- JLPT (N5-N1), HSK (1-6), Age of Acquisition databases
Implementation Guidelines
For Plugin Developers:
- Never guess; return
undefinedwhen you don’t have data - Document data requirements explicitly
- Provide validation helpers
For Language Package Maintainers:
- Use validated sources only (established corpora and word lists)
- Document data provenance (credit and link to sources)
- Quality over coverage (better 80% accurate than 100% poor)
For Application Developers:
- Handle
undefinedgracefully in UI - Don’t fill gaps with guesses
- Allow manual annotation for missing data
- Track coverage metrics
Benefits
- Trust: Users can trust displayed data is accurate
- Explicit Gaps: Missing data is visible and can be filled properly
- Quality Signal: Applications can show data confidence to users
- Focused Improvement: Developers know exactly where to improve
Exceptions
If you must use heuristics, make them explicit with confidence scores and source flags. However, it’s generally better to show nothing than “probably wrong” data, as users tend to trust any data shown.