# Dictionaries

## Adding new languages

### Brief introduction

This is not meant to be comprehensive, but it should at least touch on the main aspects of adding a language.

* [ ] **Add a dictionary file** - `./add-lang.sh de DE` (will require the relevant GNU ASpell library to be installed)
* [ ] **Attribute dictionary source** - Add a relevant `assets/dictionaries/dictionary.*.LICENSE` file
* [ ] **Make Gradle aware of the language** - Add `"de_DE",` to the `languages` array in `build.gradle`
* [ ] **Generate a trie representation of the dictionary** - `./gradlew buildDictionary_de`
* [ ] **Subclass `Language`** - Add it to the `libraries/trie` library
* [ ] **Tell Lexica about your `Language` class** - Add an entry to `Language#getAllLanguages()`
* [ ] **Generate random letter distributions** - `./gradlew analyseLanguage_de` (saves distributions to `/tmp`)
* [ ] **Choose "best" distribution** - Typically the highest "score", but boards with low min or median scores are generally poor)
* [ ] **Rename the distribution file** - `letters_LANG.txt`, e.g. `letters_de_DE.txt`
* [ ] **Copy to project** - The renamed file should exist in `app/src/test/resources/` and `app/src/main/res/raw/`
* [ ] **Add language name (in English)** - Edit `donottranslate.xml`, adding to `dict_choices_entryvalues` and `dict_choices_entries` (the latter will also require an entry in `strings.xml`)
* [ ] **Add scrabble scores** - Edit your `Language` subclass, adding letter scores from [Wikipedia - Scrabble letter distributions](https://en.wikipedia.org/wiki/Scrabble_letter_distributions)
* [ ] **Run tests** - `./gradlew check && ./gradlew connectedCheck`

### Obtaining a dictionary

This project uses the GNU Aspell project to obtain dictionaries.
The script `add_lang.sh` in the root directory of this project will:

 * Dump all words from a specific dictionary (e.g. en\_UK or de\_DE).
 * Omit words shorter than 3 characters and longer than 9.

Although it is of course technically possible for words longer than 9 to be recorded on Lexica boards,
in practice it is so unlikely as to cause problems when generating new random board generators. 
The reason is that it is hard to measure how successful a board generator is if the vast majority of
words in a language are very long (e.g. in German).

Other dictionaries have also been included, such as the Japanese dictionary. This comes to us from
the [JMdict project](http://www.edrdg.org/jmdict/j_jmdict.html) and used under the
[CC-BY-SA-3.0 license](http://www.edrdg.org/edrdg/licence.html). This was made possible from the
work of [@wichmann](https://github.com/wichmann) [here](https://github.com/lexica/lexica/issues/36#issuecomment-388008561).

### Anatomy of a random board generator in Lexica

Once a dictionary is available, the next trick is to create a set of probability distributions to be
used by random board generators, so that the generated boards tend to have nice properties (i.e. lots
of words).

The format used by these probability distributions is inherited from the original Lexic project:

```
a 12 3 2 1 1
b 5 3 1
c 3 1
d 8 4 1
e 24 12 3 1 1
...
```

Boards are generated by consulting this probability distribution, and performing a weighted random choice of letters.
This is done by looking at the first column of numbers, and performing roulette wheel selection based on this value.
The higher the value, the more likely a letter will be chosen.

Once a letter is chosen (e.g. `d` in the example board above), then the first number in that row is removed
(leaving `d 4 1` in this example. You will note that each successive number is lower, meaning the probability
of a subsequent letter being chosen again is less than the original choice of that letter.

Once a letters has had all of its numbers exhausted, that letter will not appear on the board any more.
Thus, in the example above, `c 3 1` means that the letter `c` can be chosen at most twice per board.


### Generating random board generators

The original source code of Lexic included hard coded versions of these letter frequencies,
without a way to generate them.
This version of Lexica includes a genetic algorithm which will attempt to produce these probability
distributions with the best properties, namely:

> Random boards generated from them tend to have lots of words, but not too many

To run the algorithm: `./gradlew analyseLanguage_LANG`.

The genetic algorithm will start by seeding some random genomes, where a gene is represented by a single line
(e.g. `e 24 12 3 1 1`).
The way it is seeded is by looking at all words in the dictionary, and the distribution of letters in these words.
If the dictionary doesn't have a word with more than 5 `e`'s, then it will not start with a genome of `e 24 12 3 1 1 1 1 1`,
because that has the potential to come up with some nonsense boards, e.g. those which have too many `z`'s or other
weird characteristics.

Breeding the genomes together is based on randomly chosing a row of distributions from one of two parents.

Fitness is determined by generating hundreds (or thousands) of boards from a distribution, and looking at
the min, mean, and max number of words that are possible with each of the boards.
The fitness function cares a lot about mean, and quite a lot about the minimum (e.g. boards with zero words
are useless), and only slightly about the max.

The GA can be run several times, and will output the fittest genome at the end of each run, saving it to disk.
