chatbot model pipeline
======================

# dependency graph

```
+- api service
   |
   +- embedding
   |
   +- corpus/documents.pkl
   |
   +- corpus/rerank.pkl

+- embedding
   |
   +- train.ipynb

+- corpus/documents.pkl
   |
   +- documents.ipynb

+- corpus/rerank.pkl
   |
   +- rerank.ipynb 
   
+- train.ipynb
   |
   +- corpus/GenQ.csv
   |
   +- corpus/rerank.csv

+- documents.ipynb
   |
   +- corpus/docs.csv
   |
   +- corpus/kb.csv
   |
   +- corpus/community.csv
   
+- rerank.ipynb 
   |
   +- embedding
   |
   +- corpus/rerank.csv
   
+- corpus/GenQ.csv
   |
   +- GenQ.ipynb
  
+- GenQ.ipynb
   |
   + corpus/documents.pkl
   |
   + (optional) corpus/GenQ.csv
   
+- corpus/docs.csv
   |
   +- docs.ipynb
   
+- corpus/kb.csv
   |
   +- kb.ipynb
   
+- corpus/community.csv
   |
   +- community.ipynb
```

# raw documents

`docs.ipynb`, `kb.ipynb` and `community.ipynb` generate the raw documents (`corpus/*.csv`) by splitting the corpus line by line.

- docs.ipynb: use [pingcap/docs](https://github.com/pingcap/docs) to generate `corpus/docs.csv`. You must clone [pingcap/docs](https://github.com/pingcap/docs) into `repos` directory before run this script.
- kb.ipynb: use [knowledge-base-cn](https://github.com/pingcap-inc/knowledge-base-cn) to generate `corpus/kb.csv`. You must clone [knowledge-base-cn](https://github.com/pingcap-inc/knowledge-base-cn) into `repos` directory before run this script.
- community.ipynb: use the following documents to generate `corpus/community.csv`:
    - [book.tidb.net](https://github.com/pingcap/book.tidb.net), you must clone it into `repos` directory
    - [cms-cn](https://cms-cn.pingcap.com)
    - [community blog](https://tidb.net/blog)

# vectorization

`documents.ipynb` use `corpus/docs.csv`, `corpus/kb.csv` and `corpus/community.csv` as input, and output `corpus/documents.pkl`, which append a new column named `vector`. The `vector` is the vectorization of the content column. Besides, `documents.ipynb` slide on the raw documents to create similar size chunks.

Because `documents.ipynb` use the embedding model to create vector of the content, the `train.ipynb` should be run first to create the embedding model.

# train the embedding model

The embedding model is trained by `train.ipynb`, which use the following corpus:

- corpus/GenQ.csv: the corpus created by GenQ.ipynb, which use the document fragments and let ChatGPT to generate question of sampled chunks
- corpus/rerank.csv: the corpus collected from users

# rerank

In some cases, the embedding model not works well and can't retrive correct document for some question. Then we can create a hard binding to that question with `rerank.ipynb`.

`rerank.ipynb` use `corpus/rerank.csv` as input, use embedding model to create a vector column just like `documents.ipynb`. But differently, it create vector on the question column instead of the content column, this guarantees the content will be retrived by that question.