Feature/fastapi
Merge Request: OpenAI Tokenization & Model Selection
Description
This merge request implements the following two tasks:
-
Standardizing Tokenization:
- The library
tiktoken
(or equivalent) is used to ensure accurate tokenization.
- The library
-
Model Selection Handling:
- Incorporates logic for managing model selection (e.g.,
gpt-3.5-turbo
,gpt-4
). - create config folder for later use
- Incorporates logic for managing model selection (e.g.,