Tech Stack
FastAPI
Python
Redis
PostgreSQL
Docker
Git
CI/CD
Description
FastAPI API gateway for a ChatGPT-style multi-model chat system powered by open-source LLMs (Ollama). Validates JWT, manages users/sessions/messages, enqueues inference tasks to RabbitMQ, and serves cached history from Redis with PostgreSQL as source of truth.
The backend never blocks on slow inference — messages are processed asynchronously via a task queue while the React frontend polls until the AI reply is ready.
- JWT authentication: register, login, admin model listing (GET /models).
- Session and message CRUD; POST /sessions/{id}/messages returns { task_id } immediately.
- RabbitMQ chat_tasks queue with dead-letter queue for failed jobs after 3 retries.
- Redis: last 50 messages per session, task status hashes, model config cache.
- PostgreSQL schema: users, chat_sessions, messages, model_configs.
- Design choices: FastAPI for async I/O, RabbitMQ over Kafka for task queues, polling over WebSocket for simpler frontend.
Page Info
Async message flow
Gateway returns task_id immediately (202 Accepted); frontend polls GET /tasks/{id}/status while the worker calls Ollama.
