crypto_futures/prompts/1_project_structure

===== INPUT =====

>>> """Main points:
...
... We're building a Python app for getting real-time orderbook and trade data from several cryptocurrency exchanges for archival, processing, prediction and trade execution - basically an algorithmic trading bot.
...  We'll be using the following software stack:
...  1. TimescaleDB as archive database (full credentials for connection -> postgres://postgres:DMl0h3qMnkiWWKRJSIgR@localhost:5432/trading);
...  2. Redis as RAM cache and message broker (credentials for connection -> localhost:6379);
...  3. Celery for general multiprocessing, scheduling and background tasks.
...
... Additional context:
...
... The app will be connecting to 6 cryptoexchanges (the number may change later): Binance, Bybit, OKX, Bitget, KuCoin and MEXC. We'll be working with both spot and futures markets.
... Our interest is in data of finest granularity, meaning WebSocket channels with orderbook depth and individual trades changing in real-time; OHLCV data is out of picture. Expect at least 2 client connections fo
... r every trading pair on every cryptoexchange, that makes >1000 active connections sending >10000 bits of valuable data every second. The sheer amount of data calls for event-driven architucture pattern for our
...  app.
... Usage of specific cryptoexchange libraries is discouraged for two reasons: 1. every cryptoexchange is different from another and this applies to their APIs also, and 2. our code blocks designed for handling We
... bSocket connections (e.g. connecting, receiving packages, sending subscription and heartbeat packages, reconnecting on network error) should be uniform, reusing same logic as much as possible.
... We have our own workstation on site, so that we could deploy the app locally. The hardware infrastructure is as follows:
...  - Lenovo RD450X dual-CPU motherboard (10 Gigabit NIC version);
...  - 2x Intel Xeon 2686v4;
...  - 8x 64GB DDR4 ECC REG Samsung 2400Mhz 4DRx4 LRDIMM [M386A8K40BM1-CRC];
...  - 2x Samsung PM1735 1.6 TB NVMe SSD (in ZFS RAID).
... We'll be utilizing our own spin on momentum trading strategies, analyzing data for prediction of direction and power of price movement based on the insights we can find in orderbook and trades, such as the vol
... ume of aggressive and passive trades; amount of large trades, iceberg trades, repeating trades and impulses; correlation with BTC and same coins between exchanges, and so on. Having latest orderbook data, we'l
... l be dynamically predicting the amount of slippage, looking for closest order concentrations for our stop loss and the estimated movement power for our target. And then all 3 those factors will be optimal, we'
... ll execute the deal.
... The same strategy should be applicable both to real-time and archive data for backtesting. Every point of the code should be logged for ease of debugging.
... Somewhere down the road, when the first block of data collection and second of data processing and prediction is ready, we'll be working on frontend and API.
...
... Prompt:
... Let's start at the beginning. Come up with the most optimal project structure for our needs and prepare a requirements.txt file with all the libraries we'll need."""

===== OUTPUT =====

Here's a **project structure** and a **`requirements.txt`** file tailored for your **algorithmic trading bot** that handles real-time orderbook and trade data from multiple cryptocurrency exchanges using an
event-driven architecture with TimescaleDB, Redis, and Celery.

---

## 🧱 Project Structure

```
trading_bot/
│
├── app/
│   ├── __init__.py
│   ├── main.py              # Entry point for the app
│   ├── config.py            # Configuration (DB, Redis, exchanges, etc.)
│   ├── logger.py            # Custom logging setup
│   ├── utils/
│   │   ├── __init__.py
│   │   ├── helpers.py       # Utility functions
│   │   └── decorators.py    # Custom decorators (e.g., retry, rate limit)
│   ├── exchanges/
│   │   ├── __init__.py
│   │   ├── base_exchange.py # Base class for exchange handling
│   │   ├── binance.py
│   │   ├── bybit.py
│   │   ├── okx.py
│   │   ├── bitget.py
│   │   ├── kucoin.py
│   │   └── mexc.py
│   ├── core/
│   │   ├── __init__.py
│   │   ├── data_processor.py # Processes raw data into structured form
│   │   ├── orderbook_handler.py # Handles orderbook updates
│   │   ├── trade_handler.py     # Handles individual trades
│   │   └── prediction_engine.py # Predictive models or logic
│   ├── tasks/
│   │   ├── __init__.py
│   │   ├── data_ingestion.py  # Celery tasks for data ingestion
│   │   ├── processing.py      # Celery tasks for processing
│   │   └── execution.py       # Celery tasks for trade execution
│   ├── api/
│   │   ├── __init__.py
│   │   ├── routes.py          # API endpoints (future)
│   │   └── app.py             # FastAPI app (future)
│   └── db/
│       ├── __init__.py
│       ├── timescale_client.py # TimescaleDB connection manager
│       └── schema.py           # Table definitions
│
├── tests/
│   ├── __init__.py
│   └── test_exchanges.py       # Unit tests for exchange modules
│
├── logs/
│   └── trading_bot.log         # Log output
│
├── .env                        # Environment variables
├── requirements.txt            # Dependencies
├── README.md
└── Dockerfile                  # Optional: for containerization
```

---

## 📦 `requirements.txt`

```txt
# Core
aiohttp==3.9.3
asyncio==3.4.3
celery==5.3.4
redis==5.0.1
psycopg2-binary==2.9.7
timescaledb-tune==1.0.0

# Data Processing
pandas==2.1.3
numpy==1.24.3
pyarrow==14.0.1

# WebSockets & Messaging
websockets==12.0
aioredis==2.0.1

# Logging & Monitoring
structlog==24.1.0
loguru==0.7.2

# Utility
python-dotenv==1.0.0
pydantic==2.5.0
click==8.1.7

# Optional (for future API)
fastapi==0.104.1
uvicorn[standard]==0.24.0

# Testing
pytest==7.4.2
pytest-asyncio==0.21.1
```

---

## 🔧 Notes

### 1. **Exchange Handling**
- You'll implement `base_exchange.py` to define a common interface:
  - `connect()`
  - `subscribe_orderbook()`
  - `subscribe_trades()`
  - `handle_message()`
  - `reconnect()`
- Each exchange module (`binance.py`, `bybit.py`, etc.) will inherit from `BaseExchange` and implement specific logic.

### 2. **Event-Driven Architecture**
- Use `asyncio` + `aiohttp` + `websockets` for handling multiple connections.
- `Celery` will manage background tasks like:
  - Saving to DB
  - Triggering predictions
  - Executing trades
- Redis is used for:
  - Caching latest data
  - Message queueing between components

### 3. **TimescaleDB**
- Use `psycopg2-binary` + `sqlalchemy` or `asyncpg` for async DB access.
- Store:
  - Orderbooks (as JSONB or structured tables)
  - Trades
  - Predictions
  - Execution logs

### 4. **Logging**
- Use `structlog` or `loguru` for structured logging.
- All modules should log to `logs/trading_bot.log`.

### 5. **Scalability**
- With 1000+ connections, use:
  - `asyncio` for concurrency
  - `Celery` with Redis as broker
  - Consider `gunicorn` + `gevent` or `uvicorn` if scaling API later

---

## ✅ Next Steps

1. **Start with `base_exchange.py`**
2. **Implement exchange-specific modules**
3. **Set up `timescale_client.py` and DB schema**
4. **Build `data_processor.py` and `orderbook_handler.py`**
5. **Implement Celery tasks for ingestion and processing**
6. **Add logging and error handling**

---

Would you like me to generate a `base_exchange.py` or a `timescale_client.py` next?