commit cc7f5a4f323f14b97c1e1707508d49e4dd5441ef Author: dela Date: Sun May 24 17:20:51 2026 +0800 feat: 初始化 cryptoHermes 行情网关 v1 MVP + harness 工程文档 Binance USDⓈ-M Futures 行情网关，给上游 Hermes 量化分析引擎提供单一聚合 接口 /v1/market/context（K 线 + funding + OI + 多空比 + taker volume）。 只读公共行情，不下单、不接私钥、不查账户。 ## v1 实现范围（Milestone 1-5） - Clean Architecture 4 层（controller/usecase/repo/entity），接口边界在 internal/usecase/ports.go - Binance Futures REST client（K 线 / ticker24h / funding / OI / 多空比 / taker volume 共 9 个接口），全链路 string 价格避免 float64 精度问题 - TimescaleDB 5 张 hypertable（market_klines / funding_rates / open_interest / long_short_ratio / taker_buy_sell_volume），主键含 时间维度，UpsertMany 幂等 - robfig/cron 定时采集（15m/1h/4h/1d/1w 多周期 K 线 + 衍生品 15 分钟 落库），未收线 K 线 (close_time > now) 由 mapper/repo 双重过滤 - pkg/httpclient 统一限流（默认 20 req/s, burst 40）+ 重试，避免触发 Binance 2400 weight/min IP 上限 - /v1/market/context 聚合接口：errgroup 并发拉 snapshot/funding/OI，DB K 线不足 200 根回源 Binance 异步补 - cmd/backfill CLI 支持指定 from/to 大段回填（Binance 历史 OI / 多空比 官方只保留 30 天，必须自己存） - Docker Compose + Makefile + golang-migrate，本地一键启 技术指标（support/resistance/Vegas/箱体）留待 v2，技术段返回空对象 + warning 占位。 ## Harness 工程文档 - AGENTS.md — AI agent 工作速查（10 个章节） - ai/project-map.md — 仓库结构、扩展点、控制流 - ai/risk-guardrails.md — G1-G10 守卫规则（每条带可机械验证命令） - ai/adr/0001-architecture-foundations.md — 9 条架构基础决策 - ai/task-templates.md — 6 种任务契约模板 - ai/harness-health.md — 当前 harness 健康度评估 3 个 grep 守卫已验证通过：controller / usecase 无具体实现依赖，全项目 无私钥/签名字段。 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..15f0366 --- /dev/null +++ b/.gitignore @@ -0,0 +1,22 @@ +# Binaries +/bin/ +*.exe +*.test +*.out + +# Go +vendor/ + +# Editor +.vscode/ +.idea/ +*.swp + +# Local env / secrets +.env +.env.* +!.env.example + +# OS +.DS_Store +Thumbs.db diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..f7e10ea --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,156 @@ +# AGENTS.md — cryptoHermes 操作指南 + +本文件是 AI 编码代理在此仓库工作的最小操作手册。完整项目设计见 `docs/dev.md`，用户向文档见 `README.md`。 + +--- + +## 1. 项目定位（必读，决定可做与不可做） + +cryptoHermes 是给上游 Hermes 量化分析引擎用的 **Binance Futures 行情网关**。 + +**只做**： +- 从 Binance USDⓈ-M Futures 公共接口拉行情、衍生品数据 +- 落库到 TimescaleDB +- 通过 REST 把聚合后的市场上下文给 Hermes + +**绝不做**（详见 `ai/risk-guardrails.md`）： +- 下单、撤单、查持仓 +- 接入 API Key / Secret +- 任何账户级私有接口（签名请求） +- 钱包、托管、转账 +- WebSocket 推送（v1 范围外） + +--- + +## 2. 当前状态 + +- v1 MVP 已完成（Milestone 1-5），可启动、可采集、可返回完整 `/v1/market/context` +- Technical 指标计算（Milestone 6）留待 v2 +- 详细路线图见 `docs/dev.md` 第 19 章和 README "路线图"段 + +--- + +## 3. 技术栈 + +| 类别 | 选型 | +|---|---| +| Go | 1.23（见 `go.mod`） | +| HTTP | Fiber v2 | +| DB | TimescaleDB on PG 16 | +| DB Driver | pgx v5 (pgxpool) | +| Scheduler | robfig/cron v3 | +| Config | cleanenv（yaml + env） | +| 限流 | `golang.org/x/time/rate` | +| Migration | golang-migrate CLI | + +为什么是这套选型见 `ai/adr/0001-architecture-foundations.md`。 + +--- + +## 4. 仓库结构（参见 `ai/project-map.md`） + +``` +cmd/ 二进制入口（app / backfill） +config/ cleanenv 配置加载 +internal/ 业务代码（不可被外部 import） + app/ 装配 DI、cron、graceful shutdown + controller/ Fiber 路由层 + entity/ 纯业务实体 + usecase/ 业务接口与编排（ports.go 是接口边界） + repo/ 外部依赖实现 + webapi/binance/ Binance HTTP client + persistent/postgres/ 5 个 repository + worker/ collector 实现 +migrations/ SQL（含 hypertable 转换） +pkg/ 项目内可复用基础设施（logger/httpclient/postgres） +docs/ 设计文档 +ai/ harness 工程文档（本目录及子目录） +``` + +--- + +## 5. 验证命令（必做：改完代码至少跑前两个） + +```bash +# 编译（最快反馈） +go build ./... + +# 静态分析 +go vet ./... + +# 单包测试 +go test ./internal//... + +# 全部测试 +make test # 等价于 go test ./... + +# 启动（需要先 make docker-up && make migrate-up） +make run +``` + +**触碰 binance/mapper.go 或 entity/kline.go 之后**：必须确保对接 `Binance K线数组按下标解析` 的代码不被破坏（dev.md 第 9.2 节）。 + +**触碰任何 SQL migration 后**：本地必须重跑 `make migrate-down && make migrate-up` 验证 down/up 互逆。 + +--- + +## 6. Clean Architecture 边界（自动可验证） + +```bash +# controller 不可直接 import binance/postgres +grep -r "repo/webapi/binance\|repo/persistent" internal/controller # 必须无输出 + +# usecase 不可直接 import binance/postgres +grep -r "repo/webapi/binance\|repo/persistent" internal/usecase # 必须无输出 + +# 全项目不可出现签名 / 私钥相关字段 +grep -ri "apikey\|x-mbx-apikey\|hmac\|secret_key\|api_secret" --include="*.go" . # 必须无输出（不算注释/文档里的"不接入"说明） +``` + +任何 PR 都应该满足以上三条。 + +--- + +## 7. 关键技术约束（容易犯错） + +1. **价格精度**：全链路用 `string`，落库时由 PG 转 `NUMERIC(36,18)`。**禁止用 `float64` 存储或传递价格、成交量**。 +2. **K线未收线丢弃**：从 Binance 拉到的最新一根 K 线如果 `close_time > now()` 就不能入库。Repo 的 `UpsertMany` 会过滤 `IsClosed=false`，新代码也必须遵守。 +3. **Binance 历史 OI / 多空比官方只保留 30 天**：必须依赖 collector 持续落库才能拿到长历史。 +4. **Rate limit**：默认 20 req/s（`config.Binance.RPS`）。新增 collector 任务前估算 weight 预算（Binance 2400 weight/min IP 上限）。 +5. **国内网络**：`fapi.binance.com` 直连不稳定，通过 `HTTPS_PROXY` 环境变量挂代理。 +6. **Upsert 幂等**：5 张表的主键都包含时间维度，所有 repo 的 `UpsertMany` 使用 `ON CONFLICT DO UPDATE`，可安全重复执行。 + +--- + +## 8. 任务模式 + +任何非平凡修改，建议先写一份微型任务契约（参见 `ai/task-templates.md`）： + +``` +objective: 要做的事，一句话 +scope: 会改的文件 / 目录 +out-of-scope: 不会动的范围 +constraints: 受 ai/risk-guardrails.md 约束的哪几条 +validation: 要跑哪些命令证明通过 +acceptance: 如何判断完成 +``` + +对于跨多个 Milestone 或需要新增数据源（CoinGlass、ETF Flow）的任务，必须先写 `ai/specs/.md`。 + +--- + +## 9. 提交规范 + +- 一个 PR 改一类事。Repo 层、Binance 适配、usecase 编排不要混在一个 commit。 +- Commit message 用中文或英文都行，但要写"为什么"。 +- 不要在没有跑 `go build ./...` 和 `go vet ./...` 的情况下提交。 +- `go.mod` 改动必须配套 `go.sum`（运行 `make tidy`）。 + +--- + +## 10. 何时升级 harness + +- 同一类错误第二次出现 → 写进 `ai/risk-guardrails.md` +- 重大架构决策（换 HTTP 框架、引入消息队列、跨服务通信）→ 写 ADR +- 新增数据源（CoinGlass / ETF / CME）→ 必须有 `ai/specs/.md` 才动手 +- 跨多次会话的长任务 → 写 `ai/work-status.md` diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..8fba1cd --- /dev/null +++ b/Dockerfile @@ -0,0 +1,16 @@ +FROM golang:1.23-alpine AS builder + +WORKDIR /src +COPY go.mod go.sum* ./ +RUN go mod download || true +COPY . . +RUN CGO_ENABLED=0 GOOS=linux go build -o /out/app ./cmd/app \ + && CGO_ENABLED=0 GOOS=linux go build -o /out/backfill ./cmd/backfill + +FROM alpine:3.20 +RUN apk add --no-cache ca-certificates tzdata +WORKDIR /app +COPY --from=builder /out/app /app/app +COPY --from=builder /out/backfill /app/backfill +EXPOSE 8080 +ENTRYPOINT ["/app/app"] diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..15fee5e --- /dev/null +++ b/Makefile @@ -0,0 +1,34 @@ +DATABASE_URL ?= postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable + +.PHONY: run build test lint tidy docker-up docker-down migrate-up migrate-down backfill + +run: + go run ./cmd/app + +build: + go build -o bin/app ./cmd/app + go build -o bin/backfill ./cmd/backfill + +test: + go test ./... + +lint: + golangci-lint run + +tidy: + go mod tidy + +docker-up: + docker compose up -d + +docker-down: + docker compose down + +migrate-up: + migrate -path ./migrations -database "$(DATABASE_URL)" up + +migrate-down: + migrate -path ./migrations -database "$(DATABASE_URL)" down 1 + +backfill: + go run ./cmd/backfill $(ARGS) diff --git a/README.md b/README.md new file mode 100644 index 0000000..4cc5282 --- /dev/null +++ b/README.md @@ -0,0 +1,309 @@ +# cryptoHermes — Market Data Gateway + +为上游 Hermes 量化分析引擎提供统一的币圈行情与衍生品上下文。Hermes 只需要调用本服务的 `/v1/market/context` 一个接口，就能拿到 BTC/ETH 多周期 K 线、当前价、funding、OI、多空比等用于策略分析的数据。 + +本服务只做行情采集与聚合，**不下单、不接私钥、不查账户、不托管任何资金**。 + +--- + +## 特性 + +- Binance USDⓈ-M Futures 公共接口，无需 API Key。 +- 多周期 K 线 / 24h ticker / funding / Open Interest / 全局多空比 / 大户持仓多空比 / Taker 主动买卖量。 +- 历史数据自动落库（Binance 官方历史 OI / 多空比只保留 30 天，必须自己存）。 +- Clean Architecture：controller / usecase / repo 分层，usecase 只依赖接口。 +- Fiber v2 REST API + robfig/cron 定时采集 + TimescaleDB hypertable 存时序数据。 +- Rate Limit 节流（默认 20 req/s），可重复运行的 upsert，K 线只入库已收线。 + +--- + +## 技术栈 + +| 类型 | 选型 | +|---|---| +| 语言 | Go 1.23 | +| HTTP | Fiber v2 | +| DB | TimescaleDB (PG 16) | +| DB Driver | pgx v5 (pgxpool) | +| Scheduler | robfig/cron v3 | +| Config | cleanenv (yaml + env) | +| 限流 | golang.org/x/time/rate | +| Migration | golang-migrate CLI | + +--- + +## 快速开始 + +### 1. 依赖 + +```bash +# Go 1.23+ +go version + +# golang-migrate CLI（用于跑 migration） +go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest +``` + +### 2. 启动数据库 + +```bash +make docker-up +``` + +会用 `timescale/timescaledb:latest-pg16` 启一个 Postgres，监听本地 5432。 + +### 3. 执行 migration + +```bash +export DATABASE_URL="postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable" +make migrate-up +``` + +会建 5 张 hypertable：`market_klines` / `funding_rates` / `open_interest` / `long_short_ratio` / `taker_buy_sell_volume`。 + +### 4. （可选）回填历史 K 线 + +第一次启动时 DB 是空的，直接调 `/v1/market/context` 会触发回源 Binance。建议先回填： + +```bash +make backfill ARGS="--symbol BTCUSDT --interval 15m --limit 500" +make backfill ARGS="--symbol BTCUSDT --interval 1h --limit 500" +make backfill ARGS="--symbol BTCUSDT --interval 4h --limit 500" +make backfill ARGS="--symbol BTCUSDT --interval 1d --limit 500" +make backfill ARGS="--symbol BTCUSDT --interval 1w --limit 500" +# ETHUSDT 同上 +``` + +也可以指定起止时间做大段回填： + +```bash +make backfill ARGS="--symbol BTCUSDT --interval 1h --from 1704067200000 --to 1735689600000" +``` + +### 5. 启动服务 + +```bash +make run +``` + +服务会： + +- 监听 `:8080` +- 立刻启动 cron，每 15 分钟拉一次 funding / OI / 多空比 / taker volume / 15m K 线，整点拉 1h，每 4h 拉 4h，每日 00:05 拉 1d，每周一 00:10 拉 1w。 + +### 6. 验证 + +```bash +curl -s localhost:8080/v1/health | jq . +curl -s 'localhost:8080/v1/market/context?symbol=BTCUSDT' | jq 'keys' +``` + +--- + +## 国内网络 + +`fapi.binance.com` 国内直连不稳定，通过环境变量挂代理即可（Go 默认尊重 `HTTPS_PROXY`）： + +```bash +HTTPS_PROXY=http://127.0.0.1:7890 make run +``` + +--- + +## 配置 + +`config/config.yml`（所有字段都可被环境变量覆盖，参见 `config/config.go` 的 `env:` tag）： + +```yaml +app: + port: 8080 + env: local + +binance: + base_url: https://fapi.binance.com + timeout: 10s + retry_count: 2 + rps: 20 # IP 级限速，超过会自动 sleep + burst: 40 + +postgres: + dsn: postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable + max_conns: 10 + min_conns: 2 + +collector: + enabled: true # 设为 false 可只跑只读 API，不开 cron + symbols: [BTCUSDT, ETHUSDT] + intervals: [15m, 1h, 4h, 1d, 1w] + default_limit: 500 +``` + +常用环境变量： + +| 变量 | 说明 | +|---|---| +| `CONFIG_PATH` | 配置文件路径，默认 `config/config.yml` | +| `POSTGRES_DSN` | 覆盖 yaml 中的 DSN（Docker Compose 里就是这么干的） | +| `APP_PORT` | HTTP 端口 | +| `HTTPS_PROXY` | 走代理访问 Binance | + +--- + +## API + +### `GET /v1/health` + +```json +{ "status": "ok", "time": 1717000000000 } +``` + +### `GET /v1/market/context?symbol=BTCUSDT` + +**Hermes 主接口**。聚合返回 K 线 + 衍生品 + 数据质量信息： + +```jsonc +{ + "symbol": "BTCUSDT", + "generatedAt": 1717000000000, + "snapshot": { + "lastPrice": "108000.12", + "priceChangePercent": "2.14", + "highPrice": "109200.00", + "lowPrice": "104800.00", + "volume": "...", + "quoteVolume": "..." + }, + "klines": { + "15m": [ /* 300 根 */ ], + "1h": [ /* 300 根 */ ], + "4h": [ /* 300 根 */ ], + "1d": [ /* 300 根 */ ], + "1w": [ /* 300 根 */ ] + }, + "derivatives": { + "funding": { "current": { ... }, "history": [ ... ] }, + "openInterest": { "current": { ... }, "history": [ ... ] }, + "longShortRatio": { + "global": [ ... ], + "topTraderPosition": [ ... ] + }, + "takerBuySellVolume": [] + }, + "technical": { + "support": [], "resistance": [], + "rangeHigh": null, "rangeLow": null, "longShortLine": null + }, + "dataQuality": { + "source": "binance", + "warnings": [] + } +} +``` + +> `technical.*` 字段在 v1 留空。v2 会补支撑压力、箱体、多空线计算。 + +### `GET /v1/market/klines?symbol=BTCUSDT&interval=1h&limit=300` + +直接读 DB 返回 K 线数组。`interval` 取值 `15m/1h/4h/1d/1w`。 + +### `GET /v1/market/snapshot?symbol=BTCUSDT` + +24h ticker 实时拉取（不走 DB）。 + +### `GET /v1/market/derivatives?symbol=BTCUSDT&period=1h` + +衍生品聚合：funding / OI / 多空比。`period` 默认 `1h`。 + +--- + +## 目录结构 + +``` +cryptoHermes/ +├── cmd/ +│ ├── app/ # 主服务入口 +│ └── backfill/ # 历史 K 线回填 CLI +├── config/ # cleanenv 配置 +├── internal/ +│ ├── app/ # 装配 DI、cron scheduler、graceful shutdown +│ ├── controller/ # Fiber 路由 + middleware +│ ├── entity/ # 业务实体（不依赖 Fiber/DB/Binance） +│ ├── usecase/ # 业务接口与编排 +│ ├── repo/ +│ │ ├── webapi/binance/ # Binance HTTP client +│ │ └── persistent/postgres/ # 5 个 repository +│ └── worker/ # 实际跑采集逻辑的 collector +├── migrations/ # SQL migration（含 hypertable 转换） +├── pkg/ +│ ├── httpclient/ # 通用 HTTP client + retry + rate limit +│ ├── logger/ # slog 包装 +│ └── postgres/ # pgxpool 初始化 +├── docs/dev.md # 完整设计文档 +├── docker-compose.yml +├── Dockerfile +├── Makefile +└── go.mod +``` + +--- + +## 数据库 + +5 张 hypertable，主键设计保证 upsert 幂等： + +| 表 | 主键 | 时间列 | +|---|---|---| +| `market_klines` | `(source, symbol, interval, open_time)` | `open_time` | +| `funding_rates` | `(source, symbol, funding_time)` | `funding_time` | +| `open_interest` | `(source, symbol, period, timestamp)` | `timestamp` | +| `long_short_ratio` | `(source, symbol, period, ratio_type, timestamp)` | `timestamp` | +| `taker_buy_sell_volume` | `(source, symbol, period, timestamp)` | `timestamp` | + +K 线时间列存毫秒 epoch，Timescale chunk 间隔默认 1 周（衍生品表）/ 30 天（funding）。 + +--- + +## 常用命令 + +```bash +make run # 跑服务 +make build # 编译两个二进制到 ./bin/ +make test # 跑测试 +make tidy # go mod tidy +make docker-up # 起 Timescale +make docker-down # 停 docker +make migrate-up # apply migration +make migrate-down # 回滚 1 步 +make backfill ARGS="--symbol BTCUSDT --interval 1h --limit 500" +``` + +--- + +## Clean Architecture 边界（验收用） + +```bash +# controller 不应直接依赖 binance client +grep -r "repo/webapi/binance" internal/controller # 期望: 无输出 + +# usecase 不应直接依赖具体 binance/postgres 实现 +grep -r "repo/webapi/binance\|repo/persistent" internal/usecase # 期望: 无输出 + +# 全项目不应出现私钥/账户/签名相关字段 +grep -ri "apikey\|x-mbx-apikey\|hmac" . # 期望: 无输出 +``` + +--- + +## 路线图 + +- **v1（当前）**：Binance 行情 + 衍生品 + 落库 + `/v1/market/context` 聚合接口。 +- **v2**：技术结构计算（支撑压力 / 箱体 / 多空线 / 均线 / Vegas）、CoinGlass 清算数据、ETF Flow。 +- **v3**：CME gap、CVD、链上稳定币流动性、宏观日历、多交易所聚合。 + +完整设计参见 [`docs/dev.md`](docs/dev.md)。 + +--- + +## License + +Internal project. Not for redistribution. diff --git a/ai/adr/0001-architecture-foundations.md b/ai/adr/0001-architecture-foundations.md new file mode 100644 index 0000000..4d1cd76 --- /dev/null +++ b/ai/adr/0001-architecture-foundations.md @@ -0,0 +1,189 @@ +# ADR 0001 — 架构基础决策 + +**状态**：Accepted +**日期**：2026-05-24 +**决策者**：项目维护者 +**适用范围**：cryptoHermes v1 全部代码 + +--- + +## 背景 + +cryptoHermes 是给上游 Hermes 量化分析引擎用的 Binance Futures 行情网关。在 MVP 启动前需要锁定一组基础技术决策，避免后续 PR 反复重新评估。 + +需求边界（来自 `docs/dev.md` 第 1 章）： +- 只读公共行情，不下单、不接私钥 +- BTCUSDT / ETHUSDT，分钟到周级 K 线 + 衍生品（funding / OI / 多空比 / taker volume） +- 历史数据必须自存（Binance 历史 OI / 多空比官方只保留 30 天） +- Hermes 只调用 `/v1/market/context` 一个聚合接口 +- 部署形态：单体服务 + Postgres，国内开发机通过 HTTPS_PROXY 访问 Binance + +--- + +## 决策 + +### D1. Web 框架：Fiber v2 + +**选了 Fiber v2**（而不是 net/http 标准库、Gin、Echo、Chi）。 + +**理由**： +- 性能足够（fasthttp 底层），单体网关不需要再调优 +- API 与 Express/Gin 相近，团队上手快 +- 内置 recover/logger/req-id 中间件 +- v2 稳定，社区活跃 + +**代价**： +- 基于 fasthttp，不兼容 `net/http` 中间件生态 +- 错误类型用 `*fiber.Error`，需要在 controller 包一层 + +**何时重新评估**：如果需要 HTTP/2 server push、需要复用 `net/http` 生态中间件、或者 v2 长期不维护——届时考虑迁回标准库。 + +--- + +### D2. 数据库：TimescaleDB on PG 16 + +**选了 TimescaleDB**（而不是纯 Postgres、InfluxDB、ClickHouse）。 + +**理由**： +- 时序数据天然适合 hypertable：5 张表全部按时间分片，查询近期数据走最新 chunk，旧数据自动归档 +- SQL 接口和普通 PG 完全兼容，pgx 直连，无需新驱动 +- 没有引入新的运维栈（开发机用 Docker 起 `timescale/timescaledb:latest-pg16` 即可） +- Chunk 间隔可按表调（K 线 1 周、funding 30 天） + +**代价**： +- 必须跑 `SELECT create_hypertable(...)` 才能享受时序优化，新表容易忘 +- 升级 PG 大版本需要看 Timescale 兼容矩阵 + +**何时重新评估**：如果数据量进入 TB 级、需要列存压缩或者 OLAP 复杂聚合——考虑 ClickHouse。 + +--- + +### D3. Architecture：Clean Architecture（4 层 + ports.go 接口边界） + +**4 层**：`controller / usecase / repo / entity`。 +**接口边界**：`internal/usecase/ports.go` 是唯一定义对外接口的地方。 + +**理由**： +- 上游 Hermes 只通过 REST 调用本服务，但 Binance 这个数据源大概率会扩展到 CoinGlass / ETF / CME。把数据源抽象成 `MarketDataProvider` / `DerivativesProvider` 接口，新增数据源只需要新建 `internal/repo/webapi//`，不动业务编排。 +- 单测可以 mock 接口，不需要起真 DB / 真 Binance。 +- `internal/app/app.go` 是唯一的组合根，DI 集中可见。 + +**强制约束**（见 `ai/risk-guardrails.md` G6）： +``` +controller 不可直接 import binance/postgres +usecase 不可直接 import binance/postgres +``` + +**代价**： +- 多一层 `ports.go`，每个新接口要写两次（接口 + 实现） +- 对小型项目可能略 overkill，但本项目已经规划到 v3（多数据源），早做减少返工 + +**何时重新评估**：如果 v2 之后接口数量爆炸（> 30 个）→ 考虑按子域拆分 `ports.go` 为多个文件。 + +--- + +### D4. 传输层：REST-only（v1 不上 gRPC / 消息队列 / WebSocket） + +**选了 REST**（而不是 gRPC、NATS、WS 推送）。 + +**理由**： +- Hermes 是同语言 Go 服务，但跨服务调用频率是分钟级（拿一次 context），REST + JSON 完全够 +- gRPC 多一份 proto 维护负担，团队/工具收益不明显 +- WebSocket 推送是独立工程量（重连、消息丢失、订阅状态机），v1 不进入 +- Cron + REST 拉取模型对运维友好（任何 HTTP 监控就够） + +**代价**： +- 如果未来 Hermes 需要 sub-second 行情，REST 拉模型撑不住 + +**何时重新评估**：Hermes 接入实盘策略后，如果发现"分钟级"延迟成为瓶颈 → 写新 ADR 引入 WS。 + +--- + +### D5. 价格 / 成交量：全链路 `string` + +**所有金额字段端到端用 `string`**：Binance DTO 是 string、entity 是 string、JSON 响应是 string。落库时由 PG 转 `NUMERIC(36,18)`。 + +**理由**： +- Binance 原样返回字符串，避免 float64 转换丢精度 +- `108000.12` 这种数 float64 表示就会丢尾数；累积统计后偏差更大 +- 上游量化分析对价格精度敏感，宁可让 Hermes 自己决定怎么解析 + +**代价**： +- 加减乘除需要 PG 端做或者引入 decimal 库 +- JSON 响应里 price 是字符串，前端/客户端要适配 + +**强制约束**：见 `ai/risk-guardrails.md` G2。 + +--- + +### D6. 数据源边界：只用 Binance USDⓈ-M Futures 公共 API + +**v1 不引入 CoinGlass、Glassnode、ETF Flow、CME、链上数据**。 + +**理由**： +- Binance 一家覆盖 80% 的需求（K 线 / 衍生品 / 多空比） +- 引入新数据源等于引入新的鉴权、限流、错误模式、SLA——v1 优先把一条链路打通 +- 路线图 v2 / v3 才扩展数据源（README "路线图" 段） + +**强制约束**： +- 新增数据源前必须先写 `ai/specs/.md`（见 AGENTS.md §10） +- 不可使用 Binance 的私有签名接口（见 G1） + +--- + +### D7. 配置层：cleanenv（yaml + env override） + +**选了 cleanenv**（而不是 viper、koanf、env-only）。 + +**理由**： +- 一份 yaml 默认值 + 环境变量覆盖，开发和部署都方便 +- 比 viper 轻，没有多余抽象 +- `env-default:` tag 让默认值跟字段定义放在一起，可读性好 + +**代价**：无明显缺点。如果后续需要热重载（不需要）才会考虑换。 + +--- + +### D8. 限流：`golang.org/x/time/rate` token bucket + +挂在 `pkg/httpclient/client.go` 里，默认 RPS=20 / Burst=40，从配置读。 + +**理由**： +- 标准库附属，零外部依赖 +- token bucket 适合突发场景 +- Binance IP 限 2400 weight/min，20 req/s × 60s = 1200 req/min，留出一半余量给重试和 backfill + +**强制约束**：所有外部 HTTP 调用走 `pkg/httpclient`，见 G5。 + +--- + +### D9. Scheduler：robfig/cron v3 + +**选了 robfig/cron v3**（而不是自己写 ticker、go-co-op/gocron）。 + +**理由**： +- cron 表达式所有人都看得懂，schedule 修改是配置而非代码 +- 稳定老牌，3.0 之后无破坏性变化 +- 和 Fiber 在同一进程内运行，graceful shutdown 容易做 + +**代价**： +- 没有分布式锁——但本服务 v1 是单实例，不需要 + +--- + +## 不在本 ADR 范围 + +- **下单 / 账户 / 私钥**：永久禁止（G1）。要做必须新仓库。 +- **多交易所聚合**：v3 才考虑，届时再写 ADR。 +- **WS / 实时推送**：见 D4，需要单独写 ADR。 +- **观测性（metrics / tracing）**：MVP 阶段只用 slog 结构化日志，Prometheus / OTel 留待生产部署前评估。 + +--- + +## 后续 ADR 的触发条件（来自 AGENTS.md §10） + +- 换 HTTP 框架 → 新 ADR +- 引入消息队列 / WS → 新 ADR +- 跨服务通信改 gRPC → 新 ADR +- 数据库换型（PG → ClickHouse）→ 新 ADR +- 引入新数据源 → `ai/specs/.md`（不一定要 ADR，除非影响整体架构） diff --git a/ai/harness-health.md b/ai/harness-health.md new file mode 100644 index 0000000..eee4956 --- /dev/null +++ b/ai/harness-health.md @@ -0,0 +1,103 @@ +# Harness Health — cryptoHermes + +**日期**：2026-05-24 +**版本**：v1 MVP 刚完成（Milestone 1-5） + +按 he-maintainer 的 scoring rubric 评估当前 harness 工程状态。Rating: `strong` / `partial` / `weak`。 + +--- + +## 总览 + +| 维度 | Rating | 说明 | +|---|---|---| +| 项目地图清晰度 | strong | `ai/project-map.md` 完整覆盖 Clean Arch 4 层 + 数据流 + 扩展点 | +| 验证命令可跑 | strong | Makefile 全部命令在 README + AGENTS.md 双重列出 | +| 快速反馈环路 | strong | `go build ./...` < 2s，`go vet ./...` < 1s，单包测试支持 | +| 任务契约质量 | partial | 模板已写（`ai/task-templates.md`），但还没有真实任务用过 | +| 守卫规则清晰度 | strong | 10 条 G1-G10，每条都有可机械验证命令 | +| 架构决策耐久度 | strong | ADR 0001 记录 9 条核心决策；其他 ADR 等触发 | +| Spec / test-plan | partial | 还没有 specs/，因为 v1 没有需要 spec 的任务；v2 新数据源时必填 | +| 失败反馈环路 | partial | AGENTS.md §10 写了升级时机，但还没有 `ai/risk-guardrails.md` 改动史可参考 | +| 文档 vs 代码对齐 | strong | AGENTS.md / README / project-map 全部基于实际仓库结构 | +| MEMORY / 长期上下文 | n/a | 本仓库无 `MEMORY.md`，按设计——长期知识在 `docs/dev.md` 和 `ai/` 文档中 | + +**总体**：**strong**（MVP 完成时点适用），但有 2 个 partial： +- task-templates 未经过真实任务校验 +- specs/ 目录还是空的（等 v2 需要时填） + +--- + +## 已具备的工程基础 + +- **3 层 AI 文档系统**：`AGENTS.md`（速查）→ `ai/project-map.md`（结构）→ `ai/adr/`（决策）。`docs/dev.md` 作为最详细的设计源头单独存在。 +- **可机械验证的守卫**：G1-G10 全部给了 grep / make 命令，没有靠 review 的软规则。 +- **唯一的组合根**：`internal/app/app.go`，DI 一处可见。 +- **接口边界文件**：`internal/usecase/ports.go` 是唯一定义对外接口的位置，新 agent 找接口去这里。 +- **强约束 grep 在 AGENTS.md §6**：每次 PR 都可以直接跑。 + +--- + +## 已识别的薄弱点 + +### W1. task-templates 未经过实战 +**现状**：模板已写完，但没有 PR 用过这些契约。 +**影响**：模板可能太严格或太松。 +**纠正时机**：完成第一个非平凡 PR（比如 Milestone 6 的技术指标）后回看，调整模板。 + +### W2. 还没有 work-status.md +**现状**：目前是单 agent / 单会话工作，不需要 work-status 协调。 +**影响**：暂无。 +**升级时机**：开始 Milestone 6（技术指标）或 v2（新数据源）时，如果跨多次会话，建立 `ai/work-status.md`。 + +### W3. CI 完全未搭建 +**现状**：本地有 `make test` / `go vet`，但仓库无 GitHub Actions / GitLab CI。 +**影响**：守卫规则全靠人工跑 grep。 +**升级时机**：合并第一个外部 PR 前，搭一个最小 CI（`go build` + `go vet` + grep 边界检查 + migration up/down）。 + +### W4. 单元测试覆盖率未度量 +**现状**：MVP 阶段重点是端到端可用，未补单测覆盖率。 +**影响**：refactor 时风险偏高。 +**升级时机**：Milestone 6 引入指标计算（计算正确性 critical）前，先补 mapper / repo 的 table-driven 测试。 + +### W5. 部分 docs/* 未提交到 git +**现状**：`docs/dev.md` 是 untracked（git status 显示 `?? docs/`）。 +**纠正方式**：本次 commit 一起加进来。 + +--- + +## 后续升级触发器 + +| 触发条件 | 应采取的动作 | +|---|---| +| 同一类错误第二次出现 | 加进 `ai/risk-guardrails.md`，给可验证命令 | +| 重大架构决策（换 HTTP 框架 / 引入 MQ / 跨服务通信） | 写新 ADR | +| 新增数据源（CoinGlass / ETF / CME） | 必填 `ai/specs/.md` | +| 跨多次会话的长任务 | 写 `ai/work-status.md` | +| 单测/集成测试出现 flake | 加 test-plan，约定 retry / quarantine 策略 | +| 文档与代码出现漂移（grep 守卫失败但代码正常） | 立刻同步，不要"以代码为准"绕开守卫 | + +--- + +## 复评建议 + +- **3 个月后**（约 2026-08）：复评本文件。若 v2 数据源工作启动，至少需要 1 份 spec + 可能新 ADR。 +- **每次新 Milestone 完成时**：跑一次 AGENTS.md §6 的 3 个 grep，确认边界未漂移。 +- **新成员（agent 或人）首次提 PR 后**：根据他们卡住的地方反向更新 AGENTS.md / project-map（"如果他们卡过这里，下一个人也会卡"）。 + +--- + +## Harness 文件清单（截至本次 commit） + +``` +AGENTS.md ← AI agent 速查 +README.md ← 用户向使用文档 +docs/dev.md ← 完整设计文档（源头） +ai/project-map.md ← 仓库结构与扩展点 +ai/risk-guardrails.md ← G1-G10 守卫规则 +ai/adr/0001-architecture-foundations.md ← 9 条架构基础决策 +ai/task-templates.md ← 6 种任务模板 +ai/harness-health.md ← 本文件 +``` + +CLAUDE.md / GEMINI.md / .cursor/*.md 等其他 AI 工具的入口文件——**暂不需要**，因为这些工具都会读 AGENTS.md。如未来要差异化对待，再分裂。 diff --git a/ai/project-map.md b/ai/project-map.md new file mode 100644 index 0000000..45bfb9c --- /dev/null +++ b/ai/project-map.md @@ -0,0 +1,224 @@ +# Project Map — cryptoHermes + +代码 agent 工作的地形图。每个目录写明：**职责**、**可以依赖谁**、**不可以依赖谁**、**扩展点在哪**。 + +完整设计见 `docs/dev.md`。 + +--- + +## 顶层布局 + +``` +cryptoHermes/ +├── cmd/ 二进制入口 +├── config/ cleanenv 加载 +├── internal/ 业务代码（不可被外部 import） +├── migrations/ SQL（含 hypertable） +├── pkg/ 项目内可复用基础设施 +├── docs/ 设计文档（dev.md 是源头） +├── ai/ harness 工程文档（本目录） +├── docker-compose.yml Timescale + 服务 +├── Dockerfile multi-stage golang:1.23-alpine → alpine:3.20 +├── Makefile 所有验证命令 +└── go.mod Go 1.23 +``` + +--- + +## `cmd/` — 二进制入口 + +| 路径 | 职责 | +|---|---| +| `cmd/app/main.go` | 主服务。读 config → 调 `internal/app.Run(cfg)` | +| `cmd/backfill/main.go` | 历史 K 线回填 CLI。直接装配 binance client + kline repo，不走 fiber | + +**约束**：`cmd/` 里只允许装配 + 调用，不允许写业务逻辑。 + +--- + +## `config/` — 配置 + +- `config.go`：cleanenv 加载，`yaml + env` 双源。环境变量优先（见 `env:` tag）。 +- `config.yml`：默认配置。**不要在这里放敏感信息**（没有敏感信息可放，因为本服务无私钥）。 + +**扩展点**：新增配置项 → `config.go` 加字段（同时加 `yaml:` 和 `env:` tag）→ `config.yml` 加默认值 → 在 README "配置" 段补一行。 + +--- + +## `internal/` — 业务核心 + +Clean Architecture 四层。**依赖方向单向：`controller → usecase → entity ← repo`**。 + +``` + ┌──────────────┐ + │ controller │ HTTP 层：参数校验、调 usecase、序列化 + └──────┬───────┘ + │ depends on + ▼ + ┌──────────────┐ + │ usecase │ 业务编排。只依赖 ports.go 中的接口 + └──────┬───────┘ + │ depends on (interface only) + ▼ + ┌──────────────┐ ┌──────────────┐ + │ entity │ ◄────── │ repo │ 外部依赖实现 + └──────────────┘ └──────────────┘ + (binance / postgres) +``` + +### `internal/entity/` — 纯业务实体 + +- `kline.go`, `ticker.go`, `funding.go`, `open_interest.go`, `long_short_ratio.go`, `taker_volume.go`, `market_context.go` +- **价格/成交量字段全部 `string`**，避免 float64 精度问题。落库时 PG 转 `NUMERIC(36,18)`。 +- `Kline` 有 `IsClosed bool` 字段——未收线 K 线 (`close_time > now()`) 由 mapper 自动判定。 +- `LongShortRatio` 有常量 `RatioTypeGlobalAccount` / `RatioTypeTopTraderPosition` / `RatioTypeTopTraderAccount`。 + +**不依赖**：fiber、pgx、binance、任何外部库。可以从这里 import 到任何其他层。 + +### `internal/usecase/` — 业务接口与编排 + +- `ports.go`：**全部对外接口在这里定义**。这是 Clean Arch 的边界文件。 + - `MarketDataProvider`（K线、ticker24h） + - `DerivativesProvider`（funding/OI/多空比/taker volume 共 7 个方法） + - `KlineRepository` / `FundingRepository` / `OpenInterestRepository` / `LongShortRatioRepository` / `TakerVolumeRepository` +- `market_context.go`：聚合 `/v1/market/context`。errgroup 并发拉 snapshot/funding/OI，DB K 线不足 200 根回源 Binance 补，异步写回。 + +**约束**（grep 可验证）： +```bash +grep -r "repo/webapi/binance\|repo/persistent" internal/usecase # 必须无输出 +``` +usecase 不可直接 import 任何具体实现。新增业务流程 → 在 `ports.go` 加接口 → 在 usecase 编排 → 由 `internal/app/app.go` 注入实现。 + +### `internal/controller/` — HTTP 层 + +- `restapi/router.go` + `middleware.go`：fiber app、recover、logger、request-id +- `restapi/v1/health_routes.go`：`/v1/health` +- `restapi/v1/market_routes.go`：`/v1/market/{context,klines,snapshot,derivatives}` + +**约束**： +```bash +grep -r "repo/webapi/binance\|repo/persistent" internal/controller # 必须无输出 +``` +controller 只能依赖 usecase 和 entity，不能直接调 binance client 或 postgres repo。 + +**扩展点**：新增路由 → 加 handler → 通过 `MarketDeps` 注入依赖（不要直接 import repo 包）。 + +### `internal/repo/` — 外部依赖实现 + +| 子目录 | 实现哪些接口 | +|---|---| +| `webapi/binance/` | `MarketDataProvider`、`DerivativesProvider` | +| `persistent/postgres/` | 5 个 `*Repository` | + +**Binance client**（`webapi/binance/`）： +- `client.go`：包 `pkg/httpclient`，统一 base URL `https://fapi.binance.com`，错误用 `ExternalAPIError{Source, Path, StatusCode, Message}` 包装。 +- `dto.go`：贴近 Binance 原始响应（K 线是数组）。 +- `mapper.go`：DTO → entity。**K 线按下标解析**（`row[0]=openTime ... row[10]=takerBuyQuoteVolume`），自动设置 `IsClosed = close_time < now()`。 +- `market.go`：`GetKlines` / `GetKlinesRange` / `GetTicker24h` +- `derivatives.go`：funding × 2 + OI × 2 + 多空比 × 2 + taker volume + +**Postgres repos**（`persistent/postgres/`）： +- 5 个 repo，每个有 `UpsertMany`（`ON CONFLICT DO UPDATE`）和 `FindRecent`（按时间倒序拉，返回前反转成正序）。 +- `kline_repo.go` 的 `UpsertMany` 会过滤掉 `IsClosed=false` 的 K 线。 +- 使用 pgxpool + transaction。 + +**扩展点**： +- 新增数据源（CoinGlass / ETF）→ 新建 `webapi/coinglass/`，实现新的 Provider 接口（先在 `ports.go` 定义）。 +- 新增表 → 新 migration + 新 repo + 在 `ports.go` 加新接口。 + +### `internal/worker/` — Collector + +- `collector.go`：被 cron 调度的采集逻辑。每个 `Collect*` 方法遍历 `symbols × intervals/periods`，拉数据 → upsert。 +- **不直接被 controller 调用**，只通过 `internal/app/scheduler.go` 触发。 + +### `internal/app/` — 装配与生命周期 + +- `app.go`：DI 装配的唯一位置。 + - 顺序：pool → binanceClient → repos → contextUC → collector → scheduler → fiber app + - graceful shutdown：监听 SIGINT/SIGTERM，10s 超时，先停 cron、再停 fiber、最后关 pool。 +- `scheduler.go`：cron spec 在这里。 + +**这里是 Clean Arch 的"组合根"**——所有具体类型在这里被实例化并注入抽象接口。 + +--- + +## `pkg/` — 项目内可复用基础设施 + +| 包 | 职责 | +|---|---| +| `pkg/httpclient/` | 通用 HTTP client + 重试（429/5xx 最多 2 次）+ `x/time/rate` 限流 | +| `pkg/logger/` | slog 包装（按 env 选 JSON / Text） | +| `pkg/postgres/` | pgxpool 初始化 + Ping | + +**约束**：`pkg/` 不依赖 `internal/`，反向依赖。可以被外部项目 import（虽然目前没有）。 + +--- + +## `migrations/` — SQL + +10 个文件（5 张表 × up/down）。 + +每个 `.up.sql` 末尾必须有： +```sql +SELECT create_hypertable('', '', + chunk_time_interval => , + if_not_exists => TRUE, + migrate_data => TRUE); +``` + +chunk 间隔： +- K 线 / OI / LS / taker → 1 周 = 604800000 ms +- funding → 30 天 = 2592000000 ms + +**改 migration 前必读 `ai/risk-guardrails.md` 中"DB migration"段**。 + +--- + +## 数据流（控制流） + +### 写路径（采集） + +``` +cron tick + └→ internal/app/scheduler.go + └→ internal/worker/collector.go (Collect*) + ├→ binance.GetXxx (HTTP → DTO → entity) + └→ repo.UpsertMany (entity → SQL upsert) +``` + +### 读路径（API） + +``` +HTTP request + └→ internal/controller/restapi/v1/market_routes.go + └→ usecase.MarketContext.Build + ├→ marketData.GetTicker24h ┐ + ├→ derivatives.GetCurrent… ├ errgroup 并发 + ├→ derivatives.GetCurrent… ┘ + ├→ klineRepo.FindRecent × 5 周期 + │ └ DB 不足 → 回源 binance.GetKlines（异步写回） + └→ fundingRepo / oiRepo / lsRepo.FindRecent +``` + +--- + +## 不可逾越的边界（自动验证见 AGENTS.md §6） + +1. **controller ↛ binance/postgres**：必须经 usecase。 +2. **usecase ↛ binance/postgres 包**：只依赖 `ports.go` 接口。 +3. **entity ↛ 任何外部库**：可被所有层 import。 +4. **pkg ↛ internal**：基础设施不知道业务。 +5. **任何代码 ↛ 私钥/签名**：grep `apikey|hmac|x-mbx-apikey|secret_key|api_secret` 必须无 Go 代码命中。 + +--- + +## 关键扩展点速查 + +| 场景 | 改哪 | +|---|---| +| 新增 REST 路由 | `internal/controller/restapi/v1/`，依赖通过 `MarketDeps` 注入 | +| 新增业务编排 | `internal/usecase/ports.go` 加接口 → 新 usecase 文件 → `internal/app/app.go` 装配 | +| 新增数据表 | 新 migration → 新 entity（可选）→ 新 repo → `ports.go` 接口 → 装配 | +| 新增数据源 | `ai/specs/.md` → 新 `internal/repo/webapi//` → 新 Provider 接口 | +| 新增定时任务 | `internal/worker/collector.go` 加方法 → `internal/app/scheduler.go` 注册 cron | +| 改配置项 | `config/config.go` 加字段 → `config/config.yml` 加默认 → README 记录 | diff --git a/ai/risk-guardrails.md b/ai/risk-guardrails.md new file mode 100644 index 0000000..06b90e3 --- /dev/null +++ b/ai/risk-guardrails.md @@ -0,0 +1,184 @@ +# Risk Guardrails — cryptoHermes + +可机械验证 / 不可逾越的规则。每条规则给出**为什么**、**怎么验证**。 + +任何 PR 都应该满足这里全部规则。 + +--- + +## G1. 绝不接入交易能力（Hard Stop） + +**规则**：本服务只读公共行情。**禁止**任何下单、撤单、查询账户/持仓/资金、签名请求、托管、转账。 + +**为什么**：项目定位是"上游 Hermes 量化分析引擎的行情网关"。一旦引入交易能力，安全边界、合规风险、密钥管理责任全部上升一个数量级。`docs/dev.md` 第 1 章原文："本服务只提供市场数据与分析上下文，不做交易、不下单、不托管资金"。 + +**怎么验证**： +```bash +# Go 代码不可出现私钥/签名相关字段 +grep -ri --include="*.go" "apikey\|x-mbx-apikey\|hmac\|secret_key\|api_secret" . +# 期望：无输出（注释/字符串/文档中说明"不接入"不算） + +# Binance 私有接口前缀不可出现 +grep -r --include="*.go" "fapi/v1/order\|fapi/v1/account\|fapi/v2/account\|fapi/v2/positionRisk" . +# 期望：无输出 +``` + +**例外**：无。如果上游需求变化要做交易，**必须先开新仓库**或新模块（独立鉴权 / 独立部署），不能在本仓库内直接放开。 + +--- + +## G2. 价格 / 成交量绝不用 float64（Hard Stop） + +**规则**：所有金额、价格、成交量字段**全链路用 `string`**。DTO、entity、JSON 响应都是 `string`。落库时由 PG 转 `NUMERIC(36,18)`。 + +**为什么**：`108000.12` 这种价格 float64 表示就会丢精度。Binance 原样返回字符串，本服务保留字符串到 PG。这一点在所有 5 个 entity 文件里都已经做到了，新代码必须遵守。 + +**怎么验证**： +```bash +# 任何新增 entity 字段如果叫 *Price/*Volume/*Quantity/*Amount，类型必须是 string +grep -rn --include="*.go" "Price\s*float\|Volume\s*float\|Quantity\s*float\|Amount\s*float" internal/entity +# 期望：无输出 +``` + +**例外**：纯统计/分析（如指标计算的中间过程）可以用 float64，但**结果落库或返回 API 时必须转回 string**。 + +--- + +## G3. 未收线 K 线（IsClosed=false）不入库 + +**规则**：从 Binance 拉到的最新一根 K 线如果 `close_time > now()`，**不能写入 `market_klines`**。 + +**为什么**：未收线 K 线的 close/high/low/volume 还会变。如果入库，后续不更新就是脏数据，更新又会触发 upsert 风暴。`internal/repo/persistent/postgres/kline_repo.go` 的 `UpsertMany` 已经过滤 `IsClosed=false`，`internal/repo/webapi/binance/mapper.go` 已经自动设置 `IsClosed = closeTime < now()`。新代码（含 backfill）必须遵守。 + +**怎么验证**：手动跑一次 collector，立刻 `SELECT count(*) FROM market_klines WHERE close_time > extract(epoch from now()) * 1000` 应为 0。 + +--- + +## G4. UpsertMany 必须幂等 + +**规则**：所有 repo 的 `UpsertMany` 用 `INSERT ... ON CONFLICT () DO UPDATE SET ... updated_at = now()`。**禁止** `INSERT` 不带 ON CONFLICT 或用 `INSERT ... ON CONFLICT DO NOTHING`（后者错过了更新场景）。 + +**为什么**：cron 会重复跑、backfill 会和 cron 重叠、重启会重拉。重复执行必须不能产生重复行也不能丢更新（例如 OI 后修正、close 值微调）。 + +**怎么验证**： +```bash +make backfill ARGS="--symbol BTCUSDT --interval 1h --limit 100" +psql $DATABASE_URL -c "SELECT count(*) FROM market_klines WHERE symbol='BTCUSDT' AND interval='1h'" +make backfill ARGS="--symbol BTCUSDT --interval 1h --limit 100" +psql $DATABASE_URL -c "SELECT count(*) FROM market_klines WHERE symbol='BTCUSDT' AND interval='1h'" +# 期望：两次 count 一致 +``` + +--- + +## G5. Rate Limit 不可绕开 + +**规则**：所有外部 HTTP 调用走 `pkg/httpclient`，该 client 已挂 `golang.org/x/time/rate`（默认 20 req/s，burst 40）。**禁止** 在业务代码里直接用 `net/http` 调 Binance。 + +**为什么**：Binance USDⓈ-M Futures IP 级限额 2400 weight/min。一旦超限会被封 IP 几分钟到几小时。本地代理在国内是稀缺资源，封了影响整个开发环境。 + +**预算**：新增 collector 任务前估算 weight： +- `GET /fapi/v1/klines`：1（limit ≤ 100）→ 10（limit ≤ 1000） +- `GET /fapi/v1/ticker/24hr`：1（单 symbol）/ 40（全量） +- `GET /fapi/v1/openInterest`：1 +- `GET /futures/data/*`：1 + +**怎么验证**： +```bash +grep -rn --include="*.go" "net/http\"" internal/repo/webapi +# 期望：只在 pkg/httpclient 里，repo/webapi 不应直接 import net/http +``` + +--- + +## G6. Clean Architecture 边界 + +**规则**：见 `ai/project-map.md` "不可逾越的边界"段。3 个 grep 必须无输出。 + +**为什么**：一旦 controller / usecase 直接 import 具体实现，测试就需要起 DB 起 Binance；后续替换数据源（CoinGlass）也变成大手术。 + +**怎么验证**： +```bash +grep -r "repo/webapi/binance\|repo/persistent" internal/controller # 必须无输出 +grep -r "repo/webapi/binance\|repo/persistent" internal/usecase # 必须无输出 +``` + +--- + +## G7. Migration 必须 down/up 互逆 + +**规则**：任何 SQL migration 改动后，本地必须跑： +```bash +make migrate-down && make migrate-up +``` +两步都成功（不报错、不丢表）才能提交。 + +**为什么**：生产回滚是兜底手段，down 脚本错了就回不去。 + +**额外规则**： +- 新 migration **不可改已发布 migration 的内容**——只能加新文件（编号递增）。 +- 改表结构如果会丢数据（drop column / change type），必须在 PR 里显式说明。 +- 改 hypertable chunk 间隔需要先看 `docs/dev.md` 第 10 章。 + +--- + +## G8. K 线数组下标解析不可破 + +**规则**：`internal/repo/webapi/binance/mapper.go` 中 `mapKline` 的下标解析对应 Binance 官方约定： + +``` +[0] openTime ms +[1] open string +[2] high string +[3] low string +[4] close string +[5] volume string +[6] closeTime ms +[7] quoteAssetVolume string +[8] numberOfTrades int +[9] takerBuyBaseVolume string +[10] takerBuyQuoteVolume string +[11] ignore +``` + +**禁止**：调整顺序、跳过位置、改用对象解析（Binance 这个接口就是数组）。 + +**为什么**：错一位整张表数据就废了，回滚成本极高。`docs/dev.md` 第 9.2 节有完整说明。 + +**怎么验证**：动了 `mapper.go` 之后跑一次回填，对比 Binance 网页 K 线图同一根的 OHLCV 是否一致。 + +--- + +## G9. 配置项不引入秘钥 + +**规则**：`config/config.yml` 和环境变量列表里不允许出现 `api_key`、`secret`、`token`、`password`（DB 密码除外，且必须从环境变量读、不能 hardcode）。 + +**为什么**：本服务不需要任何 Binance 私有鉴权，也不向外部传秘密。一旦引入会破坏 G1。 + +**怎么验证**： +```bash +grep -i "api_key\|api_secret\|access_token" config/ +# 期望：无输出 +``` + +--- + +## G10. WebSocket / 实时推送不在 v1 范围 + +**规则**：v1 只用 REST 拉取。不引入 gorilla/websocket、不订阅 `wss://fstream.binance.com`。 + +**为什么**：WS 长连接的可观测性、重连、消息丢失处理是独立工程量。v1 用 REST + cron 已经能满足 Hermes 的"分钟级上下文"需求。要做 WS 必须先写 ADR。 + +**怎么验证**： +```bash +grep -rn --include="*.go" "websocket\|fstream.binance" . +# 期望：无输出 +``` + +--- + +## 升级守护规则的时机 + +同一类错误第二次出现 → 写一条新规则进来。修改本文件 = 改动了项目契约，需要在 PR 描述里说明原因并 @ 维护者。 + +新增 Hard Stop 规则时，**必须给出可机械验证的命令**——只能靠 review 来执行的规则会被绕过。 diff --git a/ai/task-templates.md b/ai/task-templates.md new file mode 100644 index 0000000..a5df989 --- /dev/null +++ b/ai/task-templates.md @@ -0,0 +1,205 @@ +# Task Templates — cryptoHermes + +任何非平凡修改（>1 个文件、改边界、引入新依赖、改 migration、新增 collector），先写一份微型任务契约。短即可——长度像 commit message，但目的是让你自己（和后续 agent）能在 30 秒内对齐范围。 + +--- + +## 通用契约模板 + +``` +objective: 一句话说要做的事 +scope: 会改的文件 / 目录（具体路径，不要"相关文件"） +out-of-scope: 显式不改的范围（防止 scope creep） +constraints: 引用 ai/risk-guardrails.md 的哪几条（G1-G10） +validation: 要跑哪些命令证明通过（必须可机械执行） +acceptance: 完成判据（行为可观测的描述） +``` + +**示例**： + +``` +objective: 给 /v1/market/context 加一个 ?intervals=1h,4h 参数，允许 Hermes 只拉部分周期，减少响应体积 +scope: + - internal/controller/restapi/v1/market_routes.go + - internal/usecase/market_context.go + - README.md（API 段补 query 参数说明） +out-of-scope: + - 不改 entity（响应结构不变，只是 klines map 的 key 集合可能少几个） + - 不改 collector（采集策略不变） +constraints: + - G6：不能在 controller 里直接调 repo，必须通过 usecase 接收 intervals 参数 +validation: + - go build ./... && go vet ./... + - curl 'localhost:8080/v1/market/context?symbol=BTCUSDT&intervals=1h,4h' | jq '.klines | keys' + - 期望：只返回 ["1h","4h"] +acceptance: + - 不传 intervals 时行为完全不变（默认全部 5 个周期） + - 传未知 interval 返回 400 + - 至少一个 happy-path 单测覆盖 +``` + +--- + +## 任务类型 → 模板 + +### T1. 新增 API 路由 + +``` +objective: <做什么> +scope: + - internal/controller/restapi/v1/.go（新 handler） + - internal/usecase/.go（新业务方法或扩展现有） + - internal/usecase/ports.go（如果需要新 repo 方法） +out-of-scope: + - 不改 entity 结构 + - 不改 collector +constraints: G6 +validation: + - go build ./... && go vet ./... + - curl 测 happy-path + 错误参数 +acceptance: + - 路由响应符合 JSON schema + - 参数校验完整（缺失/非法都返回 400） + - README 更新 API 段 +``` + +### T2. 新增数据表 / migration + +``` +objective: <为什么要存这个> +scope: + - migrations/0000N_.up.sql + - migrations/0000N_.down.sql + - internal/entity/.go + - internal/repo/persistent/postgres/_repo.go + - internal/usecase/ports.go（新 Repository 接口） +out-of-scope: + - 不改已发布 migration（编号≤当前 max） +constraints: + - G2（金额必须 string / NUMERIC(36,18)） + - G4（UpsertMany 必须幂等） + - G7（down/up 互逆） +validation: + - make migrate-up + - make migrate-down + - make migrate-up # 再次成功 + - go build ./... +acceptance: + - 表是 hypertable：psql 跑 SELECT * FROM timescaledb_information.hypertables 看到 + - upsert 跑两次行数不变 + - 至少 1 个 repo 单测（连本地 Timescale） +``` + +### T3. 新增数据源（CoinGlass / ETF / ...） + +**必须先写 `ai/specs/.md`**（见 AGENTS.md §8）。 + +``` +objective: 接入 提供 <什么数据> +spec: ai/specs/.md +scope: + - internal/repo/webapi//{client,dto,mapper,...}.go + - internal/usecase/ports.go（新 Provider 接口） + - internal/app/app.go（DI 装配） + - config/config.go + config.yml（新数据源的 base_url / rps / timeout） +out-of-scope: + - 不改 Binance client +constraints: + - G1（绝不接私钥） + - G2（金额 string） + - G5（必须走 pkg/httpclient 限流） +validation: + - go build ./... && go vet ./... + - 临时挂一个 debug 路由验证能拉到真数据 +acceptance: + - 新 Provider 实现接口；usecase 不直接 import 包 + - 速率预算文档化（在 spec 里） +``` + +### T4. 改 collector / scheduler + +``` +objective: <调整哪个采集，为什么> +scope: + - internal/worker/collector.go + - internal/app/scheduler.go +out-of-scope: + - 不改 binance client（如果要新方法，先开 T3 / T1） + - 不改表结构（如果要，开 T2） +constraints: + - G3（未收线丢弃） + - G4（upsert 幂等） + - G5（rate limit 不绕开） +validation: + - go build ./... && go vet ./... + - 本地起服务跑 1 个 cron 周期，psql 看新行写入 +acceptance: + - 新 cron 表达式合理（不要 0 0 * * * 这种容易和别人撞的高峰） + - 重启服务后 scheduler 能正常注册 + - weight 预算在 PR 描述里说明（Binance 2400/min IP 上限） +``` + +### T5. Bug fix + +``` +objective: 修 （issue 链接或复现命令） +scope: <最小改动范围> +constraints: <相关 G 条> +validation: + - 写一个能复现 bug 的最小测试（先红后绿） + - go build ./... && go vet ./... +acceptance: + - 测试通过 + - 不引入新的 lint warning +``` + +### T6. 重构 / Clean Architecture 修正 + +``` +objective: <消除哪个边界违反 / 抽取哪段重复> +scope: <文件列表> +out-of-scope: + - 不改行为（API 响应字节级一致 / DB schema 不变） +constraints: G6 +validation: + - go build ./... && go vet ./... + - 全部 grep 边界检查（AGENTS.md §6） + - make test +acceptance: + - diff 中不含行为变化（只是搬代码 / 重命名 / 抽接口） +``` + +--- + +## 何时升级到 spec + +micro contract 写不下、或者需要跨多个 agent / 多次会话才能完成 → 升级到 `ai/specs/.md`： + +- 多 milestone 跨越 +- 引入新数据源 +- 改变响应 schema（影响上游 Hermes） +- 引入新部署组件（消息队列、新 service） + +spec 内容：objective / non-goals / config 形状 / API 形状 / 错误模式 / 兼容性 / 验收 / 验证计划。 + +--- + +## 何时升级到 ADR + +contract 涉及"以后类似场景应该都这么做" → 写 ADR（`ai/adr/000N-.md`）： + +- 换核心库 +- 改架构层数 / 改 Clean Arch 边界 +- 引入新通信协议 +- 改部署拓扑 + +ADR 结构参考 `ai/adr/0001-architecture-foundations.md`。 + +--- + +## 反模式 + +- ❌ "顺便把这里也改一下"——顺便的部分单独开一个 contract，不要混 +- ❌ "测试很难写，先合了"——测试难写说明设计有问题，先调设计 +- ❌ "之前 review 过类似的，应该没问题"——每个 PR 独立 validate +- ❌ commit message 写"WIP"——本仓库不允许 WIP commit 进 main diff --git a/cmd/app/main.go b/cmd/app/main.go new file mode 100644 index 0000000..a8e4837 --- /dev/null +++ b/cmd/app/main.go @@ -0,0 +1,33 @@ +package main + +import ( + "flag" + "log/slog" + "os" + + "cryptoHermes/config" + "cryptoHermes/internal/app" + "cryptoHermes/pkg/logger" +) + +func main() { + var cfgPath string + flag.StringVar(&cfgPath, "config", "config/config.yml", "config file path") + flag.Parse() + if env := os.Getenv("CONFIG_PATH"); env != "" { + cfgPath = env + } + + cfg, err := config.Load(cfgPath) + if err != nil { + slog.Default().Error("config_load_failed", "err", err) + os.Exit(1) + } + + log := logger.New(cfg.App.Env) + + if err := app.Run(cfg, log); err != nil { + log.Error("app_run_failed", "err", err) + os.Exit(1) + } +} diff --git a/cmd/backfill/main.go b/cmd/backfill/main.go new file mode 100644 index 0000000..170c230 --- /dev/null +++ b/cmd/backfill/main.go @@ -0,0 +1,125 @@ +package main + +import ( + "context" + "flag" + "fmt" + "log/slog" + "os" + "time" + + "cryptoHermes/config" + "cryptoHermes/internal/entity" + "cryptoHermes/internal/repo/persistent/postgres" + "cryptoHermes/internal/repo/webapi/binance" + pkglogger "cryptoHermes/pkg/logger" + pgpkg "cryptoHermes/pkg/postgres" +) + +func main() { + var ( + cfgPath string + symbol string + interval string + fromMs int64 + toMs int64 + limit int + ) + flag.StringVar(&cfgPath, "config", "config/config.yml", "config file path") + flag.StringVar(&symbol, "symbol", "BTCUSDT", "trading pair, e.g. BTCUSDT") + flag.StringVar(&interval, "interval", "1h", "kline interval (15m/1h/4h/1d/1w)") + flag.Int64Var(&fromMs, "from", 0, "start time in ms (0 = pull most recent N bars)") + flag.Int64Var(&toMs, "to", 0, "end time in ms (0 = now)") + flag.IntVar(&limit, "limit", 1500, "bars per request, max 1500") + flag.Parse() + + if env := os.Getenv("CONFIG_PATH"); env != "" { + cfgPath = env + } + + cfg, err := config.Load(cfgPath) + if err != nil { + slog.Default().Error("config_load_failed", "err", err) + os.Exit(1) + } + log := pkglogger.New(cfg.App.Env) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) + defer cancel() + + pool, err := pgpkg.NewPool(ctx, pgpkg.Options{ + DSN: cfg.Postgres.DSN, + MaxConns: cfg.Postgres.MaxConns, + MinConns: cfg.Postgres.MinConns, + }) + if err != nil { + log.Error("pool", "err", err) + os.Exit(1) + } + defer pool.Close() + + client := binance.NewClient(binance.ClientOptions{ + BaseURL: cfg.Binance.BaseURL, + Timeout: cfg.Binance.Timeout, + RetryCount: cfg.Binance.RetryCount, + RPS: cfg.Binance.RPS, + Burst: cfg.Binance.Burst, + }) + repo := postgres.NewKlineRepo(pool) + + if toMs == 0 { + toMs = time.Now().UnixMilli() + } + + if fromMs == 0 { + ks, err := client.GetKlines(ctx, symbol, interval, limit) + if err != nil { + log.Error("fetch", "err", err) + os.Exit(1) + } + closed := filterClosed(ks) + if err := repo.UpsertMany(ctx, closed); err != nil { + log.Error("upsert", "err", err) + os.Exit(1) + } + log.Info("backfill_done", "symbol", symbol, "interval", interval, "rows", len(closed)) + return + } + + cursor := fromMs + totalRows := 0 + for cursor < toMs { + ks, err := client.GetKlinesRange(ctx, symbol, interval, cursor, toMs, limit) + if err != nil { + log.Error("fetch_range", "cursor", cursor, "err", err) + os.Exit(1) + } + if len(ks) == 0 { + break + } + closed := filterClosed(ks) + if err := repo.UpsertMany(ctx, closed); err != nil { + log.Error("upsert", "err", err) + os.Exit(1) + } + totalRows += len(closed) + lastClose := ks[len(ks)-1].CloseTime + if lastClose <= cursor { + break + } + cursor = lastClose + 1 + fmt.Printf("\rbackfill: %s/%s rows=%d cursor=%d ", symbol, interval, totalRows, cursor) + } + fmt.Println() + log.Info("backfill_done", "symbol", symbol, "interval", interval, "rows", totalRows) +} + +func filterClosed(in []entity.Kline) []entity.Kline { + out := make([]entity.Kline, 0, len(in)) + for _, k := range in { + if k.IsClosed { + out = append(out, k) + } + } + return out +} diff --git a/config/config.go b/config/config.go new file mode 100644 index 0000000..d291f5a --- /dev/null +++ b/config/config.go @@ -0,0 +1,62 @@ +package config + +import ( + "time" + + "github.com/ilyakaznacheev/cleanenv" +) + +type Config struct { + App App `yaml:"app"` + HTTP HTTP `yaml:"http"` + Binance Binance `yaml:"binance"` + Postgres Postgres `yaml:"postgres"` + Collector Collector `yaml:"collector"` +} + +type App struct { + Name string `yaml:"name" env:"APP_NAME" env-default:"crypto-hermes-gateway"` + Env string `yaml:"env" env:"APP_ENV" env-default:"local"` + Port int `yaml:"port" env:"APP_PORT" env-default:"8080"` +} + +type HTTP struct { + ReadTimeout time.Duration `yaml:"read_timeout" env-default:"5s"` + WriteTimeout time.Duration `yaml:"write_timeout" env-default:"10s"` + IdleTimeout time.Duration `yaml:"idle_timeout" env-default:"60s"` +} + +type Binance struct { + BaseURL string `yaml:"base_url" env:"BINANCE_BASE_URL" env-default:"https://fapi.binance.com"` + Timeout time.Duration `yaml:"timeout" env-default:"10s"` + RetryCount int `yaml:"retry_count" env-default:"2"` + RPS float64 `yaml:"rps" env-default:"20"` + Burst int `yaml:"burst" env-default:"40"` +} + +type Postgres struct { + DSN string `yaml:"dsn" env:"POSTGRES_DSN" env-required:"true"` + MaxConns int32 `yaml:"max_conns" env-default:"10"` + MinConns int32 `yaml:"min_conns" env-default:"2"` +} + +type Collector struct { + Enabled bool `yaml:"enabled" env-default:"true"` + Symbols []string `yaml:"symbols" env-default:"BTCUSDT,ETHUSDT"` + Intervals []string `yaml:"intervals" env-default:"15m,1h,4h,1d,1w"` + DefaultLimit int `yaml:"default_limit" env-default:"500"` +} + +func Load(path string) (*Config, error) { + var cfg Config + if path != "" { + if err := cleanenv.ReadConfig(path, &cfg); err != nil { + return nil, err + } + } else { + if err := cleanenv.ReadEnv(&cfg); err != nil { + return nil, err + } + } + return &cfg, nil +} diff --git a/config/config.yml b/config/config.yml new file mode 100644 index 0000000..5c9f084 --- /dev/null +++ b/config/config.yml @@ -0,0 +1,34 @@ +app: + name: crypto-hermes-gateway + env: local + port: 8080 + +http: + read_timeout: 5s + write_timeout: 10s + idle_timeout: 60s + +binance: + base_url: https://fapi.binance.com + timeout: 10s + retry_count: 2 + rps: 20 + burst: 40 + +postgres: + dsn: postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable + max_conns: 10 + min_conns: 2 + +collector: + enabled: true + symbols: + - BTCUSDT + - ETHUSDT + intervals: + - 15m + - 1h + - 4h + - 1d + - 1w + default_limit: 500 diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..d4448ce --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,28 @@ +services: + postgres: + image: timescale/timescaledb:latest-pg16 + container_name: crypto-hermes-postgres + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: hermes_market + ports: + - "5432:5432" + volumes: + - hermes_pg_data:/var/lib/postgresql/data + + app: + build: . + container_name: crypto-hermes-market-gateway + depends_on: + - postgres + ports: + - "8080:8080" + environment: + CONFIG_PATH: /app/config/config.yml + POSTGRES_DSN: postgres://postgres:postgres@crypto-hermes-postgres:5432/hermes_market?sslmode=disable + volumes: + - ./config:/app/config + +volumes: + hermes_pg_data: diff --git a/docs/dev.md b/docs/dev.md new file mode 100644 index 0000000..680ac4a --- /dev/null +++ b/docs/dev.md @@ -0,0 +1,1330 @@ +下面这份可以直接保存为： + +```text +docs/development.md +``` + +# Hermes Market Data Gateway 开发文档 + +## 1. 项目目标 + +本项目用于给 Hermes 提供统一的币圈行情与衍生品分析上下文。 + +核心原则： + +```text +1. Hermes 只调用本服务，不直接访问 Binance / CoinGlass / ETF 数据源 +2. 本服务只提供市场数据与分析上下文，不做交易、不下单、不托管资金 +3. 第一版只保留 REST API，不引入 gRPC / NATS / RabbitMQ +4. usecase 只依赖接口，不直接依赖 Binance client +5. Binance / CoinGlass / ETF / Macro / CME 都放在 repo/webapi +6. PostgreSQL / TimescaleDB 放在 repo/persistent +7. K线、Funding、OI、多空比长期数据必须落库 +``` + +项目架构参考 `go-clean-template` 的 Clean Architecture 思路：业务逻辑放在 usecase，外部系统通过 repo/webapi 或 repo/persistent 实现接口，避免服务变大后 controller、client、DB 逻辑混在一起。该模板本身定位就是 Go 服务的 Clean Architecture 模板，并支持 REST、gRPC、AMQP RPC、NATS RPC 等 transport；本项目第一版只启用 REST。([GitHub][1]) + +--- + +## 2. 第一版范围 + +### 2.1 必须实现 + +第一版只做 Binance Futures 基础数据。 + +```text +数据源: +- Binance USDⓈ-M Futures Public API + +币种: +- BTCUSDT +- ETHUSDT + +周期: +- 15m +- 1h +- 4h +- 1d +- 1w + +数据: +- K线 / OHLCV +- 24h ticker +- 当前 funding +- 历史 funding +- 当前 OI +- 历史 OI +- Global long/short ratio +- Top trader position ratio +- Taker buy/sell volume +``` + +Binance K线接口为 `GET /fapi/v1/klines`，K线由 open time 唯一标识，limit 默认 500、最大 1500；24h ticker 接口为 `GET /fapi/v1/ticker/24hr`，返回 lastPrice、highPrice、lowPrice、volume、quoteVolume、priceChangePercent 等字段。([Binance Developer Center][2]) + +Funding 历史接口为 `GET /fapi/v1/fundingRate`，limit 最大 1000；当前 OI 接口为 `GET /fapi/v1/openInterest`，历史 OI 接口为 `GET /futures/data/openInterestHist`，并且 Binance 官方说明历史 OI 只提供最近 1 个月数据，所以必须自己长期落库。([Binance Developer Center][3]) + +Long/Short Ratio 接口为 `GET /futures/data/globalLongShortAccountRatio`，官方说明只保留最近 30 天数据，也需要定时采集入库。([Binance Developer Center][4]) + +--- + +## 3. 非目标 + +第一版不做： + +```text +1. 不做下单 +2. 不做账户资产查询 +3. 不接私有 API Key +4. 不做交易信号自动执行 +5. 不做 NATS / RabbitMQ / gRPC +6. 不做复杂权限系统 +7. 不做多租户 +8. 不做清算热力图 +9. 不做 CVD +10. 不做链上数据 +``` + +后续版本再接： + +```text +v2: +- CoinGlass +- ETF Flow +- 清算数据 +- CME 缺口 + +v3: +- CVD +- 链上稳定币流动性 +- 宏观日历 +- 多交易所聚合 +``` + +--- + +## 4. 技术栈 + +```text +Language: +- Go 1.22+ + +HTTP: +- Gin 或 Fiber +- 如果你想贴近 go-clean-template，可以用 Fiber + +Database: +- PostgreSQL +- TimescaleDB 可选，但推荐 + +Cache: +- Redis，可选 + +Scheduler: +- robfig/cron + +DB Driver: +- pgx + +Config: +- cleanenv 或 viper + +Decimal: +- 推荐 shopspring/decimal +- 不建议用 float64 直接处理价格和成交量 +``` + +推荐依赖： + +```bash +go get github.com/gofiber/fiber/v2 +go get github.com/jackc/pgx/v5/pgxpool +go get github.com/robfig/cron/v3 +go get github.com/shopspring/decimal +go get github.com/ilyakaznacheev/cleanenv +``` + +--- + +## 5. 项目目录结构 + +```text +crypto-hermes-gateway/ + cmd/ + app/ + main.go + + config/ + config.go + config.yml + + internal/ + app/ + app.go + scheduler.go + + controller/ + restapi/ + router.go + middleware.go + v1/ + market_routes.go + health_routes.go + + entity/ + kline.go + ticker.go + funding.go + open_interest.go + long_short_ratio.go + taker_volume.go + market_context.go + technical_level.go + + usecase/ + ports.go + market_context.go + market_collector.go + indicator.go + + repo/ + webapi/ + binance/ + client.go + dto.go + mapper.go + market.go + derivatives.go + + persistent/ + postgres/ + kline_repo.go + funding_repo.go + open_interest_repo.go + long_short_ratio_repo.go + taker_volume_repo.go + + worker/ + collector.go + kline_collector.go + derivatives_collector.go + + migrations/ + 000001_market_klines.up.sql + 000001_market_klines.down.sql + 000002_funding_rates.up.sql + 000003_open_interest.up.sql + 000004_long_short_ratio.up.sql + 000005_taker_volume.up.sql + + pkg/ + httpclient/ + client.go + logger/ + logger.go + postgres/ + postgres.go + + docs/ + development.md + api.md + database.md + + docker-compose.yml + Dockerfile + Makefile + go.mod +``` + +--- + +## 6. 分层说明 + +### 6.1 Controller 层 + +目录： + +```text +internal/controller/restapi/v1 +``` + +职责： + +```text +1. 接收 HTTP 请求 +2. 校验 query/path/body +3. 调用 usecase +4. 返回 JSON +``` + +禁止： + +```text +1. 不直接调用 Binance client +2. 不写复杂业务逻辑 +3. 不直接拼数据库 SQL +4. 不计算技术指标 +``` + +示例接口： + +```http +GET /v1/market/context?symbol=BTCUSDT +GET /v1/market/klines?symbol=BTCUSDT&interval=1h&limit=300 +GET /v1/market/snapshot?symbol=BTCUSDT +GET /v1/market/derivatives?symbol=BTCUSDT&period=1h +GET /v1/health +``` + +--- + +### 6.2 Entity 层 + +目录： + +```text +internal/entity +``` + +职责： + +```text +1. 定义核心业务模型 +2. 不依赖 Gin / Fiber +3. 不依赖 Binance DTO +4. 不依赖 PostgreSQL row +``` + +示例： + +```go +package entity + +type Kline struct { + Source string `json:"source"` + Symbol string `json:"symbol"` + Interval string `json:"interval"` + OpenTime int64 `json:"openTime"` + CloseTime int64 `json:"closeTime"` + Open string `json:"open"` + High string `json:"high"` + Low string `json:"low"` + Close string `json:"close"` + Volume string `json:"volume"` + QuoteVolume string `json:"quoteVolume"` + TradeCount int64 `json:"tradeCount"` + TakerBuyBaseVolume string `json:"takerBuyBaseVolume"` + TakerBuyQuoteVolume string `json:"takerBuyQuoteVolume"` +} +``` + +--- + +### 6.3 Usecase 层 + +目录： + +```text +internal/usecase +``` + +职责： + +```text +1. 编排业务流程 +2. 聚合不同数据源 +3. 计算 Hermes 需要的市场上下文 +4. 通过接口依赖 repo，不依赖具体 Binance client +``` + +核心 usecase： + +```text +MarketContextUsecase: +- 构建 /v1/market/context 返回结果 + +MarketCollectorUsecase: +- 定时采集 K线、Funding、OI、多空比 + +IndicatorUsecase: +- 计算支撑压力、均线、布林、Vegas、箱体等 +``` + +--- + +### 6.4 Repo WebAPI 层 + +目录： + +```text +internal/repo/webapi +``` + +职责： + +```text +1. 调外部 API +2. 处理 Binance / CoinGlass / ETF 的原始 response +3. 将外部 DTO 映射成 entity +``` + +第一版只实现： + +```text +internal/repo/webapi/binance +``` + +后续扩展： + +```text +internal/repo/webapi/coinglass +internal/repo/webapi/etf +internal/repo/webapi/macro +internal/repo/webapi/cme +``` + +--- + +### 6.5 Repo Persistent 层 + +目录： + +```text +internal/repo/persistent/postgres +``` + +职责： + +```text +1. 落库 +2. 查询历史数据 +3. Upsert 去重 +4. 提供给 usecase 使用的仓储接口实现 +``` + +--- + +## 7. Usecase 接口设计 + +`internal/usecase/ports.go` + +```go +package usecase + +import ( + "context" + + "crypto-hermes-gateway/internal/entity" +) + +type MarketDataProvider interface { + GetKlines(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error) + GetTicker24h(ctx context.Context, symbol string) (*entity.Ticker24h, error) +} + +type DerivativesProvider interface { + GetCurrentFunding(ctx context.Context, symbol string) (*entity.FundingRate, error) + GetFundingHistory(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error) + + GetCurrentOpenInterest(ctx context.Context, symbol string) (*entity.OpenInterest, error) + GetOpenInterestHistory(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error) + + GetGlobalLongShortRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error) + GetTopTraderPositionRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error) + + GetTakerBuySellVolume(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error) +} + +type KlineRepository interface { + UpsertMany(ctx context.Context, items []entity.Kline) error + FindRecent(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error) +} + +type FundingRepository interface { + UpsertMany(ctx context.Context, items []entity.FundingRate) error + FindRecent(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error) +} + +type OpenInterestRepository interface { + UpsertMany(ctx context.Context, items []entity.OpenInterest) error + FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error) +} + +type LongShortRatioRepository interface { + UpsertMany(ctx context.Context, items []entity.LongShortRatio) error + FindRecent(ctx context.Context, symbol, period, ratioType string, limit int) ([]entity.LongShortRatio, error) +} +``` + +--- + +## 8. 核心聚合接口 + +### 8.1 Hermes 主接口 + +```http +GET /v1/market/context?symbol=BTCUSDT +``` + +用途： + +```text +Hermes 分析 BTC / ETH 时，只调用这个接口。 +``` + +返回结构： + +```json +{ + "symbol": "BTCUSDT", + "generatedAt": 1710000000000, + "snapshot": { + "lastPrice": "108000.12", + "priceChangePercent": "2.14", + "highPrice": "109200.00", + "lowPrice": "104800.00", + "volume": "123456.78", + "quoteVolume": "13200000000.00" + }, + "klines": { + "15m": [], + "1h": [], + "4h": [], + "1d": [], + "1w": [] + }, + "derivatives": { + "funding": { + "current": {}, + "history": [] + }, + "openInterest": { + "current": {}, + "history": [] + }, + "longShortRatio": { + "global": [], + "topTraderPosition": [] + }, + "takerBuySellVolume": [] + }, + "technical": { + "support": [], + "resistance": [], + "rangeHigh": null, + "rangeLow": null, + "longShortLine": null + }, + "dataQuality": { + "source": "binance", + "warnings": [] + } +} +``` + +--- + +## 9. Binance WebAPI 实现规范 + +### 9.1 Base URL + +```text +https://fapi.binance.com +``` + +### 9.2 必接接口 + +```text +K线: +GET /fapi/v1/klines + +24h ticker: +GET /fapi/v1/ticker/24hr + +当前 funding: +GET /fapi/v1/premiumIndex + +历史 funding: +GET /fapi/v1/fundingRate + +当前 OI: +GET /fapi/v1/openInterest + +历史 OI: +GET /futures/data/openInterestHist + +全局多空比: +GET /futures/data/globalLongShortAccountRatio + +大户持仓多空比: +GET /futures/data/topLongShortPositionRatio + +主动买卖量: +GET /futures/data/takerlongshortRatio +``` + +### 9.3 HTTP Client 要求 + +```text +1. timeout: 10s +2. retry: 最多 2 次 +3. retry only: + - 429 + - 500 + - 502 + - 503 + - 504 +4. 所有请求带 context +5. 所有外部 API error 必须包装 source、path、status code +``` + +错误格式： + +```go +type ExternalAPIError struct { + Source string + Path string + StatusCode int + Message string +} +``` + +--- + +## 10. 数据库设计 + +### 10.1 market_klines + +```sql +CREATE TABLE IF NOT EXISTS market_klines ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + interval TEXT NOT NULL, + + open_time BIGINT NOT NULL, + close_time BIGINT NOT NULL, + + open NUMERIC(36, 18) NOT NULL, + high NUMERIC(36, 18) NOT NULL, + low NUMERIC(36, 18) NOT NULL, + close NUMERIC(36, 18) NOT NULL, + + volume NUMERIC(36, 18) NOT NULL, + quote_volume NUMERIC(36, 18) NOT NULL, + + trade_count BIGINT NOT NULL, + + taker_buy_base_volume NUMERIC(36, 18) NOT NULL, + taker_buy_quote_volume NUMERIC(36, 18) NOT NULL, + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, interval, open_time) +); + +CREATE INDEX IF NOT EXISTS idx_market_klines_symbol_interval_time +ON market_klines(symbol, interval, open_time DESC); +``` + +--- + +### 10.2 funding_rates + +```sql +CREATE TABLE IF NOT EXISTS funding_rates ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + + funding_time BIGINT NOT NULL, + funding_rate NUMERIC(36, 18) NOT NULL, + mark_price NUMERIC(36, 18), + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, funding_time) +); + +CREATE INDEX IF NOT EXISTS idx_funding_rates_symbol_time +ON funding_rates(symbol, funding_time DESC); +``` + +--- + +### 10.3 open_interest + +```sql +CREATE TABLE IF NOT EXISTS open_interest ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + period TEXT NOT NULL, + + timestamp BIGINT NOT NULL, + open_interest NUMERIC(36, 18) NOT NULL, + open_interest_value NUMERIC(36, 18), + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, period, timestamp) +); + +CREATE INDEX IF NOT EXISTS idx_open_interest_symbol_period_time +ON open_interest(symbol, period, timestamp DESC); +``` + +--- + +### 10.4 long_short_ratio + +```sql +CREATE TABLE IF NOT EXISTS long_short_ratio ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + period TEXT NOT NULL, + ratio_type TEXT NOT NULL, + + timestamp BIGINT NOT NULL, + + long_short_ratio NUMERIC(36, 18) NOT NULL, + long_value NUMERIC(36, 18), + short_value NUMERIC(36, 18), + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, period, ratio_type, timestamp) +); + +CREATE INDEX IF NOT EXISTS idx_long_short_ratio_symbol_period_time +ON long_short_ratio(symbol, period, ratio_type, timestamp DESC); +``` + +`ratio_type` 可选值： + +```text +global_account +top_trader_position +top_trader_account +``` + +--- + +### 10.5 taker_buy_sell_volume + +```sql +CREATE TABLE IF NOT EXISTS taker_buy_sell_volume ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + period TEXT NOT NULL, + + timestamp BIGINT NOT NULL, + + buy_sell_ratio NUMERIC(36, 18), + buy_volume NUMERIC(36, 18), + sell_volume NUMERIC(36, 18), + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, period, timestamp) +); + +CREATE INDEX IF NOT EXISTS idx_taker_buy_sell_symbol_period_time +ON taker_buy_sell_volume(symbol, period, timestamp DESC); +``` + +--- + +## 11. 定时采集设计 + +### 11.1 支持币种 + +配置文件： + +```yaml +symbols: + - BTCUSDT + - ETHUSDT + +intervals: + - 15m + - 1h + - 4h + - 1d + - 1w +``` + +--- + +### 11.2 采集频率 + +```text +K线: +- 15m: 每 15 分钟采集一次 +- 1h: 每小时采集一次 +- 4h: 每 4 小时采集一次 +- 1d: 每天采集一次 +- 1w: 每周采集一次 + +Ticker: +- 30s 或 60s 刷新，可先不落库 + +Funding: +- 当前 funding: 每 15 分钟 +- 历史 funding: 每 8 小时后补一次 + +OI: +- 当前 OI: 每 5 分钟或 15 分钟 +- 历史 OI: 每 15 分钟 + +Long/Short Ratio: +- 每 15 分钟或 1 小时 + +Taker Buy/Sell: +- 每 15 分钟或 1 小时 +``` + +--- + +### 11.3 Collector 设计 + +```go +type Collector struct { + marketData usecase.MarketDataProvider + derivatives usecase.DerivativesProvider + + klineRepo usecase.KlineRepository + fundingRepo usecase.FundingRepository + oiRepo usecase.OpenInterestRepository + lsRepo usecase.LongShortRatioRepository + + symbols []string + intervals []string +} +``` + +核心方法： + +```go +func (c *Collector) CollectKlines(ctx context.Context) error +func (c *Collector) CollectFunding(ctx context.Context) error +func (c *Collector) CollectOpenInterest(ctx context.Context) error +func (c *Collector) CollectLongShortRatio(ctx context.Context) error +``` + +--- + +## 12. MarketContextUsecase 设计 + +### 12.1 构造方法 + +```go +type MarketContextUsecase struct { + marketData MarketDataProvider + derivatives DerivativesProvider + + klineRepo KlineRepository + fundingRepo FundingRepository + oiRepo OpenInterestRepository + lsRepo LongShortRatioRepository +} + +func NewMarketContextUsecase( + marketData MarketDataProvider, + derivatives DerivativesProvider, + klineRepo KlineRepository, + fundingRepo FundingRepository, + oiRepo OpenInterestRepository, + lsRepo LongShortRatioRepository, +) *MarketContextUsecase { + return &MarketContextUsecase{ + marketData: marketData, + derivatives: derivatives, + klineRepo: klineRepo, + fundingRepo: fundingRepo, + oiRepo: oiRepo, + lsRepo: lsRepo, + } +} +``` + +--- + +### 12.2 Build 流程 + +```text +输入: +- symbol + +流程: +1. 校验 symbol +2. 并发获取 snapshot / funding / current OI +3. 从 DB 获取 15m / 1h / 4h / 1d / 1w K线 +4. 如果 DB 数据不足，则回源 Binance +5. 从 DB 获取 funding history +6. 从 DB 获取 OI history +7. 从 DB 获取 long/short ratio +8. 计算技术结构 +9. 返回 Hermes MarketContext +``` + +--- + +## 13. API 设计 + +### 13.1 Health Check + +```http +GET /v1/health +``` + +Response: + +```json +{ + "status": "ok", + "time": 1710000000000 +} +``` + +--- + +### 13.2 Market Context + +```http +GET /v1/market/context?symbol=BTCUSDT +``` + +Query: + +```text +symbol: required, example BTCUSDT +``` + +Response: + +```json +{ + "symbol": "BTCUSDT", + "generatedAt": 1710000000000, + "snapshot": {}, + "klines": {}, + "derivatives": {}, + "technical": {}, + "dataQuality": { + "source": "binance", + "warnings": [] + } +} +``` + +--- + +### 13.3 Klines + +```http +GET /v1/market/klines?symbol=BTCUSDT&interval=1h&limit=300 +``` + +Query: + +```text +symbol: required +interval: required, enum 15m/1h/4h/1d/1w +limit: optional, default 300, max 1000 +``` + +--- + +### 13.4 Snapshot + +```http +GET /v1/market/snapshot?symbol=BTCUSDT +``` + +--- + +### 13.5 Derivatives + +```http +GET /v1/market/derivatives?symbol=BTCUSDT&period=1h +``` + +Response: + +```json +{ + "symbol": "BTCUSDT", + "funding": { + "current": {}, + "history": [] + }, + "openInterest": { + "current": {}, + "history": [] + }, + "longShortRatio": { + "global": [], + "topTraderPosition": [] + }, + "takerBuySellVolume": [] +} +``` + +--- + +## 14. 配置文件 + +`config/config.yml` + +```yaml +app: + name: crypto-hermes-gateway + env: local + port: 8080 + +http: + read_timeout: 5s + write_timeout: 10s + idle_timeout: 60s + +binance: + base_url: https://fapi.binance.com + timeout: 10s + retry_count: 2 + +postgres: + dsn: postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable + max_conns: 10 + min_conns: 2 + +collector: + enabled: true + symbols: + - BTCUSDT + - ETHUSDT + intervals: + - 15m + - 1h + - 4h + - 1d + - 1w + default_limit: 500 +``` + +--- + +## 15. Docker Compose + +```yaml +version: "3.9" + +services: + postgres: + image: timescale/timescaledb:latest-pg16 + container_name: hermes-postgres + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: hermes_market + ports: + - "5432:5432" + volumes: + - hermes_pg_data:/var/lib/postgresql/data + + app: + build: . + container_name: hermes-market-gateway + depends_on: + - postgres + ports: + - "8080:8080" + environment: + CONFIG_PATH: /app/config/config.yml + volumes: + - ./config:/app/config + +volumes: + hermes_pg_data: +``` + +--- + +## 16. Makefile + +```makefile +run: + go run ./cmd/app + +test: + go test ./... + +lint: + golangci-lint run + +docker-up: + docker compose up -d + +docker-down: + docker compose down + +migrate-up: + migrate -path ./migrations -database "$(DATABASE_URL)" up + +migrate-down: + migrate -path ./migrations -database "$(DATABASE_URL)" down +``` + +--- + +## 17. Hermes 调用方式 + +Hermes 只调用： + +```http +GET /v1/market/context?symbol=BTCUSDT +``` + +然后 Hermes prompt 使用： + +```text +你是币圈量化分析引擎。请基于以下 JSON 市场上下文分析 BTCUSDT。 + +分析框架: +1. 多周期结构: 15m / 1h / 4h / 1d / 1w +2. 支撑压力: 前高、前低、箱体、成交密集区 +3. 反弹质量: 量价、taker buy volume、OI、funding +4. 情绪拥挤: global long/short、top trader long/short +5. 判断是否诱多 / 诱空 +6. 输出: + - 当前主方向 + - 上方压力 + - 下方支撑 + - 多空分水岭 + - 高空条件 + - 低多条件 + - 风险提示 + +市场上下文 JSON: +{{market_context_json}} +``` + +--- + +## 18. 开发里程碑 + +### Milestone 1：项目骨架 + +```text +目标: +- 基于 go-clean-template 思路搭建项目 +- 删除 gRPC / NATS / RabbitMQ +- 保留 REST API + +完成标准: +- GET /v1/health 返回 ok +- 项目可本地启动 +``` + +--- + +### Milestone 2：Binance WebAPI + +```text +目标: +- 实现 Binance client +- 实现 K线、ticker、funding、OI、多空比接口 + +完成标准: +- 可以成功请求 BTCUSDT / ETHUSDT +- 外部 API 错误有统一封装 +- 不在 usecase 里直接依赖 Binance client +``` + +--- + +### Milestone 3：PostgreSQL 落库 + +```text +目标: +- 建表 +- 实现 repository +- 支持 upsert + +完成标准: +- K线可入库 +- Funding 可入库 +- OI 可入库 +- Long/Short Ratio 可入库 +``` + +--- + +### Milestone 4：定时采集 + +```text +目标: +- 实现 collector +- 定时采集 BTCUSDT / ETHUSDT + +完成标准: +- 每 15 分钟采集 15m K线 +- 每 1 小时采集 1h 级别数据 +- 采集重复数据不会插入重复记录 +``` + +--- + +### Milestone 5：Market Context API + +```text +目标: +- 实现 Hermes 主接口 + +完成标准: +- GET /v1/market/context?symbol=BTCUSDT 返回完整 JSON +- 包含 snapshot、klines、funding、OI、多空比 +- 数据不足时返回 dataQuality.warnings +``` + +--- + +### Milestone 6：技术结构计算 + +```text +目标: +- 初步计算支撑压力和趋势结构 + +完成标准: +- 返回 support +- 返回 resistance +- 返回 rangeHigh / rangeLow +- 返回 longShortLine +``` + +--- + +## 19. 后续扩展 + +### v2：CoinGlass + +```text +新增: +- 清算统计 +- 清算热力图 +- 跨交易所 OI +- 跨交易所 funding +- ETF Flow +``` + +Binance WebSocket 强平流 `@forceOrder` 只能推送每个 symbol 在 1000ms 内最大强平订单快照；如果要做清算密集区、热力图、清算地图，后续更适合接 CoinGlass / Hyblock 这类专业数据源。([Binance Developer Center][5]) + +--- + +### v3：ETF Flow + +```text +新增: +- BTC ETF daily net flow +- ETH ETF daily net flow +- IBIT / FBTC / ARKB / BITB / GBTC 分项 +- 最近 7 天 / 30 天净流入 +``` + +--- + +### v4：宏观与 CME + +```text +新增: +- CPI +- FOMC +- 非农 +- 利率决议 +- CME BTC Futures K线 +- CME gap 自动计算 +``` + +--- + +## 20. 验收标准 + +第一版验收标准： + +```text +1. 服务可启动 +2. /v1/health 正常 +3. /v1/market/context?symbol=BTCUSDT 正常返回 +4. /v1/market/context?symbol=ETHUSDT 正常返回 +5. Binance client 不出现在 controller 中 +6. usecase 只依赖接口 +7. K线、Funding、OI、多空比已经落库 +8. 采集任务可重复执行且不会产生重复数据 +9. Hermes 只需要调用 /v1/market/context +10. 没有任何下单、账户、私钥、API Key 相关逻辑 +``` + +--- + +## 21. 推荐开发顺序 + +```text +第 1 步: +搭项目骨架，跑通 /v1/health + +第 2 步: +实现 Binance client + DTO mapper + +第 3 步: +实现 usecase ports + +第 4 步: +实现 /v1/market/klines 和 /v1/market/snapshot + +第 5 步: +实现数据库 migration 和 repository + +第 6 步: +实现 collector 定时落库 + +第 7 步: +实现 /v1/market/derivatives + +第 8 步: +实现 /v1/market/context + +第 9 步: +接 Hermes + +第 10 步: +补技术指标和支撑压力计算 +``` + +--- + +## 22. 最小可运行版本定义 + +MVP 只需要这一个接口稳定： + +```http +GET /v1/market/context?symbol=BTCUSDT +``` + +只要它返回： + +```text +1. 15m / 1h / 4h / 1d / 1w K线 +2. 当前价格 +3. 24h 涨跌幅 +4. 24h 成交量 +5. funding +6. OI +7. global long/short ratio +8. top trader position ratio +9. taker buy/sell volume +``` + +Hermes 就可以开始做“加密仲达系统”第一版分析。 + +[1]: https://github.com/evrone/go-clean-template?utm_source=chatgpt.com "GitHub - evrone/go-clean-template: Clean Architecture template for ..." +[2]: https://developers.binance.com/docs/derivatives/usds-margined-futures/market-data/rest-api/Kline-Candlestick-Data?utm_source=chatgpt.com "Kline Candlestick Data | Binance Open Platform" +[3]: https://developers.binance.com/docs/derivatives/usds-margined-futures/market-data/rest-api/Get-Funding-Rate-History "Get Funding Rate History | Binance Open Platform" +[4]: https://developers.binance.com/docs/derivatives/usds-margined-futures/market-data/rest-api/Long-Short-Ratio "Long Short Ratio | Binance Open Platform" +[5]: https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Liquidation-Order-Streams?utm_source=chatgpt.com "Liquidation Order Streams | Binance Open Platform" diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..7f036ea --- /dev/null +++ b/go.mod @@ -0,0 +1,37 @@ +module cryptoHermes + +go 1.23 + +require ( + github.com/gofiber/fiber/v2 v2.52.5 + github.com/ilyakaznacheev/cleanenv v1.5.0 + github.com/jackc/pgx/v5 v5.7.1 + github.com/robfig/cron/v3 v3.0.1 + golang.org/x/sync v0.8.0 + golang.org/x/time v0.6.0 +) + +require ( + github.com/BurntSushi/toml v1.2.1 // indirect + github.com/andybalholm/brotli v1.0.5 // indirect + github.com/google/uuid v1.5.0 // indirect + github.com/jackc/pgpassfile v1.0.0 // indirect + github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect + github.com/jackc/puddle/v2 v2.2.2 // indirect + github.com/joho/godotenv v1.5.1 // indirect + github.com/klauspost/compress v1.17.0 // indirect + github.com/kr/text v0.2.0 // indirect + github.com/mattn/go-colorable v0.1.13 // indirect + github.com/mattn/go-isatty v0.0.20 // indirect + github.com/mattn/go-runewidth v0.0.15 // indirect + github.com/rivo/uniseg v0.2.0 // indirect + github.com/rogpeppe/go-internal v1.14.1 // indirect + github.com/valyala/bytebufferpool v1.0.0 // indirect + github.com/valyala/fasthttp v1.51.0 // indirect + github.com/valyala/tcplisten v1.0.0 // indirect + golang.org/x/crypto v0.27.0 // indirect + golang.org/x/sys v0.26.0 // indirect + golang.org/x/text v0.18.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect + olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 // indirect +) diff --git a/go.sum b/go.sum new file mode 100644 index 0000000..41fee35 --- /dev/null +++ b/go.sum @@ -0,0 +1,76 @@ +github.com/BurntSushi/toml v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak= +github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= +github.com/andybalholm/brotli v1.0.5 h1:8uQZIdzKmjc/iuPu7O2ioW48L81FgatrcpfFmiq/cCs= +github.com/andybalholm/brotli v1.0.5/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/gofiber/fiber/v2 v2.52.5 h1:tWoP1MJQjGEe4GB5TUGOi7P2E0ZMMRx5ZTG4rT+yGMo= +github.com/gofiber/fiber/v2 v2.52.5/go.mod h1:KEOE+cXMhXG0zHc9d8+E38hoX+ZN7bhOtgeF2oT6jrQ= +github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU= +github.com/google/uuid v1.5.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/ilyakaznacheev/cleanenv v1.5.0 h1:0VNZXggJE2OYdXE87bfSSwGxeiGt9moSR2lOrsHHvr4= +github.com/ilyakaznacheev/cleanenv v1.5.0/go.mod h1:a5aDzaJrLCQZsazHol1w8InnDcOX0OColm64SlIi6gk= +github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= +github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= +github.com/jackc/pgx/v5 v5.7.1 h1:x7SYsPBYDkHDksogeSmZZ5xzThcTgRz++I5E+ePFUcs= +github.com/jackc/pgx/v5 v5.7.1/go.mod h1:e7O26IywZZ+naJtWWos6i6fvWK+29etgITqrqHLfoZA= +github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= +github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= +github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0= +github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4= +github.com/klauspost/compress v1.17.0 h1:Rnbp4K9EjcDuVuHtd0dgA4qNuv9yKDYKK1ulpJwgrqM= +github.com/klauspost/compress v1.17.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= +github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0= +github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= +github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= +github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= +github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= +github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/mattn/go-runewidth v0.0.15 h1:UNAjwbU9l54TA3KzvqLGxwWjHmMgBUVhBiTjelZgg3U= +github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY= +github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= +github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs= +github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro= +github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= +github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw= +github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= +github.com/valyala/fasthttp v1.51.0 h1:8b30A5JlZ6C7AS81RsWjYMQmrZG6feChmgAolCl1SqA= +github.com/valyala/fasthttp v1.51.0/go.mod h1:oI2XroL+lI7vdXyYoQk03bXBThfFl2cVdIA3Xl7cH8g= +github.com/valyala/tcplisten v1.0.0 h1:rBHj/Xf+E1tRGZyWIWwJDiRY0zc1Js+CV5DqwacVSA8= +github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc= +golang.org/x/crypto v0.27.0 h1:GXm2NjJrPaiv/h1tb2UH8QfgC/hOf/+z0p6PT8o1w7A= +golang.org/x/crypto v0.27.0/go.mod h1:1Xngt8kV6Dvbssa53Ziq6Eqn0HqbZi5Z6R0ZpwQzt70= +golang.org/x/sync v0.8.0 h1:3NFvSEYkUoMifnESzZl15y791HH1qU2xm6eCJU5ZPXQ= +golang.org/x/sync v0.8.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo= +golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224= +golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= +golang.org/x/time v0.6.0 h1:eTDhh4ZXt5Qf0augr54TN6suAUudPcawVZeIAPU7D4U= +golang.org/x/time v0.6.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 h1:slmdOY3vp8a7KQbHkL+FLbvbkgMqmXojpFUO/jENuqQ= +olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3/go.mod h1:oVgVk4OWVDi43qWBEyGhXgYxt7+ED4iYNpTngSLX2Iw= diff --git a/internal/app/app.go b/internal/app/app.go new file mode 100644 index 0000000..fce8615 --- /dev/null +++ b/internal/app/app.go @@ -0,0 +1,121 @@ +package app + +import ( + "context" + "fmt" + "log/slog" + "os" + "os/signal" + "syscall" + "time" + + "github.com/gofiber/fiber/v2" + + "cryptoHermes/config" + "cryptoHermes/internal/controller/restapi" + "cryptoHermes/internal/repo/persistent/postgres" + "cryptoHermes/internal/repo/webapi/binance" + "cryptoHermes/internal/usecase" + "cryptoHermes/internal/worker" + pgpkg "cryptoHermes/pkg/postgres" +) + +func Run(cfg *config.Config, log *slog.Logger) error { + rootCtx, cancel := context.WithCancel(context.Background()) + defer cancel() + + pool, err := pgpkg.NewPool(rootCtx, pgpkg.Options{ + DSN: cfg.Postgres.DSN, + MaxConns: cfg.Postgres.MaxConns, + MinConns: cfg.Postgres.MinConns, + }) + if err != nil { + return fmt.Errorf("postgres: %w", err) + } + defer pool.Close() + + binanceClient := binance.NewClient(binance.ClientOptions{ + BaseURL: cfg.Binance.BaseURL, + Timeout: cfg.Binance.Timeout, + RetryCount: cfg.Binance.RetryCount, + RPS: cfg.Binance.RPS, + Burst: cfg.Binance.Burst, + }) + + klineRepo := postgres.NewKlineRepo(pool) + fundingRepo := postgres.NewFundingRepo(pool) + oiRepo := postgres.NewOpenInterestRepo(pool) + lsRepo := postgres.NewLongShortRatioRepo(pool) + takerRepo := postgres.NewTakerVolumeRepo(pool) + + contextUC := usecase.NewMarketContextUsecase( + binanceClient, + binanceClient, + klineRepo, + fundingRepo, + oiRepo, + lsRepo, + log, + ) + + collector := worker.NewCollector(worker.Deps{ + MarketData: binanceClient, + Derivatives: binanceClient, + KlineRepo: klineRepo, + FundingRepo: fundingRepo, + OIRepo: oiRepo, + LSRepo: lsRepo, + TakerRepo: takerRepo, + Symbols: cfg.Collector.Symbols, + Intervals: cfg.Collector.Intervals, + Limit: cfg.Collector.DefaultLimit, + Logger: log, + }) + + var sched *Scheduler + if cfg.Collector.Enabled { + sched = NewScheduler(collector, log) + sched.Start(rootCtx) + defer sched.Stop() + } + + fiberApp := fiber.New(fiber.Config{ + AppName: cfg.App.Name, + ReadTimeout: cfg.HTTP.ReadTimeout, + WriteTimeout: cfg.HTTP.WriteTimeout, + IdleTimeout: cfg.HTTP.IdleTimeout, + ErrorHandler: restapi.ErrorHandler, + }) + + restapi.Register(fiberApp, restapi.Deps{ + Logger: log, + MarketContext: contextUC, + MarketData: binanceClient, + Derivatives: binanceClient, + KlineRepo: klineRepo, + FundingRepo: fundingRepo, + OIRepo: oiRepo, + LSRepo: lsRepo, + }) + + listenErr := make(chan error, 1) + go func() { + addr := fmt.Sprintf(":%d", cfg.App.Port) + log.Info("http_listening", "addr", addr) + listenErr <- fiberApp.Listen(addr) + }() + + stop := make(chan os.Signal, 1) + signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM) + + select { + case err := <-listenErr: + return err + case sig := <-stop: + log.Info("shutdown_signal", "signal", sig.String()) + } + + shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second) + defer shutdownCancel() + return fiberApp.ShutdownWithContext(shutdownCtx) +} diff --git a/internal/app/scheduler.go b/internal/app/scheduler.go new file mode 100644 index 0000000..55eee55 --- /dev/null +++ b/internal/app/scheduler.go @@ -0,0 +1,58 @@ +package app + +import ( + "context" + "log/slog" + + "github.com/robfig/cron/v3" + + "cryptoHermes/internal/worker" +) + +type Scheduler struct { + cron *cron.Cron + collector *worker.Collector + log *slog.Logger + rootCtx context.Context +} + +func NewScheduler(c *worker.Collector, log *slog.Logger) *Scheduler { + return &Scheduler{ + cron: cron.New(), + collector: c, + log: log, + } +} + +func (s *Scheduler) Start(ctx context.Context) { + s.rootCtx = ctx + + s.add("*/15 * * * *", "klines_15m", func(c context.Context) { _ = s.collector.CollectKlines(c, "15m") }) + s.add("0 * * * *", "klines_1h", func(c context.Context) { _ = s.collector.CollectKlines(c, "1h") }) + s.add("0 */4 * * *", "klines_4h", func(c context.Context) { _ = s.collector.CollectKlines(c, "4h") }) + s.add("5 0 * * *", "klines_1d", func(c context.Context) { _ = s.collector.CollectKlines(c, "1d") }) + s.add("10 0 * * 1", "klines_1w", func(c context.Context) { _ = s.collector.CollectKlines(c, "1w") }) + + s.add("*/15 * * * *", "funding", func(c context.Context) { _ = s.collector.CollectFunding(c) }) + s.add("*/15 * * * *", "oi", func(c context.Context) { _ = s.collector.CollectOpenInterest(c) }) + s.add("*/15 * * * *", "ls", func(c context.Context) { _ = s.collector.CollectLongShortRatio(c) }) + s.add("*/15 * * * *", "taker", func(c context.Context) { _ = s.collector.CollectTakerVolume(c) }) + + s.cron.Start() + s.log.Info("scheduler_started") +} + +func (s *Scheduler) add(spec, name string, fn func(context.Context)) { + _, err := s.cron.AddFunc(spec, func() { + s.log.Info("cron_tick", "job", name) + fn(s.rootCtx) + }) + if err != nil { + s.log.Error("cron_add_failed", "job", name, "err", err) + } +} + +func (s *Scheduler) Stop() { + s.log.Info("scheduler_stopping") + <-s.cron.Stop().Done() +} diff --git a/internal/controller/restapi/middleware.go b/internal/controller/restapi/middleware.go new file mode 100644 index 0000000..40bed9f --- /dev/null +++ b/internal/controller/restapi/middleware.go @@ -0,0 +1,38 @@ +package restapi + +import ( + "log/slog" + "time" + + "github.com/gofiber/fiber/v2" +) + +func RequestLogger(log *slog.Logger) fiber.Handler { + return func(c *fiber.Ctx) error { + start := time.Now() + err := c.Next() + log.Info("http_request", + "method", c.Method(), + "path", c.Path(), + "status", c.Response().StatusCode(), + "duration_ms", time.Since(start).Milliseconds(), + "ip", c.IP(), + ) + return err + } +} + +func ErrorHandler(c *fiber.Ctx, err error) error { + code := fiber.StatusInternalServerError + msg := "internal error" + if fe, ok := err.(*fiber.Error); ok { + code = fe.Code + msg = fe.Message + } + return c.Status(code).JSON(fiber.Map{ + "error": fiber.Map{ + "code": code, + "message": msg, + }, + }) +} diff --git a/internal/controller/restapi/router.go b/internal/controller/restapi/router.go new file mode 100644 index 0000000..9768270 --- /dev/null +++ b/internal/controller/restapi/router.go @@ -0,0 +1,39 @@ +package restapi + +import ( + "log/slog" + + "github.com/gofiber/fiber/v2" + "github.com/gofiber/fiber/v2/middleware/recover" + + v1 "cryptoHermes/internal/controller/restapi/v1" + "cryptoHermes/internal/usecase" +) + +type Deps struct { + Logger *slog.Logger + MarketContext *usecase.MarketContextUsecase + MarketData usecase.MarketDataProvider + Derivatives usecase.DerivativesProvider + KlineRepo usecase.KlineRepository + FundingRepo usecase.FundingRepository + OIRepo usecase.OpenInterestRepository + LSRepo usecase.LongShortRatioRepository +} + +func Register(app *fiber.App, deps Deps) { + app.Use(recover.New()) + app.Use(RequestLogger(deps.Logger)) + + api := app.Group("/v1") + v1.RegisterHealth(api) + v1.RegisterMarket(api, v1.MarketDeps{ + MarketContext: deps.MarketContext, + MarketData: deps.MarketData, + Derivatives: deps.Derivatives, + KlineRepo: deps.KlineRepo, + FundingRepo: deps.FundingRepo, + OIRepo: deps.OIRepo, + LSRepo: deps.LSRepo, + }) +} diff --git a/internal/controller/restapi/v1/health_routes.go b/internal/controller/restapi/v1/health_routes.go new file mode 100644 index 0000000..33f61d1 --- /dev/null +++ b/internal/controller/restapi/v1/health_routes.go @@ -0,0 +1,16 @@ +package v1 + +import ( + "time" + + "github.com/gofiber/fiber/v2" +) + +func RegisterHealth(r fiber.Router) { + r.Get("/health", func(c *fiber.Ctx) error { + return c.JSON(fiber.Map{ + "status": "ok", + "time": time.Now().UnixMilli(), + }) + }) +} diff --git a/internal/controller/restapi/v1/market_routes.go b/internal/controller/restapi/v1/market_routes.go new file mode 100644 index 0000000..605bd40 --- /dev/null +++ b/internal/controller/restapi/v1/market_routes.go @@ -0,0 +1,118 @@ +package v1 + +import ( + "strconv" + "strings" + + "github.com/gofiber/fiber/v2" + + "cryptoHermes/internal/entity" + "cryptoHermes/internal/usecase" +) + +type MarketDeps struct { + MarketContext *usecase.MarketContextUsecase + MarketData usecase.MarketDataProvider + Derivatives usecase.DerivativesProvider + KlineRepo usecase.KlineRepository + FundingRepo usecase.FundingRepository + OIRepo usecase.OpenInterestRepository + LSRepo usecase.LongShortRatioRepository +} + +var allowedIntervals = map[string]bool{ + "15m": true, "1h": true, "4h": true, "1d": true, "1w": true, +} + +func RegisterMarket(r fiber.Router, d MarketDeps) { + g := r.Group("/market") + + g.Get("/context", func(c *fiber.Ctx) error { + symbol := strings.ToUpper(c.Query("symbol")) + if symbol == "" { + return fiber.NewError(fiber.StatusBadRequest, "symbol is required") + } + out, err := d.MarketContext.Build(c.UserContext(), symbol) + if err != nil && out == nil { + return fiber.NewError(fiber.StatusBadRequest, err.Error()) + } + return c.JSON(out) + }) + + g.Get("/klines", func(c *fiber.Ctx) error { + symbol := strings.ToUpper(c.Query("symbol")) + interval := c.Query("interval") + if symbol == "" || interval == "" { + return fiber.NewError(fiber.StatusBadRequest, "symbol and interval are required") + } + if !allowedIntervals[interval] { + return fiber.NewError(fiber.StatusBadRequest, "interval must be one of 15m/1h/4h/1d/1w") + } + limit := 300 + if l := c.Query("limit"); l != "" { + if v, err := strconv.Atoi(l); err == nil && v > 0 && v <= 1000 { + limit = v + } + } + rows, err := d.KlineRepo.FindRecent(c.UserContext(), symbol, interval, limit) + if err != nil { + return fiber.NewError(fiber.StatusInternalServerError, err.Error()) + } + return c.JSON(rows) + }) + + g.Get("/snapshot", func(c *fiber.Ctx) error { + symbol := strings.ToUpper(c.Query("symbol")) + if symbol == "" { + return fiber.NewError(fiber.StatusBadRequest, "symbol is required") + } + t, err := d.MarketData.GetTicker24h(c.UserContext(), symbol) + if err != nil { + return fiber.NewError(fiber.StatusBadGateway, err.Error()) + } + return c.JSON(t) + }) + + g.Get("/derivatives", func(c *fiber.Ctx) error { + symbol := strings.ToUpper(c.Query("symbol")) + period := c.Query("period", "1h") + if symbol == "" { + return fiber.NewError(fiber.StatusBadRequest, "symbol is required") + } + + fundCur, ferr := d.Derivatives.GetCurrentFunding(c.UserContext(), symbol) + oiCur, oerr := d.Derivatives.GetCurrentOpenInterest(c.UserContext(), symbol) + + fundHist, _ := d.FundingRepo.FindRecent(c.UserContext(), symbol, 100) + oiHist, _ := d.OIRepo.FindRecent(c.UserContext(), symbol, period, 200) + globalLS, _ := d.LSRepo.FindRecent(c.UserContext(), symbol, period, entity.RatioTypeGlobalAccount, 200) + topLS, _ := d.LSRepo.FindRecent(c.UserContext(), symbol, period, entity.RatioTypeTopTraderPosition, 200) + + resp := fiber.Map{ + "symbol": symbol, + "funding": fiber.Map{ + "current": fundCur, + "history": fundHist, + "error": errString(ferr), + }, + "openInterest": fiber.Map{ + "current": oiCur, + "history": oiHist, + "error": errString(oerr), + }, + "longShortRatio": fiber.Map{ + "global": globalLS, + "topTraderPosition": topLS, + }, + "takerBuySellVolume": []any{}, + } + return c.JSON(resp) + }) +} + +func errString(e error) string { + if e == nil { + return "" + } + return e.Error() +} diff --git a/internal/entity/funding.go b/internal/entity/funding.go new file mode 100644 index 0000000..9c2cc21 --- /dev/null +++ b/internal/entity/funding.go @@ -0,0 +1,9 @@ +package entity + +type FundingRate struct { + Source string `json:"source"` + Symbol string `json:"symbol"` + FundingTime int64 `json:"fundingTime"` + FundingRate string `json:"fundingRate"` + MarkPrice string `json:"markPrice,omitempty"` +} diff --git a/internal/entity/kline.go b/internal/entity/kline.go new file mode 100644 index 0000000..c4d7a96 --- /dev/null +++ b/internal/entity/kline.go @@ -0,0 +1,19 @@ +package entity + +type Kline struct { + Source string `json:"source"` + Symbol string `json:"symbol"` + Interval string `json:"interval"` + OpenTime int64 `json:"openTime"` + CloseTime int64 `json:"closeTime"` + Open string `json:"open"` + High string `json:"high"` + Low string `json:"low"` + Close string `json:"close"` + Volume string `json:"volume"` + QuoteVolume string `json:"quoteVolume"` + TradeCount int64 `json:"tradeCount"` + TakerBuyBaseVolume string `json:"takerBuyBaseVolume"` + TakerBuyQuoteVolume string `json:"takerBuyQuoteVolume"` + IsClosed bool `json:"-"` +} diff --git a/internal/entity/long_short_ratio.go b/internal/entity/long_short_ratio.go new file mode 100644 index 0000000..b968fc2 --- /dev/null +++ b/internal/entity/long_short_ratio.go @@ -0,0 +1,18 @@ +package entity + +const ( + RatioTypeGlobalAccount = "global_account" + RatioTypeTopTraderPosition = "top_trader_position" + RatioTypeTopTraderAccount = "top_trader_account" +) + +type LongShortRatio struct { + Source string `json:"source"` + Symbol string `json:"symbol"` + Period string `json:"period"` + RatioType string `json:"ratioType"` + Timestamp int64 `json:"timestamp"` + LongShortRatio string `json:"longShortRatio"` + LongValue string `json:"longValue,omitempty"` + ShortValue string `json:"shortValue,omitempty"` +} diff --git a/internal/entity/market_context.go b/internal/entity/market_context.go new file mode 100644 index 0000000..601e611 --- /dev/null +++ b/internal/entity/market_context.go @@ -0,0 +1,52 @@ +package entity + +type MarketContext struct { + Symbol string `json:"symbol"` + GeneratedAt int64 `json:"generatedAt"` + Snapshot *Ticker24h `json:"snapshot"` + Klines map[string][]Kline `json:"klines"` + Derivatives DerivativesBundle `json:"derivatives"` + Technical TechnicalStructure `json:"technical"` + DataQuality DataQuality `json:"dataQuality"` +} + +type DerivativesBundle struct { + Funding FundingBundle `json:"funding"` + OpenInterest OpenInterestBundle `json:"openInterest"` + LongShortRatio LongShortBundle `json:"longShortRatio"` + TakerBuySellVolume []TakerBuySellVolume `json:"takerBuySellVolume"` +} + +type FundingBundle struct { + Current *FundingRate `json:"current"` + History []FundingRate `json:"history"` +} + +type OpenInterestBundle struct { + Current *OpenInterest `json:"current"` + History []OpenInterest `json:"history"` +} + +type LongShortBundle struct { + Global []LongShortRatio `json:"global"` + TopTraderPosition []LongShortRatio `json:"topTraderPosition"` +} + +type TechnicalStructure struct { + Support []TechnicalLevel `json:"support"` + Resistance []TechnicalLevel `json:"resistance"` + RangeHigh *string `json:"rangeHigh"` + RangeLow *string `json:"rangeLow"` + LongShortLine *string `json:"longShortLine"` +} + +type TechnicalLevel struct { + Price string `json:"price"` + Strength string `json:"strength,omitempty"` + Source string `json:"source,omitempty"` +} + +type DataQuality struct { + Source string `json:"source"` + Warnings []string `json:"warnings"` +} diff --git a/internal/entity/open_interest.go b/internal/entity/open_interest.go new file mode 100644 index 0000000..c227d54 --- /dev/null +++ b/internal/entity/open_interest.go @@ -0,0 +1,10 @@ +package entity + +type OpenInterest struct { + Source string `json:"source"` + Symbol string `json:"symbol"` + Period string `json:"period"` + Timestamp int64 `json:"timestamp"` + OpenInterest string `json:"openInterest"` + OpenInterestValue string `json:"openInterestValue,omitempty"` +} diff --git a/internal/entity/taker_volume.go b/internal/entity/taker_volume.go new file mode 100644 index 0000000..9e58ebe --- /dev/null +++ b/internal/entity/taker_volume.go @@ -0,0 +1,11 @@ +package entity + +type TakerBuySellVolume struct { + Source string `json:"source"` + Symbol string `json:"symbol"` + Period string `json:"period"` + Timestamp int64 `json:"timestamp"` + BuySellRatio string `json:"buySellRatio,omitempty"` + BuyVolume string `json:"buyVolume,omitempty"` + SellVolume string `json:"sellVolume,omitempty"` +} diff --git a/internal/entity/ticker.go b/internal/entity/ticker.go new file mode 100644 index 0000000..2e9861d --- /dev/null +++ b/internal/entity/ticker.go @@ -0,0 +1,14 @@ +package entity + +type Ticker24h struct { + Symbol string `json:"symbol"` + LastPrice string `json:"lastPrice"` + PriceChange string `json:"priceChange"` + PriceChangePercent string `json:"priceChangePercent"` + HighPrice string `json:"highPrice"` + LowPrice string `json:"lowPrice"` + Volume string `json:"volume"` + QuoteVolume string `json:"quoteVolume"` + OpenTime int64 `json:"openTime"` + CloseTime int64 `json:"closeTime"` +} diff --git a/internal/repo/persistent/postgres/funding_repo.go b/internal/repo/persistent/postgres/funding_repo.go new file mode 100644 index 0000000..da9bac4 --- /dev/null +++ b/internal/repo/persistent/postgres/funding_repo.go @@ -0,0 +1,86 @@ +package postgres + +import ( + "context" + "database/sql" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" + + "cryptoHermes/internal/entity" +) + +type FundingRepo struct { + pool *pgxpool.Pool +} + +func NewFundingRepo(pool *pgxpool.Pool) *FundingRepo { + return &FundingRepo{pool: pool} +} + +const fundingUpsertSQL = ` +INSERT INTO funding_rates (source, symbol, funding_time, funding_rate, mark_price) +VALUES ($1, $2, $3, $4, NULLIF($5, '')::NUMERIC) +ON CONFLICT (source, symbol, funding_time) DO UPDATE SET + funding_rate = EXCLUDED.funding_rate, + mark_price = EXCLUDED.mark_price +` + +func (r *FundingRepo) UpsertMany(ctx context.Context, items []entity.FundingRate) error { + if len(items) == 0 { + return nil + } + tx, err := r.pool.Begin(ctx) + if err != nil { + return err + } + defer tx.Rollback(ctx) + + for _, f := range items { + if f.FundingRate == "" || f.FundingTime == 0 { + continue + } + if _, err := tx.Exec(ctx, fundingUpsertSQL, + f.Source, f.Symbol, f.FundingTime, f.FundingRate, f.MarkPrice, + ); err != nil { + return fmt.Errorf("upsert funding %s@%d: %w", f.Symbol, f.FundingTime, err) + } + } + return tx.Commit(ctx) +} + +const fundingFindSQL = ` +SELECT source, symbol, funding_time, funding_rate::text, COALESCE(mark_price::text, '') +FROM funding_rates +WHERE symbol = $1 +ORDER BY funding_time DESC +LIMIT $2 +` + +func (r *FundingRepo) FindRecent(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error) { + if limit <= 0 { + limit = 100 + } + rows, err := r.pool.Query(ctx, fundingFindSQL, symbol, limit) + if err != nil { + return nil, err + } + defer rows.Close() + + out := make([]entity.FundingRate, 0, limit) + for rows.Next() { + var f entity.FundingRate + var mark sql.NullString + if err := rows.Scan(&f.Source, &f.Symbol, &f.FundingTime, &f.FundingRate, &mark); err != nil { + return nil, err + } + if mark.Valid { + f.MarkPrice = mark.String + } + out = append(out, f) + } + for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 { + out[i], out[j] = out[j], out[i] + } + return out, rows.Err() +} diff --git a/internal/repo/persistent/postgres/kline_repo.go b/internal/repo/persistent/postgres/kline_repo.go new file mode 100644 index 0000000..473de43 --- /dev/null +++ b/internal/repo/persistent/postgres/kline_repo.go @@ -0,0 +1,114 @@ +package postgres + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" + + "cryptoHermes/internal/entity" +) + +type KlineRepo struct { + pool *pgxpool.Pool +} + +func NewKlineRepo(pool *pgxpool.Pool) *KlineRepo { + return &KlineRepo{pool: pool} +} + +const klineUpsertSQL = ` +INSERT INTO market_klines ( + source, symbol, interval, open_time, close_time, + open, high, low, close, + volume, quote_volume, + trade_count, taker_buy_base_volume, taker_buy_quote_volume, + updated_at +) VALUES ( + $1, $2, $3, $4, $5, + $6, $7, $8, $9, + $10, $11, + $12, $13, $14, + now() +) +ON CONFLICT (source, symbol, interval, open_time) DO UPDATE SET + close_time = EXCLUDED.close_time, + open = EXCLUDED.open, + high = EXCLUDED.high, + low = EXCLUDED.low, + close = EXCLUDED.close, + volume = EXCLUDED.volume, + quote_volume = EXCLUDED.quote_volume, + trade_count = EXCLUDED.trade_count, + taker_buy_base_volume = EXCLUDED.taker_buy_base_volume, + taker_buy_quote_volume = EXCLUDED.taker_buy_quote_volume, + updated_at = now() +` + +func (r *KlineRepo) UpsertMany(ctx context.Context, items []entity.Kline) error { + if len(items) == 0 { + return nil + } + tx, err := r.pool.Begin(ctx) + if err != nil { + return fmt.Errorf("begin: %w", err) + } + defer tx.Rollback(ctx) + + for _, k := range items { + if !k.IsClosed { + continue + } + _, err := tx.Exec(ctx, klineUpsertSQL, + k.Source, k.Symbol, k.Interval, k.OpenTime, k.CloseTime, + k.Open, k.High, k.Low, k.Close, + k.Volume, k.QuoteVolume, + k.TradeCount, k.TakerBuyBaseVolume, k.TakerBuyQuoteVolume, + ) + if err != nil { + return fmt.Errorf("upsert kline %s/%s@%d: %w", k.Symbol, k.Interval, k.OpenTime, err) + } + } + return tx.Commit(ctx) +} + +const klineFindSQL = ` +SELECT source, symbol, interval, open_time, close_time, + open::text, high::text, low::text, close::text, + volume::text, quote_volume::text, + trade_count, taker_buy_base_volume::text, taker_buy_quote_volume::text +FROM market_klines +WHERE symbol = $1 AND interval = $2 +ORDER BY open_time DESC +LIMIT $3 +` + +func (r *KlineRepo) FindRecent(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error) { + if limit <= 0 { + limit = 300 + } + rows, err := r.pool.Query(ctx, klineFindSQL, symbol, interval, limit) + if err != nil { + return nil, fmt.Errorf("query: %w", err) + } + defer rows.Close() + + out := make([]entity.Kline, 0, limit) + for rows.Next() { + var k entity.Kline + if err := rows.Scan( + &k.Source, &k.Symbol, &k.Interval, &k.OpenTime, &k.CloseTime, + &k.Open, &k.High, &k.Low, &k.Close, + &k.Volume, &k.QuoteVolume, + &k.TradeCount, &k.TakerBuyBaseVolume, &k.TakerBuyQuoteVolume, + ); err != nil { + return nil, fmt.Errorf("scan: %w", err) + } + k.IsClosed = true + out = append(out, k) + } + for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 { + out[i], out[j] = out[j], out[i] + } + return out, rows.Err() +} diff --git a/internal/repo/persistent/postgres/long_short_ratio_repo.go b/internal/repo/persistent/postgres/long_short_ratio_repo.go new file mode 100644 index 0000000..ed9e8ed --- /dev/null +++ b/internal/repo/persistent/postgres/long_short_ratio_repo.go @@ -0,0 +1,94 @@ +package postgres + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" + + "cryptoHermes/internal/entity" +) + +type LongShortRatioRepo struct { + pool *pgxpool.Pool +} + +func NewLongShortRatioRepo(pool *pgxpool.Pool) *LongShortRatioRepo { + return &LongShortRatioRepo{pool: pool} +} + +const lsUpsertSQL = ` +INSERT INTO long_short_ratio ( + source, symbol, period, ratio_type, timestamp, + long_short_ratio, long_value, short_value +) VALUES ( + $1, $2, $3, $4, $5, + $6, + NULLIF($7, '')::NUMERIC, + NULLIF($8, '')::NUMERIC +) +ON CONFLICT (source, symbol, period, ratio_type, timestamp) DO UPDATE SET + long_short_ratio = EXCLUDED.long_short_ratio, + long_value = EXCLUDED.long_value, + short_value = EXCLUDED.short_value +` + +func (r *LongShortRatioRepo) UpsertMany(ctx context.Context, items []entity.LongShortRatio) error { + if len(items) == 0 { + return nil + } + tx, err := r.pool.Begin(ctx) + if err != nil { + return err + } + defer tx.Rollback(ctx) + + for _, ls := range items { + if ls.LongShortRatio == "" || ls.Timestamp == 0 { + continue + } + if _, err := tx.Exec(ctx, lsUpsertSQL, + ls.Source, ls.Symbol, ls.Period, ls.RatioType, ls.Timestamp, + ls.LongShortRatio, ls.LongValue, ls.ShortValue, + ); err != nil { + return fmt.Errorf("upsert ls %s/%s/%s@%d: %w", ls.Symbol, ls.Period, ls.RatioType, ls.Timestamp, err) + } + } + return tx.Commit(ctx) +} + +const lsFindSQL = ` +SELECT source, symbol, period, ratio_type, timestamp, + long_short_ratio::text, + COALESCE(long_value::text, ''), + COALESCE(short_value::text, '') +FROM long_short_ratio +WHERE symbol = $1 AND period = $2 AND ratio_type = $3 +ORDER BY timestamp DESC +LIMIT $4 +` + +func (r *LongShortRatioRepo) FindRecent(ctx context.Context, symbol, period, ratioType string, limit int) ([]entity.LongShortRatio, error) { + if limit <= 0 { + limit = 100 + } + rows, err := r.pool.Query(ctx, lsFindSQL, symbol, period, ratioType, limit) + if err != nil { + return nil, err + } + defer rows.Close() + + out := make([]entity.LongShortRatio, 0, limit) + for rows.Next() { + var ls entity.LongShortRatio + if err := rows.Scan(&ls.Source, &ls.Symbol, &ls.Period, &ls.RatioType, &ls.Timestamp, + &ls.LongShortRatio, &ls.LongValue, &ls.ShortValue); err != nil { + return nil, err + } + out = append(out, ls) + } + for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 { + out[i], out[j] = out[j], out[i] + } + return out, rows.Err() +} diff --git a/internal/repo/persistent/postgres/open_interest_repo.go b/internal/repo/persistent/postgres/open_interest_repo.go new file mode 100644 index 0000000..8592948 --- /dev/null +++ b/internal/repo/persistent/postgres/open_interest_repo.go @@ -0,0 +1,82 @@ +package postgres + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" + + "cryptoHermes/internal/entity" +) + +type OpenInterestRepo struct { + pool *pgxpool.Pool +} + +func NewOpenInterestRepo(pool *pgxpool.Pool) *OpenInterestRepo { + return &OpenInterestRepo{pool: pool} +} + +const oiUpsertSQL = ` +INSERT INTO open_interest (source, symbol, period, timestamp, open_interest, open_interest_value) +VALUES ($1, $2, $3, $4, $5, NULLIF($6, '')::NUMERIC) +ON CONFLICT (source, symbol, period, timestamp) DO UPDATE SET + open_interest = EXCLUDED.open_interest, + open_interest_value = EXCLUDED.open_interest_value +` + +func (r *OpenInterestRepo) UpsertMany(ctx context.Context, items []entity.OpenInterest) error { + if len(items) == 0 { + return nil + } + tx, err := r.pool.Begin(ctx) + if err != nil { + return err + } + defer tx.Rollback(ctx) + + for _, o := range items { + if o.OpenInterest == "" || o.Timestamp == 0 { + continue + } + if _, err := tx.Exec(ctx, oiUpsertSQL, + o.Source, o.Symbol, o.Period, o.Timestamp, o.OpenInterest, o.OpenInterestValue, + ); err != nil { + return fmt.Errorf("upsert oi %s/%s@%d: %w", o.Symbol, o.Period, o.Timestamp, err) + } + } + return tx.Commit(ctx) +} + +const oiFindSQL = ` +SELECT source, symbol, period, timestamp, + open_interest::text, COALESCE(open_interest_value::text, '') +FROM open_interest +WHERE symbol = $1 AND period = $2 +ORDER BY timestamp DESC +LIMIT $3 +` + +func (r *OpenInterestRepo) FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error) { + if limit <= 0 { + limit = 100 + } + rows, err := r.pool.Query(ctx, oiFindSQL, symbol, period, limit) + if err != nil { + return nil, err + } + defer rows.Close() + + out := make([]entity.OpenInterest, 0, limit) + for rows.Next() { + var o entity.OpenInterest + if err := rows.Scan(&o.Source, &o.Symbol, &o.Period, &o.Timestamp, &o.OpenInterest, &o.OpenInterestValue); err != nil { + return nil, err + } + out = append(out, o) + } + for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 { + out[i], out[j] = out[j], out[i] + } + return out, rows.Err() +} diff --git a/internal/repo/persistent/postgres/taker_volume_repo.go b/internal/repo/persistent/postgres/taker_volume_repo.go new file mode 100644 index 0000000..16c1686 --- /dev/null +++ b/internal/repo/persistent/postgres/taker_volume_repo.go @@ -0,0 +1,90 @@ +package postgres + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" + + "cryptoHermes/internal/entity" +) + +type TakerVolumeRepo struct { + pool *pgxpool.Pool +} + +func NewTakerVolumeRepo(pool *pgxpool.Pool) *TakerVolumeRepo { + return &TakerVolumeRepo{pool: pool} +} + +const takerUpsertSQL = ` +INSERT INTO taker_buy_sell_volume (source, symbol, period, timestamp, buy_sell_ratio, buy_volume, sell_volume) +VALUES ($1, $2, $3, $4, + NULLIF($5, '')::NUMERIC, + NULLIF($6, '')::NUMERIC, + NULLIF($7, '')::NUMERIC) +ON CONFLICT (source, symbol, period, timestamp) DO UPDATE SET + buy_sell_ratio = EXCLUDED.buy_sell_ratio, + buy_volume = EXCLUDED.buy_volume, + sell_volume = EXCLUDED.sell_volume +` + +func (r *TakerVolumeRepo) UpsertMany(ctx context.Context, items []entity.TakerBuySellVolume) error { + if len(items) == 0 { + return nil + } + tx, err := r.pool.Begin(ctx) + if err != nil { + return err + } + defer tx.Rollback(ctx) + + for _, t := range items { + if t.Timestamp == 0 { + continue + } + if _, err := tx.Exec(ctx, takerUpsertSQL, + t.Source, t.Symbol, t.Period, t.Timestamp, + t.BuySellRatio, t.BuyVolume, t.SellVolume, + ); err != nil { + return fmt.Errorf("upsert taker %s/%s@%d: %w", t.Symbol, t.Period, t.Timestamp, err) + } + } + return tx.Commit(ctx) +} + +const takerFindSQL = ` +SELECT source, symbol, period, timestamp, + COALESCE(buy_sell_ratio::text, ''), + COALESCE(buy_volume::text, ''), + COALESCE(sell_volume::text, '') +FROM taker_buy_sell_volume +WHERE symbol = $1 AND period = $2 +ORDER BY timestamp DESC +LIMIT $3 +` + +func (r *TakerVolumeRepo) FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error) { + if limit <= 0 { + limit = 100 + } + rows, err := r.pool.Query(ctx, takerFindSQL, symbol, period, limit) + if err != nil { + return nil, err + } + defer rows.Close() + + out := make([]entity.TakerBuySellVolume, 0, limit) + for rows.Next() { + var t entity.TakerBuySellVolume + if err := rows.Scan(&t.Source, &t.Symbol, &t.Period, &t.Timestamp, + &t.BuySellRatio, &t.BuyVolume, &t.SellVolume); err != nil { + return nil, err + } + out = append(out, t) + } + for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 { + out[i], out[j] = out[j], out[i] + } + return out, rows.Err() +} diff --git a/internal/repo/webapi/binance/client.go b/internal/repo/webapi/binance/client.go new file mode 100644 index 0000000..20b7f3b --- /dev/null +++ b/internal/repo/webapi/binance/client.go @@ -0,0 +1,72 @@ +package binance + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" + "time" + + "cryptoHermes/pkg/httpclient" +) + +const sourceName = "binance" + +type Client struct { + http *httpclient.Client + source string +} + +type ClientOptions struct { + BaseURL string + Timeout time.Duration + RetryCount int + RPS float64 + Burst int +} + +func NewClient(opts ClientOptions) *Client { + return &Client{ + http: httpclient.New(httpclient.Options{ + BaseURL: opts.BaseURL, + Timeout: opts.Timeout, + RetryCount: opts.RetryCount, + RPS: opts.RPS, + Burst: opts.Burst, + }), + source: sourceName, + } +} + +type ExternalAPIError struct { + Source string + Path string + StatusCode int + Message string +} + +func (e *ExternalAPIError) Error() string { + return fmt.Sprintf("%s %s status=%d: %s", e.Source, e.Path, e.StatusCode, e.Message) +} + +func (c *Client) get(ctx context.Context, path string, q url.Values, out any) error { + status, body, _, err := c.http.Do(ctx, httpclient.Request{ + Method: http.MethodGet, + Path: path, + Query: q, + }) + if err != nil { + return &ExternalAPIError{Source: c.source, Path: path, StatusCode: 0, Message: err.Error()} + } + if status >= 400 { + return &ExternalAPIError{Source: c.source, Path: path, StatusCode: status, Message: string(body)} + } + if out == nil { + return nil + } + if err := json.Unmarshal(body, out); err != nil { + return &ExternalAPIError{Source: c.source, Path: path, StatusCode: status, Message: "decode: " + err.Error()} + } + return nil +} diff --git a/internal/repo/webapi/binance/derivatives.go b/internal/repo/webapi/binance/derivatives.go new file mode 100644 index 0000000..dbd70ba --- /dev/null +++ b/internal/repo/webapi/binance/derivatives.go @@ -0,0 +1,116 @@ +package binance + +import ( + "context" + "net/url" + "strconv" + + "cryptoHermes/internal/entity" +) + +func (c *Client) GetCurrentFunding(ctx context.Context, symbol string) (*entity.FundingRate, error) { + q := url.Values{} + q.Set("symbol", symbol) + var dto premiumIndexDTO + if err := c.get(ctx, "/fapi/v1/premiumIndex", q, &dto); err != nil { + return nil, err + } + return mapPremiumIndex(&dto), nil +} + +func (c *Client) GetFundingHistory(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error) { + if limit <= 0 { + limit = 100 + } + if limit > 1000 { + limit = 1000 + } + q := url.Values{} + q.Set("symbol", symbol) + q.Set("limit", strconv.Itoa(limit)) + var dto []fundingRateDTO + if err := c.get(ctx, "/fapi/v1/fundingRate", q, &dto); err != nil { + return nil, err + } + return mapFundingHistory(dto), nil +} + +func (c *Client) GetCurrentOpenInterest(ctx context.Context, symbol string) (*entity.OpenInterest, error) { + q := url.Values{} + q.Set("symbol", symbol) + var dto openInterestDTO + if err := c.get(ctx, "/fapi/v1/openInterest", q, &dto); err != nil { + return nil, err + } + return mapOpenInterest(&dto, "current"), nil +} + +func (c *Client) GetOpenInterestHistory(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error) { + if limit <= 0 { + limit = 30 + } + if limit > 500 { + limit = 500 + } + q := url.Values{} + q.Set("symbol", symbol) + q.Set("period", period) + q.Set("limit", strconv.Itoa(limit)) + var dto []openInterestHistDTO + if err := c.get(ctx, "/futures/data/openInterestHist", q, &dto); err != nil { + return nil, err + } + return mapOpenInterestHistory(dto, period), nil +} + +func (c *Client) GetGlobalLongShortRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error) { + dto, err := c.fetchLongShort(ctx, "/futures/data/globalLongShortAccountRatio", symbol, period, limit) + if err != nil { + return nil, err + } + return mapLongShort(dto, period, entity.RatioTypeGlobalAccount), nil +} + +func (c *Client) GetTopTraderPositionRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error) { + dto, err := c.fetchLongShort(ctx, "/futures/data/topLongShortPositionRatio", symbol, period, limit) + if err != nil { + return nil, err + } + return mapLongShort(dto, period, entity.RatioTypeTopTraderPosition), nil +} + +func (c *Client) fetchLongShort(ctx context.Context, path, symbol, period string, limit int) ([]longShortRatioDTO, error) { + if limit <= 0 { + limit = 30 + } + if limit > 500 { + limit = 500 + } + q := url.Values{} + q.Set("symbol", symbol) + q.Set("period", period) + q.Set("limit", strconv.Itoa(limit)) + var dto []longShortRatioDTO + if err := c.get(ctx, path, q, &dto); err != nil { + return nil, err + } + return dto, nil +} + +func (c *Client) GetTakerBuySellVolume(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error) { + if limit <= 0 { + limit = 30 + } + if limit > 500 { + limit = 500 + } + q := url.Values{} + q.Set("symbol", symbol) + q.Set("period", period) + q.Set("limit", strconv.Itoa(limit)) + var dto []takerVolumeDTO + if err := c.get(ctx, "/futures/data/takerlongshortRatio", q, &dto); err != nil { + return nil, err + } + return mapTakerVolume(dto, symbol, period), nil +} diff --git a/internal/repo/webapi/binance/dto.go b/internal/repo/webapi/binance/dto.go new file mode 100644 index 0000000..7606cb3 --- /dev/null +++ b/internal/repo/webapi/binance/dto.go @@ -0,0 +1,60 @@ +package binance + +type tickerDTO struct { + Symbol string `json:"symbol"` + LastPrice string `json:"lastPrice"` + PriceChange string `json:"priceChange"` + PriceChangePercent string `json:"priceChangePercent"` + HighPrice string `json:"highPrice"` + LowPrice string `json:"lowPrice"` + Volume string `json:"volume"` + QuoteVolume string `json:"quoteVolume"` + OpenTime int64 `json:"openTime"` + CloseTime int64 `json:"closeTime"` +} + +type premiumIndexDTO struct { + Symbol string `json:"symbol"` + MarkPrice string `json:"markPrice"` + IndexPrice string `json:"indexPrice"` + LastFundingRate string `json:"lastFundingRate"` + NextFundingTime int64 `json:"nextFundingTime"` + Time int64 `json:"time"` +} + +type fundingRateDTO struct { + Symbol string `json:"symbol"` + FundingTime int64 `json:"fundingTime"` + FundingRate string `json:"fundingRate"` + MarkPrice string `json:"markPrice"` +} + +type openInterestDTO struct { + OpenInterest string `json:"openInterest"` + Symbol string `json:"symbol"` + Time int64 `json:"time"` +} + +type openInterestHistDTO struct { + Symbol string `json:"symbol"` + SumOpenInterest string `json:"sumOpenInterest"` + SumOpenInterestValue string `json:"sumOpenInterestValue"` + Timestamp int64 `json:"timestamp"` +} + +type longShortRatioDTO struct { + Symbol string `json:"symbol"` + LongShortRatio string `json:"longShortRatio"` + LongAccount string `json:"longAccount"` + ShortAccount string `json:"shortAccount"` + LongPosition string `json:"longPosition"` + ShortPosition string `json:"shortPosition"` + Timestamp int64 `json:"timestamp"` +} + +type takerVolumeDTO struct { + BuySellRatio string `json:"buySellRatio"` + BuyVol string `json:"buyVol"` + SellVol string `json:"sellVol"` + Timestamp int64 `json:"timestamp"` +} diff --git a/internal/repo/webapi/binance/mapper.go b/internal/repo/webapi/binance/mapper.go new file mode 100644 index 0000000..cde59fd --- /dev/null +++ b/internal/repo/webapi/binance/mapper.go @@ -0,0 +1,170 @@ +package binance + +import ( + "fmt" + "time" + + "cryptoHermes/internal/entity" +) + +func parseKlineRow(symbol, interval string, row []any) (entity.Kline, error) { + if len(row) < 12 { + return entity.Kline{}, fmt.Errorf("binance kline row has %d fields, expected >=12", len(row)) + } + openTime, err := toInt64(row[0]) + if err != nil { + return entity.Kline{}, fmt.Errorf("openTime: %w", err) + } + closeTime, err := toInt64(row[6]) + if err != nil { + return entity.Kline{}, fmt.Errorf("closeTime: %w", err) + } + tradeCount, err := toInt64(row[8]) + if err != nil { + return entity.Kline{}, fmt.Errorf("tradeCount: %w", err) + } + + open, _ := row[1].(string) + high, _ := row[2].(string) + low, _ := row[3].(string) + closeP, _ := row[4].(string) + volume, _ := row[5].(string) + quoteVol, _ := row[7].(string) + takerBase, _ := row[9].(string) + takerQuote, _ := row[10].(string) + + nowMs := time.Now().UnixMilli() + return entity.Kline{ + Source: sourceName, + Symbol: symbol, + Interval: interval, + OpenTime: openTime, + CloseTime: closeTime, + Open: open, + High: high, + Low: low, + Close: closeP, + Volume: volume, + QuoteVolume: quoteVol, + TradeCount: tradeCount, + TakerBuyBaseVolume: takerBase, + TakerBuyQuoteVolume: takerQuote, + IsClosed: closeTime < nowMs, + }, nil +} + +func toInt64(v any) (int64, error) { + switch x := v.(type) { + case float64: + return int64(x), nil + case int64: + return x, nil + case int: + return int64(x), nil + default: + return 0, fmt.Errorf("unexpected type %T", v) + } +} + +func mapTicker(d *tickerDTO) *entity.Ticker24h { + return &entity.Ticker24h{ + Symbol: d.Symbol, + LastPrice: d.LastPrice, + PriceChange: d.PriceChange, + PriceChangePercent: d.PriceChangePercent, + HighPrice: d.HighPrice, + LowPrice: d.LowPrice, + Volume: d.Volume, + QuoteVolume: d.QuoteVolume, + OpenTime: d.OpenTime, + CloseTime: d.CloseTime, + } +} + +func mapPremiumIndex(d *premiumIndexDTO) *entity.FundingRate { + return &entity.FundingRate{ + Source: sourceName, + Symbol: d.Symbol, + FundingTime: d.NextFundingTime, + FundingRate: d.LastFundingRate, + MarkPrice: d.MarkPrice, + } +} + +func mapFundingHistory(d []fundingRateDTO) []entity.FundingRate { + out := make([]entity.FundingRate, 0, len(d)) + for _, x := range d { + out = append(out, entity.FundingRate{ + Source: sourceName, + Symbol: x.Symbol, + FundingTime: x.FundingTime, + FundingRate: x.FundingRate, + MarkPrice: x.MarkPrice, + }) + } + return out +} + +func mapOpenInterest(d *openInterestDTO, period string) *entity.OpenInterest { + return &entity.OpenInterest{ + Source: sourceName, + Symbol: d.Symbol, + Period: period, + Timestamp: d.Time, + OpenInterest: d.OpenInterest, + } +} + +func mapOpenInterestHistory(d []openInterestHistDTO, period string) []entity.OpenInterest { + out := make([]entity.OpenInterest, 0, len(d)) + for _, x := range d { + out = append(out, entity.OpenInterest{ + Source: sourceName, + Symbol: x.Symbol, + Period: period, + Timestamp: x.Timestamp, + OpenInterest: x.SumOpenInterest, + OpenInterestValue: x.SumOpenInterestValue, + }) + } + return out +} + +func mapLongShort(d []longShortRatioDTO, period, ratioType string) []entity.LongShortRatio { + out := make([]entity.LongShortRatio, 0, len(d)) + for _, x := range d { + longVal := x.LongAccount + shortVal := x.ShortAccount + if ratioType == entity.RatioTypeTopTraderPosition { + longVal = x.LongPosition + shortVal = x.ShortPosition + } + out = append(out, entity.LongShortRatio{ + Source: sourceName, + Symbol: x.Symbol, + Period: period, + RatioType: ratioType, + Timestamp: x.Timestamp, + LongShortRatio: x.LongShortRatio, + LongValue: longVal, + ShortValue: shortVal, + }) + } + return out +} + +func mapTakerVolume(d []takerVolumeDTO, symbol, period string) []entity.TakerBuySellVolume { + out := make([]entity.TakerBuySellVolume, 0, len(d)) + for _, x := range d { + out = append(out, entity.TakerBuySellVolume{ + Source: sourceName, + Symbol: symbol, + Period: period, + Timestamp: x.Timestamp, + BuySellRatio: x.BuySellRatio, + BuyVolume: x.BuyVol, + SellVolume: x.SellVol, + }) + } + return out +} diff --git a/internal/repo/webapi/binance/market.go b/internal/repo/webapi/binance/market.go new file mode 100644 index 0000000..929fa27 --- /dev/null +++ b/internal/repo/webapi/binance/market.go @@ -0,0 +1,57 @@ +package binance + +import ( + "context" + "net/url" + "strconv" + + "cryptoHermes/internal/entity" +) + +func (c *Client) GetKlines(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error) { + return c.GetKlinesRange(ctx, symbol, interval, 0, 0, limit) +} + +func (c *Client) GetKlinesRange(ctx context.Context, symbol, interval string, startMs, endMs int64, limit int) ([]entity.Kline, error) { + if limit <= 0 { + limit = 500 + } + if limit > 1500 { + limit = 1500 + } + q := url.Values{} + q.Set("symbol", symbol) + q.Set("interval", interval) + q.Set("limit", strconv.Itoa(limit)) + if startMs > 0 { + q.Set("startTime", strconv.FormatInt(startMs, 10)) + } + if endMs > 0 { + q.Set("endTime", strconv.FormatInt(endMs, 10)) + } + + var raw [][]any + if err := c.get(ctx, "/fapi/v1/klines", q, &raw); err != nil { + return nil, err + } + + out := make([]entity.Kline, 0, len(raw)) + for _, row := range raw { + k, err := parseKlineRow(symbol, interval, row) + if err != nil { + return nil, err + } + out = append(out, k) + } + return out, nil +} + +func (c *Client) GetTicker24h(ctx context.Context, symbol string) (*entity.Ticker24h, error) { + q := url.Values{} + q.Set("symbol", symbol) + var dto tickerDTO + if err := c.get(ctx, "/fapi/v1/ticker/24hr", q, &dto); err != nil { + return nil, err + } + return mapTicker(&dto), nil +} diff --git a/internal/usecase/market_collector.go b/internal/usecase/market_collector.go new file mode 100644 index 0000000..557db73 --- /dev/null +++ b/internal/usecase/market_collector.go @@ -0,0 +1,18 @@ +package usecase + +import ( + "context" + "log/slog" +) + +type MarketCollectorUsecase struct { + log *slog.Logger +} + +func NewMarketCollectorUsecase(log *slog.Logger) *MarketCollectorUsecase { + return &MarketCollectorUsecase{log: log} +} + +func (u *MarketCollectorUsecase) Tick(ctx context.Context) error { + return nil +} diff --git a/internal/usecase/market_context.go b/internal/usecase/market_context.go new file mode 100644 index 0000000..ad1bda2 --- /dev/null +++ b/internal/usecase/market_context.go @@ -0,0 +1,204 @@ +package usecase + +import ( + "context" + "errors" + "fmt" + "log/slog" + "sync" + "time" + + "golang.org/x/sync/errgroup" + + "cryptoHermes/internal/entity" +) + +var supportedSymbols = map[string]bool{ + "BTCUSDT": true, + "ETHUSDT": true, +} + +var supportedIntervals = []string{"15m", "1h", "4h", "1d", "1w"} + +const ( + klineWindowSize = 300 + klineMinForOK = 200 + derivativePeriod = "1h" + fundingHistoryLen = 100 + oiHistoryLen = 200 + longShortLen = 200 + takerHistoryLen = 200 +) + +type MarketContextUsecase struct { + marketData MarketDataProvider + derivatives DerivativesProvider + + klineRepo KlineRepository + fundingRepo FundingRepository + oiRepo OpenInterestRepository + lsRepo LongShortRatioRepository + + log *slog.Logger +} + +func NewMarketContextUsecase( + marketData MarketDataProvider, + derivatives DerivativesProvider, + klineRepo KlineRepository, + fundingRepo FundingRepository, + oiRepo OpenInterestRepository, + lsRepo LongShortRatioRepository, + log *slog.Logger, +) *MarketContextUsecase { + return &MarketContextUsecase{ + marketData: marketData, + derivatives: derivatives, + klineRepo: klineRepo, + fundingRepo: fundingRepo, + oiRepo: oiRepo, + lsRepo: lsRepo, + log: log, + } +} + +func (u *MarketContextUsecase) Build(ctx context.Context, symbol string) (*entity.MarketContext, error) { + if !supportedSymbols[symbol] { + return nil, fmt.Errorf("unsupported symbol: %s", symbol) + } + + var ( + snapshot *entity.Ticker24h + currentFund *entity.FundingRate + currentOI *entity.OpenInterest + warnMu sync.Mutex + warnings []string + ) + addWarn := func(w string) { + warnMu.Lock() + warnings = append(warnings, w) + warnMu.Unlock() + } + + g, gctx := errgroup.WithContext(ctx) + + g.Go(func() error { + s, err := u.marketData.GetTicker24h(gctx, symbol) + if err != nil { + addWarn("snapshot fetch failed: " + err.Error()) + return nil + } + snapshot = s + return nil + }) + g.Go(func() error { + f, err := u.derivatives.GetCurrentFunding(gctx, symbol) + if err != nil { + addWarn("current funding fetch failed: " + err.Error()) + return nil + } + currentFund = f + return nil + }) + g.Go(func() error { + oi, err := u.derivatives.GetCurrentOpenInterest(gctx, symbol) + if err != nil { + addWarn("current OI fetch failed: " + err.Error()) + return nil + } + currentOI = oi + return nil + }) + + _ = g.Wait() + + klines := make(map[string][]entity.Kline, len(supportedIntervals)) + for _, iv := range supportedIntervals { + rows, err := u.klineRepo.FindRecent(ctx, symbol, iv, klineWindowSize) + if err != nil { + addWarn(fmt.Sprintf("klines DB query failed for %s: %v", iv, err)) + rows = nil + } + if len(rows) < klineMinForOK { + addWarn(fmt.Sprintf("klines %s only %d in DB, falling back to Binance", iv, len(rows))) + fresh, ferr := u.marketData.GetKlines(ctx, symbol, iv, klineWindowSize) + if ferr != nil { + addWarn(fmt.Sprintf("klines fallback %s failed: %v", iv, ferr)) + } else { + closed := make([]entity.Kline, 0, len(fresh)) + for _, k := range fresh { + if k.IsClosed { + closed = append(closed, k) + } + } + go func(items []entity.Kline) { + bgCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if err := u.klineRepo.UpsertMany(bgCtx, items); err != nil { + u.log.Error("background_kline_upsert_failed", "err", err) + } + }(closed) + rows = closed + } + } + klines[iv] = rows + } + + fundingHist, err := u.fundingRepo.FindRecent(ctx, symbol, fundingHistoryLen) + if err != nil { + addWarn("funding history query failed: " + err.Error()) + } + + oiHist, err := u.oiRepo.FindRecent(ctx, symbol, derivativePeriod, oiHistoryLen) + if err != nil { + addWarn("OI history query failed: " + err.Error()) + } + + globalLS, err := u.lsRepo.FindRecent(ctx, symbol, derivativePeriod, entity.RatioTypeGlobalAccount, longShortLen) + if err != nil { + addWarn("global long/short query failed: " + err.Error()) + } + topLS, err := u.lsRepo.FindRecent(ctx, symbol, derivativePeriod, entity.RatioTypeTopTraderPosition, longShortLen) + if err != nil { + addWarn("top trader position long/short query failed: " + err.Error()) + } + + out := &entity.MarketContext{ + Symbol: symbol, + GeneratedAt: time.Now().UnixMilli(), + Snapshot: snapshot, + Klines: klines, + Derivatives: entity.DerivativesBundle{ + Funding: entity.FundingBundle{ + Current: currentFund, + History: fundingHist, + }, + OpenInterest: entity.OpenInterestBundle{ + Current: currentOI, + History: oiHist, + }, + LongShortRatio: entity.LongShortBundle{ + Global: globalLS, + TopTraderPosition: topLS, + }, + TakerBuySellVolume: nil, + }, + Technical: entity.TechnicalStructure{ + Support: []entity.TechnicalLevel{}, + Resistance: []entity.TechnicalLevel{}, + }, + DataQuality: entity.DataQuality{ + Source: "binance", + Warnings: warnings, + }, + } + + if out.DataQuality.Warnings == nil { + out.DataQuality.Warnings = []string{} + } + + if snapshot == nil { + return out, errors.New("market snapshot unavailable") + } + return out, nil +} diff --git a/internal/usecase/ports.go b/internal/usecase/ports.go new file mode 100644 index 0000000..6e7730c --- /dev/null +++ b/internal/usecase/ports.go @@ -0,0 +1,51 @@ +package usecase + +import ( + "context" + + "cryptoHermes/internal/entity" +) + +type MarketDataProvider interface { + GetKlines(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error) + GetKlinesRange(ctx context.Context, symbol, interval string, startMs, endMs int64, limit int) ([]entity.Kline, error) + GetTicker24h(ctx context.Context, symbol string) (*entity.Ticker24h, error) +} + +type DerivativesProvider interface { + GetCurrentFunding(ctx context.Context, symbol string) (*entity.FundingRate, error) + GetFundingHistory(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error) + + GetCurrentOpenInterest(ctx context.Context, symbol string) (*entity.OpenInterest, error) + GetOpenInterestHistory(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error) + + GetGlobalLongShortRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error) + GetTopTraderPositionRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error) + + GetTakerBuySellVolume(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error) +} + +type KlineRepository interface { + UpsertMany(ctx context.Context, items []entity.Kline) error + FindRecent(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error) +} + +type FundingRepository interface { + UpsertMany(ctx context.Context, items []entity.FundingRate) error + FindRecent(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error) +} + +type OpenInterestRepository interface { + UpsertMany(ctx context.Context, items []entity.OpenInterest) error + FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error) +} + +type LongShortRatioRepository interface { + UpsertMany(ctx context.Context, items []entity.LongShortRatio) error + FindRecent(ctx context.Context, symbol, period, ratioType string, limit int) ([]entity.LongShortRatio, error) +} + +type TakerVolumeRepository interface { + UpsertMany(ctx context.Context, items []entity.TakerBuySellVolume) error + FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error) +} diff --git a/internal/worker/collector.go b/internal/worker/collector.go new file mode 100644 index 0000000..b2adc68 --- /dev/null +++ b/internal/worker/collector.go @@ -0,0 +1,152 @@ +package worker + +import ( + "context" + "log/slog" + + "cryptoHermes/internal/entity" + "cryptoHermes/internal/usecase" +) + +type Deps struct { + MarketData usecase.MarketDataProvider + Derivatives usecase.DerivativesProvider + + KlineRepo usecase.KlineRepository + FundingRepo usecase.FundingRepository + OIRepo usecase.OpenInterestRepository + LSRepo usecase.LongShortRatioRepository + TakerRepo usecase.TakerVolumeRepository + + Symbols []string + Intervals []string + Limit int + Logger *slog.Logger +} + +type Collector struct { + d Deps +} + +func NewCollector(d Deps) *Collector { + if d.Limit <= 0 { + d.Limit = 500 + } + return &Collector{d: d} +} + +func (c *Collector) Symbols() []string { return c.d.Symbols } +func (c *Collector) Intervals() []string { return c.d.Intervals } + +func (c *Collector) CollectKlines(ctx context.Context, interval string) error { + for _, sym := range c.d.Symbols { + ks, err := c.d.MarketData.GetKlines(ctx, sym, interval, c.d.Limit) + if err != nil { + c.d.Logger.Error("collect_klines_failed", "symbol", sym, "interval", interval, "err", err) + continue + } + closed := make([]entity.Kline, 0, len(ks)) + for _, k := range ks { + if k.IsClosed { + closed = append(closed, k) + } + } + if err := c.d.KlineRepo.UpsertMany(ctx, closed); err != nil { + c.d.Logger.Error("upsert_klines_failed", "symbol", sym, "interval", interval, "err", err) + continue + } + c.d.Logger.Info("collect_klines_ok", "symbol", sym, "interval", interval, "rows", len(closed)) + } + return nil +} + +func (c *Collector) CollectAllKlines(ctx context.Context) error { + for _, iv := range c.d.Intervals { + _ = c.CollectKlines(ctx, iv) + } + return nil +} + +func (c *Collector) CollectFunding(ctx context.Context) error { + for _, sym := range c.d.Symbols { + hist, err := c.d.Derivatives.GetFundingHistory(ctx, sym, 100) + if err != nil { + c.d.Logger.Error("collect_funding_failed", "symbol", sym, "err", err) + continue + } + if err := c.d.FundingRepo.UpsertMany(ctx, hist); err != nil { + c.d.Logger.Error("upsert_funding_failed", "symbol", sym, "err", err) + continue + } + c.d.Logger.Info("collect_funding_ok", "symbol", sym, "rows", len(hist)) + } + return nil +} + +func (c *Collector) CollectOpenInterest(ctx context.Context) error { + periods := []string{"5m", "15m", "1h", "4h", "1d"} + for _, sym := range c.d.Symbols { + for _, p := range periods { + hist, err := c.d.Derivatives.GetOpenInterestHistory(ctx, sym, p, 500) + if err != nil { + c.d.Logger.Error("collect_oi_failed", "symbol", sym, "period", p, "err", err) + continue + } + if err := c.d.OIRepo.UpsertMany(ctx, hist); err != nil { + c.d.Logger.Error("upsert_oi_failed", "symbol", sym, "period", p, "err", err) + continue + } + c.d.Logger.Info("collect_oi_ok", "symbol", sym, "period", p, "rows", len(hist)) + } + } + return nil +} + +func (c *Collector) CollectLongShortRatio(ctx context.Context) error { + periods := []string{"5m", "15m", "1h", "4h", "1d"} + for _, sym := range c.d.Symbols { + for _, p := range periods { + g, err := c.d.Derivatives.GetGlobalLongShortRatio(ctx, sym, p, 500) + if err == nil { + if err := c.d.LSRepo.UpsertMany(ctx, g); err != nil { + c.d.Logger.Error("upsert_global_ls_failed", "symbol", sym, "period", p, "err", err) + } else { + c.d.Logger.Info("collect_global_ls_ok", "symbol", sym, "period", p, "rows", len(g)) + } + } else { + c.d.Logger.Error("collect_global_ls_failed", "symbol", sym, "period", p, "err", err) + } + + t, err := c.d.Derivatives.GetTopTraderPositionRatio(ctx, sym, p, 500) + if err == nil { + if err := c.d.LSRepo.UpsertMany(ctx, t); err != nil { + c.d.Logger.Error("upsert_top_ls_failed", "symbol", sym, "period", p, "err", err) + } else { + c.d.Logger.Info("collect_top_ls_ok", "symbol", sym, "period", p, "rows", len(t)) + } + } else { + c.d.Logger.Error("collect_top_ls_failed", "symbol", sym, "period", p, "err", err) + } + } + } + return nil +} + +func (c *Collector) CollectTakerVolume(ctx context.Context) error { + periods := []string{"5m", "15m", "1h", "4h", "1d"} + for _, sym := range c.d.Symbols { + for _, p := range periods { + items, err := c.d.Derivatives.GetTakerBuySellVolume(ctx, sym, p, 500) + if err != nil { + c.d.Logger.Error("collect_taker_failed", "symbol", sym, "period", p, "err", err) + continue + } + if err := c.d.TakerRepo.UpsertMany(ctx, items); err != nil { + c.d.Logger.Error("upsert_taker_failed", "symbol", sym, "period", p, "err", err) + continue + } + c.d.Logger.Info("collect_taker_ok", "symbol", sym, "period", p, "rows", len(items)) + } + } + return nil +} diff --git a/migrations/000001_market_klines.down.sql b/migrations/000001_market_klines.down.sql new file mode 100644 index 0000000..994f0b6 --- /dev/null +++ b/migrations/000001_market_klines.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS market_klines; diff --git a/migrations/000001_market_klines.up.sql b/migrations/000001_market_klines.up.sql new file mode 100644 index 0000000..a4043ad --- /dev/null +++ b/migrations/000001_market_klines.up.sql @@ -0,0 +1,34 @@ +CREATE TABLE IF NOT EXISTS market_klines ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + interval TEXT NOT NULL, + + open_time BIGINT NOT NULL, + close_time BIGINT NOT NULL, + + open NUMERIC(36, 18) NOT NULL, + high NUMERIC(36, 18) NOT NULL, + low NUMERIC(36, 18) NOT NULL, + close NUMERIC(36, 18) NOT NULL, + + volume NUMERIC(36, 18) NOT NULL, + quote_volume NUMERIC(36, 18) NOT NULL, + + trade_count BIGINT NOT NULL, + + taker_buy_base_volume NUMERIC(36, 18) NOT NULL, + taker_buy_quote_volume NUMERIC(36, 18) NOT NULL, + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, interval, open_time) +); + +CREATE INDEX IF NOT EXISTS idx_market_klines_symbol_interval_time +ON market_klines(symbol, interval, open_time DESC); + +SELECT create_hypertable('market_klines', 'open_time', + chunk_time_interval => 604800000, + if_not_exists => TRUE, + migrate_data => TRUE); diff --git a/migrations/000002_funding_rates.down.sql b/migrations/000002_funding_rates.down.sql new file mode 100644 index 0000000..8ab7cb0 --- /dev/null +++ b/migrations/000002_funding_rates.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS funding_rates; diff --git a/migrations/000002_funding_rates.up.sql b/migrations/000002_funding_rates.up.sql new file mode 100644 index 0000000..970e5eb --- /dev/null +++ b/migrations/000002_funding_rates.up.sql @@ -0,0 +1,20 @@ +CREATE TABLE IF NOT EXISTS funding_rates ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + + funding_time BIGINT NOT NULL, + funding_rate NUMERIC(36, 18) NOT NULL, + mark_price NUMERIC(36, 18), + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, funding_time) +); + +CREATE INDEX IF NOT EXISTS idx_funding_rates_symbol_time +ON funding_rates(symbol, funding_time DESC); + +SELECT create_hypertable('funding_rates', 'funding_time', + chunk_time_interval => 2592000000, + if_not_exists => TRUE, + migrate_data => TRUE); diff --git a/migrations/000003_open_interest.down.sql b/migrations/000003_open_interest.down.sql new file mode 100644 index 0000000..6ea96c5 --- /dev/null +++ b/migrations/000003_open_interest.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS open_interest; diff --git a/migrations/000003_open_interest.up.sql b/migrations/000003_open_interest.up.sql new file mode 100644 index 0000000..ae3df96 --- /dev/null +++ b/migrations/000003_open_interest.up.sql @@ -0,0 +1,21 @@ +CREATE TABLE IF NOT EXISTS open_interest ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + period TEXT NOT NULL, + + timestamp BIGINT NOT NULL, + open_interest NUMERIC(36, 18) NOT NULL, + open_interest_value NUMERIC(36, 18), + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, period, timestamp) +); + +CREATE INDEX IF NOT EXISTS idx_open_interest_symbol_period_time +ON open_interest(symbol, period, timestamp DESC); + +SELECT create_hypertable('open_interest', 'timestamp', + chunk_time_interval => 604800000, + if_not_exists => TRUE, + migrate_data => TRUE); diff --git a/migrations/000004_long_short_ratio.down.sql b/migrations/000004_long_short_ratio.down.sql new file mode 100644 index 0000000..a95c6e8 --- /dev/null +++ b/migrations/000004_long_short_ratio.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS long_short_ratio; diff --git a/migrations/000004_long_short_ratio.up.sql b/migrations/000004_long_short_ratio.up.sql new file mode 100644 index 0000000..31d885a --- /dev/null +++ b/migrations/000004_long_short_ratio.up.sql @@ -0,0 +1,24 @@ +CREATE TABLE IF NOT EXISTS long_short_ratio ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + period TEXT NOT NULL, + ratio_type TEXT NOT NULL, + + timestamp BIGINT NOT NULL, + + long_short_ratio NUMERIC(36, 18) NOT NULL, + long_value NUMERIC(36, 18), + short_value NUMERIC(36, 18), + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, period, ratio_type, timestamp) +); + +CREATE INDEX IF NOT EXISTS idx_long_short_ratio_symbol_period_time +ON long_short_ratio(symbol, period, ratio_type, timestamp DESC); + +SELECT create_hypertable('long_short_ratio', 'timestamp', + chunk_time_interval => 604800000, + if_not_exists => TRUE, + migrate_data => TRUE); diff --git a/migrations/000005_taker_volume.down.sql b/migrations/000005_taker_volume.down.sql new file mode 100644 index 0000000..754cf8c --- /dev/null +++ b/migrations/000005_taker_volume.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS taker_buy_sell_volume; diff --git a/migrations/000005_taker_volume.up.sql b/migrations/000005_taker_volume.up.sql new file mode 100644 index 0000000..c9302d0 --- /dev/null +++ b/migrations/000005_taker_volume.up.sql @@ -0,0 +1,23 @@ +CREATE TABLE IF NOT EXISTS taker_buy_sell_volume ( + source TEXT NOT NULL, + symbol TEXT NOT NULL, + period TEXT NOT NULL, + + timestamp BIGINT NOT NULL, + + buy_sell_ratio NUMERIC(36, 18), + buy_volume NUMERIC(36, 18), + sell_volume NUMERIC(36, 18), + + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + PRIMARY KEY (source, symbol, period, timestamp) +); + +CREATE INDEX IF NOT EXISTS idx_taker_buy_sell_symbol_period_time +ON taker_buy_sell_volume(symbol, period, timestamp DESC); + +SELECT create_hypertable('taker_buy_sell_volume', 'timestamp', + chunk_time_interval => 604800000, + if_not_exists => TRUE, + migrate_data => TRUE); diff --git a/pkg/httpclient/client.go b/pkg/httpclient/client.go new file mode 100644 index 0000000..cd9bbe1 --- /dev/null +++ b/pkg/httpclient/client.go @@ -0,0 +1,120 @@ +package httpclient + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "time" + + "golang.org/x/time/rate" +) + +type Client struct { + httpClient *http.Client + baseURL string + limiter *rate.Limiter + retryCount int +} + +type Options struct { + BaseURL string + Timeout time.Duration + RetryCount int + RPS float64 + Burst int +} + +func New(opts Options) *Client { + if opts.Timeout == 0 { + opts.Timeout = 10 * time.Second + } + if opts.RPS <= 0 { + opts.RPS = 20 + } + if opts.Burst <= 0 { + opts.Burst = 40 + } + return &Client{ + httpClient: &http.Client{Timeout: opts.Timeout}, + baseURL: opts.BaseURL, + retryCount: opts.RetryCount, + limiter: rate.NewLimiter(rate.Limit(opts.RPS), opts.Burst), + } +} + +type Request struct { + Method string + Path string + Query url.Values + Header http.Header +} + +func (c *Client) Do(ctx context.Context, req Request) (int, []byte, http.Header, error) { + if err := c.limiter.Wait(ctx); err != nil { + return 0, nil, nil, err + } + + fullURL := c.baseURL + req.Path + if len(req.Query) > 0 { + fullURL += "?" + req.Query.Encode() + } + + var lastErr error + for attempt := 0; attempt <= c.retryCount; attempt++ { + if attempt > 0 { + select { + case <-ctx.Done(): + return 0, nil, nil, ctx.Err() + case <-time.After(time.Duration(attempt) * 500 * time.Millisecond): + } + } + + httpReq, err := http.NewRequestWithContext(ctx, req.Method, fullURL, nil) + if err != nil { + return 0, nil, nil, err + } + for k, vs := range req.Header { + for _, v := range vs { + httpReq.Header.Add(k, v) + } + } + + resp, err := c.httpClient.Do(httpReq) + if err != nil { + lastErr = err + continue + } + body, readErr := io.ReadAll(resp.Body) + resp.Body.Close() + if readErr != nil { + lastErr = readErr + continue + } + + if shouldRetry(resp.StatusCode) && attempt < c.retryCount { + lastErr = fmt.Errorf("retryable status %d", resp.StatusCode) + continue + } + return resp.StatusCode, body, resp.Header, nil + } + + if lastErr == nil { + lastErr = errors.New("unknown http error") + } + return 0, nil, nil, lastErr +} + +func shouldRetry(status int) bool { + switch status { + case http.StatusTooManyRequests, + http.StatusInternalServerError, + http.StatusBadGateway, + http.StatusServiceUnavailable, + http.StatusGatewayTimeout: + return true + } + return false +} diff --git a/pkg/logger/logger.go b/pkg/logger/logger.go new file mode 100644 index 0000000..097354a --- /dev/null +++ b/pkg/logger/logger.go @@ -0,0 +1,17 @@ +package logger + +import ( + "log/slog" + "os" +) + +func New(env string) *slog.Logger { + var handler slog.Handler + opts := &slog.HandlerOptions{Level: slog.LevelInfo} + if env == "local" || env == "dev" { + handler = slog.NewTextHandler(os.Stdout, opts) + } else { + handler = slog.NewJSONHandler(os.Stdout, opts) + } + return slog.New(handler) +} diff --git a/pkg/postgres/postgres.go b/pkg/postgres/postgres.go new file mode 100644 index 0000000..e7d844f --- /dev/null +++ b/pkg/postgres/postgres.go @@ -0,0 +1,36 @@ +package postgres + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" +) + +type Options struct { + DSN string + MaxConns int32 + MinConns int32 +} + +func NewPool(ctx context.Context, opts Options) (*pgxpool.Pool, error) { + cfg, err := pgxpool.ParseConfig(opts.DSN) + if err != nil { + return nil, fmt.Errorf("parse dsn: %w", err) + } + if opts.MaxConns > 0 { + cfg.MaxConns = opts.MaxConns + } + if opts.MinConns > 0 { + cfg.MinConns = opts.MinConns + } + pool, err := pgxpool.NewWithConfig(ctx, cfg) + if err != nil { + return nil, fmt.Errorf("pool: %w", err) + } + if err := pool.Ping(ctx); err != nil { + pool.Close() + return nil, fmt.Errorf("ping: %w", err) + } + return pool, nil +}