feat: 初始化 cryptoHermes 行情网关 v1 MVP + harness 工程文档
Binance USDⓈ-M Futures 行情网关，给上游 Hermes 量化分析引擎提供单一聚合 接口 /v1/market/context（K 线 + funding + OI + 多空比 + taker volume）。 只读公共行情，不下单、不接私钥、不查账户。 ## v1 实现范围（Milestone 1-5） - Clean Architecture 4 层（controller/usecase/repo/entity），接口边界在 internal/usecase/ports.go - Binance Futures REST client（K 线 / ticker24h / funding / OI / 多空比 / taker volume 共 9 个接口），全链路 string 价格避免 float64 精度问题 - TimescaleDB 5 张 hypertable（market_klines / funding_rates / open_interest / long_short_ratio / taker_buy_sell_volume），主键含 时间维度，UpsertMany 幂等 - robfig/cron 定时采集（15m/1h/4h/1d/1w 多周期 K 线 + 衍生品 15 分钟 落库），未收线 K 线 (close_time > now) 由 mapper/repo 双重过滤 - pkg/httpclient 统一限流（默认 20 req/s, burst 40）+ 重试，避免触发 Binance 2400 weight/min IP 上限 - /v1/market/context 聚合接口：errgroup 并发拉 snapshot/funding/OI，DB K 线不足 200 根回源 Binance 异步补 - cmd/backfill CLI 支持指定 from/to 大段回填（Binance 历史 OI / 多空比 官方只保留 30 天，必须自己存） - Docker Compose + Makefile + golang-migrate，本地一键启 技术指标（support/resistance/Vegas/箱体）留待 v2，技术段返回空对象 + warning 占位。 ## Harness 工程文档 - AGENTS.md — AI agent 工作速查（10 个章节） - ai/project-map.md — 仓库结构、扩展点、控制流 - ai/risk-guardrails.md — G1-G10 守卫规则（每条带可机械验证命令） - ai/adr/0001-architecture-foundations.md — 9 条架构基础决策 - ai/task-templates.md — 6 种任务契约模板 - ai/harness-health.md — 当前 harness 健康度评估 3 个 grep 守卫已验证通过：controller / usecase 无具体实现依赖，全项目 无私钥/签名字段。
This commit is contained in:
dela
22
.gitignore
vendored
Normal file
22
.gitignore
vendored
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
# Binaries
|
||||||
|
/bin/
|
||||||
|
*.exe
|
||||||
|
*.test
|
||||||
|
*.out
|
||||||
|
|
||||||
|
# Go
|
||||||
|
vendor/
|
||||||
|
|
||||||
|
# Editor
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
|
||||||
|
# Local env / secrets
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
156
AGENTS.md
Normal file
156
AGENTS.md
Normal file
@@ -0,0 +1,156 @@
|
|||||||
|
# AGENTS.md — cryptoHermes 操作指南
|
||||||
|
|
||||||
|
本文件是 AI 编码代理在此仓库工作的最小操作手册。完整项目设计见 `docs/dev.md`，用户向文档见 `README.md`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 项目定位（必读，决定可做与不可做）
|
||||||
|
|
||||||
|
cryptoHermes 是给上游 Hermes 量化分析引擎用的 **Binance Futures 行情网关**。
|
||||||
|
|
||||||
|
**只做**：
|
||||||
|
- 从 Binance USDⓈ-M Futures 公共接口拉行情、衍生品数据
|
||||||
|
- 落库到 TimescaleDB
|
||||||
|
- 通过 REST 把聚合后的市场上下文给 Hermes
|
||||||
|
|
||||||
|
**绝不做**（详见 `ai/risk-guardrails.md`）：
|
||||||
|
- 下单、撤单、查持仓
|
||||||
|
- 接入 API Key / Secret
|
||||||
|
- 任何账户级私有接口（签名请求）
|
||||||
|
- 钱包、托管、转账
|
||||||
|
- WebSocket 推送（v1 范围外）
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 当前状态
|
||||||
|
|
||||||
|
- v1 MVP 已完成（Milestone 1-5），可启动、可采集、可返回完整 `/v1/market/context`
|
||||||
|
- Technical 指标计算（Milestone 6）留待 v2
|
||||||
|
- 详细路线图见 `docs/dev.md` 第 19 章和 README "路线图"段
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 技术栈
|
||||||
|
|
||||||
|
| 类别 | 选型 |
|
||||||
|
|---|---|
|
||||||
|
| Go | 1.23（见 `go.mod`） |
|
||||||
|
| HTTP | Fiber v2 |
|
||||||
|
| DB | TimescaleDB on PG 16 |
|
||||||
|
| DB Driver | pgx v5 (pgxpool) |
|
||||||
|
| Scheduler | robfig/cron v3 |
|
||||||
|
| Config | cleanenv（yaml + env） |
|
||||||
|
| 限流 | `golang.org/x/time/rate` |
|
||||||
|
| Migration | golang-migrate CLI |
|
||||||
|
|
||||||
|
为什么是这套选型见 `ai/adr/0001-architecture-foundations.md`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 仓库结构（参见 `ai/project-map.md`）
|
||||||
|
|
||||||
|
```
|
||||||
|
cmd/ 二进制入口（app / backfill）
|
||||||
|
config/ cleanenv 配置加载
|
||||||
|
internal/ 业务代码（不可被外部 import）
|
||||||
|
app/ 装配 DI、cron、graceful shutdown
|
||||||
|
controller/ Fiber 路由层
|
||||||
|
entity/ 纯业务实体
|
||||||
|
usecase/ 业务接口与编排（ports.go 是接口边界）
|
||||||
|
repo/ 外部依赖实现
|
||||||
|
webapi/binance/ Binance HTTP client
|
||||||
|
persistent/postgres/ 5 个 repository
|
||||||
|
worker/ collector 实现
|
||||||
|
migrations/ SQL（含 hypertable 转换）
|
||||||
|
pkg/ 项目内可复用基础设施（logger/httpclient/postgres）
|
||||||
|
docs/ 设计文档
|
||||||
|
ai/ harness 工程文档（本目录及子目录）
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 验证命令（必做：改完代码至少跑前两个）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 编译（最快反馈）
|
||||||
|
go build ./...
|
||||||
|
|
||||||
|
# 静态分析
|
||||||
|
go vet ./...
|
||||||
|
|
||||||
|
# 单包测试
|
||||||
|
go test ./internal/<pkg>/...
|
||||||
|
|
||||||
|
# 全部测试
|
||||||
|
make test # 等价于 go test ./...
|
||||||
|
|
||||||
|
# 启动（需要先 make docker-up && make migrate-up）
|
||||||
|
make run
|
||||||
|
```
|
||||||
|
|
||||||
|
**触碰 binance/mapper.go 或 entity/kline.go 之后**：必须确保对接 `Binance K线数组按下标解析` 的代码不被破坏（dev.md 第 9.2 节）。
|
||||||
|
|
||||||
|
**触碰任何 SQL migration 后**：本地必须重跑 `make migrate-down && make migrate-up` 验证 down/up 互逆。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Clean Architecture 边界（自动可验证）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# controller 不可直接 import binance/postgres
|
||||||
|
grep -r "repo/webapi/binance\|repo/persistent" internal/controller # 必须无输出
|
||||||
|
|
||||||
|
# usecase 不可直接 import binance/postgres
|
||||||
|
grep -r "repo/webapi/binance\|repo/persistent" internal/usecase # 必须无输出
|
||||||
|
|
||||||
|
# 全项目不可出现签名 / 私钥相关字段
|
||||||
|
grep -ri "apikey\|x-mbx-apikey\|hmac\|secret_key\|api_secret" --include="*.go" . # 必须无输出（不算注释/文档里的"不接入"说明）
|
||||||
|
```
|
||||||
|
|
||||||
|
任何 PR 都应该满足以上三条。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 关键技术约束（容易犯错）
|
||||||
|
|
||||||
|
1. **价格精度**：全链路用 `string`，落库时由 PG 转 `NUMERIC(36,18)`。**禁止用 `float64` 存储或传递价格、成交量**。
|
||||||
|
2. **K线未收线丢弃**：从 Binance 拉到的最新一根 K 线如果 `close_time > now()` 就不能入库。Repo 的 `UpsertMany` 会过滤 `IsClosed=false`，新代码也必须遵守。
|
||||||
|
3. **Binance 历史 OI / 多空比官方只保留 30 天**：必须依赖 collector 持续落库才能拿到长历史。
|
||||||
|
4. **Rate limit**：默认 20 req/s（`config.Binance.RPS`）。新增 collector 任务前估算 weight 预算（Binance 2400 weight/min IP 上限）。
|
||||||
|
5. **国内网络**：`fapi.binance.com` 直连不稳定，通过 `HTTPS_PROXY` 环境变量挂代理。
|
||||||
|
6. **Upsert 幂等**：5 张表的主键都包含时间维度，所有 repo 的 `UpsertMany` 使用 `ON CONFLICT DO UPDATE`，可安全重复执行。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 任务模式
|
||||||
|
|
||||||
|
任何非平凡修改，建议先写一份微型任务契约（参见 `ai/task-templates.md`）：
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: 要做的事，一句话
|
||||||
|
scope: 会改的文件 / 目录
|
||||||
|
out-of-scope: 不会动的范围
|
||||||
|
constraints: 受 ai/risk-guardrails.md 约束的哪几条
|
||||||
|
validation: 要跑哪些命令证明通过
|
||||||
|
acceptance: 如何判断完成
|
||||||
|
```
|
||||||
|
|
||||||
|
对于跨多个 Milestone 或需要新增数据源（CoinGlass、ETF Flow）的任务，必须先写 `ai/specs/<task>.md`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 提交规范
|
||||||
|
|
||||||
|
- 一个 PR 改一类事。Repo 层、Binance 适配、usecase 编排不要混在一个 commit。
|
||||||
|
- Commit message 用中文或英文都行，但要写"为什么"。
|
||||||
|
- 不要在没有跑 `go build ./...` 和 `go vet ./...` 的情况下提交。
|
||||||
|
- `go.mod` 改动必须配套 `go.sum`（运行 `make tidy`）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. 何时升级 harness
|
||||||
|
|
||||||
|
- 同一类错误第二次出现 → 写进 `ai/risk-guardrails.md`
|
||||||
|
- 重大架构决策（换 HTTP 框架、引入消息队列、跨服务通信）→ 写 ADR
|
||||||
|
- 新增数据源（CoinGlass / ETF / CME）→ 必须有 `ai/specs/<source>.md` 才动手
|
||||||
|
- 跨多次会话的长任务 → 写 `ai/work-status.md`
|
||||||
16
Dockerfile
Normal file
16
Dockerfile
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
FROM golang:1.23-alpine AS builder
|
||||||
|
|
||||||
|
WORKDIR /src
|
||||||
|
COPY go.mod go.sum* ./
|
||||||
|
RUN go mod download || true
|
||||||
|
COPY . .
|
||||||
|
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/app ./cmd/app \
|
||||||
|
&& CGO_ENABLED=0 GOOS=linux go build -o /out/backfill ./cmd/backfill
|
||||||
|
|
||||||
|
FROM alpine:3.20
|
||||||
|
RUN apk add --no-cache ca-certificates tzdata
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=builder /out/app /app/app
|
||||||
|
COPY --from=builder /out/backfill /app/backfill
|
||||||
|
EXPOSE 8080
|
||||||
|
ENTRYPOINT ["/app/app"]
|
||||||
34
Makefile
Normal file
34
Makefile
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
DATABASE_URL ?= postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable
|
||||||
|
|
||||||
|
.PHONY: run build test lint tidy docker-up docker-down migrate-up migrate-down backfill
|
||||||
|
|
||||||
|
run:
|
||||||
|
go run ./cmd/app
|
||||||
|
|
||||||
|
build:
|
||||||
|
go build -o bin/app ./cmd/app
|
||||||
|
go build -o bin/backfill ./cmd/backfill
|
||||||
|
|
||||||
|
test:
|
||||||
|
go test ./...
|
||||||
|
|
||||||
|
lint:
|
||||||
|
golangci-lint run
|
||||||
|
|
||||||
|
tidy:
|
||||||
|
go mod tidy
|
||||||
|
|
||||||
|
docker-up:
|
||||||
|
docker compose up -d
|
||||||
|
|
||||||
|
docker-down:
|
||||||
|
docker compose down
|
||||||
|
|
||||||
|
migrate-up:
|
||||||
|
migrate -path ./migrations -database "$(DATABASE_URL)" up
|
||||||
|
|
||||||
|
migrate-down:
|
||||||
|
migrate -path ./migrations -database "$(DATABASE_URL)" down 1
|
||||||
|
|
||||||
|
backfill:
|
||||||
|
go run ./cmd/backfill $(ARGS)
|
||||||
309
README.md
Normal file
309
README.md
Normal file
@@ -0,0 +1,309 @@
|
|||||||
|
# cryptoHermes — Market Data Gateway
|
||||||
|
|
||||||
|
为上游 Hermes 量化分析引擎提供统一的币圈行情与衍生品上下文。Hermes 只需要调用本服务的 `/v1/market/context` 一个接口，就能拿到 BTC/ETH 多周期 K 线、当前价、funding、OI、多空比等用于策略分析的数据。
|
||||||
|
|
||||||
|
本服务只做行情采集与聚合，**不下单、不接私钥、不查账户、不托管任何资金**。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 特性
|
||||||
|
|
||||||
|
- Binance USDⓈ-M Futures 公共接口，无需 API Key。
|
||||||
|
- 多周期 K 线 / 24h ticker / funding / Open Interest / 全局多空比 / 大户持仓多空比 / Taker 主动买卖量。
|
||||||
|
- 历史数据自动落库（Binance 官方历史 OI / 多空比只保留 30 天，必须自己存）。
|
||||||
|
- Clean Architecture：controller / usecase / repo 分层，usecase 只依赖接口。
|
||||||
|
- Fiber v2 REST API + robfig/cron 定时采集 + TimescaleDB hypertable 存时序数据。
|
||||||
|
- Rate Limit 节流（默认 20 req/s），可重复运行的 upsert，K 线只入库已收线。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 技术栈
|
||||||
|
|
||||||
|
| 类型 | 选型 |
|
||||||
|
|---|---|
|
||||||
|
| 语言 | Go 1.23 |
|
||||||
|
| HTTP | Fiber v2 |
|
||||||
|
| DB | TimescaleDB (PG 16) |
|
||||||
|
| DB Driver | pgx v5 (pgxpool) |
|
||||||
|
| Scheduler | robfig/cron v3 |
|
||||||
|
| Config | cleanenv (yaml + env) |
|
||||||
|
| 限流 | golang.org/x/time/rate |
|
||||||
|
| Migration | golang-migrate CLI |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 快速开始
|
||||||
|
|
||||||
|
### 1. 依赖
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Go 1.23+
|
||||||
|
go version
|
||||||
|
|
||||||
|
# golang-migrate CLI（用于跑 migration）
|
||||||
|
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. 启动数据库
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make docker-up
|
||||||
|
```
|
||||||
|
|
||||||
|
会用 `timescale/timescaledb:latest-pg16` 启一个 Postgres，监听本地 5432。
|
||||||
|
|
||||||
|
### 3. 执行 migration
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export DATABASE_URL="postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable"
|
||||||
|
make migrate-up
|
||||||
|
```
|
||||||
|
|
||||||
|
会建 5 张 hypertable：`market_klines` / `funding_rates` / `open_interest` / `long_short_ratio` / `taker_buy_sell_volume`。
|
||||||
|
|
||||||
|
### 4. （可选）回填历史 K 线
|
||||||
|
|
||||||
|
第一次启动时 DB 是空的，直接调 `/v1/market/context` 会触发回源 Binance。建议先回填：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 15m --limit 500"
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 1h --limit 500"
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 4h --limit 500"
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 1d --limit 500"
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 1w --limit 500"
|
||||||
|
# ETHUSDT 同上
|
||||||
|
```
|
||||||
|
|
||||||
|
也可以指定起止时间做大段回填：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 1h --from 1704067200000 --to 1735689600000"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. 启动服务
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make run
|
||||||
|
```
|
||||||
|
|
||||||
|
服务会：
|
||||||
|
|
||||||
|
- 监听 `:8080`
|
||||||
|
- 立刻启动 cron，每 15 分钟拉一次 funding / OI / 多空比 / taker volume / 15m K 线，整点拉 1h，每 4h 拉 4h，每日 00:05 拉 1d，每周一 00:10 拉 1w。
|
||||||
|
|
||||||
|
### 6. 验证
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s localhost:8080/v1/health | jq .
|
||||||
|
curl -s 'localhost:8080/v1/market/context?symbol=BTCUSDT' | jq 'keys'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 国内网络
|
||||||
|
|
||||||
|
`fapi.binance.com` 国内直连不稳定，通过环境变量挂代理即可（Go 默认尊重 `HTTPS_PROXY`）：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
HTTPS_PROXY=http://127.0.0.1:7890 make run
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 配置
|
||||||
|
|
||||||
|
`config/config.yml`（所有字段都可被环境变量覆盖，参见 `config/config.go` 的 `env:` tag）：
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
app:
|
||||||
|
port: 8080
|
||||||
|
env: local
|
||||||
|
|
||||||
|
binance:
|
||||||
|
base_url: https://fapi.binance.com
|
||||||
|
timeout: 10s
|
||||||
|
retry_count: 2
|
||||||
|
rps: 20 # IP 级限速，超过会自动 sleep
|
||||||
|
burst: 40
|
||||||
|
|
||||||
|
postgres:
|
||||||
|
dsn: postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable
|
||||||
|
max_conns: 10
|
||||||
|
min_conns: 2
|
||||||
|
|
||||||
|
collector:
|
||||||
|
enabled: true # 设为 false 可只跑只读 API，不开 cron
|
||||||
|
symbols: [BTCUSDT, ETHUSDT]
|
||||||
|
intervals: [15m, 1h, 4h, 1d, 1w]
|
||||||
|
default_limit: 500
|
||||||
|
```
|
||||||
|
|
||||||
|
常用环境变量：
|
||||||
|
|
||||||
|
| 变量 | 说明 |
|
||||||
|
|---|---|
|
||||||
|
| `CONFIG_PATH` | 配置文件路径，默认 `config/config.yml` |
|
||||||
|
| `POSTGRES_DSN` | 覆盖 yaml 中的 DSN（Docker Compose 里就是这么干的） |
|
||||||
|
| `APP_PORT` | HTTP 端口 |
|
||||||
|
| `HTTPS_PROXY` | 走代理访问 Binance |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## API
|
||||||
|
|
||||||
|
### `GET /v1/health`
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "status": "ok", "time": 1717000000000 }
|
||||||
|
```
|
||||||
|
|
||||||
|
### `GET /v1/market/context?symbol=BTCUSDT`
|
||||||
|
|
||||||
|
**Hermes 主接口**。聚合返回 K 线 + 衍生品 + 数据质量信息：
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{
|
||||||
|
"symbol": "BTCUSDT",
|
||||||
|
"generatedAt": 1717000000000,
|
||||||
|
"snapshot": {
|
||||||
|
"lastPrice": "108000.12",
|
||||||
|
"priceChangePercent": "2.14",
|
||||||
|
"highPrice": "109200.00",
|
||||||
|
"lowPrice": "104800.00",
|
||||||
|
"volume": "...",
|
||||||
|
"quoteVolume": "..."
|
||||||
|
},
|
||||||
|
"klines": {
|
||||||
|
"15m": [ /* 300 根 */ ],
|
||||||
|
"1h": [ /* 300 根 */ ],
|
||||||
|
"4h": [ /* 300 根 */ ],
|
||||||
|
"1d": [ /* 300 根 */ ],
|
||||||
|
"1w": [ /* 300 根 */ ]
|
||||||
|
},
|
||||||
|
"derivatives": {
|
||||||
|
"funding": { "current": { ... }, "history": [ ... ] },
|
||||||
|
"openInterest": { "current": { ... }, "history": [ ... ] },
|
||||||
|
"longShortRatio": {
|
||||||
|
"global": [ ... ],
|
||||||
|
"topTraderPosition": [ ... ]
|
||||||
|
},
|
||||||
|
"takerBuySellVolume": []
|
||||||
|
},
|
||||||
|
"technical": {
|
||||||
|
"support": [], "resistance": [],
|
||||||
|
"rangeHigh": null, "rangeLow": null, "longShortLine": null
|
||||||
|
},
|
||||||
|
"dataQuality": {
|
||||||
|
"source": "binance",
|
||||||
|
"warnings": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
> `technical.*` 字段在 v1 留空。v2 会补支撑压力、箱体、多空线计算。
|
||||||
|
|
||||||
|
### `GET /v1/market/klines?symbol=BTCUSDT&interval=1h&limit=300`
|
||||||
|
|
||||||
|
直接读 DB 返回 K 线数组。`interval` 取值 `15m/1h/4h/1d/1w`。
|
||||||
|
|
||||||
|
### `GET /v1/market/snapshot?symbol=BTCUSDT`
|
||||||
|
|
||||||
|
24h ticker 实时拉取（不走 DB）。
|
||||||
|
|
||||||
|
### `GET /v1/market/derivatives?symbol=BTCUSDT&period=1h`
|
||||||
|
|
||||||
|
衍生品聚合：funding / OI / 多空比。`period` 默认 `1h`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 目录结构
|
||||||
|
|
||||||
|
```
|
||||||
|
cryptoHermes/
|
||||||
|
├── cmd/
|
||||||
|
│ ├── app/ # 主服务入口
|
||||||
|
│ └── backfill/ # 历史 K 线回填 CLI
|
||||||
|
├── config/ # cleanenv 配置
|
||||||
|
├── internal/
|
||||||
|
│ ├── app/ # 装配 DI、cron scheduler、graceful shutdown
|
||||||
|
│ ├── controller/ # Fiber 路由 + middleware
|
||||||
|
│ ├── entity/ # 业务实体（不依赖 Fiber/DB/Binance）
|
||||||
|
│ ├── usecase/ # 业务接口与编排
|
||||||
|
│ ├── repo/
|
||||||
|
│ │ ├── webapi/binance/ # Binance HTTP client
|
||||||
|
│ │ └── persistent/postgres/ # 5 个 repository
|
||||||
|
│ └── worker/ # 实际跑采集逻辑的 collector
|
||||||
|
├── migrations/ # SQL migration（含 hypertable 转换）
|
||||||
|
├── pkg/
|
||||||
|
│ ├── httpclient/ # 通用 HTTP client + retry + rate limit
|
||||||
|
│ ├── logger/ # slog 包装
|
||||||
|
│ └── postgres/ # pgxpool 初始化
|
||||||
|
├── docs/dev.md # 完整设计文档
|
||||||
|
├── docker-compose.yml
|
||||||
|
├── Dockerfile
|
||||||
|
├── Makefile
|
||||||
|
└── go.mod
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 数据库
|
||||||
|
|
||||||
|
5 张 hypertable，主键设计保证 upsert 幂等：
|
||||||
|
|
||||||
|
| 表 | 主键 | 时间列 |
|
||||||
|
|---|---|---|
|
||||||
|
| `market_klines` | `(source, symbol, interval, open_time)` | `open_time` |
|
||||||
|
| `funding_rates` | `(source, symbol, funding_time)` | `funding_time` |
|
||||||
|
| `open_interest` | `(source, symbol, period, timestamp)` | `timestamp` |
|
||||||
|
| `long_short_ratio` | `(source, symbol, period, ratio_type, timestamp)` | `timestamp` |
|
||||||
|
| `taker_buy_sell_volume` | `(source, symbol, period, timestamp)` | `timestamp` |
|
||||||
|
|
||||||
|
K 线时间列存毫秒 epoch，Timescale chunk 间隔默认 1 周（衍生品表）/ 30 天（funding）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 常用命令
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make run # 跑服务
|
||||||
|
make build # 编译两个二进制到 ./bin/
|
||||||
|
make test # 跑测试
|
||||||
|
make tidy # go mod tidy
|
||||||
|
make docker-up # 起 Timescale
|
||||||
|
make docker-down # 停 docker
|
||||||
|
make migrate-up # apply migration
|
||||||
|
make migrate-down # 回滚 1 步
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 1h --limit 500"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Clean Architecture 边界（验收用）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# controller 不应直接依赖 binance client
|
||||||
|
grep -r "repo/webapi/binance" internal/controller # 期望: 无输出
|
||||||
|
|
||||||
|
# usecase 不应直接依赖具体 binance/postgres 实现
|
||||||
|
grep -r "repo/webapi/binance\|repo/persistent" internal/usecase # 期望: 无输出
|
||||||
|
|
||||||
|
# 全项目不应出现私钥/账户/签名相关字段
|
||||||
|
grep -ri "apikey\|x-mbx-apikey\|hmac" . # 期望: 无输出
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 路线图
|
||||||
|
|
||||||
|
- **v1（当前）**：Binance 行情 + 衍生品 + 落库 + `/v1/market/context` 聚合接口。
|
||||||
|
- **v2**：技术结构计算（支撑压力 / 箱体 / 多空线 / 均线 / Vegas）、CoinGlass 清算数据、ETF Flow。
|
||||||
|
- **v3**：CME gap、CVD、链上稳定币流动性、宏观日历、多交易所聚合。
|
||||||
|
|
||||||
|
完整设计参见 [`docs/dev.md`](docs/dev.md)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Internal project. Not for redistribution.
|
||||||
189
ai/adr/0001-architecture-foundations.md
Normal file
189
ai/adr/0001-architecture-foundations.md
Normal file
@@ -0,0 +1,189 @@
|
|||||||
|
# ADR 0001 — 架构基础决策
|
||||||
|
|
||||||
|
**状态**：Accepted
|
||||||
|
**日期**：2026-05-24
|
||||||
|
**决策者**：项目维护者
|
||||||
|
**适用范围**：cryptoHermes v1 全部代码
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 背景
|
||||||
|
|
||||||
|
cryptoHermes 是给上游 Hermes 量化分析引擎用的 Binance Futures 行情网关。在 MVP 启动前需要锁定一组基础技术决策，避免后续 PR 反复重新评估。
|
||||||
|
|
||||||
|
需求边界（来自 `docs/dev.md` 第 1 章）：
|
||||||
|
- 只读公共行情，不下单、不接私钥
|
||||||
|
- BTCUSDT / ETHUSDT，分钟到周级 K 线 + 衍生品（funding / OI / 多空比 / taker volume）
|
||||||
|
- 历史数据必须自存（Binance 历史 OI / 多空比官方只保留 30 天）
|
||||||
|
- Hermes 只调用 `/v1/market/context` 一个聚合接口
|
||||||
|
- 部署形态：单体服务 + Postgres，国内开发机通过 HTTPS_PROXY 访问 Binance
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 决策
|
||||||
|
|
||||||
|
### D1. Web 框架：Fiber v2
|
||||||
|
|
||||||
|
**选了 Fiber v2**（而不是 net/http 标准库、Gin、Echo、Chi）。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- 性能足够（fasthttp 底层），单体网关不需要再调优
|
||||||
|
- API 与 Express/Gin 相近，团队上手快
|
||||||
|
- 内置 recover/logger/req-id 中间件
|
||||||
|
- v2 稳定，社区活跃
|
||||||
|
|
||||||
|
**代价**：
|
||||||
|
- 基于 fasthttp，不兼容 `net/http` 中间件生态
|
||||||
|
- 错误类型用 `*fiber.Error`，需要在 controller 包一层
|
||||||
|
|
||||||
|
**何时重新评估**：如果需要 HTTP/2 server push、需要复用 `net/http` 生态中间件、或者 v2 长期不维护——届时考虑迁回标准库。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D2. 数据库：TimescaleDB on PG 16
|
||||||
|
|
||||||
|
**选了 TimescaleDB**（而不是纯 Postgres、InfluxDB、ClickHouse）。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- 时序数据天然适合 hypertable：5 张表全部按时间分片，查询近期数据走最新 chunk，旧数据自动归档
|
||||||
|
- SQL 接口和普通 PG 完全兼容，pgx 直连，无需新驱动
|
||||||
|
- 没有引入新的运维栈（开发机用 Docker 起 `timescale/timescaledb:latest-pg16` 即可）
|
||||||
|
- Chunk 间隔可按表调（K 线 1 周、funding 30 天）
|
||||||
|
|
||||||
|
**代价**：
|
||||||
|
- 必须跑 `SELECT create_hypertable(...)` 才能享受时序优化，新表容易忘
|
||||||
|
- 升级 PG 大版本需要看 Timescale 兼容矩阵
|
||||||
|
|
||||||
|
**何时重新评估**：如果数据量进入 TB 级、需要列存压缩或者 OLAP 复杂聚合——考虑 ClickHouse。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D3. Architecture：Clean Architecture（4 层 + ports.go 接口边界）
|
||||||
|
|
||||||
|
**4 层**：`controller / usecase / repo / entity`。
|
||||||
|
**接口边界**：`internal/usecase/ports.go` 是唯一定义对外接口的地方。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- 上游 Hermes 只通过 REST 调用本服务，但 Binance 这个数据源大概率会扩展到 CoinGlass / ETF / CME。把数据源抽象成 `MarketDataProvider` / `DerivativesProvider` 接口，新增数据源只需要新建 `internal/repo/webapi/<source>/`，不动业务编排。
|
||||||
|
- 单测可以 mock 接口，不需要起真 DB / 真 Binance。
|
||||||
|
- `internal/app/app.go` 是唯一的组合根，DI 集中可见。
|
||||||
|
|
||||||
|
**强制约束**（见 `ai/risk-guardrails.md` G6）：
|
||||||
|
```
|
||||||
|
controller 不可直接 import binance/postgres
|
||||||
|
usecase 不可直接 import binance/postgres
|
||||||
|
```
|
||||||
|
|
||||||
|
**代价**：
|
||||||
|
- 多一层 `ports.go`，每个新接口要写两次（接口 + 实现）
|
||||||
|
- 对小型项目可能略 overkill，但本项目已经规划到 v3（多数据源），早做减少返工
|
||||||
|
|
||||||
|
**何时重新评估**：如果 v2 之后接口数量爆炸（> 30 个）→ 考虑按子域拆分 `ports.go` 为多个文件。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D4. 传输层：REST-only（v1 不上 gRPC / 消息队列 / WebSocket）
|
||||||
|
|
||||||
|
**选了 REST**（而不是 gRPC、NATS、WS 推送）。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- Hermes 是同语言 Go 服务，但跨服务调用频率是分钟级（拿一次 context），REST + JSON 完全够
|
||||||
|
- gRPC 多一份 proto 维护负担，团队/工具收益不明显
|
||||||
|
- WebSocket 推送是独立工程量（重连、消息丢失、订阅状态机），v1 不进入
|
||||||
|
- Cron + REST 拉取模型对运维友好（任何 HTTP 监控就够）
|
||||||
|
|
||||||
|
**代价**：
|
||||||
|
- 如果未来 Hermes 需要 sub-second 行情，REST 拉模型撑不住
|
||||||
|
|
||||||
|
**何时重新评估**：Hermes 接入实盘策略后，如果发现"分钟级"延迟成为瓶颈 → 写新 ADR 引入 WS。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D5. 价格 / 成交量：全链路 `string`
|
||||||
|
|
||||||
|
**所有金额字段端到端用 `string`**：Binance DTO 是 string、entity 是 string、JSON 响应是 string。落库时由 PG 转 `NUMERIC(36,18)`。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- Binance 原样返回字符串，避免 float64 转换丢精度
|
||||||
|
- `108000.12` 这种数 float64 表示就会丢尾数；累积统计后偏差更大
|
||||||
|
- 上游量化分析对价格精度敏感，宁可让 Hermes 自己决定怎么解析
|
||||||
|
|
||||||
|
**代价**：
|
||||||
|
- 加减乘除需要 PG 端做或者引入 decimal 库
|
||||||
|
- JSON 响应里 price 是字符串，前端/客户端要适配
|
||||||
|
|
||||||
|
**强制约束**：见 `ai/risk-guardrails.md` G2。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D6. 数据源边界：只用 Binance USDⓈ-M Futures 公共 API
|
||||||
|
|
||||||
|
**v1 不引入 CoinGlass、Glassnode、ETF Flow、CME、链上数据**。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- Binance 一家覆盖 80% 的需求（K 线 / 衍生品 / 多空比）
|
||||||
|
- 引入新数据源等于引入新的鉴权、限流、错误模式、SLA——v1 优先把一条链路打通
|
||||||
|
- 路线图 v2 / v3 才扩展数据源（README "路线图" 段）
|
||||||
|
|
||||||
|
**强制约束**：
|
||||||
|
- 新增数据源前必须先写 `ai/specs/<source>.md`（见 AGENTS.md §10）
|
||||||
|
- 不可使用 Binance 的私有签名接口（见 G1）
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D7. 配置层：cleanenv（yaml + env override）
|
||||||
|
|
||||||
|
**选了 cleanenv**（而不是 viper、koanf、env-only）。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- 一份 yaml 默认值 + 环境变量覆盖，开发和部署都方便
|
||||||
|
- 比 viper 轻，没有多余抽象
|
||||||
|
- `env-default:` tag 让默认值跟字段定义放在一起，可读性好
|
||||||
|
|
||||||
|
**代价**：无明显缺点。如果后续需要热重载（不需要）才会考虑换。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D8. 限流：`golang.org/x/time/rate` token bucket
|
||||||
|
|
||||||
|
挂在 `pkg/httpclient/client.go` 里，默认 RPS=20 / Burst=40，从配置读。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- 标准库附属，零外部依赖
|
||||||
|
- token bucket 适合突发场景
|
||||||
|
- Binance IP 限 2400 weight/min，20 req/s × 60s = 1200 req/min，留出一半余量给重试和 backfill
|
||||||
|
|
||||||
|
**强制约束**：所有外部 HTTP 调用走 `pkg/httpclient`，见 G5。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D9. Scheduler：robfig/cron v3
|
||||||
|
|
||||||
|
**选了 robfig/cron v3**（而不是自己写 ticker、go-co-op/gocron）。
|
||||||
|
|
||||||
|
**理由**：
|
||||||
|
- cron 表达式所有人都看得懂，schedule 修改是配置而非代码
|
||||||
|
- 稳定老牌，3.0 之后无破坏性变化
|
||||||
|
- 和 Fiber 在同一进程内运行，graceful shutdown 容易做
|
||||||
|
|
||||||
|
**代价**：
|
||||||
|
- 没有分布式锁——但本服务 v1 是单实例，不需要
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 不在本 ADR 范围
|
||||||
|
|
||||||
|
- **下单 / 账户 / 私钥**：永久禁止（G1）。要做必须新仓库。
|
||||||
|
- **多交易所聚合**：v3 才考虑，届时再写 ADR。
|
||||||
|
- **WS / 实时推送**：见 D4，需要单独写 ADR。
|
||||||
|
- **观测性（metrics / tracing）**：MVP 阶段只用 slog 结构化日志，Prometheus / OTel 留待生产部署前评估。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 后续 ADR 的触发条件（来自 AGENTS.md §10）
|
||||||
|
|
||||||
|
- 换 HTTP 框架 → 新 ADR
|
||||||
|
- 引入消息队列 / WS → 新 ADR
|
||||||
|
- 跨服务通信改 gRPC → 新 ADR
|
||||||
|
- 数据库换型（PG → ClickHouse）→ 新 ADR
|
||||||
|
- 引入新数据源 → `ai/specs/<source>.md`（不一定要 ADR，除非影响整体架构）
|
||||||
103
ai/harness-health.md
Normal file
103
ai/harness-health.md
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
# Harness Health — cryptoHermes
|
||||||
|
|
||||||
|
**日期**：2026-05-24
|
||||||
|
**版本**：v1 MVP 刚完成（Milestone 1-5）
|
||||||
|
|
||||||
|
按 he-maintainer 的 scoring rubric 评估当前 harness 工程状态。Rating: `strong` / `partial` / `weak`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 总览
|
||||||
|
|
||||||
|
| 维度 | Rating | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| 项目地图清晰度 | strong | `ai/project-map.md` 完整覆盖 Clean Arch 4 层 + 数据流 + 扩展点 |
|
||||||
|
| 验证命令可跑 | strong | Makefile 全部命令在 README + AGENTS.md 双重列出 |
|
||||||
|
| 快速反馈环路 | strong | `go build ./...` < 2s，`go vet ./...` < 1s，单包测试支持 |
|
||||||
|
| 任务契约质量 | partial | 模板已写（`ai/task-templates.md`），但还没有真实任务用过 |
|
||||||
|
| 守卫规则清晰度 | strong | 10 条 G1-G10，每条都有可机械验证命令 |
|
||||||
|
| 架构决策耐久度 | strong | ADR 0001 记录 9 条核心决策；其他 ADR 等触发 |
|
||||||
|
| Spec / test-plan | partial | 还没有 specs/，因为 v1 没有需要 spec 的任务；v2 新数据源时必填 |
|
||||||
|
| 失败反馈环路 | partial | AGENTS.md §10 写了升级时机，但还没有 `ai/risk-guardrails.md` 改动史可参考 |
|
||||||
|
| 文档 vs 代码对齐 | strong | AGENTS.md / README / project-map 全部基于实际仓库结构 |
|
||||||
|
| MEMORY / 长期上下文 | n/a | 本仓库无 `MEMORY.md`，按设计——长期知识在 `docs/dev.md` 和 `ai/` 文档中 |
|
||||||
|
|
||||||
|
**总体**：**strong**（MVP 完成时点适用），但有 2 个 partial：
|
||||||
|
- task-templates 未经过真实任务校验
|
||||||
|
- specs/ 目录还是空的（等 v2 需要时填）
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 已具备的工程基础
|
||||||
|
|
||||||
|
- **3 层 AI 文档系统**：`AGENTS.md`（速查）→ `ai/project-map.md`（结构）→ `ai/adr/`（决策）。`docs/dev.md` 作为最详细的设计源头单独存在。
|
||||||
|
- **可机械验证的守卫**：G1-G10 全部给了 grep / make 命令，没有靠 review 的软规则。
|
||||||
|
- **唯一的组合根**：`internal/app/app.go`，DI 一处可见。
|
||||||
|
- **接口边界文件**：`internal/usecase/ports.go` 是唯一定义对外接口的位置，新 agent 找接口去这里。
|
||||||
|
- **强约束 grep 在 AGENTS.md §6**：每次 PR 都可以直接跑。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 已识别的薄弱点
|
||||||
|
|
||||||
|
### W1. task-templates 未经过实战
|
||||||
|
**现状**：模板已写完，但没有 PR 用过这些契约。
|
||||||
|
**影响**：模板可能太严格或太松。
|
||||||
|
**纠正时机**：完成第一个非平凡 PR（比如 Milestone 6 的技术指标）后回看，调整模板。
|
||||||
|
|
||||||
|
### W2. 还没有 work-status.md
|
||||||
|
**现状**：目前是单 agent / 单会话工作，不需要 work-status 协调。
|
||||||
|
**影响**：暂无。
|
||||||
|
**升级时机**：开始 Milestone 6（技术指标）或 v2（新数据源）时，如果跨多次会话，建立 `ai/work-status.md`。
|
||||||
|
|
||||||
|
### W3. CI 完全未搭建
|
||||||
|
**现状**：本地有 `make test` / `go vet`，但仓库无 GitHub Actions / GitLab CI。
|
||||||
|
**影响**：守卫规则全靠人工跑 grep。
|
||||||
|
**升级时机**：合并第一个外部 PR 前，搭一个最小 CI（`go build` + `go vet` + grep 边界检查 + migration up/down）。
|
||||||
|
|
||||||
|
### W4. 单元测试覆盖率未度量
|
||||||
|
**现状**：MVP 阶段重点是端到端可用，未补单测覆盖率。
|
||||||
|
**影响**：refactor 时风险偏高。
|
||||||
|
**升级时机**：Milestone 6 引入指标计算（计算正确性 critical）前，先补 mapper / repo 的 table-driven 测试。
|
||||||
|
|
||||||
|
### W5. 部分 docs/* 未提交到 git
|
||||||
|
**现状**：`docs/dev.md` 是 untracked（git status 显示 `?? docs/`）。
|
||||||
|
**纠正方式**：本次 commit 一起加进来。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 后续升级触发器
|
||||||
|
|
||||||
|
| 触发条件 | 应采取的动作 |
|
||||||
|
|---|---|
|
||||||
|
| 同一类错误第二次出现 | 加进 `ai/risk-guardrails.md`，给可验证命令 |
|
||||||
|
| 重大架构决策（换 HTTP 框架 / 引入 MQ / 跨服务通信） | 写新 ADR |
|
||||||
|
| 新增数据源（CoinGlass / ETF / CME） | 必填 `ai/specs/<source>.md` |
|
||||||
|
| 跨多次会话的长任务 | 写 `ai/work-status.md` |
|
||||||
|
| 单测/集成测试出现 flake | 加 test-plan，约定 retry / quarantine 策略 |
|
||||||
|
| 文档与代码出现漂移（grep 守卫失败但代码正常） | 立刻同步，不要"以代码为准"绕开守卫 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 复评建议
|
||||||
|
|
||||||
|
- **3 个月后**（约 2026-08）：复评本文件。若 v2 数据源工作启动，至少需要 1 份 spec + 可能新 ADR。
|
||||||
|
- **每次新 Milestone 完成时**：跑一次 AGENTS.md §6 的 3 个 grep，确认边界未漂移。
|
||||||
|
- **新成员（agent 或人）首次提 PR 后**：根据他们卡住的地方反向更新 AGENTS.md / project-map（"如果他们卡过这里，下一个人也会卡"）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Harness 文件清单（截至本次 commit）
|
||||||
|
|
||||||
|
```
|
||||||
|
AGENTS.md ← AI agent 速查
|
||||||
|
README.md ← 用户向使用文档
|
||||||
|
docs/dev.md ← 完整设计文档（源头）
|
||||||
|
ai/project-map.md ← 仓库结构与扩展点
|
||||||
|
ai/risk-guardrails.md ← G1-G10 守卫规则
|
||||||
|
ai/adr/0001-architecture-foundations.md ← 9 条架构基础决策
|
||||||
|
ai/task-templates.md ← 6 种任务模板
|
||||||
|
ai/harness-health.md ← 本文件
|
||||||
|
```
|
||||||
|
|
||||||
|
CLAUDE.md / GEMINI.md / .cursor/*.md 等其他 AI 工具的入口文件——**暂不需要**，因为这些工具都会读 AGENTS.md。如未来要差异化对待，再分裂。
|
||||||
224
ai/project-map.md
Normal file
224
ai/project-map.md
Normal file
@@ -0,0 +1,224 @@
|
|||||||
|
# Project Map — cryptoHermes
|
||||||
|
|
||||||
|
代码 agent 工作的地形图。每个目录写明：**职责**、**可以依赖谁**、**不可以依赖谁**、**扩展点在哪**。
|
||||||
|
|
||||||
|
完整设计见 `docs/dev.md`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 顶层布局
|
||||||
|
|
||||||
|
```
|
||||||
|
cryptoHermes/
|
||||||
|
├── cmd/ 二进制入口
|
||||||
|
├── config/ cleanenv 加载
|
||||||
|
├── internal/ 业务代码（不可被外部 import）
|
||||||
|
├── migrations/ SQL（含 hypertable）
|
||||||
|
├── pkg/ 项目内可复用基础设施
|
||||||
|
├── docs/ 设计文档（dev.md 是源头）
|
||||||
|
├── ai/ harness 工程文档（本目录）
|
||||||
|
├── docker-compose.yml Timescale + 服务
|
||||||
|
├── Dockerfile multi-stage golang:1.23-alpine → alpine:3.20
|
||||||
|
├── Makefile 所有验证命令
|
||||||
|
└── go.mod Go 1.23
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `cmd/` — 二进制入口
|
||||||
|
|
||||||
|
| 路径 | 职责 |
|
||||||
|
|---|---|
|
||||||
|
| `cmd/app/main.go` | 主服务。读 config → 调 `internal/app.Run(cfg)` |
|
||||||
|
| `cmd/backfill/main.go` | 历史 K 线回填 CLI。直接装配 binance client + kline repo，不走 fiber |
|
||||||
|
|
||||||
|
**约束**：`cmd/` 里只允许装配 + 调用，不允许写业务逻辑。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `config/` — 配置
|
||||||
|
|
||||||
|
- `config.go`：cleanenv 加载，`yaml + env` 双源。环境变量优先（见 `env:` tag）。
|
||||||
|
- `config.yml`：默认配置。**不要在这里放敏感信息**（没有敏感信息可放，因为本服务无私钥）。
|
||||||
|
|
||||||
|
**扩展点**：新增配置项 → `config.go` 加字段（同时加 `yaml:` 和 `env:` tag）→ `config.yml` 加默认值 → 在 README "配置" 段补一行。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `internal/` — 业务核心
|
||||||
|
|
||||||
|
Clean Architecture 四层。**依赖方向单向：`controller → usecase → entity ← repo`**。
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────┐
|
||||||
|
│ controller │ HTTP 层：参数校验、调 usecase、序列化
|
||||||
|
└──────┬───────┘
|
||||||
|
│ depends on
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│ usecase │ 业务编排。只依赖 ports.go 中的接口
|
||||||
|
└──────┬───────┘
|
||||||
|
│ depends on (interface only)
|
||||||
|
▼
|
||||||
|
┌──────────────┐ ┌──────────────┐
|
||||||
|
│ entity │ ◄────── │ repo │ 外部依赖实现
|
||||||
|
└──────────────┘ └──────────────┘
|
||||||
|
(binance / postgres)
|
||||||
|
```
|
||||||
|
|
||||||
|
### `internal/entity/` — 纯业务实体
|
||||||
|
|
||||||
|
- `kline.go`, `ticker.go`, `funding.go`, `open_interest.go`, `long_short_ratio.go`, `taker_volume.go`, `market_context.go`
|
||||||
|
- **价格/成交量字段全部 `string`**，避免 float64 精度问题。落库时 PG 转 `NUMERIC(36,18)`。
|
||||||
|
- `Kline` 有 `IsClosed bool` 字段——未收线 K 线 (`close_time > now()`) 由 mapper 自动判定。
|
||||||
|
- `LongShortRatio` 有常量 `RatioTypeGlobalAccount` / `RatioTypeTopTraderPosition` / `RatioTypeTopTraderAccount`。
|
||||||
|
|
||||||
|
**不依赖**：fiber、pgx、binance、任何外部库。可以从这里 import 到任何其他层。
|
||||||
|
|
||||||
|
### `internal/usecase/` — 业务接口与编排
|
||||||
|
|
||||||
|
- `ports.go`：**全部对外接口在这里定义**。这是 Clean Arch 的边界文件。
|
||||||
|
- `MarketDataProvider`（K线、ticker24h）
|
||||||
|
- `DerivativesProvider`（funding/OI/多空比/taker volume 共 7 个方法）
|
||||||
|
- `KlineRepository` / `FundingRepository` / `OpenInterestRepository` / `LongShortRatioRepository` / `TakerVolumeRepository`
|
||||||
|
- `market_context.go`：聚合 `/v1/market/context`。errgroup 并发拉 snapshot/funding/OI，DB K 线不足 200 根回源 Binance 补，异步写回。
|
||||||
|
|
||||||
|
**约束**（grep 可验证）：
|
||||||
|
```bash
|
||||||
|
grep -r "repo/webapi/binance\|repo/persistent" internal/usecase # 必须无输出
|
||||||
|
```
|
||||||
|
usecase 不可直接 import 任何具体实现。新增业务流程 → 在 `ports.go` 加接口 → 在 usecase 编排 → 由 `internal/app/app.go` 注入实现。
|
||||||
|
|
||||||
|
### `internal/controller/` — HTTP 层
|
||||||
|
|
||||||
|
- `restapi/router.go` + `middleware.go`：fiber app、recover、logger、request-id
|
||||||
|
- `restapi/v1/health_routes.go`：`/v1/health`
|
||||||
|
- `restapi/v1/market_routes.go`：`/v1/market/{context,klines,snapshot,derivatives}`
|
||||||
|
|
||||||
|
**约束**：
|
||||||
|
```bash
|
||||||
|
grep -r "repo/webapi/binance\|repo/persistent" internal/controller # 必须无输出
|
||||||
|
```
|
||||||
|
controller 只能依赖 usecase 和 entity，不能直接调 binance client 或 postgres repo。
|
||||||
|
|
||||||
|
**扩展点**：新增路由 → 加 handler → 通过 `MarketDeps` 注入依赖（不要直接 import repo 包）。
|
||||||
|
|
||||||
|
### `internal/repo/` — 外部依赖实现
|
||||||
|
|
||||||
|
| 子目录 | 实现哪些接口 |
|
||||||
|
|---|---|
|
||||||
|
| `webapi/binance/` | `MarketDataProvider`、`DerivativesProvider` |
|
||||||
|
| `persistent/postgres/` | 5 个 `*Repository` |
|
||||||
|
|
||||||
|
**Binance client**（`webapi/binance/`）：
|
||||||
|
- `client.go`：包 `pkg/httpclient`，统一 base URL `https://fapi.binance.com`，错误用 `ExternalAPIError{Source, Path, StatusCode, Message}` 包装。
|
||||||
|
- `dto.go`：贴近 Binance 原始响应（K 线是数组）。
|
||||||
|
- `mapper.go`：DTO → entity。**K 线按下标解析**（`row[0]=openTime ... row[10]=takerBuyQuoteVolume`），自动设置 `IsClosed = close_time < now()`。
|
||||||
|
- `market.go`：`GetKlines` / `GetKlinesRange` / `GetTicker24h`
|
||||||
|
- `derivatives.go`：funding × 2 + OI × 2 + 多空比 × 2 + taker volume
|
||||||
|
|
||||||
|
**Postgres repos**（`persistent/postgres/`）：
|
||||||
|
- 5 个 repo，每个有 `UpsertMany`（`ON CONFLICT DO UPDATE`）和 `FindRecent`（按时间倒序拉，返回前反转成正序）。
|
||||||
|
- `kline_repo.go` 的 `UpsertMany` 会过滤掉 `IsClosed=false` 的 K 线。
|
||||||
|
- 使用 pgxpool + transaction。
|
||||||
|
|
||||||
|
**扩展点**：
|
||||||
|
- 新增数据源（CoinGlass / ETF）→ 新建 `webapi/coinglass/`，实现新的 Provider 接口（先在 `ports.go` 定义）。
|
||||||
|
- 新增表 → 新 migration + 新 repo + 在 `ports.go` 加新接口。
|
||||||
|
|
||||||
|
### `internal/worker/` — Collector
|
||||||
|
|
||||||
|
- `collector.go`：被 cron 调度的采集逻辑。每个 `Collect*` 方法遍历 `symbols × intervals/periods`，拉数据 → upsert。
|
||||||
|
- **不直接被 controller 调用**，只通过 `internal/app/scheduler.go` 触发。
|
||||||
|
|
||||||
|
### `internal/app/` — 装配与生命周期
|
||||||
|
|
||||||
|
- `app.go`：DI 装配的唯一位置。
|
||||||
|
- 顺序：pool → binanceClient → repos → contextUC → collector → scheduler → fiber app
|
||||||
|
- graceful shutdown：监听 SIGINT/SIGTERM，10s 超时，先停 cron、再停 fiber、最后关 pool。
|
||||||
|
- `scheduler.go`：cron spec 在这里。
|
||||||
|
|
||||||
|
**这里是 Clean Arch 的"组合根"**——所有具体类型在这里被实例化并注入抽象接口。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `pkg/` — 项目内可复用基础设施
|
||||||
|
|
||||||
|
| 包 | 职责 |
|
||||||
|
|---|---|
|
||||||
|
| `pkg/httpclient/` | 通用 HTTP client + 重试（429/5xx 最多 2 次）+ `x/time/rate` 限流 |
|
||||||
|
| `pkg/logger/` | slog 包装（按 env 选 JSON / Text） |
|
||||||
|
| `pkg/postgres/` | pgxpool 初始化 + Ping |
|
||||||
|
|
||||||
|
**约束**：`pkg/` 不依赖 `internal/`，反向依赖。可以被外部项目 import（虽然目前没有）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `migrations/` — SQL
|
||||||
|
|
||||||
|
10 个文件（5 张表 × up/down）。
|
||||||
|
|
||||||
|
每个 `.up.sql` 末尾必须有：
|
||||||
|
```sql
|
||||||
|
SELECT create_hypertable('<table>', '<time_col>',
|
||||||
|
chunk_time_interval => <ms>,
|
||||||
|
if_not_exists => TRUE,
|
||||||
|
migrate_data => TRUE);
|
||||||
|
```
|
||||||
|
|
||||||
|
chunk 间隔：
|
||||||
|
- K 线 / OI / LS / taker → 1 周 = 604800000 ms
|
||||||
|
- funding → 30 天 = 2592000000 ms
|
||||||
|
|
||||||
|
**改 migration 前必读 `ai/risk-guardrails.md` 中"DB migration"段**。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 数据流（控制流）
|
||||||
|
|
||||||
|
### 写路径（采集）
|
||||||
|
|
||||||
|
```
|
||||||
|
cron tick
|
||||||
|
└→ internal/app/scheduler.go
|
||||||
|
└→ internal/worker/collector.go (Collect*)
|
||||||
|
├→ binance.GetXxx (HTTP → DTO → entity)
|
||||||
|
└→ repo.UpsertMany (entity → SQL upsert)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 读路径（API）
|
||||||
|
|
||||||
|
```
|
||||||
|
HTTP request
|
||||||
|
└→ internal/controller/restapi/v1/market_routes.go
|
||||||
|
└→ usecase.MarketContext.Build
|
||||||
|
├→ marketData.GetTicker24h ┐
|
||||||
|
├→ derivatives.GetCurrent… ├ errgroup 并发
|
||||||
|
├→ derivatives.GetCurrent… ┘
|
||||||
|
├→ klineRepo.FindRecent × 5 周期
|
||||||
|
│ └ DB 不足 → 回源 binance.GetKlines（异步写回）
|
||||||
|
└→ fundingRepo / oiRepo / lsRepo.FindRecent
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 不可逾越的边界（自动验证见 AGENTS.md §6）
|
||||||
|
|
||||||
|
1. **controller ↛ binance/postgres**：必须经 usecase。
|
||||||
|
2. **usecase ↛ binance/postgres 包**：只依赖 `ports.go` 接口。
|
||||||
|
3. **entity ↛ 任何外部库**：可被所有层 import。
|
||||||
|
4. **pkg ↛ internal**：基础设施不知道业务。
|
||||||
|
5. **任何代码 ↛ 私钥/签名**：grep `apikey|hmac|x-mbx-apikey|secret_key|api_secret` 必须无 Go 代码命中。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 关键扩展点速查
|
||||||
|
|
||||||
|
| 场景 | 改哪 |
|
||||||
|
|---|---|
|
||||||
|
| 新增 REST 路由 | `internal/controller/restapi/v1/`，依赖通过 `MarketDeps` 注入 |
|
||||||
|
| 新增业务编排 | `internal/usecase/ports.go` 加接口 → 新 usecase 文件 → `internal/app/app.go` 装配 |
|
||||||
|
| 新增数据表 | 新 migration → 新 entity（可选）→ 新 repo → `ports.go` 接口 → 装配 |
|
||||||
|
| 新增数据源 | `ai/specs/<source>.md` → 新 `internal/repo/webapi/<source>/` → 新 Provider 接口 |
|
||||||
|
| 新增定时任务 | `internal/worker/collector.go` 加方法 → `internal/app/scheduler.go` 注册 cron |
|
||||||
|
| 改配置项 | `config/config.go` 加字段 → `config/config.yml` 加默认 → README 记录 |
|
||||||
184
ai/risk-guardrails.md
Normal file
184
ai/risk-guardrails.md
Normal file
@@ -0,0 +1,184 @@
|
|||||||
|
# Risk Guardrails — cryptoHermes
|
||||||
|
|
||||||
|
可机械验证 / 不可逾越的规则。每条规则给出**为什么**、**怎么验证**。
|
||||||
|
|
||||||
|
任何 PR 都应该满足这里全部规则。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G1. 绝不接入交易能力（Hard Stop）
|
||||||
|
|
||||||
|
**规则**：本服务只读公共行情。**禁止**任何下单、撤单、查询账户/持仓/资金、签名请求、托管、转账。
|
||||||
|
|
||||||
|
**为什么**：项目定位是"上游 Hermes 量化分析引擎的行情网关"。一旦引入交易能力，安全边界、合规风险、密钥管理责任全部上升一个数量级。`docs/dev.md` 第 1 章原文："本服务只提供市场数据与分析上下文，不做交易、不下单、不托管资金"。
|
||||||
|
|
||||||
|
**怎么验证**：
|
||||||
|
```bash
|
||||||
|
# Go 代码不可出现私钥/签名相关字段
|
||||||
|
grep -ri --include="*.go" "apikey\|x-mbx-apikey\|hmac\|secret_key\|api_secret" .
|
||||||
|
# 期望：无输出（注释/字符串/文档中说明"不接入"不算）
|
||||||
|
|
||||||
|
# Binance 私有接口前缀不可出现
|
||||||
|
grep -r --include="*.go" "fapi/v1/order\|fapi/v1/account\|fapi/v2/account\|fapi/v2/positionRisk" .
|
||||||
|
# 期望：无输出
|
||||||
|
```
|
||||||
|
|
||||||
|
**例外**：无。如果上游需求变化要做交易，**必须先开新仓库**或新模块（独立鉴权 / 独立部署），不能在本仓库内直接放开。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G2. 价格 / 成交量绝不用 float64（Hard Stop）
|
||||||
|
|
||||||
|
**规则**：所有金额、价格、成交量字段**全链路用 `string`**。DTO、entity、JSON 响应都是 `string`。落库时由 PG 转 `NUMERIC(36,18)`。
|
||||||
|
|
||||||
|
**为什么**：`108000.12` 这种价格 float64 表示就会丢精度。Binance 原样返回字符串，本服务保留字符串到 PG。这一点在所有 5 个 entity 文件里都已经做到了，新代码必须遵守。
|
||||||
|
|
||||||
|
**怎么验证**：
|
||||||
|
```bash
|
||||||
|
# 任何新增 entity 字段如果叫 *Price/*Volume/*Quantity/*Amount，类型必须是 string
|
||||||
|
grep -rn --include="*.go" "Price\s*float\|Volume\s*float\|Quantity\s*float\|Amount\s*float" internal/entity
|
||||||
|
# 期望：无输出
|
||||||
|
```
|
||||||
|
|
||||||
|
**例外**：纯统计/分析（如指标计算的中间过程）可以用 float64，但**结果落库或返回 API 时必须转回 string**。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G3. 未收线 K 线（IsClosed=false）不入库
|
||||||
|
|
||||||
|
**规则**：从 Binance 拉到的最新一根 K 线如果 `close_time > now()`，**不能写入 `market_klines`**。
|
||||||
|
|
||||||
|
**为什么**：未收线 K 线的 close/high/low/volume 还会变。如果入库，后续不更新就是脏数据，更新又会触发 upsert 风暴。`internal/repo/persistent/postgres/kline_repo.go` 的 `UpsertMany` 已经过滤 `IsClosed=false`，`internal/repo/webapi/binance/mapper.go` 已经自动设置 `IsClosed = closeTime < now()`。新代码（含 backfill）必须遵守。
|
||||||
|
|
||||||
|
**怎么验证**：手动跑一次 collector，立刻 `SELECT count(*) FROM market_klines WHERE close_time > extract(epoch from now()) * 1000` 应为 0。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G4. UpsertMany 必须幂等
|
||||||
|
|
||||||
|
**规则**：所有 repo 的 `UpsertMany` 用 `INSERT ... ON CONFLICT (<PK>) DO UPDATE SET ... updated_at = now()`。**禁止** `INSERT` 不带 ON CONFLICT 或用 `INSERT ... ON CONFLICT DO NOTHING`（后者错过了更新场景）。
|
||||||
|
|
||||||
|
**为什么**：cron 会重复跑、backfill 会和 cron 重叠、重启会重拉。重复执行必须不能产生重复行也不能丢更新（例如 OI 后修正、close 值微调）。
|
||||||
|
|
||||||
|
**怎么验证**：
|
||||||
|
```bash
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 1h --limit 100"
|
||||||
|
psql $DATABASE_URL -c "SELECT count(*) FROM market_klines WHERE symbol='BTCUSDT' AND interval='1h'"
|
||||||
|
make backfill ARGS="--symbol BTCUSDT --interval 1h --limit 100"
|
||||||
|
psql $DATABASE_URL -c "SELECT count(*) FROM market_klines WHERE symbol='BTCUSDT' AND interval='1h'"
|
||||||
|
# 期望：两次 count 一致
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G5. Rate Limit 不可绕开
|
||||||
|
|
||||||
|
**规则**：所有外部 HTTP 调用走 `pkg/httpclient`，该 client 已挂 `golang.org/x/time/rate`（默认 20 req/s，burst 40）。**禁止** 在业务代码里直接用 `net/http` 调 Binance。
|
||||||
|
|
||||||
|
**为什么**：Binance USDⓈ-M Futures IP 级限额 2400 weight/min。一旦超限会被封 IP 几分钟到几小时。本地代理在国内是稀缺资源，封了影响整个开发环境。
|
||||||
|
|
||||||
|
**预算**：新增 collector 任务前估算 weight：
|
||||||
|
- `GET /fapi/v1/klines`：1（limit ≤ 100）→ 10（limit ≤ 1000）
|
||||||
|
- `GET /fapi/v1/ticker/24hr`：1（单 symbol）/ 40（全量）
|
||||||
|
- `GET /fapi/v1/openInterest`：1
|
||||||
|
- `GET /futures/data/*`：1
|
||||||
|
|
||||||
|
**怎么验证**：
|
||||||
|
```bash
|
||||||
|
grep -rn --include="*.go" "net/http\"" internal/repo/webapi
|
||||||
|
# 期望：只在 pkg/httpclient 里，repo/webapi 不应直接 import net/http
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G6. Clean Architecture 边界
|
||||||
|
|
||||||
|
**规则**：见 `ai/project-map.md` "不可逾越的边界"段。3 个 grep 必须无输出。
|
||||||
|
|
||||||
|
**为什么**：一旦 controller / usecase 直接 import 具体实现，测试就需要起 DB 起 Binance；后续替换数据源（CoinGlass）也变成大手术。
|
||||||
|
|
||||||
|
**怎么验证**：
|
||||||
|
```bash
|
||||||
|
grep -r "repo/webapi/binance\|repo/persistent" internal/controller # 必须无输出
|
||||||
|
grep -r "repo/webapi/binance\|repo/persistent" internal/usecase # 必须无输出
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G7. Migration 必须 down/up 互逆
|
||||||
|
|
||||||
|
**规则**：任何 SQL migration 改动后，本地必须跑：
|
||||||
|
```bash
|
||||||
|
make migrate-down && make migrate-up
|
||||||
|
```
|
||||||
|
两步都成功（不报错、不丢表）才能提交。
|
||||||
|
|
||||||
|
**为什么**：生产回滚是兜底手段，down 脚本错了就回不去。
|
||||||
|
|
||||||
|
**额外规则**：
|
||||||
|
- 新 migration **不可改已发布 migration 的内容**——只能加新文件（编号递增）。
|
||||||
|
- 改表结构如果会丢数据（drop column / change type），必须在 PR 里显式说明。
|
||||||
|
- 改 hypertable chunk 间隔需要先看 `docs/dev.md` 第 10 章。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G8. K 线数组下标解析不可破
|
||||||
|
|
||||||
|
**规则**：`internal/repo/webapi/binance/mapper.go` 中 `mapKline` 的下标解析对应 Binance 官方约定：
|
||||||
|
|
||||||
|
```
|
||||||
|
[0] openTime ms
|
||||||
|
[1] open string
|
||||||
|
[2] high string
|
||||||
|
[3] low string
|
||||||
|
[4] close string
|
||||||
|
[5] volume string
|
||||||
|
[6] closeTime ms
|
||||||
|
[7] quoteAssetVolume string
|
||||||
|
[8] numberOfTrades int
|
||||||
|
[9] takerBuyBaseVolume string
|
||||||
|
[10] takerBuyQuoteVolume string
|
||||||
|
[11] ignore
|
||||||
|
```
|
||||||
|
|
||||||
|
**禁止**：调整顺序、跳过位置、改用对象解析（Binance 这个接口就是数组）。
|
||||||
|
|
||||||
|
**为什么**：错一位整张表数据就废了，回滚成本极高。`docs/dev.md` 第 9.2 节有完整说明。
|
||||||
|
|
||||||
|
**怎么验证**：动了 `mapper.go` 之后跑一次回填，对比 Binance 网页 K 线图同一根的 OHLCV 是否一致。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G9. 配置项不引入秘钥
|
||||||
|
|
||||||
|
**规则**：`config/config.yml` 和环境变量列表里不允许出现 `api_key`、`secret`、`token`、`password`（DB 密码除外，且必须从环境变量读、不能 hardcode）。
|
||||||
|
|
||||||
|
**为什么**：本服务不需要任何 Binance 私有鉴权，也不向外部传秘密。一旦引入会破坏 G1。
|
||||||
|
|
||||||
|
**怎么验证**：
|
||||||
|
```bash
|
||||||
|
grep -i "api_key\|api_secret\|access_token" config/
|
||||||
|
# 期望：无输出
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## G10. WebSocket / 实时推送不在 v1 范围
|
||||||
|
|
||||||
|
**规则**：v1 只用 REST 拉取。不引入 gorilla/websocket、不订阅 `wss://fstream.binance.com`。
|
||||||
|
|
||||||
|
**为什么**：WS 长连接的可观测性、重连、消息丢失处理是独立工程量。v1 用 REST + cron 已经能满足 Hermes 的"分钟级上下文"需求。要做 WS 必须先写 ADR。
|
||||||
|
|
||||||
|
**怎么验证**：
|
||||||
|
```bash
|
||||||
|
grep -rn --include="*.go" "websocket\|fstream.binance" .
|
||||||
|
# 期望：无输出
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 升级守护规则的时机
|
||||||
|
|
||||||
|
同一类错误第二次出现 → 写一条新规则进来。修改本文件 = 改动了项目契约，需要在 PR 描述里说明原因并 @ 维护者。
|
||||||
|
|
||||||
|
新增 Hard Stop 规则时，**必须给出可机械验证的命令**——只能靠 review 来执行的规则会被绕过。
|
||||||
205
ai/task-templates.md
Normal file
205
ai/task-templates.md
Normal file
@@ -0,0 +1,205 @@
|
|||||||
|
# Task Templates — cryptoHermes
|
||||||
|
|
||||||
|
任何非平凡修改（>1 个文件、改边界、引入新依赖、改 migration、新增 collector），先写一份微型任务契约。短即可——长度像 commit message，但目的是让你自己（和后续 agent）能在 30 秒内对齐范围。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 通用契约模板
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: 一句话说要做的事
|
||||||
|
scope: 会改的文件 / 目录（具体路径，不要"相关文件"）
|
||||||
|
out-of-scope: 显式不改的范围（防止 scope creep）
|
||||||
|
constraints: 引用 ai/risk-guardrails.md 的哪几条（G1-G10）
|
||||||
|
validation: 要跑哪些命令证明通过（必须可机械执行）
|
||||||
|
acceptance: 完成判据（行为可观测的描述）
|
||||||
|
```
|
||||||
|
|
||||||
|
**示例**：
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: 给 /v1/market/context 加一个 ?intervals=1h,4h 参数，允许 Hermes 只拉部分周期，减少响应体积
|
||||||
|
scope:
|
||||||
|
- internal/controller/restapi/v1/market_routes.go
|
||||||
|
- internal/usecase/market_context.go
|
||||||
|
- README.md（API 段补 query 参数说明）
|
||||||
|
out-of-scope:
|
||||||
|
- 不改 entity（响应结构不变，只是 klines map 的 key 集合可能少几个）
|
||||||
|
- 不改 collector（采集策略不变）
|
||||||
|
constraints:
|
||||||
|
- G6：不能在 controller 里直接调 repo，必须通过 usecase 接收 intervals 参数
|
||||||
|
validation:
|
||||||
|
- go build ./... && go vet ./...
|
||||||
|
- curl 'localhost:8080/v1/market/context?symbol=BTCUSDT&intervals=1h,4h' | jq '.klines | keys'
|
||||||
|
- 期望：只返回 ["1h","4h"]
|
||||||
|
acceptance:
|
||||||
|
- 不传 intervals 时行为完全不变（默认全部 5 个周期）
|
||||||
|
- 传未知 interval 返回 400
|
||||||
|
- 至少一个 happy-path 单测覆盖
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 任务类型 → 模板
|
||||||
|
|
||||||
|
### T1. 新增 API 路由
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: <做什么>
|
||||||
|
scope:
|
||||||
|
- internal/controller/restapi/v1/<file>.go（新 handler）
|
||||||
|
- internal/usecase/<file>.go（新业务方法或扩展现有）
|
||||||
|
- internal/usecase/ports.go（如果需要新 repo 方法）
|
||||||
|
out-of-scope:
|
||||||
|
- 不改 entity 结构
|
||||||
|
- 不改 collector
|
||||||
|
constraints: G6
|
||||||
|
validation:
|
||||||
|
- go build ./... && go vet ./...
|
||||||
|
- curl 测 happy-path + 错误参数
|
||||||
|
acceptance:
|
||||||
|
- 路由响应符合 JSON schema
|
||||||
|
- 参数校验完整（缺失/非法都返回 400）
|
||||||
|
- README 更新 API 段
|
||||||
|
```
|
||||||
|
|
||||||
|
### T2. 新增数据表 / migration
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: <为什么要存这个>
|
||||||
|
scope:
|
||||||
|
- migrations/0000N_<name>.up.sql
|
||||||
|
- migrations/0000N_<name>.down.sql
|
||||||
|
- internal/entity/<name>.go
|
||||||
|
- internal/repo/persistent/postgres/<name>_repo.go
|
||||||
|
- internal/usecase/ports.go（新 Repository 接口）
|
||||||
|
out-of-scope:
|
||||||
|
- 不改已发布 migration（编号≤当前 max）
|
||||||
|
constraints:
|
||||||
|
- G2（金额必须 string / NUMERIC(36,18)）
|
||||||
|
- G4（UpsertMany 必须幂等）
|
||||||
|
- G7（down/up 互逆）
|
||||||
|
validation:
|
||||||
|
- make migrate-up
|
||||||
|
- make migrate-down
|
||||||
|
- make migrate-up # 再次成功
|
||||||
|
- go build ./...
|
||||||
|
acceptance:
|
||||||
|
- 表是 hypertable：psql 跑 SELECT * FROM timescaledb_information.hypertables 看到
|
||||||
|
- upsert 跑两次行数不变
|
||||||
|
- 至少 1 个 repo 单测（连本地 Timescale）
|
||||||
|
```
|
||||||
|
|
||||||
|
### T3. 新增数据源（CoinGlass / ETF / ...）
|
||||||
|
|
||||||
|
**必须先写 `ai/specs/<source>.md`**（见 AGENTS.md §8）。
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: 接入 <source> 提供 <什么数据>
|
||||||
|
spec: ai/specs/<source>.md
|
||||||
|
scope:
|
||||||
|
- internal/repo/webapi/<source>/{client,dto,mapper,...}.go
|
||||||
|
- internal/usecase/ports.go（新 Provider 接口）
|
||||||
|
- internal/app/app.go（DI 装配）
|
||||||
|
- config/config.go + config.yml（新数据源的 base_url / rps / timeout）
|
||||||
|
out-of-scope:
|
||||||
|
- 不改 Binance client
|
||||||
|
constraints:
|
||||||
|
- G1（绝不接私钥）
|
||||||
|
- G2（金额 string）
|
||||||
|
- G5（必须走 pkg/httpclient 限流）
|
||||||
|
validation:
|
||||||
|
- go build ./... && go vet ./...
|
||||||
|
- 临时挂一个 debug 路由验证能拉到真数据
|
||||||
|
acceptance:
|
||||||
|
- 新 Provider 实现接口；usecase 不直接 import <source> 包
|
||||||
|
- 速率预算文档化（在 spec 里）
|
||||||
|
```
|
||||||
|
|
||||||
|
### T4. 改 collector / scheduler
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: <调整哪个采集，为什么>
|
||||||
|
scope:
|
||||||
|
- internal/worker/collector.go
|
||||||
|
- internal/app/scheduler.go
|
||||||
|
out-of-scope:
|
||||||
|
- 不改 binance client（如果要新方法，先开 T3 / T1）
|
||||||
|
- 不改表结构（如果要，开 T2）
|
||||||
|
constraints:
|
||||||
|
- G3（未收线丢弃）
|
||||||
|
- G4（upsert 幂等）
|
||||||
|
- G5（rate limit 不绕开）
|
||||||
|
validation:
|
||||||
|
- go build ./... && go vet ./...
|
||||||
|
- 本地起服务跑 1 个 cron 周期，psql 看新行写入
|
||||||
|
acceptance:
|
||||||
|
- 新 cron 表达式合理（不要 0 0 * * * 这种容易和别人撞的高峰）
|
||||||
|
- 重启服务后 scheduler 能正常注册
|
||||||
|
- weight 预算在 PR 描述里说明（Binance 2400/min IP 上限）
|
||||||
|
```
|
||||||
|
|
||||||
|
### T5. Bug fix
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: 修 <bug 简述>（issue 链接或复现命令）
|
||||||
|
scope: <最小改动范围>
|
||||||
|
constraints: <相关 G 条>
|
||||||
|
validation:
|
||||||
|
- 写一个能复现 bug 的最小测试（先红后绿）
|
||||||
|
- go build ./... && go vet ./...
|
||||||
|
acceptance:
|
||||||
|
- 测试通过
|
||||||
|
- 不引入新的 lint warning
|
||||||
|
```
|
||||||
|
|
||||||
|
### T6. 重构 / Clean Architecture 修正
|
||||||
|
|
||||||
|
```
|
||||||
|
objective: <消除哪个边界违反 / 抽取哪段重复>
|
||||||
|
scope: <文件列表>
|
||||||
|
out-of-scope:
|
||||||
|
- 不改行为（API 响应字节级一致 / DB schema 不变）
|
||||||
|
constraints: G6
|
||||||
|
validation:
|
||||||
|
- go build ./... && go vet ./...
|
||||||
|
- 全部 grep 边界检查（AGENTS.md §6）
|
||||||
|
- make test
|
||||||
|
acceptance:
|
||||||
|
- diff 中不含行为变化（只是搬代码 / 重命名 / 抽接口）
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 何时升级到 spec
|
||||||
|
|
||||||
|
micro contract 写不下、或者需要跨多个 agent / 多次会话才能完成 → 升级到 `ai/specs/<task>.md`：
|
||||||
|
|
||||||
|
- 多 milestone 跨越
|
||||||
|
- 引入新数据源
|
||||||
|
- 改变响应 schema（影响上游 Hermes）
|
||||||
|
- 引入新部署组件（消息队列、新 service）
|
||||||
|
|
||||||
|
spec 内容：objective / non-goals / config 形状 / API 形状 / 错误模式 / 兼容性 / 验收 / 验证计划。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 何时升级到 ADR
|
||||||
|
|
||||||
|
contract 涉及"以后类似场景应该都这么做" → 写 ADR（`ai/adr/000N-<decision>.md`）：
|
||||||
|
|
||||||
|
- 换核心库
|
||||||
|
- 改架构层数 / 改 Clean Arch 边界
|
||||||
|
- 引入新通信协议
|
||||||
|
- 改部署拓扑
|
||||||
|
|
||||||
|
ADR 结构参考 `ai/adr/0001-architecture-foundations.md`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 反模式
|
||||||
|
|
||||||
|
- ❌ "顺便把这里也改一下"——顺便的部分单独开一个 contract，不要混
|
||||||
|
- ❌ "测试很难写，先合了"——测试难写说明设计有问题，先调设计
|
||||||
|
- ❌ "之前 review 过类似的，应该没问题"——每个 PR 独立 validate
|
||||||
|
- ❌ commit message 写"WIP"——本仓库不允许 WIP commit 进 main
|
||||||
33
cmd/app/main.go
Normal file
33
cmd/app/main.go
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"cryptoHermes/config"
|
||||||
|
"cryptoHermes/internal/app"
|
||||||
|
"cryptoHermes/pkg/logger"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
var cfgPath string
|
||||||
|
flag.StringVar(&cfgPath, "config", "config/config.yml", "config file path")
|
||||||
|
flag.Parse()
|
||||||
|
if env := os.Getenv("CONFIG_PATH"); env != "" {
|
||||||
|
cfgPath = env
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg, err := config.Load(cfgPath)
|
||||||
|
if err != nil {
|
||||||
|
slog.Default().Error("config_load_failed", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
log := logger.New(cfg.App.Env)
|
||||||
|
|
||||||
|
if err := app.Run(cfg, log); err != nil {
|
||||||
|
log.Error("app_run_failed", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
}
|
||||||
125
cmd/backfill/main.go
Normal file
125
cmd/backfill/main.go
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"cryptoHermes/config"
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
"cryptoHermes/internal/repo/persistent/postgres"
|
||||||
|
"cryptoHermes/internal/repo/webapi/binance"
|
||||||
|
pkglogger "cryptoHermes/pkg/logger"
|
||||||
|
pgpkg "cryptoHermes/pkg/postgres"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
var (
|
||||||
|
cfgPath string
|
||||||
|
symbol string
|
||||||
|
interval string
|
||||||
|
fromMs int64
|
||||||
|
toMs int64
|
||||||
|
limit int
|
||||||
|
)
|
||||||
|
flag.StringVar(&cfgPath, "config", "config/config.yml", "config file path")
|
||||||
|
flag.StringVar(&symbol, "symbol", "BTCUSDT", "trading pair, e.g. BTCUSDT")
|
||||||
|
flag.StringVar(&interval, "interval", "1h", "kline interval (15m/1h/4h/1d/1w)")
|
||||||
|
flag.Int64Var(&fromMs, "from", 0, "start time in ms (0 = pull most recent N bars)")
|
||||||
|
flag.Int64Var(&toMs, "to", 0, "end time in ms (0 = now)")
|
||||||
|
flag.IntVar(&limit, "limit", 1500, "bars per request, max 1500")
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
if env := os.Getenv("CONFIG_PATH"); env != "" {
|
||||||
|
cfgPath = env
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg, err := config.Load(cfgPath)
|
||||||
|
if err != nil {
|
||||||
|
slog.Default().Error("config_load_failed", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
log := pkglogger.New(cfg.App.Env)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
pool, err := pgpkg.NewPool(ctx, pgpkg.Options{
|
||||||
|
DSN: cfg.Postgres.DSN,
|
||||||
|
MaxConns: cfg.Postgres.MaxConns,
|
||||||
|
MinConns: cfg.Postgres.MinConns,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
log.Error("pool", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer pool.Close()
|
||||||
|
|
||||||
|
client := binance.NewClient(binance.ClientOptions{
|
||||||
|
BaseURL: cfg.Binance.BaseURL,
|
||||||
|
Timeout: cfg.Binance.Timeout,
|
||||||
|
RetryCount: cfg.Binance.RetryCount,
|
||||||
|
RPS: cfg.Binance.RPS,
|
||||||
|
Burst: cfg.Binance.Burst,
|
||||||
|
})
|
||||||
|
repo := postgres.NewKlineRepo(pool)
|
||||||
|
|
||||||
|
if toMs == 0 {
|
||||||
|
toMs = time.Now().UnixMilli()
|
||||||
|
}
|
||||||
|
|
||||||
|
if fromMs == 0 {
|
||||||
|
ks, err := client.GetKlines(ctx, symbol, interval, limit)
|
||||||
|
if err != nil {
|
||||||
|
log.Error("fetch", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
closed := filterClosed(ks)
|
||||||
|
if err := repo.UpsertMany(ctx, closed); err != nil {
|
||||||
|
log.Error("upsert", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
log.Info("backfill_done", "symbol", symbol, "interval", interval, "rows", len(closed))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
cursor := fromMs
|
||||||
|
totalRows := 0
|
||||||
|
for cursor < toMs {
|
||||||
|
ks, err := client.GetKlinesRange(ctx, symbol, interval, cursor, toMs, limit)
|
||||||
|
if err != nil {
|
||||||
|
log.Error("fetch_range", "cursor", cursor, "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
if len(ks) == 0 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
closed := filterClosed(ks)
|
||||||
|
if err := repo.UpsertMany(ctx, closed); err != nil {
|
||||||
|
log.Error("upsert", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
totalRows += len(closed)
|
||||||
|
lastClose := ks[len(ks)-1].CloseTime
|
||||||
|
if lastClose <= cursor {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
cursor = lastClose + 1
|
||||||
|
fmt.Printf("\rbackfill: %s/%s rows=%d cursor=%d ", symbol, interval, totalRows, cursor)
|
||||||
|
}
|
||||||
|
fmt.Println()
|
||||||
|
log.Info("backfill_done", "symbol", symbol, "interval", interval, "rows", totalRows)
|
||||||
|
}
|
||||||
|
|
||||||
|
func filterClosed(in []entity.Kline) []entity.Kline {
|
||||||
|
out := make([]entity.Kline, 0, len(in))
|
||||||
|
for _, k := range in {
|
||||||
|
if k.IsClosed {
|
||||||
|
out = append(out, k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
62
config/config.go
Normal file
62
config/config.go
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/ilyakaznacheev/cleanenv"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Config struct {
|
||||||
|
App App `yaml:"app"`
|
||||||
|
HTTP HTTP `yaml:"http"`
|
||||||
|
Binance Binance `yaml:"binance"`
|
||||||
|
Postgres Postgres `yaml:"postgres"`
|
||||||
|
Collector Collector `yaml:"collector"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type App struct {
|
||||||
|
Name string `yaml:"name" env:"APP_NAME" env-default:"crypto-hermes-gateway"`
|
||||||
|
Env string `yaml:"env" env:"APP_ENV" env-default:"local"`
|
||||||
|
Port int `yaml:"port" env:"APP_PORT" env-default:"8080"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type HTTP struct {
|
||||||
|
ReadTimeout time.Duration `yaml:"read_timeout" env-default:"5s"`
|
||||||
|
WriteTimeout time.Duration `yaml:"write_timeout" env-default:"10s"`
|
||||||
|
IdleTimeout time.Duration `yaml:"idle_timeout" env-default:"60s"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type Binance struct {
|
||||||
|
BaseURL string `yaml:"base_url" env:"BINANCE_BASE_URL" env-default:"https://fapi.binance.com"`
|
||||||
|
Timeout time.Duration `yaml:"timeout" env-default:"10s"`
|
||||||
|
RetryCount int `yaml:"retry_count" env-default:"2"`
|
||||||
|
RPS float64 `yaml:"rps" env-default:"20"`
|
||||||
|
Burst int `yaml:"burst" env-default:"40"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type Postgres struct {
|
||||||
|
DSN string `yaml:"dsn" env:"POSTGRES_DSN" env-required:"true"`
|
||||||
|
MaxConns int32 `yaml:"max_conns" env-default:"10"`
|
||||||
|
MinConns int32 `yaml:"min_conns" env-default:"2"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type Collector struct {
|
||||||
|
Enabled bool `yaml:"enabled" env-default:"true"`
|
||||||
|
Symbols []string `yaml:"symbols" env-default:"BTCUSDT,ETHUSDT"`
|
||||||
|
Intervals []string `yaml:"intervals" env-default:"15m,1h,4h,1d,1w"`
|
||||||
|
DefaultLimit int `yaml:"default_limit" env-default:"500"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func Load(path string) (*Config, error) {
|
||||||
|
var cfg Config
|
||||||
|
if path != "" {
|
||||||
|
if err := cleanenv.ReadConfig(path, &cfg); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if err := cleanenv.ReadEnv(&cfg); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return &cfg, nil
|
||||||
|
}
|
||||||
34
config/config.yml
Normal file
34
config/config.yml
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
app:
|
||||||
|
name: crypto-hermes-gateway
|
||||||
|
env: local
|
||||||
|
port: 8080
|
||||||
|
|
||||||
|
http:
|
||||||
|
read_timeout: 5s
|
||||||
|
write_timeout: 10s
|
||||||
|
idle_timeout: 60s
|
||||||
|
|
||||||
|
binance:
|
||||||
|
base_url: https://fapi.binance.com
|
||||||
|
timeout: 10s
|
||||||
|
retry_count: 2
|
||||||
|
rps: 20
|
||||||
|
burst: 40
|
||||||
|
|
||||||
|
postgres:
|
||||||
|
dsn: postgres://postgres:postgres@localhost:5432/hermes_market?sslmode=disable
|
||||||
|
max_conns: 10
|
||||||
|
min_conns: 2
|
||||||
|
|
||||||
|
collector:
|
||||||
|
enabled: true
|
||||||
|
symbols:
|
||||||
|
- BTCUSDT
|
||||||
|
- ETHUSDT
|
||||||
|
intervals:
|
||||||
|
- 15m
|
||||||
|
- 1h
|
||||||
|
- 4h
|
||||||
|
- 1d
|
||||||
|
- 1w
|
||||||
|
default_limit: 500
|
||||||
28
docker-compose.yml
Normal file
28
docker-compose.yml
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: timescale/timescaledb:latest-pg16
|
||||||
|
container_name: crypto-hermes-postgres
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: postgres
|
||||||
|
POSTGRES_PASSWORD: postgres
|
||||||
|
POSTGRES_DB: hermes_market
|
||||||
|
ports:
|
||||||
|
- "5432:5432"
|
||||||
|
volumes:
|
||||||
|
- hermes_pg_data:/var/lib/postgresql/data
|
||||||
|
|
||||||
|
app:
|
||||||
|
build: .
|
||||||
|
container_name: crypto-hermes-market-gateway
|
||||||
|
depends_on:
|
||||||
|
- postgres
|
||||||
|
ports:
|
||||||
|
- "8080:8080"
|
||||||
|
environment:
|
||||||
|
CONFIG_PATH: /app/config/config.yml
|
||||||
|
POSTGRES_DSN: postgres://postgres:postgres@crypto-hermes-postgres:5432/hermes_market?sslmode=disable
|
||||||
|
volumes:
|
||||||
|
- ./config:/app/config
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
hermes_pg_data:
|
||||||
1330
docs/dev.md
Normal file
1330
docs/dev.md
Normal file
File diff suppressed because it is too large
Load Diff
37
go.mod
Normal file
37
go.mod
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
module cryptoHermes
|
||||||
|
|
||||||
|
go 1.23
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/gofiber/fiber/v2 v2.52.5
|
||||||
|
github.com/ilyakaznacheev/cleanenv v1.5.0
|
||||||
|
github.com/jackc/pgx/v5 v5.7.1
|
||||||
|
github.com/robfig/cron/v3 v3.0.1
|
||||||
|
golang.org/x/sync v0.8.0
|
||||||
|
golang.org/x/time v0.6.0
|
||||||
|
)
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/BurntSushi/toml v1.2.1 // indirect
|
||||||
|
github.com/andybalholm/brotli v1.0.5 // indirect
|
||||||
|
github.com/google/uuid v1.5.0 // indirect
|
||||||
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||||
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||||
|
github.com/joho/godotenv v1.5.1 // indirect
|
||||||
|
github.com/klauspost/compress v1.17.0 // indirect
|
||||||
|
github.com/kr/text v0.2.0 // indirect
|
||||||
|
github.com/mattn/go-colorable v0.1.13 // indirect
|
||||||
|
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||||
|
github.com/mattn/go-runewidth v0.0.15 // indirect
|
||||||
|
github.com/rivo/uniseg v0.2.0 // indirect
|
||||||
|
github.com/rogpeppe/go-internal v1.14.1 // indirect
|
||||||
|
github.com/valyala/bytebufferpool v1.0.0 // indirect
|
||||||
|
github.com/valyala/fasthttp v1.51.0 // indirect
|
||||||
|
github.com/valyala/tcplisten v1.0.0 // indirect
|
||||||
|
golang.org/x/crypto v0.27.0 // indirect
|
||||||
|
golang.org/x/sys v0.26.0 // indirect
|
||||||
|
golang.org/x/text v0.18.0 // indirect
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||||
|
olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 // indirect
|
||||||
|
)
|
||||||
76
go.sum
Normal file
76
go.sum
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
github.com/BurntSushi/toml v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=
|
||||||
|
github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
|
||||||
|
github.com/andybalholm/brotli v1.0.5 h1:8uQZIdzKmjc/iuPu7O2ioW48L81FgatrcpfFmiq/cCs=
|
||||||
|
github.com/andybalholm/brotli v1.0.5/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
|
||||||
|
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||||
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/gofiber/fiber/v2 v2.52.5 h1:tWoP1MJQjGEe4GB5TUGOi7P2E0ZMMRx5ZTG4rT+yGMo=
|
||||||
|
github.com/gofiber/fiber/v2 v2.52.5/go.mod h1:KEOE+cXMhXG0zHc9d8+E38hoX+ZN7bhOtgeF2oT6jrQ=
|
||||||
|
github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU=
|
||||||
|
github.com/google/uuid v1.5.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
github.com/ilyakaznacheev/cleanenv v1.5.0 h1:0VNZXggJE2OYdXE87bfSSwGxeiGt9moSR2lOrsHHvr4=
|
||||||
|
github.com/ilyakaznacheev/cleanenv v1.5.0/go.mod h1:a5aDzaJrLCQZsazHol1w8InnDcOX0OColm64SlIi6gk=
|
||||||
|
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||||
|
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||||
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||||
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||||
|
github.com/jackc/pgx/v5 v5.7.1 h1:x7SYsPBYDkHDksogeSmZZ5xzThcTgRz++I5E+ePFUcs=
|
||||||
|
github.com/jackc/pgx/v5 v5.7.1/go.mod h1:e7O26IywZZ+naJtWWos6i6fvWK+29etgITqrqHLfoZA=
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||||
|
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
|
||||||
|
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
|
||||||
|
github.com/klauspost/compress v1.17.0 h1:Rnbp4K9EjcDuVuHtd0dgA4qNuv9yKDYKK1ulpJwgrqM=
|
||||||
|
github.com/klauspost/compress v1.17.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
|
||||||
|
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
|
||||||
|
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
|
||||||
|
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||||
|
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||||
|
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
|
||||||
|
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
|
||||||
|
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
|
||||||
|
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||||
|
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||||
|
github.com/mattn/go-runewidth v0.0.15 h1:UNAjwbU9l54TA3KzvqLGxwWjHmMgBUVhBiTjelZgg3U=
|
||||||
|
github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
|
||||||
|
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
|
||||||
|
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
|
||||||
|
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
|
||||||
|
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
|
||||||
|
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
|
||||||
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
|
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
|
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
|
||||||
|
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
||||||
|
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
|
||||||
|
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
|
||||||
|
github.com/valyala/fasthttp v1.51.0 h1:8b30A5JlZ6C7AS81RsWjYMQmrZG6feChmgAolCl1SqA=
|
||||||
|
github.com/valyala/fasthttp v1.51.0/go.mod h1:oI2XroL+lI7vdXyYoQk03bXBThfFl2cVdIA3Xl7cH8g=
|
||||||
|
github.com/valyala/tcplisten v1.0.0 h1:rBHj/Xf+E1tRGZyWIWwJDiRY0zc1Js+CV5DqwacVSA8=
|
||||||
|
github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc=
|
||||||
|
golang.org/x/crypto v0.27.0 h1:GXm2NjJrPaiv/h1tb2UH8QfgC/hOf/+z0p6PT8o1w7A=
|
||||||
|
golang.org/x/crypto v0.27.0/go.mod h1:1Xngt8kV6Dvbssa53Ziq6Eqn0HqbZi5Z6R0ZpwQzt70=
|
||||||
|
golang.org/x/sync v0.8.0 h1:3NFvSEYkUoMifnESzZl15y791HH1qU2xm6eCJU5ZPXQ=
|
||||||
|
golang.org/x/sync v0.8.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||||
|
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
|
||||||
|
golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||||
|
golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224=
|
||||||
|
golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
|
||||||
|
golang.org/x/time v0.6.0 h1:eTDhh4ZXt5Qf0augr54TN6suAUudPcawVZeIAPU7D4U=
|
||||||
|
golang.org/x/time v0.6.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
|
||||||
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||||
|
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
|
olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 h1:slmdOY3vp8a7KQbHkL+FLbvbkgMqmXojpFUO/jENuqQ=
|
||||||
|
olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3/go.mod h1:oVgVk4OWVDi43qWBEyGhXgYxt7+ED4iYNpTngSLX2Iw=
|
||||||
121
internal/app/app.go
Normal file
121
internal/app/app.go
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/gofiber/fiber/v2"
|
||||||
|
|
||||||
|
"cryptoHermes/config"
|
||||||
|
"cryptoHermes/internal/controller/restapi"
|
||||||
|
"cryptoHermes/internal/repo/persistent/postgres"
|
||||||
|
"cryptoHermes/internal/repo/webapi/binance"
|
||||||
|
"cryptoHermes/internal/usecase"
|
||||||
|
"cryptoHermes/internal/worker"
|
||||||
|
pgpkg "cryptoHermes/pkg/postgres"
|
||||||
|
)
|
||||||
|
|
||||||
|
func Run(cfg *config.Config, log *slog.Logger) error {
|
||||||
|
rootCtx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
pool, err := pgpkg.NewPool(rootCtx, pgpkg.Options{
|
||||||
|
DSN: cfg.Postgres.DSN,
|
||||||
|
MaxConns: cfg.Postgres.MaxConns,
|
||||||
|
MinConns: cfg.Postgres.MinConns,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("postgres: %w", err)
|
||||||
|
}
|
||||||
|
defer pool.Close()
|
||||||
|
|
||||||
|
binanceClient := binance.NewClient(binance.ClientOptions{
|
||||||
|
BaseURL: cfg.Binance.BaseURL,
|
||||||
|
Timeout: cfg.Binance.Timeout,
|
||||||
|
RetryCount: cfg.Binance.RetryCount,
|
||||||
|
RPS: cfg.Binance.RPS,
|
||||||
|
Burst: cfg.Binance.Burst,
|
||||||
|
})
|
||||||
|
|
||||||
|
klineRepo := postgres.NewKlineRepo(pool)
|
||||||
|
fundingRepo := postgres.NewFundingRepo(pool)
|
||||||
|
oiRepo := postgres.NewOpenInterestRepo(pool)
|
||||||
|
lsRepo := postgres.NewLongShortRatioRepo(pool)
|
||||||
|
takerRepo := postgres.NewTakerVolumeRepo(pool)
|
||||||
|
|
||||||
|
contextUC := usecase.NewMarketContextUsecase(
|
||||||
|
binanceClient,
|
||||||
|
binanceClient,
|
||||||
|
klineRepo,
|
||||||
|
fundingRepo,
|
||||||
|
oiRepo,
|
||||||
|
lsRepo,
|
||||||
|
log,
|
||||||
|
)
|
||||||
|
|
||||||
|
collector := worker.NewCollector(worker.Deps{
|
||||||
|
MarketData: binanceClient,
|
||||||
|
Derivatives: binanceClient,
|
||||||
|
KlineRepo: klineRepo,
|
||||||
|
FundingRepo: fundingRepo,
|
||||||
|
OIRepo: oiRepo,
|
||||||
|
LSRepo: lsRepo,
|
||||||
|
TakerRepo: takerRepo,
|
||||||
|
Symbols: cfg.Collector.Symbols,
|
||||||
|
Intervals: cfg.Collector.Intervals,
|
||||||
|
Limit: cfg.Collector.DefaultLimit,
|
||||||
|
Logger: log,
|
||||||
|
})
|
||||||
|
|
||||||
|
var sched *Scheduler
|
||||||
|
if cfg.Collector.Enabled {
|
||||||
|
sched = NewScheduler(collector, log)
|
||||||
|
sched.Start(rootCtx)
|
||||||
|
defer sched.Stop()
|
||||||
|
}
|
||||||
|
|
||||||
|
fiberApp := fiber.New(fiber.Config{
|
||||||
|
AppName: cfg.App.Name,
|
||||||
|
ReadTimeout: cfg.HTTP.ReadTimeout,
|
||||||
|
WriteTimeout: cfg.HTTP.WriteTimeout,
|
||||||
|
IdleTimeout: cfg.HTTP.IdleTimeout,
|
||||||
|
ErrorHandler: restapi.ErrorHandler,
|
||||||
|
})
|
||||||
|
|
||||||
|
restapi.Register(fiberApp, restapi.Deps{
|
||||||
|
Logger: log,
|
||||||
|
MarketContext: contextUC,
|
||||||
|
MarketData: binanceClient,
|
||||||
|
Derivatives: binanceClient,
|
||||||
|
KlineRepo: klineRepo,
|
||||||
|
FundingRepo: fundingRepo,
|
||||||
|
OIRepo: oiRepo,
|
||||||
|
LSRepo: lsRepo,
|
||||||
|
})
|
||||||
|
|
||||||
|
listenErr := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
addr := fmt.Sprintf(":%d", cfg.App.Port)
|
||||||
|
log.Info("http_listening", "addr", addr)
|
||||||
|
listenErr <- fiberApp.Listen(addr)
|
||||||
|
}()
|
||||||
|
|
||||||
|
stop := make(chan os.Signal, 1)
|
||||||
|
signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
|
||||||
|
|
||||||
|
select {
|
||||||
|
case err := <-listenErr:
|
||||||
|
return err
|
||||||
|
case sig := <-stop:
|
||||||
|
log.Info("shutdown_signal", "signal", sig.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer shutdownCancel()
|
||||||
|
return fiberApp.ShutdownWithContext(shutdownCtx)
|
||||||
|
}
|
||||||
58
internal/app/scheduler.go
Normal file
58
internal/app/scheduler.go
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
|
||||||
|
"github.com/robfig/cron/v3"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/worker"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Scheduler struct {
|
||||||
|
cron *cron.Cron
|
||||||
|
collector *worker.Collector
|
||||||
|
log *slog.Logger
|
||||||
|
rootCtx context.Context
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewScheduler(c *worker.Collector, log *slog.Logger) *Scheduler {
|
||||||
|
return &Scheduler{
|
||||||
|
cron: cron.New(),
|
||||||
|
collector: c,
|
||||||
|
log: log,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Scheduler) Start(ctx context.Context) {
|
||||||
|
s.rootCtx = ctx
|
||||||
|
|
||||||
|
s.add("*/15 * * * *", "klines_15m", func(c context.Context) { _ = s.collector.CollectKlines(c, "15m") })
|
||||||
|
s.add("0 * * * *", "klines_1h", func(c context.Context) { _ = s.collector.CollectKlines(c, "1h") })
|
||||||
|
s.add("0 */4 * * *", "klines_4h", func(c context.Context) { _ = s.collector.CollectKlines(c, "4h") })
|
||||||
|
s.add("5 0 * * *", "klines_1d", func(c context.Context) { _ = s.collector.CollectKlines(c, "1d") })
|
||||||
|
s.add("10 0 * * 1", "klines_1w", func(c context.Context) { _ = s.collector.CollectKlines(c, "1w") })
|
||||||
|
|
||||||
|
s.add("*/15 * * * *", "funding", func(c context.Context) { _ = s.collector.CollectFunding(c) })
|
||||||
|
s.add("*/15 * * * *", "oi", func(c context.Context) { _ = s.collector.CollectOpenInterest(c) })
|
||||||
|
s.add("*/15 * * * *", "ls", func(c context.Context) { _ = s.collector.CollectLongShortRatio(c) })
|
||||||
|
s.add("*/15 * * * *", "taker", func(c context.Context) { _ = s.collector.CollectTakerVolume(c) })
|
||||||
|
|
||||||
|
s.cron.Start()
|
||||||
|
s.log.Info("scheduler_started")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Scheduler) add(spec, name string, fn func(context.Context)) {
|
||||||
|
_, err := s.cron.AddFunc(spec, func() {
|
||||||
|
s.log.Info("cron_tick", "job", name)
|
||||||
|
fn(s.rootCtx)
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
s.log.Error("cron_add_failed", "job", name, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Scheduler) Stop() {
|
||||||
|
s.log.Info("scheduler_stopping")
|
||||||
|
<-s.cron.Stop().Done()
|
||||||
|
}
|
||||||
38
internal/controller/restapi/middleware.go
Normal file
38
internal/controller/restapi/middleware.go
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
package restapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/gofiber/fiber/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
func RequestLogger(log *slog.Logger) fiber.Handler {
|
||||||
|
return func(c *fiber.Ctx) error {
|
||||||
|
start := time.Now()
|
||||||
|
err := c.Next()
|
||||||
|
log.Info("http_request",
|
||||||
|
"method", c.Method(),
|
||||||
|
"path", c.Path(),
|
||||||
|
"status", c.Response().StatusCode(),
|
||||||
|
"duration_ms", time.Since(start).Milliseconds(),
|
||||||
|
"ip", c.IP(),
|
||||||
|
)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func ErrorHandler(c *fiber.Ctx, err error) error {
|
||||||
|
code := fiber.StatusInternalServerError
|
||||||
|
msg := "internal error"
|
||||||
|
if fe, ok := err.(*fiber.Error); ok {
|
||||||
|
code = fe.Code
|
||||||
|
msg = fe.Message
|
||||||
|
}
|
||||||
|
return c.Status(code).JSON(fiber.Map{
|
||||||
|
"error": fiber.Map{
|
||||||
|
"code": code,
|
||||||
|
"message": msg,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
39
internal/controller/restapi/router.go
Normal file
39
internal/controller/restapi/router.go
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
package restapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
|
||||||
|
"github.com/gofiber/fiber/v2"
|
||||||
|
"github.com/gofiber/fiber/v2/middleware/recover"
|
||||||
|
|
||||||
|
v1 "cryptoHermes/internal/controller/restapi/v1"
|
||||||
|
"cryptoHermes/internal/usecase"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Deps struct {
|
||||||
|
Logger *slog.Logger
|
||||||
|
MarketContext *usecase.MarketContextUsecase
|
||||||
|
MarketData usecase.MarketDataProvider
|
||||||
|
Derivatives usecase.DerivativesProvider
|
||||||
|
KlineRepo usecase.KlineRepository
|
||||||
|
FundingRepo usecase.FundingRepository
|
||||||
|
OIRepo usecase.OpenInterestRepository
|
||||||
|
LSRepo usecase.LongShortRatioRepository
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(app *fiber.App, deps Deps) {
|
||||||
|
app.Use(recover.New())
|
||||||
|
app.Use(RequestLogger(deps.Logger))
|
||||||
|
|
||||||
|
api := app.Group("/v1")
|
||||||
|
v1.RegisterHealth(api)
|
||||||
|
v1.RegisterMarket(api, v1.MarketDeps{
|
||||||
|
MarketContext: deps.MarketContext,
|
||||||
|
MarketData: deps.MarketData,
|
||||||
|
Derivatives: deps.Derivatives,
|
||||||
|
KlineRepo: deps.KlineRepo,
|
||||||
|
FundingRepo: deps.FundingRepo,
|
||||||
|
OIRepo: deps.OIRepo,
|
||||||
|
LSRepo: deps.LSRepo,
|
||||||
|
})
|
||||||
|
}
|
||||||
16
internal/controller/restapi/v1/health_routes.go
Normal file
16
internal/controller/restapi/v1/health_routes.go
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
package v1
|
||||||
|
|
||||||
|
import (
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/gofiber/fiber/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
func RegisterHealth(r fiber.Router) {
|
||||||
|
r.Get("/health", func(c *fiber.Ctx) error {
|
||||||
|
return c.JSON(fiber.Map{
|
||||||
|
"status": "ok",
|
||||||
|
"time": time.Now().UnixMilli(),
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
118
internal/controller/restapi/v1/market_routes.go
Normal file
118
internal/controller/restapi/v1/market_routes.go
Normal file
@@ -0,0 +1,118 @@
|
|||||||
|
package v1
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/gofiber/fiber/v2"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
"cryptoHermes/internal/usecase"
|
||||||
|
)
|
||||||
|
|
||||||
|
type MarketDeps struct {
|
||||||
|
MarketContext *usecase.MarketContextUsecase
|
||||||
|
MarketData usecase.MarketDataProvider
|
||||||
|
Derivatives usecase.DerivativesProvider
|
||||||
|
KlineRepo usecase.KlineRepository
|
||||||
|
FundingRepo usecase.FundingRepository
|
||||||
|
OIRepo usecase.OpenInterestRepository
|
||||||
|
LSRepo usecase.LongShortRatioRepository
|
||||||
|
}
|
||||||
|
|
||||||
|
var allowedIntervals = map[string]bool{
|
||||||
|
"15m": true, "1h": true, "4h": true, "1d": true, "1w": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
func RegisterMarket(r fiber.Router, d MarketDeps) {
|
||||||
|
g := r.Group("/market")
|
||||||
|
|
||||||
|
g.Get("/context", func(c *fiber.Ctx) error {
|
||||||
|
symbol := strings.ToUpper(c.Query("symbol"))
|
||||||
|
if symbol == "" {
|
||||||
|
return fiber.NewError(fiber.StatusBadRequest, "symbol is required")
|
||||||
|
}
|
||||||
|
out, err := d.MarketContext.Build(c.UserContext(), symbol)
|
||||||
|
if err != nil && out == nil {
|
||||||
|
return fiber.NewError(fiber.StatusBadRequest, err.Error())
|
||||||
|
}
|
||||||
|
return c.JSON(out)
|
||||||
|
})
|
||||||
|
|
||||||
|
g.Get("/klines", func(c *fiber.Ctx) error {
|
||||||
|
symbol := strings.ToUpper(c.Query("symbol"))
|
||||||
|
interval := c.Query("interval")
|
||||||
|
if symbol == "" || interval == "" {
|
||||||
|
return fiber.NewError(fiber.StatusBadRequest, "symbol and interval are required")
|
||||||
|
}
|
||||||
|
if !allowedIntervals[interval] {
|
||||||
|
return fiber.NewError(fiber.StatusBadRequest, "interval must be one of 15m/1h/4h/1d/1w")
|
||||||
|
}
|
||||||
|
limit := 300
|
||||||
|
if l := c.Query("limit"); l != "" {
|
||||||
|
if v, err := strconv.Atoi(l); err == nil && v > 0 && v <= 1000 {
|
||||||
|
limit = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rows, err := d.KlineRepo.FindRecent(c.UserContext(), symbol, interval, limit)
|
||||||
|
if err != nil {
|
||||||
|
return fiber.NewError(fiber.StatusInternalServerError, err.Error())
|
||||||
|
}
|
||||||
|
return c.JSON(rows)
|
||||||
|
})
|
||||||
|
|
||||||
|
g.Get("/snapshot", func(c *fiber.Ctx) error {
|
||||||
|
symbol := strings.ToUpper(c.Query("symbol"))
|
||||||
|
if symbol == "" {
|
||||||
|
return fiber.NewError(fiber.StatusBadRequest, "symbol is required")
|
||||||
|
}
|
||||||
|
t, err := d.MarketData.GetTicker24h(c.UserContext(), symbol)
|
||||||
|
if err != nil {
|
||||||
|
return fiber.NewError(fiber.StatusBadGateway, err.Error())
|
||||||
|
}
|
||||||
|
return c.JSON(t)
|
||||||
|
})
|
||||||
|
|
||||||
|
g.Get("/derivatives", func(c *fiber.Ctx) error {
|
||||||
|
symbol := strings.ToUpper(c.Query("symbol"))
|
||||||
|
period := c.Query("period", "1h")
|
||||||
|
if symbol == "" {
|
||||||
|
return fiber.NewError(fiber.StatusBadRequest, "symbol is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
fundCur, ferr := d.Derivatives.GetCurrentFunding(c.UserContext(), symbol)
|
||||||
|
oiCur, oerr := d.Derivatives.GetCurrentOpenInterest(c.UserContext(), symbol)
|
||||||
|
|
||||||
|
fundHist, _ := d.FundingRepo.FindRecent(c.UserContext(), symbol, 100)
|
||||||
|
oiHist, _ := d.OIRepo.FindRecent(c.UserContext(), symbol, period, 200)
|
||||||
|
globalLS, _ := d.LSRepo.FindRecent(c.UserContext(), symbol, period, entity.RatioTypeGlobalAccount, 200)
|
||||||
|
topLS, _ := d.LSRepo.FindRecent(c.UserContext(), symbol, period, entity.RatioTypeTopTraderPosition, 200)
|
||||||
|
|
||||||
|
resp := fiber.Map{
|
||||||
|
"symbol": symbol,
|
||||||
|
"funding": fiber.Map{
|
||||||
|
"current": fundCur,
|
||||||
|
"history": fundHist,
|
||||||
|
"error": errString(ferr),
|
||||||
|
},
|
||||||
|
"openInterest": fiber.Map{
|
||||||
|
"current": oiCur,
|
||||||
|
"history": oiHist,
|
||||||
|
"error": errString(oerr),
|
||||||
|
},
|
||||||
|
"longShortRatio": fiber.Map{
|
||||||
|
"global": globalLS,
|
||||||
|
"topTraderPosition": topLS,
|
||||||
|
},
|
||||||
|
"takerBuySellVolume": []any{},
|
||||||
|
}
|
||||||
|
return c.JSON(resp)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func errString(e error) string {
|
||||||
|
if e == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return e.Error()
|
||||||
|
}
|
||||||
9
internal/entity/funding.go
Normal file
9
internal/entity/funding.go
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type FundingRate struct {
|
||||||
|
Source string `json:"source"`
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
FundingTime int64 `json:"fundingTime"`
|
||||||
|
FundingRate string `json:"fundingRate"`
|
||||||
|
MarkPrice string `json:"markPrice,omitempty"`
|
||||||
|
}
|
||||||
19
internal/entity/kline.go
Normal file
19
internal/entity/kline.go
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type Kline struct {
|
||||||
|
Source string `json:"source"`
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
Interval string `json:"interval"`
|
||||||
|
OpenTime int64 `json:"openTime"`
|
||||||
|
CloseTime int64 `json:"closeTime"`
|
||||||
|
Open string `json:"open"`
|
||||||
|
High string `json:"high"`
|
||||||
|
Low string `json:"low"`
|
||||||
|
Close string `json:"close"`
|
||||||
|
Volume string `json:"volume"`
|
||||||
|
QuoteVolume string `json:"quoteVolume"`
|
||||||
|
TradeCount int64 `json:"tradeCount"`
|
||||||
|
TakerBuyBaseVolume string `json:"takerBuyBaseVolume"`
|
||||||
|
TakerBuyQuoteVolume string `json:"takerBuyQuoteVolume"`
|
||||||
|
IsClosed bool `json:"-"`
|
||||||
|
}
|
||||||
18
internal/entity/long_short_ratio.go
Normal file
18
internal/entity/long_short_ratio.go
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
const (
|
||||||
|
RatioTypeGlobalAccount = "global_account"
|
||||||
|
RatioTypeTopTraderPosition = "top_trader_position"
|
||||||
|
RatioTypeTopTraderAccount = "top_trader_account"
|
||||||
|
)
|
||||||
|
|
||||||
|
type LongShortRatio struct {
|
||||||
|
Source string `json:"source"`
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
Period string `json:"period"`
|
||||||
|
RatioType string `json:"ratioType"`
|
||||||
|
Timestamp int64 `json:"timestamp"`
|
||||||
|
LongShortRatio string `json:"longShortRatio"`
|
||||||
|
LongValue string `json:"longValue,omitempty"`
|
||||||
|
ShortValue string `json:"shortValue,omitempty"`
|
||||||
|
}
|
||||||
52
internal/entity/market_context.go
Normal file
52
internal/entity/market_context.go
Normal file
@@ -0,0 +1,52 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type MarketContext struct {
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
GeneratedAt int64 `json:"generatedAt"`
|
||||||
|
Snapshot *Ticker24h `json:"snapshot"`
|
||||||
|
Klines map[string][]Kline `json:"klines"`
|
||||||
|
Derivatives DerivativesBundle `json:"derivatives"`
|
||||||
|
Technical TechnicalStructure `json:"technical"`
|
||||||
|
DataQuality DataQuality `json:"dataQuality"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type DerivativesBundle struct {
|
||||||
|
Funding FundingBundle `json:"funding"`
|
||||||
|
OpenInterest OpenInterestBundle `json:"openInterest"`
|
||||||
|
LongShortRatio LongShortBundle `json:"longShortRatio"`
|
||||||
|
TakerBuySellVolume []TakerBuySellVolume `json:"takerBuySellVolume"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type FundingBundle struct {
|
||||||
|
Current *FundingRate `json:"current"`
|
||||||
|
History []FundingRate `json:"history"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type OpenInterestBundle struct {
|
||||||
|
Current *OpenInterest `json:"current"`
|
||||||
|
History []OpenInterest `json:"history"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type LongShortBundle struct {
|
||||||
|
Global []LongShortRatio `json:"global"`
|
||||||
|
TopTraderPosition []LongShortRatio `json:"topTraderPosition"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type TechnicalStructure struct {
|
||||||
|
Support []TechnicalLevel `json:"support"`
|
||||||
|
Resistance []TechnicalLevel `json:"resistance"`
|
||||||
|
RangeHigh *string `json:"rangeHigh"`
|
||||||
|
RangeLow *string `json:"rangeLow"`
|
||||||
|
LongShortLine *string `json:"longShortLine"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type TechnicalLevel struct {
|
||||||
|
Price string `json:"price"`
|
||||||
|
Strength string `json:"strength,omitempty"`
|
||||||
|
Source string `json:"source,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type DataQuality struct {
|
||||||
|
Source string `json:"source"`
|
||||||
|
Warnings []string `json:"warnings"`
|
||||||
|
}
|
||||||
10
internal/entity/open_interest.go
Normal file
10
internal/entity/open_interest.go
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type OpenInterest struct {
|
||||||
|
Source string `json:"source"`
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
Period string `json:"period"`
|
||||||
|
Timestamp int64 `json:"timestamp"`
|
||||||
|
OpenInterest string `json:"openInterest"`
|
||||||
|
OpenInterestValue string `json:"openInterestValue,omitempty"`
|
||||||
|
}
|
||||||
11
internal/entity/taker_volume.go
Normal file
11
internal/entity/taker_volume.go
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type TakerBuySellVolume struct {
|
||||||
|
Source string `json:"source"`
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
Period string `json:"period"`
|
||||||
|
Timestamp int64 `json:"timestamp"`
|
||||||
|
BuySellRatio string `json:"buySellRatio,omitempty"`
|
||||||
|
BuyVolume string `json:"buyVolume,omitempty"`
|
||||||
|
SellVolume string `json:"sellVolume,omitempty"`
|
||||||
|
}
|
||||||
14
internal/entity/ticker.go
Normal file
14
internal/entity/ticker.go
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type Ticker24h struct {
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
LastPrice string `json:"lastPrice"`
|
||||||
|
PriceChange string `json:"priceChange"`
|
||||||
|
PriceChangePercent string `json:"priceChangePercent"`
|
||||||
|
HighPrice string `json:"highPrice"`
|
||||||
|
LowPrice string `json:"lowPrice"`
|
||||||
|
Volume string `json:"volume"`
|
||||||
|
QuoteVolume string `json:"quoteVolume"`
|
||||||
|
OpenTime int64 `json:"openTime"`
|
||||||
|
CloseTime int64 `json:"closeTime"`
|
||||||
|
}
|
||||||
86
internal/repo/persistent/postgres/funding_repo.go
Normal file
86
internal/repo/persistent/postgres/funding_repo.go
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
type FundingRepo struct {
|
||||||
|
pool *pgxpool.Pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewFundingRepo(pool *pgxpool.Pool) *FundingRepo {
|
||||||
|
return &FundingRepo{pool: pool}
|
||||||
|
}
|
||||||
|
|
||||||
|
const fundingUpsertSQL = `
|
||||||
|
INSERT INTO funding_rates (source, symbol, funding_time, funding_rate, mark_price)
|
||||||
|
VALUES ($1, $2, $3, $4, NULLIF($5, '')::NUMERIC)
|
||||||
|
ON CONFLICT (source, symbol, funding_time) DO UPDATE SET
|
||||||
|
funding_rate = EXCLUDED.funding_rate,
|
||||||
|
mark_price = EXCLUDED.mark_price
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *FundingRepo) UpsertMany(ctx context.Context, items []entity.FundingRate) error {
|
||||||
|
if len(items) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
tx, err := r.pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
for _, f := range items {
|
||||||
|
if f.FundingRate == "" || f.FundingTime == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := tx.Exec(ctx, fundingUpsertSQL,
|
||||||
|
f.Source, f.Symbol, f.FundingTime, f.FundingRate, f.MarkPrice,
|
||||||
|
); err != nil {
|
||||||
|
return fmt.Errorf("upsert funding %s@%d: %w", f.Symbol, f.FundingTime, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return tx.Commit(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
const fundingFindSQL = `
|
||||||
|
SELECT source, symbol, funding_time, funding_rate::text, COALESCE(mark_price::text, '')
|
||||||
|
FROM funding_rates
|
||||||
|
WHERE symbol = $1
|
||||||
|
ORDER BY funding_time DESC
|
||||||
|
LIMIT $2
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *FundingRepo) FindRecent(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
rows, err := r.pool.Query(ctx, fundingFindSQL, symbol, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := make([]entity.FundingRate, 0, limit)
|
||||||
|
for rows.Next() {
|
||||||
|
var f entity.FundingRate
|
||||||
|
var mark sql.NullString
|
||||||
|
if err := rows.Scan(&f.Source, &f.Symbol, &f.FundingTime, &f.FundingRate, &mark); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if mark.Valid {
|
||||||
|
f.MarkPrice = mark.String
|
||||||
|
}
|
||||||
|
out = append(out, f)
|
||||||
|
}
|
||||||
|
for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
|
||||||
|
out[i], out[j] = out[j], out[i]
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
114
internal/repo/persistent/postgres/kline_repo.go
Normal file
114
internal/repo/persistent/postgres/kline_repo.go
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
type KlineRepo struct {
|
||||||
|
pool *pgxpool.Pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewKlineRepo(pool *pgxpool.Pool) *KlineRepo {
|
||||||
|
return &KlineRepo{pool: pool}
|
||||||
|
}
|
||||||
|
|
||||||
|
const klineUpsertSQL = `
|
||||||
|
INSERT INTO market_klines (
|
||||||
|
source, symbol, interval, open_time, close_time,
|
||||||
|
open, high, low, close,
|
||||||
|
volume, quote_volume,
|
||||||
|
trade_count, taker_buy_base_volume, taker_buy_quote_volume,
|
||||||
|
updated_at
|
||||||
|
) VALUES (
|
||||||
|
$1, $2, $3, $4, $5,
|
||||||
|
$6, $7, $8, $9,
|
||||||
|
$10, $11,
|
||||||
|
$12, $13, $14,
|
||||||
|
now()
|
||||||
|
)
|
||||||
|
ON CONFLICT (source, symbol, interval, open_time) DO UPDATE SET
|
||||||
|
close_time = EXCLUDED.close_time,
|
||||||
|
open = EXCLUDED.open,
|
||||||
|
high = EXCLUDED.high,
|
||||||
|
low = EXCLUDED.low,
|
||||||
|
close = EXCLUDED.close,
|
||||||
|
volume = EXCLUDED.volume,
|
||||||
|
quote_volume = EXCLUDED.quote_volume,
|
||||||
|
trade_count = EXCLUDED.trade_count,
|
||||||
|
taker_buy_base_volume = EXCLUDED.taker_buy_base_volume,
|
||||||
|
taker_buy_quote_volume = EXCLUDED.taker_buy_quote_volume,
|
||||||
|
updated_at = now()
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *KlineRepo) UpsertMany(ctx context.Context, items []entity.Kline) error {
|
||||||
|
if len(items) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
tx, err := r.pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("begin: %w", err)
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
for _, k := range items {
|
||||||
|
if !k.IsClosed {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
_, err := tx.Exec(ctx, klineUpsertSQL,
|
||||||
|
k.Source, k.Symbol, k.Interval, k.OpenTime, k.CloseTime,
|
||||||
|
k.Open, k.High, k.Low, k.Close,
|
||||||
|
k.Volume, k.QuoteVolume,
|
||||||
|
k.TradeCount, k.TakerBuyBaseVolume, k.TakerBuyQuoteVolume,
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("upsert kline %s/%s@%d: %w", k.Symbol, k.Interval, k.OpenTime, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return tx.Commit(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
const klineFindSQL = `
|
||||||
|
SELECT source, symbol, interval, open_time, close_time,
|
||||||
|
open::text, high::text, low::text, close::text,
|
||||||
|
volume::text, quote_volume::text,
|
||||||
|
trade_count, taker_buy_base_volume::text, taker_buy_quote_volume::text
|
||||||
|
FROM market_klines
|
||||||
|
WHERE symbol = $1 AND interval = $2
|
||||||
|
ORDER BY open_time DESC
|
||||||
|
LIMIT $3
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *KlineRepo) FindRecent(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 300
|
||||||
|
}
|
||||||
|
rows, err := r.pool.Query(ctx, klineFindSQL, symbol, interval, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("query: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := make([]entity.Kline, 0, limit)
|
||||||
|
for rows.Next() {
|
||||||
|
var k entity.Kline
|
||||||
|
if err := rows.Scan(
|
||||||
|
&k.Source, &k.Symbol, &k.Interval, &k.OpenTime, &k.CloseTime,
|
||||||
|
&k.Open, &k.High, &k.Low, &k.Close,
|
||||||
|
&k.Volume, &k.QuoteVolume,
|
||||||
|
&k.TradeCount, &k.TakerBuyBaseVolume, &k.TakerBuyQuoteVolume,
|
||||||
|
); err != nil {
|
||||||
|
return nil, fmt.Errorf("scan: %w", err)
|
||||||
|
}
|
||||||
|
k.IsClosed = true
|
||||||
|
out = append(out, k)
|
||||||
|
}
|
||||||
|
for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
|
||||||
|
out[i], out[j] = out[j], out[i]
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
94
internal/repo/persistent/postgres/long_short_ratio_repo.go
Normal file
94
internal/repo/persistent/postgres/long_short_ratio_repo.go
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
type LongShortRatioRepo struct {
|
||||||
|
pool *pgxpool.Pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewLongShortRatioRepo(pool *pgxpool.Pool) *LongShortRatioRepo {
|
||||||
|
return &LongShortRatioRepo{pool: pool}
|
||||||
|
}
|
||||||
|
|
||||||
|
const lsUpsertSQL = `
|
||||||
|
INSERT INTO long_short_ratio (
|
||||||
|
source, symbol, period, ratio_type, timestamp,
|
||||||
|
long_short_ratio, long_value, short_value
|
||||||
|
) VALUES (
|
||||||
|
$1, $2, $3, $4, $5,
|
||||||
|
$6,
|
||||||
|
NULLIF($7, '')::NUMERIC,
|
||||||
|
NULLIF($8, '')::NUMERIC
|
||||||
|
)
|
||||||
|
ON CONFLICT (source, symbol, period, ratio_type, timestamp) DO UPDATE SET
|
||||||
|
long_short_ratio = EXCLUDED.long_short_ratio,
|
||||||
|
long_value = EXCLUDED.long_value,
|
||||||
|
short_value = EXCLUDED.short_value
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *LongShortRatioRepo) UpsertMany(ctx context.Context, items []entity.LongShortRatio) error {
|
||||||
|
if len(items) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
tx, err := r.pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
for _, ls := range items {
|
||||||
|
if ls.LongShortRatio == "" || ls.Timestamp == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := tx.Exec(ctx, lsUpsertSQL,
|
||||||
|
ls.Source, ls.Symbol, ls.Period, ls.RatioType, ls.Timestamp,
|
||||||
|
ls.LongShortRatio, ls.LongValue, ls.ShortValue,
|
||||||
|
); err != nil {
|
||||||
|
return fmt.Errorf("upsert ls %s/%s/%s@%d: %w", ls.Symbol, ls.Period, ls.RatioType, ls.Timestamp, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return tx.Commit(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
const lsFindSQL = `
|
||||||
|
SELECT source, symbol, period, ratio_type, timestamp,
|
||||||
|
long_short_ratio::text,
|
||||||
|
COALESCE(long_value::text, ''),
|
||||||
|
COALESCE(short_value::text, '')
|
||||||
|
FROM long_short_ratio
|
||||||
|
WHERE symbol = $1 AND period = $2 AND ratio_type = $3
|
||||||
|
ORDER BY timestamp DESC
|
||||||
|
LIMIT $4
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *LongShortRatioRepo) FindRecent(ctx context.Context, symbol, period, ratioType string, limit int) ([]entity.LongShortRatio, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
rows, err := r.pool.Query(ctx, lsFindSQL, symbol, period, ratioType, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := make([]entity.LongShortRatio, 0, limit)
|
||||||
|
for rows.Next() {
|
||||||
|
var ls entity.LongShortRatio
|
||||||
|
if err := rows.Scan(&ls.Source, &ls.Symbol, &ls.Period, &ls.RatioType, &ls.Timestamp,
|
||||||
|
&ls.LongShortRatio, &ls.LongValue, &ls.ShortValue); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, ls)
|
||||||
|
}
|
||||||
|
for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
|
||||||
|
out[i], out[j] = out[j], out[i]
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
82
internal/repo/persistent/postgres/open_interest_repo.go
Normal file
82
internal/repo/persistent/postgres/open_interest_repo.go
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
type OpenInterestRepo struct {
|
||||||
|
pool *pgxpool.Pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewOpenInterestRepo(pool *pgxpool.Pool) *OpenInterestRepo {
|
||||||
|
return &OpenInterestRepo{pool: pool}
|
||||||
|
}
|
||||||
|
|
||||||
|
const oiUpsertSQL = `
|
||||||
|
INSERT INTO open_interest (source, symbol, period, timestamp, open_interest, open_interest_value)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, NULLIF($6, '')::NUMERIC)
|
||||||
|
ON CONFLICT (source, symbol, period, timestamp) DO UPDATE SET
|
||||||
|
open_interest = EXCLUDED.open_interest,
|
||||||
|
open_interest_value = EXCLUDED.open_interest_value
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *OpenInterestRepo) UpsertMany(ctx context.Context, items []entity.OpenInterest) error {
|
||||||
|
if len(items) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
tx, err := r.pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
for _, o := range items {
|
||||||
|
if o.OpenInterest == "" || o.Timestamp == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := tx.Exec(ctx, oiUpsertSQL,
|
||||||
|
o.Source, o.Symbol, o.Period, o.Timestamp, o.OpenInterest, o.OpenInterestValue,
|
||||||
|
); err != nil {
|
||||||
|
return fmt.Errorf("upsert oi %s/%s@%d: %w", o.Symbol, o.Period, o.Timestamp, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return tx.Commit(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
const oiFindSQL = `
|
||||||
|
SELECT source, symbol, period, timestamp,
|
||||||
|
open_interest::text, COALESCE(open_interest_value::text, '')
|
||||||
|
FROM open_interest
|
||||||
|
WHERE symbol = $1 AND period = $2
|
||||||
|
ORDER BY timestamp DESC
|
||||||
|
LIMIT $3
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *OpenInterestRepo) FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
rows, err := r.pool.Query(ctx, oiFindSQL, symbol, period, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := make([]entity.OpenInterest, 0, limit)
|
||||||
|
for rows.Next() {
|
||||||
|
var o entity.OpenInterest
|
||||||
|
if err := rows.Scan(&o.Source, &o.Symbol, &o.Period, &o.Timestamp, &o.OpenInterest, &o.OpenInterestValue); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, o)
|
||||||
|
}
|
||||||
|
for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
|
||||||
|
out[i], out[j] = out[j], out[i]
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
90
internal/repo/persistent/postgres/taker_volume_repo.go
Normal file
90
internal/repo/persistent/postgres/taker_volume_repo.go
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
type TakerVolumeRepo struct {
|
||||||
|
pool *pgxpool.Pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewTakerVolumeRepo(pool *pgxpool.Pool) *TakerVolumeRepo {
|
||||||
|
return &TakerVolumeRepo{pool: pool}
|
||||||
|
}
|
||||||
|
|
||||||
|
const takerUpsertSQL = `
|
||||||
|
INSERT INTO taker_buy_sell_volume (source, symbol, period, timestamp, buy_sell_ratio, buy_volume, sell_volume)
|
||||||
|
VALUES ($1, $2, $3, $4,
|
||||||
|
NULLIF($5, '')::NUMERIC,
|
||||||
|
NULLIF($6, '')::NUMERIC,
|
||||||
|
NULLIF($7, '')::NUMERIC)
|
||||||
|
ON CONFLICT (source, symbol, period, timestamp) DO UPDATE SET
|
||||||
|
buy_sell_ratio = EXCLUDED.buy_sell_ratio,
|
||||||
|
buy_volume = EXCLUDED.buy_volume,
|
||||||
|
sell_volume = EXCLUDED.sell_volume
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *TakerVolumeRepo) UpsertMany(ctx context.Context, items []entity.TakerBuySellVolume) error {
|
||||||
|
if len(items) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
tx, err := r.pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
for _, t := range items {
|
||||||
|
if t.Timestamp == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := tx.Exec(ctx, takerUpsertSQL,
|
||||||
|
t.Source, t.Symbol, t.Period, t.Timestamp,
|
||||||
|
t.BuySellRatio, t.BuyVolume, t.SellVolume,
|
||||||
|
); err != nil {
|
||||||
|
return fmt.Errorf("upsert taker %s/%s@%d: %w", t.Symbol, t.Period, t.Timestamp, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return tx.Commit(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
const takerFindSQL = `
|
||||||
|
SELECT source, symbol, period, timestamp,
|
||||||
|
COALESCE(buy_sell_ratio::text, ''),
|
||||||
|
COALESCE(buy_volume::text, ''),
|
||||||
|
COALESCE(sell_volume::text, '')
|
||||||
|
FROM taker_buy_sell_volume
|
||||||
|
WHERE symbol = $1 AND period = $2
|
||||||
|
ORDER BY timestamp DESC
|
||||||
|
LIMIT $3
|
||||||
|
`
|
||||||
|
|
||||||
|
func (r *TakerVolumeRepo) FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
rows, err := r.pool.Query(ctx, takerFindSQL, symbol, period, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := make([]entity.TakerBuySellVolume, 0, limit)
|
||||||
|
for rows.Next() {
|
||||||
|
var t entity.TakerBuySellVolume
|
||||||
|
if err := rows.Scan(&t.Source, &t.Symbol, &t.Period, &t.Timestamp,
|
||||||
|
&t.BuySellRatio, &t.BuyVolume, &t.SellVolume); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, t)
|
||||||
|
}
|
||||||
|
for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
|
||||||
|
out[i], out[j] = out[j], out[i]
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
72
internal/repo/webapi/binance/client.go
Normal file
72
internal/repo/webapi/binance/client.go
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
package binance
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"cryptoHermes/pkg/httpclient"
|
||||||
|
)
|
||||||
|
|
||||||
|
const sourceName = "binance"
|
||||||
|
|
||||||
|
type Client struct {
|
||||||
|
http *httpclient.Client
|
||||||
|
source string
|
||||||
|
}
|
||||||
|
|
||||||
|
type ClientOptions struct {
|
||||||
|
BaseURL string
|
||||||
|
Timeout time.Duration
|
||||||
|
RetryCount int
|
||||||
|
RPS float64
|
||||||
|
Burst int
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewClient(opts ClientOptions) *Client {
|
||||||
|
return &Client{
|
||||||
|
http: httpclient.New(httpclient.Options{
|
||||||
|
BaseURL: opts.BaseURL,
|
||||||
|
Timeout: opts.Timeout,
|
||||||
|
RetryCount: opts.RetryCount,
|
||||||
|
RPS: opts.RPS,
|
||||||
|
Burst: opts.Burst,
|
||||||
|
}),
|
||||||
|
source: sourceName,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type ExternalAPIError struct {
|
||||||
|
Source string
|
||||||
|
Path string
|
||||||
|
StatusCode int
|
||||||
|
Message string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *ExternalAPIError) Error() string {
|
||||||
|
return fmt.Sprintf("%s %s status=%d: %s", e.Source, e.Path, e.StatusCode, e.Message)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) get(ctx context.Context, path string, q url.Values, out any) error {
|
||||||
|
status, body, _, err := c.http.Do(ctx, httpclient.Request{
|
||||||
|
Method: http.MethodGet,
|
||||||
|
Path: path,
|
||||||
|
Query: q,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return &ExternalAPIError{Source: c.source, Path: path, StatusCode: 0, Message: err.Error()}
|
||||||
|
}
|
||||||
|
if status >= 400 {
|
||||||
|
return &ExternalAPIError{Source: c.source, Path: path, StatusCode: status, Message: string(body)}
|
||||||
|
}
|
||||||
|
if out == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(body, out); err != nil {
|
||||||
|
return &ExternalAPIError{Source: c.source, Path: path, StatusCode: status, Message: "decode: " + err.Error()}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
116
internal/repo/webapi/binance/derivatives.go
Normal file
116
internal/repo/webapi/binance/derivatives.go
Normal file
@@ -0,0 +1,116 @@
|
|||||||
|
package binance
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/url"
|
||||||
|
"strconv"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (c *Client) GetCurrentFunding(ctx context.Context, symbol string) (*entity.FundingRate, error) {
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("symbol", symbol)
|
||||||
|
var dto premiumIndexDTO
|
||||||
|
if err := c.get(ctx, "/fapi/v1/premiumIndex", q, &dto); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return mapPremiumIndex(&dto), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) GetFundingHistory(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
if limit > 1000 {
|
||||||
|
limit = 1000
|
||||||
|
}
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("symbol", symbol)
|
||||||
|
q.Set("limit", strconv.Itoa(limit))
|
||||||
|
var dto []fundingRateDTO
|
||||||
|
if err := c.get(ctx, "/fapi/v1/fundingRate", q, &dto); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return mapFundingHistory(dto), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) GetCurrentOpenInterest(ctx context.Context, symbol string) (*entity.OpenInterest, error) {
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("symbol", symbol)
|
||||||
|
var dto openInterestDTO
|
||||||
|
if err := c.get(ctx, "/fapi/v1/openInterest", q, &dto); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return mapOpenInterest(&dto, "current"), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) GetOpenInterestHistory(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 30
|
||||||
|
}
|
||||||
|
if limit > 500 {
|
||||||
|
limit = 500
|
||||||
|
}
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("symbol", symbol)
|
||||||
|
q.Set("period", period)
|
||||||
|
q.Set("limit", strconv.Itoa(limit))
|
||||||
|
var dto []openInterestHistDTO
|
||||||
|
if err := c.get(ctx, "/futures/data/openInterestHist", q, &dto); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return mapOpenInterestHistory(dto, period), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) GetGlobalLongShortRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error) {
|
||||||
|
dto, err := c.fetchLongShort(ctx, "/futures/data/globalLongShortAccountRatio", symbol, period, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return mapLongShort(dto, period, entity.RatioTypeGlobalAccount), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) GetTopTraderPositionRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error) {
|
||||||
|
dto, err := c.fetchLongShort(ctx, "/futures/data/topLongShortPositionRatio", symbol, period, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return mapLongShort(dto, period, entity.RatioTypeTopTraderPosition), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) fetchLongShort(ctx context.Context, path, symbol, period string, limit int) ([]longShortRatioDTO, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 30
|
||||||
|
}
|
||||||
|
if limit > 500 {
|
||||||
|
limit = 500
|
||||||
|
}
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("symbol", symbol)
|
||||||
|
q.Set("period", period)
|
||||||
|
q.Set("limit", strconv.Itoa(limit))
|
||||||
|
var dto []longShortRatioDTO
|
||||||
|
if err := c.get(ctx, path, q, &dto); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return dto, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) GetTakerBuySellVolume(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 30
|
||||||
|
}
|
||||||
|
if limit > 500 {
|
||||||
|
limit = 500
|
||||||
|
}
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("symbol", symbol)
|
||||||
|
q.Set("period", period)
|
||||||
|
q.Set("limit", strconv.Itoa(limit))
|
||||||
|
var dto []takerVolumeDTO
|
||||||
|
if err := c.get(ctx, "/futures/data/takerlongshortRatio", q, &dto); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return mapTakerVolume(dto, symbol, period), nil
|
||||||
|
}
|
||||||
60
internal/repo/webapi/binance/dto.go
Normal file
60
internal/repo/webapi/binance/dto.go
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
package binance
|
||||||
|
|
||||||
|
type tickerDTO struct {
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
LastPrice string `json:"lastPrice"`
|
||||||
|
PriceChange string `json:"priceChange"`
|
||||||
|
PriceChangePercent string `json:"priceChangePercent"`
|
||||||
|
HighPrice string `json:"highPrice"`
|
||||||
|
LowPrice string `json:"lowPrice"`
|
||||||
|
Volume string `json:"volume"`
|
||||||
|
QuoteVolume string `json:"quoteVolume"`
|
||||||
|
OpenTime int64 `json:"openTime"`
|
||||||
|
CloseTime int64 `json:"closeTime"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type premiumIndexDTO struct {
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
MarkPrice string `json:"markPrice"`
|
||||||
|
IndexPrice string `json:"indexPrice"`
|
||||||
|
LastFundingRate string `json:"lastFundingRate"`
|
||||||
|
NextFundingTime int64 `json:"nextFundingTime"`
|
||||||
|
Time int64 `json:"time"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type fundingRateDTO struct {
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
FundingTime int64 `json:"fundingTime"`
|
||||||
|
FundingRate string `json:"fundingRate"`
|
||||||
|
MarkPrice string `json:"markPrice"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type openInterestDTO struct {
|
||||||
|
OpenInterest string `json:"openInterest"`
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
Time int64 `json:"time"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type openInterestHistDTO struct {
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
SumOpenInterest string `json:"sumOpenInterest"`
|
||||||
|
SumOpenInterestValue string `json:"sumOpenInterestValue"`
|
||||||
|
Timestamp int64 `json:"timestamp"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type longShortRatioDTO struct {
|
||||||
|
Symbol string `json:"symbol"`
|
||||||
|
LongShortRatio string `json:"longShortRatio"`
|
||||||
|
LongAccount string `json:"longAccount"`
|
||||||
|
ShortAccount string `json:"shortAccount"`
|
||||||
|
LongPosition string `json:"longPosition"`
|
||||||
|
ShortPosition string `json:"shortPosition"`
|
||||||
|
Timestamp int64 `json:"timestamp"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type takerVolumeDTO struct {
|
||||||
|
BuySellRatio string `json:"buySellRatio"`
|
||||||
|
BuyVol string `json:"buyVol"`
|
||||||
|
SellVol string `json:"sellVol"`
|
||||||
|
Timestamp int64 `json:"timestamp"`
|
||||||
|
}
|
||||||
170
internal/repo/webapi/binance/mapper.go
Normal file
170
internal/repo/webapi/binance/mapper.go
Normal file
@@ -0,0 +1,170 @@
|
|||||||
|
package binance
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
func parseKlineRow(symbol, interval string, row []any) (entity.Kline, error) {
|
||||||
|
if len(row) < 12 {
|
||||||
|
return entity.Kline{}, fmt.Errorf("binance kline row has %d fields, expected >=12", len(row))
|
||||||
|
}
|
||||||
|
openTime, err := toInt64(row[0])
|
||||||
|
if err != nil {
|
||||||
|
return entity.Kline{}, fmt.Errorf("openTime: %w", err)
|
||||||
|
}
|
||||||
|
closeTime, err := toInt64(row[6])
|
||||||
|
if err != nil {
|
||||||
|
return entity.Kline{}, fmt.Errorf("closeTime: %w", err)
|
||||||
|
}
|
||||||
|
tradeCount, err := toInt64(row[8])
|
||||||
|
if err != nil {
|
||||||
|
return entity.Kline{}, fmt.Errorf("tradeCount: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
open, _ := row[1].(string)
|
||||||
|
high, _ := row[2].(string)
|
||||||
|
low, _ := row[3].(string)
|
||||||
|
closeP, _ := row[4].(string)
|
||||||
|
volume, _ := row[5].(string)
|
||||||
|
quoteVol, _ := row[7].(string)
|
||||||
|
takerBase, _ := row[9].(string)
|
||||||
|
takerQuote, _ := row[10].(string)
|
||||||
|
|
||||||
|
nowMs := time.Now().UnixMilli()
|
||||||
|
return entity.Kline{
|
||||||
|
Source: sourceName,
|
||||||
|
Symbol: symbol,
|
||||||
|
Interval: interval,
|
||||||
|
OpenTime: openTime,
|
||||||
|
CloseTime: closeTime,
|
||||||
|
Open: open,
|
||||||
|
High: high,
|
||||||
|
Low: low,
|
||||||
|
Close: closeP,
|
||||||
|
Volume: volume,
|
||||||
|
QuoteVolume: quoteVol,
|
||||||
|
TradeCount: tradeCount,
|
||||||
|
TakerBuyBaseVolume: takerBase,
|
||||||
|
TakerBuyQuoteVolume: takerQuote,
|
||||||
|
IsClosed: closeTime < nowMs,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func toInt64(v any) (int64, error) {
|
||||||
|
switch x := v.(type) {
|
||||||
|
case float64:
|
||||||
|
return int64(x), nil
|
||||||
|
case int64:
|
||||||
|
return x, nil
|
||||||
|
case int:
|
||||||
|
return int64(x), nil
|
||||||
|
default:
|
||||||
|
return 0, fmt.Errorf("unexpected type %T", v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapTicker(d *tickerDTO) *entity.Ticker24h {
|
||||||
|
return &entity.Ticker24h{
|
||||||
|
Symbol: d.Symbol,
|
||||||
|
LastPrice: d.LastPrice,
|
||||||
|
PriceChange: d.PriceChange,
|
||||||
|
PriceChangePercent: d.PriceChangePercent,
|
||||||
|
HighPrice: d.HighPrice,
|
||||||
|
LowPrice: d.LowPrice,
|
||||||
|
Volume: d.Volume,
|
||||||
|
QuoteVolume: d.QuoteVolume,
|
||||||
|
OpenTime: d.OpenTime,
|
||||||
|
CloseTime: d.CloseTime,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapPremiumIndex(d *premiumIndexDTO) *entity.FundingRate {
|
||||||
|
return &entity.FundingRate{
|
||||||
|
Source: sourceName,
|
||||||
|
Symbol: d.Symbol,
|
||||||
|
FundingTime: d.NextFundingTime,
|
||||||
|
FundingRate: d.LastFundingRate,
|
||||||
|
MarkPrice: d.MarkPrice,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapFundingHistory(d []fundingRateDTO) []entity.FundingRate {
|
||||||
|
out := make([]entity.FundingRate, 0, len(d))
|
||||||
|
for _, x := range d {
|
||||||
|
out = append(out, entity.FundingRate{
|
||||||
|
Source: sourceName,
|
||||||
|
Symbol: x.Symbol,
|
||||||
|
FundingTime: x.FundingTime,
|
||||||
|
FundingRate: x.FundingRate,
|
||||||
|
MarkPrice: x.MarkPrice,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapOpenInterest(d *openInterestDTO, period string) *entity.OpenInterest {
|
||||||
|
return &entity.OpenInterest{
|
||||||
|
Source: sourceName,
|
||||||
|
Symbol: d.Symbol,
|
||||||
|
Period: period,
|
||||||
|
Timestamp: d.Time,
|
||||||
|
OpenInterest: d.OpenInterest,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapOpenInterestHistory(d []openInterestHistDTO, period string) []entity.OpenInterest {
|
||||||
|
out := make([]entity.OpenInterest, 0, len(d))
|
||||||
|
for _, x := range d {
|
||||||
|
out = append(out, entity.OpenInterest{
|
||||||
|
Source: sourceName,
|
||||||
|
Symbol: x.Symbol,
|
||||||
|
Period: period,
|
||||||
|
Timestamp: x.Timestamp,
|
||||||
|
OpenInterest: x.SumOpenInterest,
|
||||||
|
OpenInterestValue: x.SumOpenInterestValue,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapLongShort(d []longShortRatioDTO, period, ratioType string) []entity.LongShortRatio {
|
||||||
|
out := make([]entity.LongShortRatio, 0, len(d))
|
||||||
|
for _, x := range d {
|
||||||
|
longVal := x.LongAccount
|
||||||
|
shortVal := x.ShortAccount
|
||||||
|
if ratioType == entity.RatioTypeTopTraderPosition {
|
||||||
|
longVal = x.LongPosition
|
||||||
|
shortVal = x.ShortPosition
|
||||||
|
}
|
||||||
|
out = append(out, entity.LongShortRatio{
|
||||||
|
Source: sourceName,
|
||||||
|
Symbol: x.Symbol,
|
||||||
|
Period: period,
|
||||||
|
RatioType: ratioType,
|
||||||
|
Timestamp: x.Timestamp,
|
||||||
|
LongShortRatio: x.LongShortRatio,
|
||||||
|
LongValue: longVal,
|
||||||
|
ShortValue: shortVal,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapTakerVolume(d []takerVolumeDTO, symbol, period string) []entity.TakerBuySellVolume {
|
||||||
|
out := make([]entity.TakerBuySellVolume, 0, len(d))
|
||||||
|
for _, x := range d {
|
||||||
|
out = append(out, entity.TakerBuySellVolume{
|
||||||
|
Source: sourceName,
|
||||||
|
Symbol: symbol,
|
||||||
|
Period: period,
|
||||||
|
Timestamp: x.Timestamp,
|
||||||
|
BuySellRatio: x.BuySellRatio,
|
||||||
|
BuyVolume: x.BuyVol,
|
||||||
|
SellVolume: x.SellVol,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
57
internal/repo/webapi/binance/market.go
Normal file
57
internal/repo/webapi/binance/market.go
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
package binance
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/url"
|
||||||
|
"strconv"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (c *Client) GetKlines(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error) {
|
||||||
|
return c.GetKlinesRange(ctx, symbol, interval, 0, 0, limit)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) GetKlinesRange(ctx context.Context, symbol, interval string, startMs, endMs int64, limit int) ([]entity.Kline, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 500
|
||||||
|
}
|
||||||
|
if limit > 1500 {
|
||||||
|
limit = 1500
|
||||||
|
}
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("symbol", symbol)
|
||||||
|
q.Set("interval", interval)
|
||||||
|
q.Set("limit", strconv.Itoa(limit))
|
||||||
|
if startMs > 0 {
|
||||||
|
q.Set("startTime", strconv.FormatInt(startMs, 10))
|
||||||
|
}
|
||||||
|
if endMs > 0 {
|
||||||
|
q.Set("endTime", strconv.FormatInt(endMs, 10))
|
||||||
|
}
|
||||||
|
|
||||||
|
var raw [][]any
|
||||||
|
if err := c.get(ctx, "/fapi/v1/klines", q, &raw); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
out := make([]entity.Kline, 0, len(raw))
|
||||||
|
for _, row := range raw {
|
||||||
|
k, err := parseKlineRow(symbol, interval, row)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, k)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) GetTicker24h(ctx context.Context, symbol string) (*entity.Ticker24h, error) {
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("symbol", symbol)
|
||||||
|
var dto tickerDTO
|
||||||
|
if err := c.get(ctx, "/fapi/v1/ticker/24hr", q, &dto); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return mapTicker(&dto), nil
|
||||||
|
}
|
||||||
18
internal/usecase/market_collector.go
Normal file
18
internal/usecase/market_collector.go
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
package usecase
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
)
|
||||||
|
|
||||||
|
type MarketCollectorUsecase struct {
|
||||||
|
log *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewMarketCollectorUsecase(log *slog.Logger) *MarketCollectorUsecase {
|
||||||
|
return &MarketCollectorUsecase{log: log}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *MarketCollectorUsecase) Tick(ctx context.Context) error {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
204
internal/usecase/market_context.go
Normal file
204
internal/usecase/market_context.go
Normal file
@@ -0,0 +1,204 @@
|
|||||||
|
package usecase
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/sync/errgroup"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
var supportedSymbols = map[string]bool{
|
||||||
|
"BTCUSDT": true,
|
||||||
|
"ETHUSDT": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
var supportedIntervals = []string{"15m", "1h", "4h", "1d", "1w"}
|
||||||
|
|
||||||
|
const (
|
||||||
|
klineWindowSize = 300
|
||||||
|
klineMinForOK = 200
|
||||||
|
derivativePeriod = "1h"
|
||||||
|
fundingHistoryLen = 100
|
||||||
|
oiHistoryLen = 200
|
||||||
|
longShortLen = 200
|
||||||
|
takerHistoryLen = 200
|
||||||
|
)
|
||||||
|
|
||||||
|
type MarketContextUsecase struct {
|
||||||
|
marketData MarketDataProvider
|
||||||
|
derivatives DerivativesProvider
|
||||||
|
|
||||||
|
klineRepo KlineRepository
|
||||||
|
fundingRepo FundingRepository
|
||||||
|
oiRepo OpenInterestRepository
|
||||||
|
lsRepo LongShortRatioRepository
|
||||||
|
|
||||||
|
log *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewMarketContextUsecase(
|
||||||
|
marketData MarketDataProvider,
|
||||||
|
derivatives DerivativesProvider,
|
||||||
|
klineRepo KlineRepository,
|
||||||
|
fundingRepo FundingRepository,
|
||||||
|
oiRepo OpenInterestRepository,
|
||||||
|
lsRepo LongShortRatioRepository,
|
||||||
|
log *slog.Logger,
|
||||||
|
) *MarketContextUsecase {
|
||||||
|
return &MarketContextUsecase{
|
||||||
|
marketData: marketData,
|
||||||
|
derivatives: derivatives,
|
||||||
|
klineRepo: klineRepo,
|
||||||
|
fundingRepo: fundingRepo,
|
||||||
|
oiRepo: oiRepo,
|
||||||
|
lsRepo: lsRepo,
|
||||||
|
log: log,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *MarketContextUsecase) Build(ctx context.Context, symbol string) (*entity.MarketContext, error) {
|
||||||
|
if !supportedSymbols[symbol] {
|
||||||
|
return nil, fmt.Errorf("unsupported symbol: %s", symbol)
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
snapshot *entity.Ticker24h
|
||||||
|
currentFund *entity.FundingRate
|
||||||
|
currentOI *entity.OpenInterest
|
||||||
|
warnMu sync.Mutex
|
||||||
|
warnings []string
|
||||||
|
)
|
||||||
|
addWarn := func(w string) {
|
||||||
|
warnMu.Lock()
|
||||||
|
warnings = append(warnings, w)
|
||||||
|
warnMu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
g, gctx := errgroup.WithContext(ctx)
|
||||||
|
|
||||||
|
g.Go(func() error {
|
||||||
|
s, err := u.marketData.GetTicker24h(gctx, symbol)
|
||||||
|
if err != nil {
|
||||||
|
addWarn("snapshot fetch failed: " + err.Error())
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
snapshot = s
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
g.Go(func() error {
|
||||||
|
f, err := u.derivatives.GetCurrentFunding(gctx, symbol)
|
||||||
|
if err != nil {
|
||||||
|
addWarn("current funding fetch failed: " + err.Error())
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
currentFund = f
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
g.Go(func() error {
|
||||||
|
oi, err := u.derivatives.GetCurrentOpenInterest(gctx, symbol)
|
||||||
|
if err != nil {
|
||||||
|
addWarn("current OI fetch failed: " + err.Error())
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
currentOI = oi
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
|
||||||
|
_ = g.Wait()
|
||||||
|
|
||||||
|
klines := make(map[string][]entity.Kline, len(supportedIntervals))
|
||||||
|
for _, iv := range supportedIntervals {
|
||||||
|
rows, err := u.klineRepo.FindRecent(ctx, symbol, iv, klineWindowSize)
|
||||||
|
if err != nil {
|
||||||
|
addWarn(fmt.Sprintf("klines DB query failed for %s: %v", iv, err))
|
||||||
|
rows = nil
|
||||||
|
}
|
||||||
|
if len(rows) < klineMinForOK {
|
||||||
|
addWarn(fmt.Sprintf("klines %s only %d in DB, falling back to Binance", iv, len(rows)))
|
||||||
|
fresh, ferr := u.marketData.GetKlines(ctx, symbol, iv, klineWindowSize)
|
||||||
|
if ferr != nil {
|
||||||
|
addWarn(fmt.Sprintf("klines fallback %s failed: %v", iv, ferr))
|
||||||
|
} else {
|
||||||
|
closed := make([]entity.Kline, 0, len(fresh))
|
||||||
|
for _, k := range fresh {
|
||||||
|
if k.IsClosed {
|
||||||
|
closed = append(closed, k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
go func(items []entity.Kline) {
|
||||||
|
bgCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
if err := u.klineRepo.UpsertMany(bgCtx, items); err != nil {
|
||||||
|
u.log.Error("background_kline_upsert_failed", "err", err)
|
||||||
|
}
|
||||||
|
}(closed)
|
||||||
|
rows = closed
|
||||||
|
}
|
||||||
|
}
|
||||||
|
klines[iv] = rows
|
||||||
|
}
|
||||||
|
|
||||||
|
fundingHist, err := u.fundingRepo.FindRecent(ctx, symbol, fundingHistoryLen)
|
||||||
|
if err != nil {
|
||||||
|
addWarn("funding history query failed: " + err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
oiHist, err := u.oiRepo.FindRecent(ctx, symbol, derivativePeriod, oiHistoryLen)
|
||||||
|
if err != nil {
|
||||||
|
addWarn("OI history query failed: " + err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
globalLS, err := u.lsRepo.FindRecent(ctx, symbol, derivativePeriod, entity.RatioTypeGlobalAccount, longShortLen)
|
||||||
|
if err != nil {
|
||||||
|
addWarn("global long/short query failed: " + err.Error())
|
||||||
|
}
|
||||||
|
topLS, err := u.lsRepo.FindRecent(ctx, symbol, derivativePeriod, entity.RatioTypeTopTraderPosition, longShortLen)
|
||||||
|
if err != nil {
|
||||||
|
addWarn("top trader position long/short query failed: " + err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
out := &entity.MarketContext{
|
||||||
|
Symbol: symbol,
|
||||||
|
GeneratedAt: time.Now().UnixMilli(),
|
||||||
|
Snapshot: snapshot,
|
||||||
|
Klines: klines,
|
||||||
|
Derivatives: entity.DerivativesBundle{
|
||||||
|
Funding: entity.FundingBundle{
|
||||||
|
Current: currentFund,
|
||||||
|
History: fundingHist,
|
||||||
|
},
|
||||||
|
OpenInterest: entity.OpenInterestBundle{
|
||||||
|
Current: currentOI,
|
||||||
|
History: oiHist,
|
||||||
|
},
|
||||||
|
LongShortRatio: entity.LongShortBundle{
|
||||||
|
Global: globalLS,
|
||||||
|
TopTraderPosition: topLS,
|
||||||
|
},
|
||||||
|
TakerBuySellVolume: nil,
|
||||||
|
},
|
||||||
|
Technical: entity.TechnicalStructure{
|
||||||
|
Support: []entity.TechnicalLevel{},
|
||||||
|
Resistance: []entity.TechnicalLevel{},
|
||||||
|
},
|
||||||
|
DataQuality: entity.DataQuality{
|
||||||
|
Source: "binance",
|
||||||
|
Warnings: warnings,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
if out.DataQuality.Warnings == nil {
|
||||||
|
out.DataQuality.Warnings = []string{}
|
||||||
|
}
|
||||||
|
|
||||||
|
if snapshot == nil {
|
||||||
|
return out, errors.New("market snapshot unavailable")
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
51
internal/usecase/ports.go
Normal file
51
internal/usecase/ports.go
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
package usecase
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
type MarketDataProvider interface {
|
||||||
|
GetKlines(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error)
|
||||||
|
GetKlinesRange(ctx context.Context, symbol, interval string, startMs, endMs int64, limit int) ([]entity.Kline, error)
|
||||||
|
GetTicker24h(ctx context.Context, symbol string) (*entity.Ticker24h, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type DerivativesProvider interface {
|
||||||
|
GetCurrentFunding(ctx context.Context, symbol string) (*entity.FundingRate, error)
|
||||||
|
GetFundingHistory(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error)
|
||||||
|
|
||||||
|
GetCurrentOpenInterest(ctx context.Context, symbol string) (*entity.OpenInterest, error)
|
||||||
|
GetOpenInterestHistory(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error)
|
||||||
|
|
||||||
|
GetGlobalLongShortRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error)
|
||||||
|
GetTopTraderPositionRatio(ctx context.Context, symbol, period string, limit int) ([]entity.LongShortRatio, error)
|
||||||
|
|
||||||
|
GetTakerBuySellVolume(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type KlineRepository interface {
|
||||||
|
UpsertMany(ctx context.Context, items []entity.Kline) error
|
||||||
|
FindRecent(ctx context.Context, symbol, interval string, limit int) ([]entity.Kline, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type FundingRepository interface {
|
||||||
|
UpsertMany(ctx context.Context, items []entity.FundingRate) error
|
||||||
|
FindRecent(ctx context.Context, symbol string, limit int) ([]entity.FundingRate, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type OpenInterestRepository interface {
|
||||||
|
UpsertMany(ctx context.Context, items []entity.OpenInterest) error
|
||||||
|
FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.OpenInterest, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type LongShortRatioRepository interface {
|
||||||
|
UpsertMany(ctx context.Context, items []entity.LongShortRatio) error
|
||||||
|
FindRecent(ctx context.Context, symbol, period, ratioType string, limit int) ([]entity.LongShortRatio, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type TakerVolumeRepository interface {
|
||||||
|
UpsertMany(ctx context.Context, items []entity.TakerBuySellVolume) error
|
||||||
|
FindRecent(ctx context.Context, symbol, period string, limit int) ([]entity.TakerBuySellVolume, error)
|
||||||
|
}
|
||||||
152
internal/worker/collector.go
Normal file
152
internal/worker/collector.go
Normal file
@@ -0,0 +1,152 @@
|
|||||||
|
package worker
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
|
||||||
|
"cryptoHermes/internal/entity"
|
||||||
|
"cryptoHermes/internal/usecase"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Deps struct {
|
||||||
|
MarketData usecase.MarketDataProvider
|
||||||
|
Derivatives usecase.DerivativesProvider
|
||||||
|
|
||||||
|
KlineRepo usecase.KlineRepository
|
||||||
|
FundingRepo usecase.FundingRepository
|
||||||
|
OIRepo usecase.OpenInterestRepository
|
||||||
|
LSRepo usecase.LongShortRatioRepository
|
||||||
|
TakerRepo usecase.TakerVolumeRepository
|
||||||
|
|
||||||
|
Symbols []string
|
||||||
|
Intervals []string
|
||||||
|
Limit int
|
||||||
|
Logger *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
type Collector struct {
|
||||||
|
d Deps
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewCollector(d Deps) *Collector {
|
||||||
|
if d.Limit <= 0 {
|
||||||
|
d.Limit = 500
|
||||||
|
}
|
||||||
|
return &Collector{d: d}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) Symbols() []string { return c.d.Symbols }
|
||||||
|
func (c *Collector) Intervals() []string { return c.d.Intervals }
|
||||||
|
|
||||||
|
func (c *Collector) CollectKlines(ctx context.Context, interval string) error {
|
||||||
|
for _, sym := range c.d.Symbols {
|
||||||
|
ks, err := c.d.MarketData.GetKlines(ctx, sym, interval, c.d.Limit)
|
||||||
|
if err != nil {
|
||||||
|
c.d.Logger.Error("collect_klines_failed", "symbol", sym, "interval", interval, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
closed := make([]entity.Kline, 0, len(ks))
|
||||||
|
for _, k := range ks {
|
||||||
|
if k.IsClosed {
|
||||||
|
closed = append(closed, k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := c.d.KlineRepo.UpsertMany(ctx, closed); err != nil {
|
||||||
|
c.d.Logger.Error("upsert_klines_failed", "symbol", sym, "interval", interval, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
c.d.Logger.Info("collect_klines_ok", "symbol", sym, "interval", interval, "rows", len(closed))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) CollectAllKlines(ctx context.Context) error {
|
||||||
|
for _, iv := range c.d.Intervals {
|
||||||
|
_ = c.CollectKlines(ctx, iv)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) CollectFunding(ctx context.Context) error {
|
||||||
|
for _, sym := range c.d.Symbols {
|
||||||
|
hist, err := c.d.Derivatives.GetFundingHistory(ctx, sym, 100)
|
||||||
|
if err != nil {
|
||||||
|
c.d.Logger.Error("collect_funding_failed", "symbol", sym, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := c.d.FundingRepo.UpsertMany(ctx, hist); err != nil {
|
||||||
|
c.d.Logger.Error("upsert_funding_failed", "symbol", sym, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
c.d.Logger.Info("collect_funding_ok", "symbol", sym, "rows", len(hist))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) CollectOpenInterest(ctx context.Context) error {
|
||||||
|
periods := []string{"5m", "15m", "1h", "4h", "1d"}
|
||||||
|
for _, sym := range c.d.Symbols {
|
||||||
|
for _, p := range periods {
|
||||||
|
hist, err := c.d.Derivatives.GetOpenInterestHistory(ctx, sym, p, 500)
|
||||||
|
if err != nil {
|
||||||
|
c.d.Logger.Error("collect_oi_failed", "symbol", sym, "period", p, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := c.d.OIRepo.UpsertMany(ctx, hist); err != nil {
|
||||||
|
c.d.Logger.Error("upsert_oi_failed", "symbol", sym, "period", p, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
c.d.Logger.Info("collect_oi_ok", "symbol", sym, "period", p, "rows", len(hist))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) CollectLongShortRatio(ctx context.Context) error {
|
||||||
|
periods := []string{"5m", "15m", "1h", "4h", "1d"}
|
||||||
|
for _, sym := range c.d.Symbols {
|
||||||
|
for _, p := range periods {
|
||||||
|
g, err := c.d.Derivatives.GetGlobalLongShortRatio(ctx, sym, p, 500)
|
||||||
|
if err == nil {
|
||||||
|
if err := c.d.LSRepo.UpsertMany(ctx, g); err != nil {
|
||||||
|
c.d.Logger.Error("upsert_global_ls_failed", "symbol", sym, "period", p, "err", err)
|
||||||
|
} else {
|
||||||
|
c.d.Logger.Info("collect_global_ls_ok", "symbol", sym, "period", p, "rows", len(g))
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
c.d.Logger.Error("collect_global_ls_failed", "symbol", sym, "period", p, "err", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t, err := c.d.Derivatives.GetTopTraderPositionRatio(ctx, sym, p, 500)
|
||||||
|
if err == nil {
|
||||||
|
if err := c.d.LSRepo.UpsertMany(ctx, t); err != nil {
|
||||||
|
c.d.Logger.Error("upsert_top_ls_failed", "symbol", sym, "period", p, "err", err)
|
||||||
|
} else {
|
||||||
|
c.d.Logger.Info("collect_top_ls_ok", "symbol", sym, "period", p, "rows", len(t))
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
c.d.Logger.Error("collect_top_ls_failed", "symbol", sym, "period", p, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Collector) CollectTakerVolume(ctx context.Context) error {
|
||||||
|
periods := []string{"5m", "15m", "1h", "4h", "1d"}
|
||||||
|
for _, sym := range c.d.Symbols {
|
||||||
|
for _, p := range periods {
|
||||||
|
items, err := c.d.Derivatives.GetTakerBuySellVolume(ctx, sym, p, 500)
|
||||||
|
if err != nil {
|
||||||
|
c.d.Logger.Error("collect_taker_failed", "symbol", sym, "period", p, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := c.d.TakerRepo.UpsertMany(ctx, items); err != nil {
|
||||||
|
c.d.Logger.Error("upsert_taker_failed", "symbol", sym, "period", p, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
c.d.Logger.Info("collect_taker_ok", "symbol", sym, "period", p, "rows", len(items))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
1
migrations/000001_market_klines.down.sql
Normal file
1
migrations/000001_market_klines.down.sql
Normal file
@@ -0,0 +1 @@
|
|||||||
|
DROP TABLE IF EXISTS market_klines;
|
||||||
34
migrations/000001_market_klines.up.sql
Normal file
34
migrations/000001_market_klines.up.sql
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS market_klines (
|
||||||
|
source TEXT NOT NULL,
|
||||||
|
symbol TEXT NOT NULL,
|
||||||
|
interval TEXT NOT NULL,
|
||||||
|
|
||||||
|
open_time BIGINT NOT NULL,
|
||||||
|
close_time BIGINT NOT NULL,
|
||||||
|
|
||||||
|
open NUMERIC(36, 18) NOT NULL,
|
||||||
|
high NUMERIC(36, 18) NOT NULL,
|
||||||
|
low NUMERIC(36, 18) NOT NULL,
|
||||||
|
close NUMERIC(36, 18) NOT NULL,
|
||||||
|
|
||||||
|
volume NUMERIC(36, 18) NOT NULL,
|
||||||
|
quote_volume NUMERIC(36, 18) NOT NULL,
|
||||||
|
|
||||||
|
trade_count BIGINT NOT NULL,
|
||||||
|
|
||||||
|
taker_buy_base_volume NUMERIC(36, 18) NOT NULL,
|
||||||
|
taker_buy_quote_volume NUMERIC(36, 18) NOT NULL,
|
||||||
|
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
|
||||||
|
PRIMARY KEY (source, symbol, interval, open_time)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_market_klines_symbol_interval_time
|
||||||
|
ON market_klines(symbol, interval, open_time DESC);
|
||||||
|
|
||||||
|
SELECT create_hypertable('market_klines', 'open_time',
|
||||||
|
chunk_time_interval => 604800000,
|
||||||
|
if_not_exists => TRUE,
|
||||||
|
migrate_data => TRUE);
|
||||||
1
migrations/000002_funding_rates.down.sql
Normal file
1
migrations/000002_funding_rates.down.sql
Normal file
@@ -0,0 +1 @@
|
|||||||
|
DROP TABLE IF EXISTS funding_rates;
|
||||||
20
migrations/000002_funding_rates.up.sql
Normal file
20
migrations/000002_funding_rates.up.sql
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS funding_rates (
|
||||||
|
source TEXT NOT NULL,
|
||||||
|
symbol TEXT NOT NULL,
|
||||||
|
|
||||||
|
funding_time BIGINT NOT NULL,
|
||||||
|
funding_rate NUMERIC(36, 18) NOT NULL,
|
||||||
|
mark_price NUMERIC(36, 18),
|
||||||
|
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
|
||||||
|
PRIMARY KEY (source, symbol, funding_time)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_funding_rates_symbol_time
|
||||||
|
ON funding_rates(symbol, funding_time DESC);
|
||||||
|
|
||||||
|
SELECT create_hypertable('funding_rates', 'funding_time',
|
||||||
|
chunk_time_interval => 2592000000,
|
||||||
|
if_not_exists => TRUE,
|
||||||
|
migrate_data => TRUE);
|
||||||
1
migrations/000003_open_interest.down.sql
Normal file
1
migrations/000003_open_interest.down.sql
Normal file
@@ -0,0 +1 @@
|
|||||||
|
DROP TABLE IF EXISTS open_interest;
|
||||||
21
migrations/000003_open_interest.up.sql
Normal file
21
migrations/000003_open_interest.up.sql
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS open_interest (
|
||||||
|
source TEXT NOT NULL,
|
||||||
|
symbol TEXT NOT NULL,
|
||||||
|
period TEXT NOT NULL,
|
||||||
|
|
||||||
|
timestamp BIGINT NOT NULL,
|
||||||
|
open_interest NUMERIC(36, 18) NOT NULL,
|
||||||
|
open_interest_value NUMERIC(36, 18),
|
||||||
|
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
|
||||||
|
PRIMARY KEY (source, symbol, period, timestamp)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_open_interest_symbol_period_time
|
||||||
|
ON open_interest(symbol, period, timestamp DESC);
|
||||||
|
|
||||||
|
SELECT create_hypertable('open_interest', 'timestamp',
|
||||||
|
chunk_time_interval => 604800000,
|
||||||
|
if_not_exists => TRUE,
|
||||||
|
migrate_data => TRUE);
|
||||||
1
migrations/000004_long_short_ratio.down.sql
Normal file
1
migrations/000004_long_short_ratio.down.sql
Normal file
@@ -0,0 +1 @@
|
|||||||
|
DROP TABLE IF EXISTS long_short_ratio;
|
||||||
24
migrations/000004_long_short_ratio.up.sql
Normal file
24
migrations/000004_long_short_ratio.up.sql
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS long_short_ratio (
|
||||||
|
source TEXT NOT NULL,
|
||||||
|
symbol TEXT NOT NULL,
|
||||||
|
period TEXT NOT NULL,
|
||||||
|
ratio_type TEXT NOT NULL,
|
||||||
|
|
||||||
|
timestamp BIGINT NOT NULL,
|
||||||
|
|
||||||
|
long_short_ratio NUMERIC(36, 18) NOT NULL,
|
||||||
|
long_value NUMERIC(36, 18),
|
||||||
|
short_value NUMERIC(36, 18),
|
||||||
|
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
|
||||||
|
PRIMARY KEY (source, symbol, period, ratio_type, timestamp)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_long_short_ratio_symbol_period_time
|
||||||
|
ON long_short_ratio(symbol, period, ratio_type, timestamp DESC);
|
||||||
|
|
||||||
|
SELECT create_hypertable('long_short_ratio', 'timestamp',
|
||||||
|
chunk_time_interval => 604800000,
|
||||||
|
if_not_exists => TRUE,
|
||||||
|
migrate_data => TRUE);
|
||||||
1
migrations/000005_taker_volume.down.sql
Normal file
1
migrations/000005_taker_volume.down.sql
Normal file
@@ -0,0 +1 @@
|
|||||||
|
DROP TABLE IF EXISTS taker_buy_sell_volume;
|
||||||
23
migrations/000005_taker_volume.up.sql
Normal file
23
migrations/000005_taker_volume.up.sql
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS taker_buy_sell_volume (
|
||||||
|
source TEXT NOT NULL,
|
||||||
|
symbol TEXT NOT NULL,
|
||||||
|
period TEXT NOT NULL,
|
||||||
|
|
||||||
|
timestamp BIGINT NOT NULL,
|
||||||
|
|
||||||
|
buy_sell_ratio NUMERIC(36, 18),
|
||||||
|
buy_volume NUMERIC(36, 18),
|
||||||
|
sell_volume NUMERIC(36, 18),
|
||||||
|
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
|
||||||
|
PRIMARY KEY (source, symbol, period, timestamp)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_taker_buy_sell_symbol_period_time
|
||||||
|
ON taker_buy_sell_volume(symbol, period, timestamp DESC);
|
||||||
|
|
||||||
|
SELECT create_hypertable('taker_buy_sell_volume', 'timestamp',
|
||||||
|
chunk_time_interval => 604800000,
|
||||||
|
if_not_exists => TRUE,
|
||||||
|
migrate_data => TRUE);
|
||||||
120
pkg/httpclient/client.go
Normal file
120
pkg/httpclient/client.go
Normal file
@@ -0,0 +1,120 @@
|
|||||||
|
package httpclient
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/time/rate"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Client struct {
|
||||||
|
httpClient *http.Client
|
||||||
|
baseURL string
|
||||||
|
limiter *rate.Limiter
|
||||||
|
retryCount int
|
||||||
|
}
|
||||||
|
|
||||||
|
type Options struct {
|
||||||
|
BaseURL string
|
||||||
|
Timeout time.Duration
|
||||||
|
RetryCount int
|
||||||
|
RPS float64
|
||||||
|
Burst int
|
||||||
|
}
|
||||||
|
|
||||||
|
func New(opts Options) *Client {
|
||||||
|
if opts.Timeout == 0 {
|
||||||
|
opts.Timeout = 10 * time.Second
|
||||||
|
}
|
||||||
|
if opts.RPS <= 0 {
|
||||||
|
opts.RPS = 20
|
||||||
|
}
|
||||||
|
if opts.Burst <= 0 {
|
||||||
|
opts.Burst = 40
|
||||||
|
}
|
||||||
|
return &Client{
|
||||||
|
httpClient: &http.Client{Timeout: opts.Timeout},
|
||||||
|
baseURL: opts.BaseURL,
|
||||||
|
retryCount: opts.RetryCount,
|
||||||
|
limiter: rate.NewLimiter(rate.Limit(opts.RPS), opts.Burst),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type Request struct {
|
||||||
|
Method string
|
||||||
|
Path string
|
||||||
|
Query url.Values
|
||||||
|
Header http.Header
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Client) Do(ctx context.Context, req Request) (int, []byte, http.Header, error) {
|
||||||
|
if err := c.limiter.Wait(ctx); err != nil {
|
||||||
|
return 0, nil, nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
fullURL := c.baseURL + req.Path
|
||||||
|
if len(req.Query) > 0 {
|
||||||
|
fullURL += "?" + req.Query.Encode()
|
||||||
|
}
|
||||||
|
|
||||||
|
var lastErr error
|
||||||
|
for attempt := 0; attempt <= c.retryCount; attempt++ {
|
||||||
|
if attempt > 0 {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return 0, nil, nil, ctx.Err()
|
||||||
|
case <-time.After(time.Duration(attempt) * 500 * time.Millisecond):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
httpReq, err := http.NewRequestWithContext(ctx, req.Method, fullURL, nil)
|
||||||
|
if err != nil {
|
||||||
|
return 0, nil, nil, err
|
||||||
|
}
|
||||||
|
for k, vs := range req.Header {
|
||||||
|
for _, v := range vs {
|
||||||
|
httpReq.Header.Add(k, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := c.httpClient.Do(httpReq)
|
||||||
|
if err != nil {
|
||||||
|
lastErr = err
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
body, readErr := io.ReadAll(resp.Body)
|
||||||
|
resp.Body.Close()
|
||||||
|
if readErr != nil {
|
||||||
|
lastErr = readErr
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if shouldRetry(resp.StatusCode) && attempt < c.retryCount {
|
||||||
|
lastErr = fmt.Errorf("retryable status %d", resp.StatusCode)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
return resp.StatusCode, body, resp.Header, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if lastErr == nil {
|
||||||
|
lastErr = errors.New("unknown http error")
|
||||||
|
}
|
||||||
|
return 0, nil, nil, lastErr
|
||||||
|
}
|
||||||
|
|
||||||
|
func shouldRetry(status int) bool {
|
||||||
|
switch status {
|
||||||
|
case http.StatusTooManyRequests,
|
||||||
|
http.StatusInternalServerError,
|
||||||
|
http.StatusBadGateway,
|
||||||
|
http.StatusServiceUnavailable,
|
||||||
|
http.StatusGatewayTimeout:
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
17
pkg/logger/logger.go
Normal file
17
pkg/logger/logger.go
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
package logger
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
)
|
||||||
|
|
||||||
|
func New(env string) *slog.Logger {
|
||||||
|
var handler slog.Handler
|
||||||
|
opts := &slog.HandlerOptions{Level: slog.LevelInfo}
|
||||||
|
if env == "local" || env == "dev" {
|
||||||
|
handler = slog.NewTextHandler(os.Stdout, opts)
|
||||||
|
} else {
|
||||||
|
handler = slog.NewJSONHandler(os.Stdout, opts)
|
||||||
|
}
|
||||||
|
return slog.New(handler)
|
||||||
|
}
|
||||||
36
pkg/postgres/postgres.go
Normal file
36
pkg/postgres/postgres.go
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Options struct {
|
||||||
|
DSN string
|
||||||
|
MaxConns int32
|
||||||
|
MinConns int32
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewPool(ctx context.Context, opts Options) (*pgxpool.Pool, error) {
|
||||||
|
cfg, err := pgxpool.ParseConfig(opts.DSN)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("parse dsn: %w", err)
|
||||||
|
}
|
||||||
|
if opts.MaxConns > 0 {
|
||||||
|
cfg.MaxConns = opts.MaxConns
|
||||||
|
}
|
||||||
|
if opts.MinConns > 0 {
|
||||||
|
cfg.MinConns = opts.MinConns
|
||||||
|
}
|
||||||
|
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("pool: %w", err)
|
||||||
|
}
|
||||||
|
if err := pool.Ping(ctx); err != nil {
|
||||||
|
pool.Close()
|
||||||
|
return nil, fmt.Errorf("ping: %w", err)
|
||||||
|
}
|
||||||
|
return pool, nil
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user