First Commit
ci / go (push) Waiting to run
ci / go-db (agent) (push) Waiting to run
ci / go-db (config) (push) Waiting to run
ci / go-db (db) (push) Waiting to run
ci / go-db (evidence) (push) Waiting to run
ci / go-db (llmrec) (push) Waiting to run
ci / go-db (server) (push) Waiting to run
web / web (push) Waiting to run
docs / links (push) Canceled after 0s
detections / detections (push) Canceled after 0s
ci / go (push) Waiting to run
ci / go-db (agent) (push) Waiting to run
ci / go-db (config) (push) Waiting to run
ci / go-db (db) (push) Waiting to run
ci / go-db (evidence) (push) Waiting to run
ci / go-db (llmrec) (push) Waiting to run
ci / go-db (server) (push) Waiting to run
web / web (push) Waiting to run
docs / links (push) Canceled after 0s
detections / detections (push) Canceled after 0s
This commit is contained in:
dela
@@ -0,0 +1,425 @@
|
||||
---
|
||||
name: api-recon
|
||||
description: 收集网站API接口时调用该skill。
|
||||
---
|
||||
|
||||
# API Recon（前端接口侦察）
|
||||
|
||||
在**已授权**前提下，尽可能完整地发现：**后端 API**（路径、方法、参数、响应体）、**前端路由**、**UI 功能触发点**（Tab、弹窗、表格操作等）。
|
||||
|
||||
---
|
||||
|
||||
## 边界与禁止（Agent 必读 · 违反即越界）
|
||||
|
||||
本 skill **仅做 API / 参数面侦察**，不是漏洞挖掘或渗透利用阶段。
|
||||
|
||||
### 任务边界
|
||||
|
||||
| 范围 | 允许 | 禁止 |
|
||||
|---|---|---|
|
||||
| **目标** | 枚举 path、method、参数、路由、UI 触发点 | SQLi/XSS/越权/爆破/fuzz 漏洞、改包攻击、破坏性操作 |
|
||||
| **鉴权** | Hook + stub/mock 绕过**客户端**登录门 | 向用户索要或猜测账号密码；尝试真实登录表单提交 |
|
||||
| **运行时** | 无凭据下 hook 接口，用 mock 响应让 SPA 进入登录后壳层 | 依赖真实后端会话才能继续的流程 |
|
||||
|
||||
### 无凭据动态分析（Phase 3 默认）
|
||||
|
||||
1. 通过 `preload.js` / `runtime_harvest.js` **拦截并 stub** 登录、权限、菜单等 bootstrap 接口；
|
||||
2. 对业务查询接口返回 **结构正确、业务码成功、数据可为空** 的 mock body；
|
||||
3. 使前端在无后端或 401 环境下仍能渲染登录后页面，从而触发更多 XHR/fetch/WebSocket；
|
||||
4. **空数据、空白表格、占位 UI 均属预期**——勿为此转向真实登录或漏洞测试。
|
||||
|
||||
**一句话**：用 mock 撑开前端路由与组件挂载，**只录 outbound 请求**；后端返回什么不重要，重要的是前端**还会发哪些接口**。
|
||||
|
||||
### 流程硬禁止
|
||||
|
||||
| 禁止 | 替代做法 |
|
||||
|---|---|
|
||||
| Phase 1 完成前 grep/curl/Read 主 entry `index-*.js` 提取 API path | 跑 `OUTDIR/harvest_static.py` |
|
||||
| 手写 `extract_apis.py` 等替代 harvest 的脚本 | 改 `OUTDIR/harvest_static.py` 后重跑 |
|
||||
| 同一 grep/命令失败 ≥2 次仍重复 | 换策略：读 tool_logs、改 harvest、查 reference |
|
||||
| 跳过门禁 A/B，直接跑 `scripts/` 原版 | 复制到 OUTDIR 并按目标改 |
|
||||
| 真实用户名/密码、OTP、OAuth 等鉴权 | stub/mock（见上文） |
|
||||
| 以「拿真实数据」为由跳过 stub，做越权/注入测试 | 只录 outbound，属 recon 边界 |
|
||||
| 删除、导出敏感数据、批量写等不可逆操作 | coverage 点击亦同 |
|
||||
| 未完成 runtime + 动态枚举，声称已获全部页面和接口 | 见「完成定义」或标注局限 |
|
||||
| 未完成参数触发矩阵 + diff，声称已掌握全部参数 | Phase 3b 矩阵 + Phase 5 diff |
|
||||
| 用单一 runtime 样本推断必填/可选 | 多样本 diff 或校验规则/错误反推 |
|
||||
|
||||
---
|
||||
|
||||
## 两层模型 + 运行模式
|
||||
|
||||
| 层 | 产出 | 上限 |
|
||||
|---|---|---|
|
||||
| **静态**（JS bundle） | 全量 endpoint 路径、路由草案、组包点字段候选 | 无 HTTP 方法；参数须 Phase 1b；漏掉运行时拼接 URL |
|
||||
| **运行时**（活会话） | 方法 + body + 响应 + 动态 URL + WS/SSE；多样本 diff 补全参数 | 页面须实际渲染才会发请求；单样本不足以定必填/可选 |
|
||||
|
||||
| 运行模式 | 引擎 | 适用 |
|
||||
|---|---|---|
|
||||
| **depth** | `runtime_harvest.js`（Puppeteer） | API 清单、METHOD/params/响应体、WS/SSE、可复现批量跑 |
|
||||
| **coverage** | browser + `preload.js` | 点 Tab/弹窗/表格，功能点覆盖更深 |
|
||||
| **both** | 先 depth 再 coverage | 最完整，耗时最长 |
|
||||
|
||||
**参数方法论**（无通用脚本）：path 用 harvest/正则；参数用 **锚点扩窗 + UI 绑定链 + 多样本 diff + 错误反推**（grep 配方见 [reference.md](reference.md) J 节）。
|
||||
|
||||
---
|
||||
|
||||
## 完成定义
|
||||
|
||||
全部满足方可声称 recon 完成：
|
||||
|
||||
- [ ] **静态**：Phase 1 harvest 产出 `api_static.txt`、`routes.txt`、`js/`
|
||||
- [ ] **运行时**：至少 depth 或 coverage 之一；coverage/both 须 **Hook 生效 + 动态枚举环**
|
||||
- [ ] **进壳**：访问业务 path 时非 `/login`（注意 hash 路由）
|
||||
- [ ] **参数**：coverage/both 完成参数触发矩阵 + `param_samples.json`；Phase 5 合并 `params_merged.json`
|
||||
- [ ] **深度**（若模块页空白）：Phase 4 权限树还原并重跑，直至出现 **module 级 API**（非仅 locale/bootstrap）
|
||||
- [ ] **交付**：Phase 5 产出齐全（见 Phase 5 产出表）；`insert_assets` 写入服务与端点资产
|
||||
|
||||
---
|
||||
|
||||
## 脚本与门禁
|
||||
|
||||
`scripts/` 仅为参考模板，**禁止**直接跑原版并当最终结果。
|
||||
|
||||
**规则**：先读 → 按目标改 → 写入 `OUTDIR`（如 `recon/`）→ 记 `CHANGES.md`；不匹配则按方法论重写，只借结构。
|
||||
|
||||
| 门禁 | 何时 | 参考脚本 → OUTDIR 副本 | 常见必改项 |
|
||||
|---|---|---|---|
|
||||
| **A（静态）** | Phase 0 后、**第一次**跑 harvest/spider 前 | `harvest_static.py` / `spider_mpa.py` | **多数站点默认 regex 可直接跑**；仅 manifest/方言不匹配时改 endpoint 正则、webpack/Vite `publicPath`、MPA exclude/cookie |
|
||||
| **B（运行时）** | Phase 2 后、跑 depth/coverage 前 | `runtime_harvest.js` / `preload.js` + `config.json` | Cookie/localStorage 键、neutralize 成功值、stubs、login 正则、api 前缀、hash/history |
|
||||
|
||||
**SPA 强制顺序**（不可交换；Phase 编号优先于「先探索再脚本」）：
|
||||
|
||||
| 步骤 | 必须 | 禁止 |
|
||||
|---|---|---|
|
||||
| Phase 0 完成后 | 下一条 Bash = `python3 OUTDIR/harvest_static.py <URL> OUTDIR` | curl/grep/Read 主 entry `index-*.js`（通常 >500KB） |
|
||||
| 门禁 A | 复制脚本 → 按需小改 → **立刻运行** | 先手工提取 API 再决定是否 harvest |
|
||||
| Phase 1 完成前 | `wc -l` 校验产出；404 改 harvest 重试 | 手写 extract 脚本；对未下载 URL 反复 grep |
|
||||
| Phase 1b 起 | grep 仅 `OUTDIR/js/*.js` | 用主 bundle 代替 harvest |
|
||||
|
||||
- ✅ 复制 `harvest_static.py` → （可选）改 regex → **立即运行**
|
||||
- ❌ curl 主 bundle → grep 多次 → 写临时 extract → 最后才 harvest
|
||||
- **MPA**：Phase 0 后下一条 Bash = `python3 OUTDIR/spider_mpa.py ...`
|
||||
|
||||
---
|
||||
|
||||
## 工具与输出约束
|
||||
|
||||
| 约束 | 说明 |
|
||||
|---|---|
|
||||
| 大文件 | >100KB 的 `index-*.js` **禁止** Read/grep 进上下文；用 OUTDIR 脚本批处理 |
|
||||
| grep 输出 | 必须 `\| head -20` 或 `-m 5`；对话只保留 path 摘要，勿贴 bundle 片段 |
|
||||
| 校验 | 用 `wc -l`、`ls \| wc -l`；勿 Read 整目录 |
|
||||
| regex 初探 | 可选、≤1 次、仅 ≤50KB 小 chunk 或 HTML；正式静态以 harvest 为准 |
|
||||
| reference | 配方/模板/排障见 [reference.md](reference.md)，勿重复 inline 全文 |
|
||||
|
||||
---
|
||||
|
||||
## 执行路线图
|
||||
|
||||
```
|
||||
Phase 0 分类 + OUTDIR
|
||||
→ 门禁 A → Phase 1 harvest（★ 立刻运行 ★）
|
||||
→ Phase 1b 参数逆向
|
||||
→ Phase 2 鉴权三道门 → config.json
|
||||
→ 门禁 B → Phase 3 运行时 + 参数矩阵
|
||||
→ Phase 4 权限树（必要时）→ 重跑 Phase 3
|
||||
→ Phase 5 合并报告 + insert_assets批量插入所有发现的服务、端点api资产，无论如何插入时不允许漏掉已发现的资产
|
||||
```
|
||||
|
||||
按序勾选；**前一项未完成不得进入下一 Phase**。
|
||||
|
||||
1. [ ] **Phase 0**：初探 SPA/MPA；创建 `OUTDIR` → [Phase 0](#phase-0--分类)
|
||||
2. [ ] **门禁 A + Phase 1**：复制脚本 → **立刻** harvest → `wc -l` 校验 → [Phase 1](#phase-1--静态)
|
||||
3. [ ] **Phase 1b**：锚点扩窗 + 绑定层 → `param_candidates.json` → [Phase 1b](#phase-1b--参数逆向)
|
||||
4. [ ] **Phase 2**：鉴权三道门 → `config.json` → [Phase 2](#phase-2--鉴权三道门)
|
||||
5. [ ] **门禁 B**：调整 runtime 脚本 → [Phase 3](#phase-3--运行时)
|
||||
6. [ ] **Phase 3**：depth / coverage / both；确认进壳；参数触发矩阵 → `param_samples.json`
|
||||
7. [ ] **Phase 4**（若需要）：权限树 → patch stubs → 重跑 Phase 3 → [Phase 4](#phase-4--权限树还原)
|
||||
8. [ ] **Phase 5**：合并产出 + 报告 + `insert_assets` → [Phase 5](#phase-5--合并与报告)
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — 分类
|
||||
|
||||
拉取入口 HTML，**创建 `OUTDIR`**（勿改 skill 内 `scripts/`）：
|
||||
|
||||
- **SPA**：空壳 + `<div id=app>` + chunk → Phase 1–5
|
||||
- **MPA**：SSR + `<form>`、无 endpoint bundle → 门禁 A 后：
|
||||
|
||||
```bash
|
||||
python3 recon/spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "session=..."] [--max 300] [--depth 5] [--exclude "logout|delete|destroy"]
|
||||
```
|
||||
|
||||
产出 `forms.txt`、`links.txt`、`api_inline.txt`。SPA 若 forms ≈ 0 → 切 Phase 1。
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — 静态
|
||||
|
||||
遵守 [脚本与门禁](#脚本与门禁) · [工具与输出约束](#工具与输出约束)。
|
||||
|
||||
```bash
|
||||
python3 recon/harvest_static.py <BASE_URL> <OUTDIR>
|
||||
```
|
||||
|
||||
harvest：解析 HTML script → webpack/Vite manifest → 下载全部 lazy chunk → 产出 `js/`、`api_static.txt`、`routes.txt`、`chunkmap.txt`。
|
||||
|
||||
```bash
|
||||
wc -l OUTDIR/api_static.txt OUTDIR/routes.txt
|
||||
ls OUTDIR/js | wc -l
|
||||
```
|
||||
|
||||
- chunk 数 vs manifest：404 须改 harvest 重试，勿手工 curl 逐个 chunk
|
||||
- `api_static.txt` 过少 → 放宽 OUTDIR 内 endpoint 正则后重跑（见 reference）
|
||||
|
||||
### Phase 1b — 参数逆向
|
||||
|
||||
path 来自 Phase 1；参数字段须单独 recon。grep 规则见 [工具与输出约束](#工具与输出约束)。
|
||||
|
||||
**完成标准**：重要接口能答——字段名、传输位置、类型推断、是否必填、样本值、置信度。
|
||||
|
||||
#### 1b.0 — 传输形态
|
||||
|
||||
| 形态 | 参数在哪 | 静态优先看 |
|
||||
|---|---|---|
|
||||
| REST JSON | body + query | path 锚点旁 `(params\|data\|body)\s*:\s*\{` |
|
||||
| GraphQL | `variables` | gql 模板、`$page: Int` |
|
||||
| 传统 form | urlencoded | `<form>`、`FormData` |
|
||||
| 文件上传 | multipart | `FormData.append` |
|
||||
| 路径参数 | `/user/:id` | 路由表 + `useParams` / `$route.params` |
|
||||
| 加密/签名 | 包进 `sign`/`data` | Hook 加密函数入参（reference D 节） |
|
||||
|
||||
产出：每接口标注 `transport: query|json|form|graphql|encrypted`。
|
||||
|
||||
#### 1b.1 — 锚点扩窗
|
||||
|
||||
以已知 path 为锚，扩窗口找组包对象：
|
||||
|
||||
```bash
|
||||
grep -n '"/api/user/list"' OUTDIR/js/*.js | head -20
|
||||
grep -rhoaE '.{0,120}("/api[^"]+").{0,200}' OUTDIR/js/*.js | head -20
|
||||
grep -rhoaE '(params|data|body|payload)\s*:\s*\{' OUTDIR/js/*.js | head -20
|
||||
```
|
||||
|
||||
| 包装层 | 参数线索 |
|
||||
|---|---|
|
||||
| axios 实例 | `data` / `params` |
|
||||
| 统一 request | 拦截器注入全局字段 |
|
||||
| OpenAPI 客户端 | 生成 method 签名 |
|
||||
| React Query / SWR | hook 第二参数 |
|
||||
| Vue composable | composable 入参 |
|
||||
|
||||
类型残留：`yup`/`zod`/rules、`Form.Item name=`、内嵌 Swagger。
|
||||
|
||||
→ `param_candidates.json`：`{ path, fields[], source: "static-callsite", confidence }`
|
||||
|
||||
#### 1b.2 — 绑定层
|
||||
|
||||
```
|
||||
Form field → onFinish/handleSubmit → transform → API payload
|
||||
```
|
||||
|
||||
| 绑定源 | 手法 |
|
||||
|---|---|
|
||||
| 表单 submit | 跟 submit → transform → API |
|
||||
| 表格搜索 | `getFieldsValue()` → `params` |
|
||||
| 路由 | `:id` / `?tab=` |
|
||||
| 拦截器 | 全局 `tenantId`、分页、sign |
|
||||
| 枚举 select | `options` → API 枚举值 |
|
||||
|
||||
DevTools call stack 从 `fetch`/`XHR.send` 往上追组包函数。
|
||||
|
||||
#### 1b.3 — 组包三问（≠ Phase 2 鉴权三门）
|
||||
|
||||
| 问 | 要答什么 |
|
||||
|---|---|
|
||||
| **组装** | payload 在哪 build、transform 痕迹 |
|
||||
| **校验** | required、pattern、enum |
|
||||
| **传输** | path / query / body / multipart / 头 |
|
||||
|
||||
拦截器门（Phase 2）顺带读全局注入字段（Authorization、`X-Tenant-Id`、sign）。
|
||||
|
||||
#### 1b.4 — 与 Phase 3 衔接
|
||||
|
||||
候选字段来自静态/绑定层；**必填/可选/条件依赖**须 Phase 3 参数矩阵 + diff + Phase 5 错误反推。
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — 鉴权三道门
|
||||
|
||||
在 `OUTDIR/js/` grep（带 `head`），写入 `config.json`（配方见 reference）：
|
||||
|
||||
| 门 | 问题 | 关键词 |
|
||||
|---|---|---|
|
||||
| **渲染门** | 如何判断已登录？ | `isLogin`、`getToken`、Cookie/localStorage |
|
||||
| **拦截器门** | 什么触发跳 `/login`？ | `response_code`、`errno`、axios interceptor |
|
||||
| **内容门** | 菜单/权限从哪来？ | `menu`、`permission`、`role`、`acl`、`routes` |
|
||||
|
||||
禁止把 localStorage 键名当凭据——须从 chunk/请求链确认。
|
||||
|
||||
**出口 = 门禁 B**：结论落到 `config.json`，并改 `OUTDIR/runtime_harvest.js` / `preload.js`。
|
||||
|
||||
### Phase 2b — API 观察（可选）
|
||||
|
||||
用 OUTDIR 内 `preload.js` 确认会话键名、Authorization、嵌套 API URL：
|
||||
|
||||
| 配置 | 产出 |
|
||||
|---|---|
|
||||
| `recordDetail: true` | `__API_RECON_DETAIL__` |
|
||||
| `observe.xhrHeaders: true` | headers 观察 |
|
||||
| `extractUrlsFromResponse: true` | 响应内子 API |
|
||||
| `observe.storageReads/cookieReads: true` | 回填 config |
|
||||
| `neutralizeVueRouter: true` | `__API_RECON_ROUTES__` |
|
||||
|
||||
coverage 每轮导出：`__API_RECON_LOG__`、`__API_RECON_DETAIL__`、`__API_RECON_ROUTES__`、`__API_RECON_OBSERVE__`。
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — 运行时
|
||||
|
||||
须已过门禁 B；遵守 [边界与禁止](#边界与禁止agent-必读--违反即越界) · 无凭据 mock 策略。
|
||||
|
||||
`config.json` 设置 `"runtimeMode": "depth" | "coverage" | "both"`（模板见 reference）。
|
||||
|
||||
### Hook 与 stub（depth + coverage 共用）
|
||||
|
||||
| 层 | 范围 | 目的 |
|
||||
|---|---|---|
|
||||
| L1 精确 | auth/权限/bootstrap stub | 过首屏鉴权 |
|
||||
| L2 负向修正 | 所有 JSON 响应 | 未登录码 → 成功 |
|
||||
| L3 兜底 | 未命中 L1 的 `/api` 等 | 空成功体，撑开 UI |
|
||||
|
||||
- **depth**：fake auth + `forward` 改业务码 + `stubs`；遍历 `routes`（hash/history）；产出 `runtime_api.json`
|
||||
- **coverage**：**document-start** 注入 `preload.js`（CDP `addScriptToEvaluateOnNewDocument` 或 Userscript）
|
||||
|
||||
验证：`window.__API_RECON_PRELOAD__` 存在；业务 path 不回 `/login`。
|
||||
|
||||
```bash
|
||||
cd recon && npm install
|
||||
node runtime_harvest.js config.json
|
||||
```
|
||||
|
||||
### 3b — coverage 动态枚举（必做）
|
||||
|
||||
1. 主导航/侧栏 — 每项点击，等网络 1–3s
|
||||
2. Tab — `role=tab`、`.ant-tabs-tab`
|
||||
3. 表格 — 首行查看/编辑/详情
|
||||
4. 工具栏 — 导出、筛选、新建（**避免不可逆删除**）
|
||||
5. 每进模块 — 合并 API/路由
|
||||
6. SPA — 对 `routes.txt` 未覆盖 path 受控 `pushState`（MPA 禁止）
|
||||
|
||||
**参数触发矩阵**（必做）：每模块按操作类型各录一次，**diff 多样本**：
|
||||
|
||||
| 操作 | 通常多出的参数 |
|
||||
|---|---|
|
||||
| 列表首屏 | 分页 + 默认筛选 |
|
||||
| 点搜索 | keyword、filter |
|
||||
| 高级筛选 | 更多 optional |
|
||||
| 新建/编辑 | 完整 entity |
|
||||
| 批量/导出/排序 | `ids[]`、`exportType`、`sortField` |
|
||||
|
||||
**stub 下 outbound body/headers 仍真实**——以请求为准。录制 → `scan_raw.json`、`param_samples.json`、`api_detail.json`。
|
||||
|
||||
- **Vue**：`neutralizeVueRouter: true` + document-start preload
|
||||
- **React**：`routes.txt` + 侧栏点击 + `pushState`
|
||||
- **both**：先 3a depth，再 3b coverage
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — 权限树还原
|
||||
|
||||
**触发**：模块页空白 / 每路由仅 bootstrap（如 locale）→ 内容门未过。
|
||||
|
||||
| 现象 | 含义 |
|
||||
|---|---|
|
||||
| 进壳成功 | 渲染门 + 拦截器门已过 |
|
||||
| 侧栏缺项/点击空白 | stub shape 或权限码不全 |
|
||||
| 每路由 API 相同且极少 | `v-if permission` 未通过 |
|
||||
| `routes.txt` 远少于 bundle | 须从 auth 模块补全 |
|
||||
|
||||
```bash
|
||||
grep -rhoaE '"/api[^"]*(permission|perm|role|menu|acl)[^"]*"' OUTDIR/js/*.js | sort -u | head -30
|
||||
grep -rhoaE 'userRouteAuth|getResultTree|routeMap|routeLink|menuList|authList' OUTDIR/js/*.js | head -20
|
||||
```
|
||||
|
||||
典型链：`role_permissions`（flat codes）+ `permissions/all`（tree）→ `getResultTree` → `userRouteAuth[CODE].url`。
|
||||
|
||||
```bash
|
||||
python3 recon/extract_route_map.py recon/js recon/
|
||||
python3 recon/build_perm_tree.py recon/js recon/ --config recon/config.json
|
||||
```
|
||||
|
||||
中间产出：`route_map.json`、`userRouteAuth.json`、`permissions_tree.json`、`*_stub.json`、`perm_codes_all.txt`。
|
||||
|
||||
stub 检查：外层 `response_code` 与拦截器门一致；flat codes 与 tree 对齐；`routes` 覆盖 `route_map` 全部 link。
|
||||
|
||||
更新 `config.json` 后**重跑 Phase 3**。大型 SPA 可调 `waitUntil`、`routeTimeout`、`perRouteMs`（见 reference A3/I 节）。
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — 合并与报告
|
||||
|
||||
### 产出表
|
||||
|
||||
| 文件 | 阶段 | 内容 |
|
||||
|---|---|---|
|
||||
| `js/`、`api_static.txt`、`routes.txt`、`chunkmap.txt` | 1 | 静态 bundle 与 path |
|
||||
| `param_candidates.json` | 1b | 静态参数字段候选 |
|
||||
| `config.json` | 2 | 三道门 + runtime 配置 |
|
||||
| `runtime_api.json` | 3a | depth 详细录制（含 WS/SSE） |
|
||||
| `param_samples.json`、`scan_raw.json`、`api_detail.json` | 3b | 多样本、点击日志、detail |
|
||||
| `route_map.json` 等 | 4 | 权限树中间文件（若执行） |
|
||||
| `params_merged.json` | 5 | 合并参数字段 + 置信度 |
|
||||
| `api_merged.txt` | 5 | `METHOD /path [params] [static\|runtime\|both]` |
|
||||
| `site_map.json` | 5 | 路由、API、params、功能点、局限 |
|
||||
| **insert_assets** | 5 | 将所有服务、端点资产写入资产库 |
|
||||
|
||||
### 5b — 参数合并
|
||||
|
||||
从 `param_samples.json` diff，**无通用合并脚本**。置信度规则见 reference J7（高/中/低/待触发）。
|
||||
|
||||
### 5c — 错误反推
|
||||
|
||||
授权范围内可发不完整请求读 400（**属参数 recon，非漏洞测试**）：`field 'x' is required`、枚举错误等。注意 `data` 包装、`variables`、加密前 `bizData`。
|
||||
|
||||
报告须注明：runtimeMode、静态/运行时 API 数、参数置信度、未覆盖模块、相对参考脚本的 `CHANGES.md` 摘要。
|
||||
|
||||
`site_map.json` 建议结构：
|
||||
|
||||
```json
|
||||
{
|
||||
"site": "https://example.com",
|
||||
"runtimeMode": "both",
|
||||
"appType": "vue-spa",
|
||||
"routeGuardStrategy": ["nav-neutralize", "L1-auth", "L2-patch", "forward"],
|
||||
"apisFromStatic": [],
|
||||
"apisFromRuntime": [],
|
||||
"apis": [],
|
||||
"params": [{ "method": "POST", "path": "/api/user/list", "transport": "json", "fields": [] }],
|
||||
"frontendRoutes": [],
|
||||
"routesVerifiedByClick": [],
|
||||
"featuresTriggered": [],
|
||||
"limitations": ""
|
||||
}
|
||||
```
|
||||
|
||||
更多字段与 grep 配方见 [reference.md](reference.md)。
|
||||
|
||||
---
|
||||
|
||||
## 通用说明
|
||||
|
||||
- **框架无关**：webpack/Vite/Angular lazy load 方法相同
|
||||
- **传输**：REST/JSON、GraphQL、WebSocket、SSE；gRPC-web 不在范围
|
||||
- **SSR**：客户端 fetch 可录；RSC/Server Actions 不完全可枚举
|
||||
- **盲区**：JSVMP、WASM、HMAC/mTLS 强校验 → 静态 + 标注局限
|
||||
- **参数盲区**：条件联动、hidden params、WASM 组包 → 「待触发」/「不可达」
|
||||
- **静态是安全网**：runtime 被挡时静态仍能枚举 endpoint
|
||||
|
||||
---
|
||||
|
||||
## 附加资源
|
||||
|
||||
- Grep 配方、`config.json` 模板、排障、Hook、参数逆向 J 节、site_map 模板：**[reference.md](reference.md)**
|
||||
- 参考脚本路径见 [脚本与门禁](#脚本与门禁) 表
|
||||
@@ -0,0 +1,500 @@
|
||||
# api-recon — 参考手册
|
||||
|
||||
Grep 配方、`config.json` 模板与排障。所有 grep 针对 `js/` 目录执行。bundle 单行时可先 `js-beautify` 或 `sed 's/}/}\n/g'`，通常带上下文窗口的 raw grep 即可。
|
||||
|
||||
## 脚本说明
|
||||
|
||||
`scripts/` 内所有文件均为**参考模板**，执行前必须按目标站点调整。典型改动点：
|
||||
|
||||
| 脚本 | 常见需调整项 |
|
||||
|---|---|
|
||||
| `harvest_static.py` | endpoint 正则、webpack/Vite manifest 解析、微前端 publicPath、重试/并发 |
|
||||
| `runtime_harvest.js` | neutralize 字段名与成功值、stub 匹配规则与 body 结构、routes 来源、WS 录制、`waitUntil`/`routeTimeout`/`proxy` |
|
||||
| `preload.js` | `loginPathRe`、L1 stubs、`neutralize.fields`、`apiPattern`、是否启用 L3、`recordDetail`、`observe.*`、`neutralizeVueRouter` |
|
||||
| `spider_mpa.py` | `--exclude` 破坏性链接、cookie、depth/max、同域过滤 |
|
||||
| `extract_route_map.py` | `routeMap` / `routeLink` 正则、KEY 命名模式 |
|
||||
| `build_perm_tree.py` | `userRouteAuth` 解析、`ROOTS`/`PREFIX_PARENT` 层级启发式、stub 外层字段名 |
|
||||
| `config.json` | 以上全部站点专属参数的统一入口 |
|
||||
|
||||
调整后的文件建议放在任务工作目录（如 `recon/`），报告中注明相对参考脚本的具体改动。
|
||||
|
||||
---
|
||||
|
||||
## A. 逆向三道门
|
||||
|
||||
### A1. 渲染门 — 「如何判断已登录？」
|
||||
|
||||
```bash
|
||||
grep -rhoaE '.{0,40}(isLogin|isAuthenticated|loggedIn|hasLogin|requireAuth)\b.{0,80}' js | head
|
||||
grep -rhoaE 'function (getUser|getToken|getAuth)[0-9]?\([^)]*\)\{.{0,200}' js | head
|
||||
grep -rhoaE '(localStorage|sessionStorage)\.getItem\("[^"]+"\)' js | sort -u
|
||||
grep -rhoaE '(Cookies?|cookie)\.(get|load)\("[^"]+"\)' js | sort -u
|
||||
grep -rhoaE '\batob\(|JSON\.parse\(|jwt|decode' js | head
|
||||
```
|
||||
|
||||
找链路 `isLogin = f(getUser())` → `getUser = decode(storage.read(KEY))`，确定 **存储键**、**容器**（Cookie vs localStorage）、**编码**：
|
||||
|
||||
| 编码 | config 伪造方式 |
|
||||
|---|---|
|
||||
| 明文字符串 / `"1"` / token | `"value": "anything-truthy"` |
|
||||
| `JSON.parse(x)` | `"value": "json:{\"id\":1,\"username\":\"admin\"}"` |
|
||||
| `JSON.parse(atob(x))` | `"value": "b64json:{\"id\":1,\"username\":\"admin\"}"` |
|
||||
| JWT | 无签名/`alg:none` JWT，或 bundle 内密钥签名 |
|
||||
| 加密（SM2/AES/RSA） | 找硬编码密钥；渲染门仅需可解码 blob 时可 forge；否则静态兜底 |
|
||||
|
||||
→ 写入 `cookies` / `localStorage`。
|
||||
|
||||
### A2. 拦截器门 — 「什么触发跳 /login？」
|
||||
|
||||
```bash
|
||||
grep -rhoaE '.{0,60}(interceptors\.response|axios|request\.use).{0,120}' js | head
|
||||
grep -rhoaE '.{0,40}(response_code|errcode|errno|\bcode\b|\bret\b|\bstatus\b)\s*[=!]==?\s*[\-0-9]{1,4}.{0,60}' js | head -20
|
||||
grep -rhoaE '.{0,40}(未登录|请重新登录|登录已过期|unauthorized|登录失效|授权|token.{0,10}invalid).{0,40}' js | head
|
||||
grep -rhoaE '.{0,30}(location\.href|router\.(push|replace)|navigate)\([^)]*login[^)]*\)' js | head
|
||||
```
|
||||
|
||||
确定：**字段名**、**成功值**（通常 `0` 或 `200`）、**触发跳转的失败值**。用 junk session 验证：
|
||||
|
||||
```bash
|
||||
curl -sk -X POST -H 'Cookie: <fakekey>=junk' https://target/api/<protected> -d '{}' -H 'Content-Type: application/json'
|
||||
```
|
||||
|
||||
→ 写入 `neutralize.fields` + `neutralize.success`。
|
||||
|
||||
### A3. 内容门 — 「菜单/权限从哪来？」
|
||||
|
||||
```bash
|
||||
grep -rhoaE '"/api[^"]*(permission|perm|role|menu|acl|resource|nav)[^"]*"' js | sort -u
|
||||
grep -rhoaE '.{0,30}(menus|permissions|menuList|routeList|authList|role_permissions)\b.{0,120}' js | head
|
||||
grep -rhoaE 'userRouteAuth|getResultTree|routeMap|routeLink|hasPermission|checkAuth' js | head
|
||||
grep -rhoaE '([A-Z_][A-Z0-9_]*):\{name:"[^"]*",link:"/[^"]+"\}' js | head
|
||||
```
|
||||
|
||||
**两层数据**（常见企业后台）：
|
||||
|
||||
| API | 典型 payload | 消费方 |
|
||||
|---|---|---|
|
||||
| `.../role_permissions` | `{ permissions: string[], role_type }` | 路由守卫、按钮级 ACL |
|
||||
| `.../permissions/all` | `tree[{ code, position, children }]` | 侧栏菜单渲染 |
|
||||
| bundle 内 `userRouteAuth` | `{ CODE: { url, name? } }` | code → 前端 path |
|
||||
| bundle 内 `routeMap` | `{ KEY: { name, link } }` | 别名解析（webpack `o.DASHBOARD`） |
|
||||
|
||||
读消费方代码确认：`getResultTree(tree, permissions)` 如何过滤、`v-if` / `hasAuth(code)` 检查哪个字段。
|
||||
|
||||
**手工 forge**（小站点）：构建 permissive payload → `stubs`。
|
||||
|
||||
**完整权限树还原**（大站点，侧栏/子模块仍空白）：见 **I 节**。
|
||||
|
||||
---
|
||||
|
||||
## B. config.json 模板
|
||||
|
||||
```json
|
||||
{
|
||||
"baseUrl": "https://target/",
|
||||
"runtimeMode": "both",
|
||||
"chromium": "/usr/bin/chromium",
|
||||
|
||||
"cookies": [
|
||||
{ "name": "auth", "value": "b64json:{\"id\":1,\"username\":\"admin\",\"role\":\"admin\",\"func\":{},\"permissions\":[\"*\"]}" }
|
||||
],
|
||||
"localStorage": { "token": "faketoken", "isLogin": "1" },
|
||||
|
||||
"neutralize": {
|
||||
"fields": ["response_code", "code", "errno", "ret", "status"],
|
||||
"success": 0,
|
||||
"flags": { "success": true, "message": "ok" }
|
||||
},
|
||||
"forward": true,
|
||||
"loginUrlPattern": "/login",
|
||||
"apiPattern": "/api/|/rest/|/graphql",
|
||||
|
||||
"mockTier": "L1+L2",
|
||||
"recordDetail": true,
|
||||
"observe": {
|
||||
"storageReads": false,
|
||||
"cookieReads": false,
|
||||
"xhrHeaders": true
|
||||
},
|
||||
"neutralizeVueRouter": true,
|
||||
"stubs": [
|
||||
{
|
||||
"match": "permissions/all|/menu|role_permissions",
|
||||
"body": {
|
||||
"response_code": 0, "code": 0,
|
||||
"data": {
|
||||
"permissions": ["*"],
|
||||
"menus": [
|
||||
{ "name": "dashboard", "path": "/dashboard", "show": true, "children": [] },
|
||||
{ "name": "alert", "path": "/alert", "show": true, "children": [] }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
|
||||
"explore": {
|
||||
"clickTabs": true,
|
||||
"clickTables": true,
|
||||
"pushStateFallback": true,
|
||||
"maxMenuItems": 50
|
||||
},
|
||||
|
||||
"routes": ["/dashboard", "/alert", "/asset", "/device", "/report", "/config", "/system"],
|
||||
"waitMs": 1500, "perRouteMs": 900, "headless": true,
|
||||
"waitUntil": "domcontentloaded",
|
||||
"routeTimeout": 12000,
|
||||
"proxy": "",
|
||||
|
||||
"captureResponses": true, "recordWs": true, "respMax": 600
|
||||
}
|
||||
```
|
||||
|
||||
字段说明：
|
||||
- `runtimeMode`：`depth`（Puppeteer）、`coverage`（browser MCP）、`both`
|
||||
- `cookies[].value` 前缀：`b64json:` → base64(JSON)；`json:` → 原始 JSON；无前缀 → 字面量
|
||||
- `forward: true` 转发真实请求并改写码字段；`false` 完全离线 stub
|
||||
- `mockTier`：coverage 模式 preload 启用层级，如 `L1+L2`、`L1+L2+L3`
|
||||
- `routes` 来自 `routes.txt`；forge 菜单后 harness 自动追加 `<a href>`
|
||||
- `captureResponses` / `recordWs` 仅 depth 模式有效
|
||||
- `waitUntil`：大型 SPA 用 `domcontentloaded`，避免 `networkidle2` 挂起
|
||||
- `routeTimeout`：单路由 `page.goto` 超时（毫秒）
|
||||
- `proxy`：Puppeteer `--proxy-server`；也可设 `HTTP_PROXY` / `HTTPS_PROXY`
|
||||
|
||||
### B1. 双 stub 模板（role_permissions + permissions/all）
|
||||
|
||||
```json
|
||||
"stubs": [
|
||||
{
|
||||
"match": "role_permissions",
|
||||
"body": {
|
||||
"response_code": 0,
|
||||
"data": {
|
||||
"permissions": ["MONITOR", "MONITOR_ALERT", "THREAT", "ASSETS_RISK"],
|
||||
"role_type": "SUPER_ADMIN"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"match": "permissions/all",
|
||||
"body": {
|
||||
"response_code": 0,
|
||||
"data": [
|
||||
{
|
||||
"code": "MONITOR",
|
||||
"position": 1,
|
||||
"children": [
|
||||
{ "code": "MONITOR_ALERT", "position": 1, "children": [] }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
外层字段名（`response_code` / `code` / `data`）须与 A2 拦截器门一致；`permissions` 须覆盖 tree 中所有 leaf code。
|
||||
|
||||
---
|
||||
|
||||
## C. coverage 模式：preload 配置
|
||||
|
||||
编辑 `scripts/preload.js` 顶部 `CONFIG` 对象，或通过 CDP 注入前替换：
|
||||
|
||||
```javascript
|
||||
const CONFIG = {
|
||||
loginPathRe: /\/(login|signin)(\/|$|\?)/i,
|
||||
mockTier: 'L1+L2',
|
||||
forward: true,
|
||||
recordDetail: true,
|
||||
extractUrlsFromResponse: true,
|
||||
neutralizeVueRouter: true,
|
||||
observe: { storageReads: false, cookieReads: false, xhrHeaders: true },
|
||||
neutralize: { fields: ['response_code', 'code'], success: 0 },
|
||||
stubs: [ /* 同 config.json stubs */ ],
|
||||
apiPattern: /\/(api|apis|v\d+|dev|internal|graphql)\//i,
|
||||
};
|
||||
```
|
||||
|
||||
验证：`window.__API_RECON_PRELOAD__ === true` 且 pathname 稳定。
|
||||
|
||||
导出录制结果：
|
||||
|
||||
```javascript
|
||||
JSON.stringify({
|
||||
apis: [...window.__API_RECON_LOG__],
|
||||
detail: window.__API_RECON_DETAIL__,
|
||||
routes: [...(window.__API_RECON_ROUTES__ || [])],
|
||||
observe: window.__API_RECON_OBSERVE__,
|
||||
}, null, 2)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## D. preload / runtime Hook 能力
|
||||
|
||||
preload（coverage）与 runtime_harvest（depth）内置的浏览器 Hook 能力及覆盖范围：
|
||||
|
||||
| Hook 能力 | 对 API 发现的价值 | 覆盖 |
|
||||
|---|---|---|
|
||||
| Hook fetch / XHR.open | 录请求 URL/方法 | ✅ `recordDetail` + `__API_RECON_LOG__` |
|
||||
| Hook XHR.setRequestHeader | 发现 Authorization 等头 | ✅ `observe.xhrHeaders` |
|
||||
| Hook localStorage/cookie 读 | 确认会话键名 | ⚠️ 可选 `observe.storageReads/cookieReads` |
|
||||
| Vue 获取路由 | 补全 frontendRoutes | ✅ `__API_RECON_ROUTES__`（已加载路由） |
|
||||
| Vue 路由守卫中和 / 登录跳转阻断 | 撑开模块触发 API | ✅ `neutralizeVueRouter` + 原生跳转中和 |
|
||||
| React 获取路由 | 补路由 | ⚠️ 静态 + 点击；无专用 Hook |
|
||||
| 页面跳转阻断（登录 path） | 留页分析 | ⚠️ 仅阻断登录 path，避免挡业务导航 |
|
||||
| Hook 加密库（CryptoJS/SM 等） | 加密参数 → 明文 API body | ❌ 须手工 Hook 加密函数入参；结论写 config |
|
||||
| 反调试 bypass | 否则 runtime 录不到 API | ❌ 须手工处理；静态仍可用 |
|
||||
|
||||
---
|
||||
|
||||
## E. Endpoint 提取正则（静态过少时）
|
||||
|
||||
在 `harvest_static.py` 的 `extract_endpoints` 放宽，或手动：
|
||||
|
||||
```bash
|
||||
grep -rhoaE '"/[a-z][A-Za-z0-9_/\-]{3,}"' js | sort -u
|
||||
grep -rhoaE '/api/[a-zA-Z0-9_./-]+' js | sort -u
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## F. 排障
|
||||
|
||||
| 现象 | 原因 → 处理 |
|
||||
|---|---|
|
||||
| 静态 API 很少 | endpoint 方言不匹配 → 放宽正则（D 节） |
|
||||
| chunk 数 ≪ manifest | CSS-only 或未部署 chunk；404 已重试 |
|
||||
| runtime 仍显示登录页 | 渲染门错误 → 复查 A1：键名、容器、编码、domain |
|
||||
| 进壳但模块空白 | 内容门 → forge 菜单（A3）；`routes` path 可能不对 |
|
||||
| 每路由只有 bootstrap/locale | 权限码不全 → I 节权限树还原；检查 `role_permissions` + `permissions/all` 双 stub |
|
||||
| 侧栏有项但子页空白 | tree 缺 intermediate 节点或 code 与 `userRouteAuth` 不一致 |
|
||||
| 每个 API 都跳登录 | 拦截器门 → 确认 `neutralize`；嵌套字段需扩展 walk 逻辑 |
|
||||
| WS 帧为 0 | 需用户交互后才 subscribe；加长 `perRouteMs` |
|
||||
| 响应体空 | 仅 `forward: true` 时有真实响应 |
|
||||
| Chromium 缺失 | 安装 chromium 或设置 `config.chromium` / `CHROMIUM` |
|
||||
| Mock 很多仍回登录 | Hook 太晚或缺 `location.href` setter → document-start + preload |
|
||||
| 列表全空 | L3 空数组正常；继续点 Tab/设置/详情 |
|
||||
| 误把 Redux action 当路由 | 过滤含 get/set/change/clear/toggle/upload 的内部 path |
|
||||
| Vue 仍跳登录 | preload 非 document-start → 改注入时机；或 `neutralizeVueRouter: false` 时手动清守卫 |
|
||||
| 响应里有 URL 但未进 log | 开 `extractUrlsFromResponse`；或从 `__API_RECON_DETAIL__` 人工提取 |
|
||||
| 不知 Authorization 头名 | 开 `observe.xhrHeaders` 或 DevTools 查看请求头 |
|
||||
| runtime 极慢 / 超时 | 改 `waitUntil: domcontentloaded`；降 `routeTimeout`；勿用 `networkidle2` |
|
||||
| 代理连接失败 | 检查 `proxy` / 环境变量；Puppeteer 与 curl 代理端口一致 |
|
||||
|
||||
---
|
||||
|
||||
## G. hardened 目标
|
||||
|
||||
服务端逐步校验会话（不可 forge 的签名 cookie、服务端渲染且不可 stub 的菜单）时，runtime 会在 shell 处卡住。预期行为：
|
||||
|
||||
- **静态足够做 endpoint 枚举** — 模块 path 在代码里
|
||||
- 若授权允许，用**真实会话**跑同一 harness：`forward: true`、无需 neutralize，捕获真实 methods/params/responses
|
||||
|
||||
---
|
||||
|
||||
## H. 单次任务清单
|
||||
|
||||
1. 确认授权范围
|
||||
2. **阅读** `scripts/harvest_static.py` → 按目标调整 → 运行 → 审 `api_static.txt`、`routes.txt`
|
||||
3. **Phase 1b**：path 锚点扩窗 + 绑定层 → `param_candidates.json`（J 节）
|
||||
4. 逆向 A1/A2/A3 → 写站点专属 `config.json`
|
||||
5. **阅读并调整** `runtime_harvest.js` / `preload.js` 后再执行
|
||||
6. `runtimeMode=depth`：`npm install` → 运行调整后的 harvest 脚本
|
||||
7. `runtimeMode=coverage/both`：document-start 注入调整后的 preload → browser MCP 动态枚举 + **参数触发矩阵**
|
||||
8. 模块不渲染 → **I 节权限树还原** → patch stubs → 重跑
|
||||
9. 参数多样本 diff + 错误反推 → `params_merged.json`
|
||||
10. 合并 → `site_map.json` + `api_merged.txt`，诚实标注覆盖、缺口及脚本改动点
|
||||
|
||||
---
|
||||
|
||||
## I. 权限树还原（Phase 4 深化）
|
||||
|
||||
当 forge 简单 `menus: [{ path, show: true }]` 无效、子模块仍不 mount 时使用。
|
||||
|
||||
### I1. 定位 auth 模块
|
||||
|
||||
```bash
|
||||
grep -l 'userRouteAuth' js/*.js
|
||||
grep -l 'routeMap\|routeLink' js/*.js
|
||||
grep -rhoaE 'getResultTree|role_permissions|permissions/all' js | head
|
||||
```
|
||||
|
||||
记录：**权限 API path**、**响应字段名**、**消费 chunk 文件名**。
|
||||
|
||||
### I2. 提取 routeMap
|
||||
|
||||
```bash
|
||||
python3 scripts/extract_route_map.py recon/js recon/
|
||||
# 产出 recon/route_map.json
|
||||
```
|
||||
|
||||
若 `[!] no routeMap pattern found`：放宽 `extract_route_map.py` 中正则，或手工 grep：
|
||||
|
||||
```bash
|
||||
grep -rhoaE '([A-Z_][A-Z0-9_]*):\{name:"[^"]*",link:"/[^"]+"\}' js | head -20
|
||||
```
|
||||
|
||||
### I3. 构建权限树 + stub
|
||||
|
||||
```bash
|
||||
python3 scripts/build_perm_tree.py recon/js recon/ --config recon/config.json
|
||||
```
|
||||
|
||||
脚本逻辑：
|
||||
1. 解析 `userRouteAuth={MONITOR:{url:...},...}`（含 webpack 别名 `He=o.DASHBOARD`）
|
||||
2. 用 `route_map.json` 解析 alias → 真实 path
|
||||
3. 按 code 前缀推断 parent（`MONITOR_ALERT` → `MONITOR`）
|
||||
4. 输出 `permissions_tree.json`、`permissions_all_stub.json`、`role_permissions_stub.json`
|
||||
5. `--config` 时自动写入 `config.json` 的 `stubs` 与扩展 `routes`
|
||||
|
||||
**按目标调整**（在脚本顶部）：
|
||||
- `DEFAULT_ROOTS`：顶级模块 code 列表
|
||||
- `DEFAULT_PREFIX_PARENT`：`PREFIX_` → parent 映射
|
||||
- `DEFAULT_EXTRA_PARENT`：非前缀关系的 orphan 节点
|
||||
|
||||
### I4. 校验 stub 一致性
|
||||
|
||||
```bash
|
||||
# permissions 数量应 ≈ userRouteAuth 条目数
|
||||
wc -l recon/perm_codes_all.txt
|
||||
# routes 应覆盖 route_map 全部 link
|
||||
python3 -c "import json; m=json.load(open('recon/route_map.json')); r=set(json.load(open('recon/config.json'))['routes']); print('missing', [v['link'] for v in m.values() if v['link'] not in r])"
|
||||
```
|
||||
|
||||
### I5. 重跑 runtime 并对比
|
||||
|
||||
```bash
|
||||
node recon/runtime_harvest.js recon/config.json
|
||||
# 对比 forge 前后 runtime_api.json 条数；检查 /attack、/asset 等是否出现模块 API
|
||||
```
|
||||
|
||||
| forge 前 | forge 后（成功） |
|
||||
|---|---|
|
||||
| 每路由相同 3–5 条 bootstrap | 不同路由触发不同 module API |
|
||||
| 仅 `/api/locale/language` | 出现 `/api/web/...` 模块 endpoint |
|
||||
| `routes.txt` 个位路由 | `routes` 80–110+ 来自 route_map |
|
||||
|
||||
### I6. 仍失败时
|
||||
|
||||
- **coverage 模式**：点击侧栏 + Tab，权限 gating 可能在交互后才请求
|
||||
- **stub 字段**：对比真实 API（curl + 真实 session）与 stub 的 nesting
|
||||
- **额外守卫**：grep `hasPermission|checkRole|func.` 等按钮级检查，扩展 `role_permissions.permissions`
|
||||
- **静态兜底**：模块 API path 仍在 `api_static.txt`，runtime 仅补 METHOD/body；参数保留 `param_candidates.json` + 已录样本
|
||||
|
||||
---
|
||||
|
||||
## J. 参数逆向（Phase 1b / 5b / 5c）
|
||||
|
||||
**方法论，非通用脚本。** 找 path 用正则；找参数用锚点扩窗 + UI 绑定链 + 多样本 diff + 错误反推。
|
||||
|
||||
### J1. 锚点扩窗 — 从 path 找组包对象
|
||||
|
||||
```bash
|
||||
# 以 Phase 1 已知 path 为锚
|
||||
grep -n '"/api/user/list"' js/*.js
|
||||
grep -rhoaE '.{0,120}("/api[^"]+").{0,200}' js | head
|
||||
grep -rhoaE '(params|data|body|payload)\s*:\s*\{' js | head
|
||||
grep -rhoaE '(get|post|put|delete|patch)\([^,]+,\s*\{' js | head
|
||||
```
|
||||
|
||||
### J2. 包装层与传输形态
|
||||
|
||||
```bash
|
||||
# axios / 统一 request
|
||||
grep -rhoaE '(axios|request)\.(get|post|put|delete|patch)\(' js | head
|
||||
grep -rhoaE 'interceptors\.(request|response)' js | head
|
||||
|
||||
# GraphQL
|
||||
grep -rhoaE '(query|mutation)\s+\w+|gql`|graphql\(' js | head
|
||||
grep -rhoaE '\$[a-zA-Z_]+\s*:\s*(Int|String|Boolean|\[)' js | head
|
||||
|
||||
# FormData / multipart
|
||||
grep -rhoaE 'FormData|\.append\(' js | head
|
||||
|
||||
# 路径参数
|
||||
grep -rhoaE 'path:\s*"/[^"]*:[^"]+"' js | head
|
||||
grep -rhoaE 'useParams|route\.params|\$route\.params' js | head
|
||||
```
|
||||
|
||||
### J3. 校验门 — 必填 / 格式 / 枚举
|
||||
|
||||
```bash
|
||||
grep -rhoaE '(required|message|pattern|enum|validator)\s*:' js | head
|
||||
grep -rhoaE 'yup\.|zod\.|async-validator|Form\.Item|a-form-item|el-form-item' js | head
|
||||
grep -rhoaE 'rules\s*:\s*\[|name:\s*["\'][a-zA-Z_]+["\']' js | head
|
||||
grep -rhoaE 'label.*value|options\s*:\s*\[' js | head
|
||||
```
|
||||
|
||||
### J4. 绑定层 — 表单 → API
|
||||
|
||||
```bash
|
||||
grep -rhoaE 'onFinish|handleSubmit|getFieldsValue|validateFields' js | head
|
||||
grep -rhoaE '(pick|omit|transform|dayjs|moment)\(' js | head
|
||||
```
|
||||
|
||||
runtime 补位：DevTools → Network → 请求 → **发起程序**（call stack）从 `fetch`/`send` 往上追组包函数。
|
||||
|
||||
### J5. 加密参数
|
||||
|
||||
```bash
|
||||
grep -rhoaE 'encrypt|decrypt|sign|CryptoJS|sm2|sm3|sm4|RSA|AES' js | head
|
||||
```
|
||||
|
||||
**勿在密文上猜字段** — Hook 加密函数**入参**，在加密前录 plaintext payload；结论写 `config.json` / `param_candidates.json`。
|
||||
|
||||
### J6. 参数触发矩阵（Phase 3 必做）
|
||||
|
||||
对每模块按操作各录一次，diff 请求 body/query：
|
||||
|
||||
| 操作 | 关注 |
|
||||
|---|---|
|
||||
| 列表首屏 | 分页默认值 |
|
||||
| 搜索 | keyword、filters |
|
||||
| 高级筛选 | optional 字段 |
|
||||
| 新建/编辑 | 完整 entity |
|
||||
| 批量/导出 | `ids[]`、`exportType` |
|
||||
| 排序/翻页 | `sortField`、`order` |
|
||||
|
||||
产出 `param_samples.json`：`[{ "path", "method", "action": "search", "body", "query", "headers" }]`
|
||||
|
||||
### J7. 置信度规则
|
||||
|
||||
| 置信度 | 条件 |
|
||||
|---|---|
|
||||
| **高** | 静态 callsite + runtime ≥2 样本一致 |
|
||||
| **中** | 仅静态，或仅 1 次 runtime |
|
||||
| **低** | 响应/错误反推，未二次验证 |
|
||||
| **待触发** | 静态已知字段，UI/权限未跑到 |
|
||||
|
||||
### J8. 场景快配
|
||||
|
||||
| 场景 | 顺序 |
|
||||
|---|---|
|
||||
| REST 列表页 | J1 组包对象 → J6 四次 diff → J3 rules |
|
||||
| 新建/编辑表单 | J3 Form name → J4 submit 链 → runtime 提交 + 故意留空看 400 |
|
||||
| GraphQL | J2 variables 声明 → runtime 各 operation 录 variables |
|
||||
| 加密 body | J5 Hook 入参 → 加密前字段即真实 params |
|
||||
|
||||
### J9. 与 api-recon 阶段映射
|
||||
|
||||
| api-recon | 参数 recon |
|
||||
|---|---|
|
||||
| Phase 1 静态 | J1 锚点扩窗 |
|
||||
| Phase 2 A2 拦截器 | 全局注入字段（tenantId、sign） |
|
||||
| Phase 3 runtime | J6 触发矩阵 + `param_samples.json` |
|
||||
| Phase 4 权限树 | 不同模块表单不同 → 权限够才触发全字段 |
|
||||
| Phase 5 合并 | `params_merged.json` + 置信度；勿单样本定必填 |
|
||||
|
||||
### J10. 排障
|
||||
|
||||
| 现象 | 处理 |
|
||||
|---|---|
|
||||
| 静态有字段名 runtime 从未出现 | 标注「待触发」；补权限树 / 点高级筛选 / 联动 select 各 option |
|
||||
| 同 path 不同 body 形状 | 正常 — 按 `action` 分条记录，勿强行合并 schema |
|
||||
| stub 响应假但想看 params | **看 outbound 请求** body/headers，勿从 stub 响应反推 |
|
||||
| 400 报 nested field | 注意外层包装 `data`/`bizData`/`variables` |
|
||||
| GraphQL 只见 operation 名 | 展开 `variables` JSON；静态找 `$var: Type` |
|
||||
|
||||
---
|
||||
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
build_perm_tree.py <JSDIR> <OUTDIR> [--config recon/config.json]
|
||||
|
||||
Rebuild a permissive permission/menu stub from frontend auth modules.
|
||||
|
||||
Typical consumer chain (TDP / enterprise admin SPAs):
|
||||
POST /api/web/user/role_permissions -> { permissions: [...], role_type }
|
||||
POST /api/web/permissions/all -> tree[{ code, position, children }]
|
||||
getResultTree(tree, permissions) -> menu include lists
|
||||
userRouteAuth[code].url -> frontend route path
|
||||
|
||||
This script:
|
||||
1. Locates userRouteAuth={MONITOR:{url:...},...} in js/
|
||||
2. Resolves webpack alias refs (He=o.DASHBOARD) via route_map.json
|
||||
3. Infers hierarchy from code prefixes (MONITOR_ -> MONITOR)
|
||||
4. Writes permissions_tree.json, permissions_all_stub.json,
|
||||
role_permissions_stub.json, userRouteAuth.json
|
||||
5. Optionally patches config.json stubs (permissions/all + role_permissions)
|
||||
|
||||
Adjust ROOTS / PREFIX_PARENT / EXTRA_PARENT per target if heuristics miss nodes.
|
||||
"""
|
||||
import re, json, os, sys, glob, argparse
|
||||
|
||||
DEFAULT_ROOTS = [
|
||||
'MONITOR', 'THREAT', 'ASSETS_RISK', 'INVESTIGATION', 'MANAGEMENT',
|
||||
'AGENT_EVIDENCE', 'MDR', 'PLATFORM',
|
||||
]
|
||||
DEFAULT_PREFIX_PARENT = {
|
||||
'MONITOR_': 'MONITOR', 'THREAT_': 'THREAT', 'ASSETS_RISK_': 'ASSETS_RISK',
|
||||
'INVESTIGATION_': 'INVESTIGATION', 'MANAGEMENT_': 'MANAGEMENT', 'MDR_': 'MDR',
|
||||
'PLATFORM_': 'PLATFORM', 'CUSTOM_': 'PLATFORM', 'FALSE_': 'PLATFORM',
|
||||
}
|
||||
DEFAULT_EXTRA_PARENT = {
|
||||
'LINKAGE_DISPOSAL': 'MANAGEMENT',
|
||||
'ASSETS_RISK_API': 'ASSETS_RISK',
|
||||
'ASSETS_RISK_WEAK_PWD': 'ASSETS_RISK',
|
||||
}
|
||||
|
||||
|
||||
def find_auth_file(jsdir):
|
||||
best_fp, best_len = None, 0
|
||||
for fp in glob.glob(os.path.join(jsdir, '*.js')):
|
||||
try:
|
||||
txt = open(fp, encoding='utf-8', errors='ignore').read()
|
||||
except Exception:
|
||||
continue
|
||||
if 'userRouteAuth' not in txt:
|
||||
continue
|
||||
m = re.search(r'userRouteAuth=\{MONITOR:', txt)
|
||||
if m and len(txt) > best_len:
|
||||
best_fp, best_len = fp, len(m.group(0))
|
||||
elif 'userRouteAuth' in txt and best_fp is None:
|
||||
best_fp = fp
|
||||
return best_fp
|
||||
|
||||
|
||||
def parse_aliases(chunk):
|
||||
aliases = {}
|
||||
for m in re.finditer(r'([a-zA-Z_$][\w$]*)=o\.([A-Z_0-9]+)\b', chunk[:8000]):
|
||||
aliases[m.group(1)] = m.group(2)
|
||||
return aliases
|
||||
|
||||
|
||||
def resolve_ref(token, aliases, route_map):
|
||||
token = token.strip()
|
||||
if token.startswith('"'):
|
||||
return json.loads(token)
|
||||
if token in aliases:
|
||||
key = aliases[token]
|
||||
return route_map.get(key, {}).get('link', key)
|
||||
return token
|
||||
|
||||
|
||||
def parse_user_route_auth(txt, route_map):
|
||||
m = re.search(r'(?:t\.)?userRouteAuth=(\{MONITOR:.*?\})\},', txt, re.S)
|
||||
if not m:
|
||||
m = re.search(r'userRouteAuth=(\{[A-Z_0-9]+:\{url:', txt)
|
||||
if not m:
|
||||
return None, {}
|
||||
# greedy fallback — trim at next webpack module
|
||||
body = m.group(1)
|
||||
end = body.rfind('}')
|
||||
obj_src = body[: end + 1] if end > 0 else body
|
||||
else:
|
||||
obj_src = m.group(1)
|
||||
|
||||
chunk_start = txt.find('userRouteAuth')
|
||||
chunk = txt[chunk_start:chunk_start + 35000]
|
||||
aliases = parse_aliases(chunk)
|
||||
|
||||
entries = {}
|
||||
for em in re.finditer(
|
||||
r'([A-Z_0-9]+):\{url:([^,}]+)(?:,control:(\[.*?\]|[^,}]+))?\}', obj_src
|
||||
):
|
||||
key = em.group(1)
|
||||
url = resolve_ref(em.group(2).strip(), aliases, route_map)
|
||||
controls = []
|
||||
if em.group(3):
|
||||
raw = em.group(3).strip()
|
||||
vars_ = re.findall(r'([A-Za-z_$][\w$]*)', raw) if raw.startswith('[') else [raw]
|
||||
controls = [aliases.get(v, v) for v in vars_]
|
||||
entries[key] = {'code': key, 'url': url, 'control': controls}
|
||||
return obj_src, entries
|
||||
|
||||
|
||||
def parent_of(code, roots, prefix_parent, extra_parent):
|
||||
if code in extra_parent:
|
||||
return extra_parent[code]
|
||||
if code in roots:
|
||||
return None
|
||||
for pref, par in prefix_parent.items():
|
||||
if code.startswith(pref):
|
||||
return par
|
||||
return None
|
||||
|
||||
|
||||
def build_tree(entries, roots, prefix_parent, extra_parent):
|
||||
children_of = {k: [] for k in entries}
|
||||
for code in entries:
|
||||
p = parent_of(code, roots, prefix_parent, extra_parent)
|
||||
if p:
|
||||
children_of.setdefault(p, []).append(code)
|
||||
|
||||
def make_node(code):
|
||||
node = {
|
||||
'code': code,
|
||||
'position': 'top' if code in roots else 'left',
|
||||
'name': code,
|
||||
}
|
||||
kids = sorted(children_of.get(code, []))
|
||||
if kids:
|
||||
node['children'] = [make_node(c) for c in kids]
|
||||
return node
|
||||
|
||||
tree = [make_node(r) for r in roots if r in entries or children_of.get(r)]
|
||||
orphans = [c for c in entries if parent_of(c, roots, prefix_parent, extra_parent) is None and c not in roots]
|
||||
for code in sorted(orphans):
|
||||
tree.append(make_node(code))
|
||||
return tree
|
||||
|
||||
|
||||
def flat_codes(nodes):
|
||||
out = []
|
||||
for n in nodes:
|
||||
out.append(n['code'])
|
||||
out.extend(flat_codes(n.get('children', [])))
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description='Build permission tree stubs from JS auth modules')
|
||||
ap.add_argument('jsdir')
|
||||
ap.add_argument('outdir')
|
||||
ap.add_argument('--config', help='patch stubs into config.json')
|
||||
ap.add_argument('--role', default='SUPER_ADMIN', help='role_type in role_permissions stub')
|
||||
args = ap.parse_args()
|
||||
os.makedirs(args.outdir, exist_ok=True)
|
||||
|
||||
route_map_path = os.path.join(args.outdir, 'route_map.json')
|
||||
if not os.path.exists(route_map_path):
|
||||
print('[*] route_map.json missing — run extract_route_map.py first')
|
||||
route_map = {}
|
||||
else:
|
||||
route_map = json.load(open(route_map_path, encoding='utf-8'))
|
||||
|
||||
auth_fp = find_auth_file(args.jsdir)
|
||||
if not auth_fp:
|
||||
print('[!] userRouteAuth module not found in js/')
|
||||
sys.exit(1)
|
||||
print(f'[*] auth module: {os.path.basename(auth_fp)}')
|
||||
txt = open(auth_fp, encoding='utf-8', errors='ignore').read()
|
||||
_, entries = parse_user_route_auth(txt, route_map)
|
||||
if not entries:
|
||||
print('[!] failed to parse userRouteAuth object — adjust regex in script')
|
||||
sys.exit(1)
|
||||
|
||||
tree = build_tree(entries, DEFAULT_ROOTS, DEFAULT_PREFIX_PARENT, DEFAULT_EXTRA_PARENT)
|
||||
all_codes = sorted(set(flat_codes(tree) + [c for e in entries.values() for c in e.get('control', [])]))
|
||||
|
||||
perm_all = {'response_code': 0, 'verbose_msg': 'ok', 'data': tree}
|
||||
role_perm = {
|
||||
'response_code': 0,
|
||||
'verbose_msg': 'ok',
|
||||
'data': {'role_type': args.role, 'permissions': all_codes},
|
||||
}
|
||||
|
||||
json.dump(entries, open(os.path.join(args.outdir, 'userRouteAuth.json'), 'w', encoding='utf-8'), ensure_ascii=False, indent=2)
|
||||
json.dump(tree, open(os.path.join(args.outdir, 'permissions_tree.json'), 'w', encoding='utf-8'), ensure_ascii=False, indent=2)
|
||||
open(os.path.join(args.outdir, 'perm_codes_all.txt'), 'w', encoding='utf-8').write('\n'.join(all_codes))
|
||||
json.dump(perm_all, open(os.path.join(args.outdir, 'permissions_all_stub.json'), 'w', encoding='utf-8'), ensure_ascii=False, indent=2)
|
||||
json.dump(role_perm, open(os.path.join(args.outdir, 'role_permissions_stub.json'), 'w', encoding='utf-8'), ensure_ascii=False, indent=2)
|
||||
|
||||
print(f'[+] entries={len(entries)} codes={len(all_codes)} tree_roots={len(tree)}')
|
||||
|
||||
cfg_path = args.config or os.path.join(args.outdir, 'config.json')
|
||||
if os.path.exists(cfg_path):
|
||||
cfg = json.load(open(cfg_path, encoding='utf-8'))
|
||||
stubs = [s for s in cfg.get('stubs', []) if not re.search(r'permissions/all|role_permissions', s.get('match', ''))]
|
||||
stubs = [
|
||||
{'match': 'permissions/all', 'body': perm_all},
|
||||
{'match': 'role_permissions', 'body': role_perm},
|
||||
] + stubs
|
||||
cfg['stubs'] = stubs
|
||||
json.dump(cfg, open(cfg_path, 'w', encoding='utf-8'), ensure_ascii=False, indent=2)
|
||||
print(f'[+] patched stubs -> {cfg_path}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
extract_route_map.py <JSDIR> <OUTDIR>
|
||||
|
||||
Scan downloaded JS chunks for routeMap / routeLink style objects:
|
||||
KEY:{name:"...",link:"/path"}
|
||||
|
||||
Writes route_map.json — used by build_perm_tree.py to resolve alias refs.
|
||||
"""
|
||||
import re, json, os, sys, glob
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
print("usage: extract_route_map.py <JSDIR> <OUTDIR>")
|
||||
sys.exit(1)
|
||||
jsdir, outdir = sys.argv[1], sys.argv[2]
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
|
||||
best = {}
|
||||
best_file = None
|
||||
pat = re.compile(r'([A-Z_][A-Z0-9_]*):\{name:"([^"]*)",link:"([^"]+)"')
|
||||
|
||||
for fp in glob.glob(os.path.join(jsdir, '*.js')):
|
||||
try:
|
||||
txt = open(fp, encoding='utf-8', errors='ignore').read()
|
||||
except Exception:
|
||||
continue
|
||||
hits = pat.findall(txt)
|
||||
if len(hits) > len(best):
|
||||
best = {k: {'name': n, 'link': l} for k, n, l in hits}
|
||||
best_file = fp
|
||||
|
||||
if not best:
|
||||
print('[!] no routeMap pattern found — widen regex or grep manually')
|
||||
sys.exit(1)
|
||||
|
||||
out = os.path.join(outdir, 'route_map.json')
|
||||
json.dump(best, open(out, 'w', encoding='utf-8'), ensure_ascii=False, indent=2)
|
||||
print(f'[+] {len(best)} routes from {os.path.basename(best_file)} -> {out}')
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
harvest_static.py <BASE_URL> <OUTDIR>
|
||||
|
||||
参考模板 — 非通用成品。执行前须按目标站点调整，常见改动：
|
||||
- extract_endpoints 正则（endpoint 方言）
|
||||
- webpack/Vite manifest 解析逻辑
|
||||
- 微前端 publicPath、重试策略
|
||||
|
||||
Static SPA bundle harvester. Framework-agnostic; tuned for webpack + Vite.
|
||||
1. Fetch entry HTML, collect script/module references.
|
||||
2. Parse the runtime chunk manifest(s) and download EVERY chunk (not just the
|
||||
ones in HTML), looping until no new chunk ids appear. Handles multiple
|
||||
micro-frontend runtimes, each with its own publicPath.
|
||||
3. Extract API endpoints and route paths from all downloaded JS.
|
||||
|
||||
Stdlib only. TLS verification is disabled (recon against self-signed/internal hosts).
|
||||
"""
|
||||
import sys, os, re, ssl, json, urllib.request, urllib.parse
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
|
||||
UA = "Mozilla/5.0 (spa-api-recon)"
|
||||
|
||||
def fetch(url, binary=False):
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||
with urllib.request.urlopen(req, context=CTX, timeout=25) as r:
|
||||
data = r.read()
|
||||
return (data if binary else data.decode("utf-8", "ignore")), r.status
|
||||
except Exception as e:
|
||||
code = getattr(e, "code", 0)
|
||||
return "", code
|
||||
|
||||
def origin_of(u):
|
||||
p = urllib.parse.urlparse(u)
|
||||
return f"{p.scheme}://{p.netloc}"
|
||||
|
||||
# ---- chunk manifest parsing -------------------------------------------------
|
||||
def matched_block(s, end_brace_idx):
|
||||
"""Walk back from a '}' index to its matching '{' and return the object body."""
|
||||
depth = 0; j = end_brace_idx
|
||||
while j >= 0:
|
||||
if s[j] == '}': depth += 1
|
||||
elif s[j] == '{':
|
||||
depth -= 1
|
||||
if depth == 0: return s[j:end_brace_idx+1]
|
||||
j -= 1
|
||||
return ""
|
||||
|
||||
def parse_chunk_maps(js_text):
|
||||
"""Return list of (publicPath_hint, {id: hash}) for every `}[x]+".js"` map
|
||||
(webpack __webpack_require__.u) found in the text."""
|
||||
maps = []
|
||||
# publicPath hints in this file: .p="..." or publicPath="..."
|
||||
pubs = re.findall(r'(?:\.p|publicPath)\s*=\s*"([^"]*)"', js_text)
|
||||
for m in re.finditer(r'\}\[[A-Za-z_$]\]\s*\+\s*"\.js"', js_text):
|
||||
body = matched_block(js_text, m.start())
|
||||
pairs = re.findall(r'(\d+):"([0-9a-fA-F]{6,16})"', body)
|
||||
if pairs:
|
||||
maps.append((pubs, dict(pairs)))
|
||||
# Vite style: __vite__mapDeps map of "assets/xx.js"
|
||||
for fn in re.findall(r'"(assets/[^"]+\.js)"', js_text):
|
||||
maps.append((["/"], {"__vite__": fn}))
|
||||
return maps
|
||||
|
||||
def chunk_urls(base, js_text):
|
||||
"""Yield absolute chunk URLs reconstructable from this file's manifest(s)."""
|
||||
origin = origin_of(base)
|
||||
out = set()
|
||||
for pubs, idmap in parse_chunk_maps(js_text):
|
||||
paths = pubs or ["/assets/", "/"]
|
||||
for cid, h in idmap.items():
|
||||
if cid == "__vite__":
|
||||
fn = h # already "assets/xx.js"
|
||||
for pp in paths:
|
||||
out.add(urllib.parse.urljoin(origin + "/", fn))
|
||||
continue
|
||||
for pp in paths:
|
||||
if pp.startswith("http"): bareorigin = ""; prefix = pp
|
||||
else: bareorigin = origin; prefix = pp if pp.startswith("/") else "/"+pp
|
||||
if not prefix.endswith("/"): prefix += "/"
|
||||
out.add(f"{bareorigin}{prefix}{cid}.{h}.js")
|
||||
return out
|
||||
|
||||
# ---- endpoint / route extraction -------------------------------------------
|
||||
API_HINT = re.compile(r'/(?:api|rest|service|services|gateway|graphql|v\d|web|admin|backend|open)\b', re.I)
|
||||
ASSET_EXT = re.compile(r'\.(js|css|png|jpe?g|svg|gif|woff2?|ttf|ico|map|json|mp4|webp)(\?|$)', re.I)
|
||||
|
||||
def extract_endpoints(js_text):
|
||||
paths = set()
|
||||
# quoted ('/...'), double-quoted, and backtick template paths
|
||||
for m in re.findall(r'''["'`](/[A-Za-z0-9_\-./{}$:]+)["'`]''', js_text):
|
||||
paths.add(m)
|
||||
# concatenation heads: "/api/x/" + var
|
||||
for m in re.findall(r'''["'](/[A-Za-z0-9_\-./]+/)["']\s*\+''', js_text):
|
||||
paths.add(m)
|
||||
api, other = set(), set()
|
||||
for p in paths:
|
||||
if ASSET_EXT.search(p): continue
|
||||
if p.count('/') < 2 and not API_HINT.search(p): continue
|
||||
(api if API_HINT.search(p) else other).add(p)
|
||||
return api, other
|
||||
|
||||
def extract_routes(js_text):
|
||||
r = set()
|
||||
for key in ('path', 'to', 'redirect', 'href'):
|
||||
for m in re.findall(key + r'''\s*:\s*["'](/[A-Za-z0-9_\-/:]*)["']''', js_text):
|
||||
if not ASSET_EXT.search(m) and not API_HINT.search(m):
|
||||
r.add(m)
|
||||
return r
|
||||
|
||||
# ---- main -------------------------------------------------------------------
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
print("usage: harvest_static.py <BASE_URL> <OUTDIR>"); sys.exit(1)
|
||||
base, outdir = sys.argv[1], sys.argv[2]
|
||||
if not base.startswith("http"): base = "https://" + base
|
||||
jsdir = os.path.join(outdir, "js"); os.makedirs(jsdir, exist_ok=True)
|
||||
|
||||
print(f"[*] entry: {base}")
|
||||
html, status = fetch(base)
|
||||
open(os.path.join(outdir, "index.html"), "w").write(html)
|
||||
origin = origin_of(base)
|
||||
|
||||
# initial scripts from HTML
|
||||
srcs = set(re.findall(r'<script[^>]+src="([^"]+\.js[^"]*)"', html))
|
||||
srcs |= set(re.findall(r'(?:src|href)="([^"]*\.js)"', html))
|
||||
seed = set()
|
||||
for s in srcs:
|
||||
seed.add(s if s.startswith("http") else urllib.parse.urljoin(base, s))
|
||||
print(f"[*] {len(seed)} scripts referenced in HTML")
|
||||
|
||||
have = {} # url -> local path
|
||||
def dl(url):
|
||||
fn = os.path.basename(urllib.parse.urlparse(url).path)
|
||||
if not fn.endswith(".js"): return None
|
||||
lp = os.path.join(jsdir, fn)
|
||||
if url in have: return have[url]
|
||||
data, st = fetch(url, binary=True)
|
||||
if st == 200 and data and not data[:15].lstrip().startswith(b"<"):
|
||||
open(lp, "wb").write(data); have[url] = lp; return lp
|
||||
return None
|
||||
|
||||
with ThreadPoolExecutor(max_workers=20) as ex:
|
||||
list(ex.map(dl, seed))
|
||||
|
||||
# iteratively expand via chunk manifests (chunks reference more chunks)
|
||||
seen_urls = set(have.keys()); frontier = list(have.values())
|
||||
rounds = 0
|
||||
while frontier and rounds < 6:
|
||||
rounds += 1
|
||||
new_urls = set()
|
||||
for lp in frontier:
|
||||
try: txt = open(lp, encoding="utf-8", errors="ignore").read()
|
||||
except: continue
|
||||
for cu in chunk_urls(base, txt):
|
||||
if cu not in seen_urls: new_urls.add(cu)
|
||||
seen_urls |= new_urls
|
||||
if not new_urls: break
|
||||
print(f"[*] round {rounds}: {len(new_urls)} new chunk urls from manifest")
|
||||
before = set(have.values())
|
||||
with ThreadPoolExecutor(max_workers=24) as ex:
|
||||
list(ex.map(dl, new_urls))
|
||||
frontier = [p for p in have.values() if p not in before]
|
||||
|
||||
# retry-once any manifest chunk that 404'd (transient failures are real)
|
||||
all_manifest = set()
|
||||
for lp in list(have.values()):
|
||||
try: all_manifest |= chunk_urls(base, open(lp, encoding="utf-8", errors="ignore").read())
|
||||
except: pass
|
||||
missing = [u for u in all_manifest if u not in have]
|
||||
if missing:
|
||||
with ThreadPoolExecutor(max_workers=24) as ex:
|
||||
list(ex.map(dl, missing))
|
||||
still = [u for u in all_manifest if u not in have]
|
||||
print(f"[*] manifest chunks: {len(all_manifest)} | downloaded {len(have)} | "
|
||||
f"unreachable {len(still)} (CSS-only / undeployed)")
|
||||
|
||||
print(f"[+] total JS downloaded: {len(have)}")
|
||||
|
||||
# extract from everything
|
||||
api, other, routes = set(), set(), set()
|
||||
for lp in have.values():
|
||||
try: txt = open(lp, encoding="utf-8", errors="ignore").read()
|
||||
except: continue
|
||||
a, o = extract_endpoints(txt); api |= a; other |= o
|
||||
routes |= extract_routes(txt)
|
||||
|
||||
def dump(name, items):
|
||||
path = os.path.join(outdir, name)
|
||||
open(path, "w").write("\n".join(sorted(items)))
|
||||
return path
|
||||
dump("api_static.txt", api)
|
||||
dump("paths_other.txt", other)
|
||||
dump("routes.txt", routes)
|
||||
dump("chunkmap.txt", sorted(os.path.basename(u) for u in all_manifest))
|
||||
|
||||
print(f"[+] api endpoints: {len(api)} (api_static.txt)")
|
||||
print(f"[+] other paths: {len(other)} (paths_other.txt)")
|
||||
print(f"[+] route paths: {len(routes)} (routes.txt)")
|
||||
print(f"[+] outdir: {outdir}")
|
||||
print("\n[next] reverse the 3 gate facts (see reference.md), fill config.json, "
|
||||
"then: node runtime_harvest.js config.json")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+1011
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "api-recon-runtime",
|
||||
"version": "1.0.0",
|
||||
"private": true,
|
||||
"description": "Headless runtime API harvester for the api-recon skill",
|
||||
"dependencies": {
|
||||
"puppeteer-core": "^23.11.1"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,348 @@
|
||||
/**
|
||||
* api-recon coverage 模式预加载脚本 — 参考模板
|
||||
*
|
||||
* ⚠ 非通用成品：须按目标站点调整后再注入。
|
||||
* 常见改动：loginPathRe、stubs、neutralize.fields/success、apiPattern、mockTier、forward
|
||||
*
|
||||
* document-start 注入（CDP addScriptToEvaluateOnNewDocument 或 userscript）
|
||||
* CONFIG 字段应与 recon/config.json 保持一致
|
||||
*/
|
||||
(function () {
|
||||
'use strict';
|
||||
|
||||
const CONFIG = {
|
||||
loginPathRe: /\/(login|signin)(\/|$|\?)/i,
|
||||
mockTier: 'L1+L2',
|
||||
forward: true,
|
||||
neutralize: {
|
||||
fields: ['response_code', 'code', 'errno', 'ret', 'status'],
|
||||
success: 0,
|
||||
flags: { success: true, message: 'ok' },
|
||||
},
|
||||
stubs: [],
|
||||
apiPattern: /\/(api|apis|v\d+|dev|internal|graphql)\//i,
|
||||
apiFallbackRe: /^\/(api|apis|v\d+|dev|internal)\//i,
|
||||
// API 发现增强：fetch/XHR Hook、请求头与响应录制
|
||||
recordDetail: true, // 详细录制 method/url/headers/body/响应
|
||||
respMax: 600,
|
||||
extractUrlsFromResponse: true, // 从 JSON 响应里抠嵌套 URL
|
||||
neutralizeVueRouter: true, // Vue beforeEach/push 登录跳转中和
|
||||
observe: {
|
||||
storageReads: false, // 观察 localStorage.getItem（辅助确认会话键名）
|
||||
cookieReads: false, // 观察 document.cookie 读取
|
||||
xhrHeaders: true, // 录制 XHR setRequestHeader
|
||||
},
|
||||
};
|
||||
|
||||
window.__API_RECON_PRELOAD__ = true;
|
||||
window.__API_RECON_LOG__ = window.__API_RECON_LOG__ || new Set();
|
||||
window.__API_RECON_DETAIL__ = window.__API_RECON_DETAIL__ || [];
|
||||
window.__API_RECON_ROUTES__ = window.__API_RECON_ROUTES__ || new Set();
|
||||
window.__API_RECON_OBSERVE__ = window.__API_RECON_OBSERVE__ || { storage: [], headers: [] };
|
||||
|
||||
function trunc(s, n) {
|
||||
n = n || CONFIG.respMax || 600;
|
||||
s = String(s == null ? '' : s);
|
||||
return s.length > n ? s.slice(0, n) + '…' : s;
|
||||
}
|
||||
|
||||
function extractApiUrlsFromText(text) {
|
||||
if (!CONFIG.extractUrlsFromResponse || !text) return;
|
||||
const re = /["'](\/(?:api|apis|v\d+|dev|internal|graphql)[^"'`\\s]*)["'`]/gi;
|
||||
let m;
|
||||
while ((m = re.exec(text))) {
|
||||
const p = m[1].split('?')[0];
|
||||
window.__API_RECON_LOG__.add('GET ' + p);
|
||||
}
|
||||
const broad = /\/api\/[a-zA-Z0-9_./-]+/g;
|
||||
while ((m = broad.exec(text))) {
|
||||
const p = m[0].split('?')[0];
|
||||
if (!/\.(svg|png|jpg|gif|ico)$/i.test(p)) window.__API_RECON_LOG__.add('GET ' + p);
|
||||
}
|
||||
}
|
||||
|
||||
function recordApi(url, method) {
|
||||
const p = String(url || '').split('?')[0];
|
||||
if (CONFIG.apiPattern.test(p)) {
|
||||
window.__API_RECON_LOG__.add((method || 'GET').toUpperCase() + ' ' + p);
|
||||
}
|
||||
}
|
||||
|
||||
function recordApiDetail(entry) {
|
||||
recordApi(entry.url, entry.method);
|
||||
if (!CONFIG.recordDetail) return;
|
||||
window.__API_RECON_DETAIL__.push(entry);
|
||||
if (entry.responseBody) extractApiUrlsFromText(entry.responseBody);
|
||||
}
|
||||
|
||||
function blocked(url) {
|
||||
return url && CONFIG.loginPathRe.test(String(url));
|
||||
}
|
||||
|
||||
// --- 跳转中和 ---
|
||||
(function neutralizeNativeNavigation() {
|
||||
const rawAssign = Location.prototype.assign;
|
||||
const rawReplace = Location.prototype.replace;
|
||||
Location.prototype.assign = function (url) {
|
||||
if (blocked(url)) return;
|
||||
return rawAssign.call(this, url);
|
||||
};
|
||||
Location.prototype.replace = function (url) {
|
||||
if (blocked(url)) return;
|
||||
return rawReplace.call(this, url);
|
||||
};
|
||||
|
||||
const rawPush = history.pushState;
|
||||
const rawRep = history.replaceState;
|
||||
history.pushState = function (s, t, url) {
|
||||
if (blocked(url)) return;
|
||||
return rawPush.apply(this, arguments);
|
||||
};
|
||||
history.replaceState = function (s, t, url) {
|
||||
if (blocked(url)) return;
|
||||
return rawRep.apply(this, arguments);
|
||||
};
|
||||
|
||||
const hrefDesc = Object.getOwnPropertyDescriptor(Location.prototype, 'href');
|
||||
if (hrefDesc && hrefDesc.set) {
|
||||
const nativeSet = hrefDesc.set;
|
||||
Object.defineProperty(Location.prototype, 'href', {
|
||||
configurable: true,
|
||||
enumerable: hrefDesc.enumerable,
|
||||
get: hrefDesc.get,
|
||||
set(url) {
|
||||
if (blocked(url)) return;
|
||||
return nativeSet.call(this, url);
|
||||
},
|
||||
});
|
||||
}
|
||||
window.close = function () {};
|
||||
})();
|
||||
|
||||
// --- Vue Router 登录跳转中和 ---
|
||||
function neutralizeVueRouter() {
|
||||
if (!CONFIG.neutralizeVueRouter) return;
|
||||
try {
|
||||
const el = document.querySelector('[data-v-app]');
|
||||
const app = (el && el.__vue_app__) || window.__VUE__;
|
||||
if (!app) return;
|
||||
const router = app.config && app.config.globalProperties && app.config.globalProperties.$router;
|
||||
if (!router) return;
|
||||
router.beforeEach(function (to, from, next) { next(); });
|
||||
if (router.beforeResolve) router.beforeResolve(function (to, from, next) { next(); });
|
||||
function wrapNav(fn) {
|
||||
return function (loc) {
|
||||
const path = typeof loc === 'string' ? loc : (loc && (loc.path || loc.fullPath)) || '';
|
||||
if (blocked(path)) return Promise.resolve();
|
||||
return fn.apply(this, arguments);
|
||||
};
|
||||
}
|
||||
router.push = wrapNav(router.push.bind(router));
|
||||
router.replace = wrapNav(router.replace.bind(router));
|
||||
if (router.getRoutes) {
|
||||
router.getRoutes().forEach(function (r) {
|
||||
if (r.path) window.__API_RECON_ROUTES__.add(r.path);
|
||||
});
|
||||
}
|
||||
} catch (e) { /* ignore */ }
|
||||
}
|
||||
|
||||
function runPostLoadHooks() {
|
||||
neutralizeVueRouter();
|
||||
// 业务层跳转函数（goPage / navigateTo 等）
|
||||
['goPage', 'navigateTo', 'jumpTo', 'redirectTo'].forEach(function (name) {
|
||||
if (typeof window[name] !== 'function' || window[name].__apiReconWrapped) return;
|
||||
const raw = window[name];
|
||||
window[name] = function () {
|
||||
const arg = arguments[0];
|
||||
const path = typeof arg === 'string' ? arg : (arg && arg.path) || '';
|
||||
if (blocked(path)) return;
|
||||
return raw.apply(this, arguments);
|
||||
};
|
||||
window[name].__apiReconWrapped = true;
|
||||
});
|
||||
}
|
||||
document.addEventListener('DOMContentLoaded', runPostLoadHooks);
|
||||
window.addEventListener('load', runPostLoadHooks);
|
||||
|
||||
// --- 观察 Hook：辅助发现会话键名与请求头（可选，默认关 storage/cookie）---
|
||||
if (CONFIG.observe && CONFIG.observe.storageReads) {
|
||||
const rawGet = Storage.prototype.getItem;
|
||||
Storage.prototype.getItem = function (key) {
|
||||
window.__API_RECON_OBSERVE__.storage.push({ type: 'getItem', key: key, at: location.pathname });
|
||||
return rawGet.apply(this, arguments);
|
||||
};
|
||||
}
|
||||
if (CONFIG.observe && CONFIG.observe.cookieReads) {
|
||||
const cookieDesc = Object.getOwnPropertyDescriptor(Document.prototype, 'cookie');
|
||||
if (cookieDesc && cookieDesc.get) {
|
||||
const nativeGet = cookieDesc.get;
|
||||
Object.defineProperty(document, 'cookie', {
|
||||
configurable: true,
|
||||
get: function () {
|
||||
window.__API_RECON_OBSERVE__.storage.push({ type: 'cookieRead', at: location.pathname });
|
||||
return nativeGet.call(this);
|
||||
},
|
||||
set: cookieDesc.set,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// --- Mock 辅助 ---
|
||||
const NEGATIVE_RE = /未登录|未授权|授权|not\s*login|unauthorized|forbidden/i;
|
||||
const tier = CONFIG.mockTier || 'L1+L2';
|
||||
|
||||
function patchJsonBody(text) {
|
||||
if (!tier.includes('L2')) return text;
|
||||
try {
|
||||
const j = JSON.parse(text);
|
||||
if (j && typeof j === 'object') {
|
||||
const msg = String(j.message || j.msg || '');
|
||||
for (const f of CONFIG.neutralize.fields) {
|
||||
if (f in j && j[f] !== CONFIG.neutralize.success && NEGATIVE_RE.test(msg)) {
|
||||
j[f] = CONFIG.neutralize.success;
|
||||
}
|
||||
}
|
||||
Object.assign(j, CONFIG.neutralize.flags || {});
|
||||
if (j.data == null) j.data = {};
|
||||
return JSON.stringify(j);
|
||||
}
|
||||
} catch (e) { /* ignore */ }
|
||||
return text;
|
||||
}
|
||||
|
||||
function lookupPrecise(url) {
|
||||
if (!tier.includes('L1')) return null;
|
||||
const p = String(url || '');
|
||||
for (const s of CONFIG.stubs) {
|
||||
if (s._re ? s._re.test(p) : new RegExp(s.match).test(p)) return s.body;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function fallbackBody(url) {
|
||||
const p = String(url).split('?')[0];
|
||||
if (/\/(list|search|query|page|all|tree|nodes|options)/i.test(p)) {
|
||||
return { response_code: 0, data: [] };
|
||||
}
|
||||
if (/\/(get|detail|info|config|status)/i.test(p)) {
|
||||
return { response_code: 0, data: {} };
|
||||
}
|
||||
return { response_code: 0, data: {}, success: true };
|
||||
}
|
||||
|
||||
function matchMock(url) {
|
||||
const precise = lookupPrecise(url);
|
||||
if (precise) return precise;
|
||||
if (tier.includes('L3') && CONFIG.apiFallbackRe.test(String(url || '').split('?')[0])) {
|
||||
return fallbackBody(url);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
CONFIG.stubs.forEach(function (s) {
|
||||
if (s.match && !s._re) s._re = new RegExp(s.match);
|
||||
});
|
||||
|
||||
// --- fetch Hook ---
|
||||
const rawFetch = window.fetch;
|
||||
window.fetch = async function (input, init) {
|
||||
const url = typeof input === 'string' ? input : (input && input.url) || '';
|
||||
const method = ((init && init.method) || 'GET').toUpperCase();
|
||||
const reqHeaders = (init && init.headers) || {};
|
||||
const reqBody = init && init.body ? trunc(init.body) : null;
|
||||
|
||||
const mock = matchMock(url);
|
||||
if (mock && !CONFIG.forward) {
|
||||
recordApiDetail({ method: method, url: url, reqHeaders: reqHeaders, reqBody: reqBody, status: 200, responseBody: JSON.stringify(mock), source: 'mock' });
|
||||
return new Response(JSON.stringify(mock), {
|
||||
status: 200,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
}
|
||||
|
||||
const resp = await rawFetch.apply(this, arguments);
|
||||
const clone = resp.clone();
|
||||
let text = '';
|
||||
try { text = await clone.text(); } catch (e) { /* ignore */ }
|
||||
|
||||
recordApiDetail({
|
||||
method: method, url: url, reqHeaders: reqHeaders, reqBody: reqBody,
|
||||
status: resp.status, responseBody: trunc(text), source: 'fetch',
|
||||
});
|
||||
|
||||
if (!tier.includes('L2') && !mock) return resp;
|
||||
|
||||
const patched = patchJsonBody(text);
|
||||
if (patched !== text) {
|
||||
return new Response(patched, {
|
||||
status: resp.status,
|
||||
statusText: resp.statusText,
|
||||
headers: resp.headers,
|
||||
});
|
||||
}
|
||||
return resp;
|
||||
};
|
||||
|
||||
// --- XHR Hook ---
|
||||
const rawOpen = XMLHttpRequest.prototype.open;
|
||||
const rawSend = XMLHttpRequest.prototype.send;
|
||||
const rawSetHeader = XMLHttpRequest.prototype.setRequestHeader;
|
||||
|
||||
XMLHttpRequest.prototype.open = function (method, url) {
|
||||
this.__apiReconUrl = url;
|
||||
this.__apiReconMethod = method;
|
||||
this.__apiReconHeaders = {};
|
||||
return rawOpen.apply(this, arguments);
|
||||
};
|
||||
|
||||
if (CONFIG.observe && CONFIG.observe.xhrHeaders) {
|
||||
XMLHttpRequest.prototype.setRequestHeader = function (name, value) {
|
||||
if (!this.__apiReconHeaders) this.__apiReconHeaders = {};
|
||||
this.__apiReconHeaders[name] = value;
|
||||
window.__API_RECON_OBSERVE__.headers.push({ name: name, url: this.__apiReconUrl });
|
||||
return rawSetHeader.apply(this, arguments);
|
||||
};
|
||||
}
|
||||
|
||||
XMLHttpRequest.prototype.send = function (body) {
|
||||
const xhr = this;
|
||||
const url = xhr.__apiReconUrl || '';
|
||||
const method = (xhr.__apiReconMethod || 'GET').toUpperCase();
|
||||
const mock = matchMock(url);
|
||||
|
||||
if (mock && !CONFIG.forward) {
|
||||
const bodyStr = JSON.stringify(mock);
|
||||
recordApiDetail({ method: method, url: url, reqHeaders: xhr.__apiReconHeaders, reqBody: trunc(body), status: 200, responseBody: bodyStr, source: 'mock' });
|
||||
setTimeout(function () {
|
||||
Object.defineProperty(xhr, 'readyState', { configurable: true, get: function () { return 4; } });
|
||||
Object.defineProperty(xhr, 'status', { configurable: true, get: function () { return 200; } });
|
||||
Object.defineProperty(xhr, 'responseText', { configurable: true, get: function () { return bodyStr; } });
|
||||
xhr.onreadystatechange && xhr.onreadystatechange();
|
||||
xhr.onload && xhr.onload();
|
||||
}, 0);
|
||||
return;
|
||||
}
|
||||
|
||||
const orig = xhr.onreadystatechange;
|
||||
xhr.onreadystatechange = function () {
|
||||
if (xhr.readyState === 4) {
|
||||
recordApiDetail({
|
||||
method: method, url: url, reqHeaders: xhr.__apiReconHeaders,
|
||||
reqBody: trunc(body), status: xhr.status,
|
||||
responseBody: trunc(xhr.responseText), source: 'xhr',
|
||||
});
|
||||
if (tier.includes('L2') && xhr.responseText) {
|
||||
try {
|
||||
const patched = patchJsonBody(xhr.responseText);
|
||||
if (patched !== xhr.responseText) {
|
||||
Object.defineProperty(xhr, 'responseText', { configurable: true, get: function () { return patched; } });
|
||||
}
|
||||
} catch (e) { /* ignore */ }
|
||||
}
|
||||
}
|
||||
orig && orig.apply(this, arguments);
|
||||
};
|
||||
return rawSend.apply(this, arguments);
|
||||
};
|
||||
})();
|
||||
@@ -0,0 +1,228 @@
|
||||
#!/usr/bin/env node
|
||||
/*
|
||||
* runtime_harvest.js <config.json>
|
||||
*
|
||||
* 参考模板 — 非通用成品。执行前须按目标站点调整 config.json 及脚本内逻辑：
|
||||
* cookies/localStorage、neutralize 字段、stubs 结构、loginUrlPattern、apiPattern
|
||||
*
|
||||
* Drives a headless browser through an authorized SPA to capture the live API
|
||||
* surface (method + url + body) by defeating three client-side gates:
|
||||
* 1. render gate -> inject fake auth state (cookies / localStorage)
|
||||
* 2. interceptor gate-> rewrite the "unauthorized" code field to success
|
||||
* 3. content gate -> stub the menu/permission endpoint with a full-feature payload
|
||||
*
|
||||
* Requires puppeteer-core + a system Chromium. ( cd scripts && npm install )
|
||||
*
|
||||
* config.json schema (all fields optional except baseUrl):
|
||||
* {
|
||||
* "baseUrl": "https://target/",
|
||||
* "chromium": "/usr/bin/chromium", // or env CHROMIUM
|
||||
* "cookies": [{"name":"auth","value":"b64json:{\"id\":1,\"username\":\"admin\"}"}],
|
||||
* "localStorage": {"token":"x","isLogin":"1"},
|
||||
* "neutralize": { "fields":["response_code","code","errno","ret"], "success":0,
|
||||
* "flags":{"success":true,"message":"ok"} },
|
||||
* "forward": true, // forward real req then rewrite code; false = offline stub
|
||||
* "loginUrlPattern": "/login", // navigations matching this are suppressed as a fallback
|
||||
* "stubs": [ {"match":"permission|menu|role", "body": { ...full-feature menu... }} ],
|
||||
* "routes": ["/dashboard","/device", ...], // from routes.txt or the forged menu
|
||||
* "apiPattern": "/api/|/rest/|/graphql", // what counts as an API call to record
|
||||
* "proxy": "http://127.0.0.1:8080", // optional; also HTTP_PROXY / HTTPS_PROXY
|
||||
* "waitUntil": "domcontentloaded", // prefer over networkidle2 for large SPAs
|
||||
* "routeTimeout": 12000,
|
||||
* "waitMs": 1200, "perRouteMs": 900, "headless": true
|
||||
* }
|
||||
*/
|
||||
process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
let puppeteer;
|
||||
try { puppeteer = require('puppeteer-core'); }
|
||||
catch (e) { console.error("[!] run `npm install` in the scripts/ dir first (needs puppeteer-core)"); process.exit(1); }
|
||||
|
||||
function loadCfg(p) {
|
||||
const c = JSON.parse(fs.readFileSync(p, 'utf8'));
|
||||
c.baseUrl || (() => { throw new Error("config.baseUrl required"); })();
|
||||
c.chromium = c.chromium || process.env.CHROMIUM || '/usr/bin/chromium';
|
||||
c.neutralize = c.neutralize || { fields: ['response_code', 'code', 'errno', 'ret', 'status'], success: 0, flags: { success: true } };
|
||||
c.forward = c.forward !== false;
|
||||
c.apiPattern = new RegExp(c.apiPattern || '/api/|/rest/|/service/|/graphql|/gateway/');
|
||||
c.loginUrlPattern = c.loginUrlPattern || '/login';
|
||||
c.routes = c.routes || ['/'];
|
||||
c.waitMs = c.waitMs || 1200; c.perRouteMs = c.perRouteMs || 900;
|
||||
c.headless = c.headless !== false;
|
||||
c.waitUntil = c.waitUntil || 'domcontentloaded';
|
||||
c.routeTimeout = c.routeTimeout || 12000;
|
||||
c.captureResponses = c.captureResponses !== false; // record response body samples (forward mode)
|
||||
c.recordWs = c.recordWs !== false; // record WebSocket frames + SSE endpoints
|
||||
c.respMax = c.respMax || 600; // truncation length for captured bodies
|
||||
c.proxy = c.proxy || process.env.HTTP_PROXY || process.env.HTTPS_PROXY || '';
|
||||
(c.stubs || []).forEach(s => s._re = new RegExp(s.match));
|
||||
return c;
|
||||
}
|
||||
|
||||
function cookieValue(v) {
|
||||
if (typeof v === 'string' && v.startsWith('b64json:'))
|
||||
return Buffer.from(v.slice(8)).toString('base64');
|
||||
if (typeof v === 'string' && v.startsWith('json:'))
|
||||
return v.slice(5);
|
||||
return v;
|
||||
}
|
||||
|
||||
function neutralize(txt, n) {
|
||||
try {
|
||||
const j = JSON.parse(txt);
|
||||
if (j && typeof j === 'object') {
|
||||
for (const f of n.fields) if (f in j) j[f] = n.success;
|
||||
Object.assign(j, n.flags || {});
|
||||
return JSON.stringify(j);
|
||||
}
|
||||
} catch (e) {}
|
||||
return txt;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const cfg = loadCfg(process.argv[2] || 'config.json');
|
||||
const origin = new URL(cfg.baseUrl).origin;
|
||||
const host = new URL(cfg.baseUrl).hostname;
|
||||
const rec = []; // {m,u,b,resp,ct}
|
||||
const chunks = new Set();
|
||||
const ws = []; // {url,dir,data} WebSocket frames
|
||||
const sse = new Set(); // SSE (text/event-stream) endpoints
|
||||
|
||||
if (cfg.proxy) {
|
||||
process.env.HTTP_PROXY = cfg.proxy;
|
||||
process.env.HTTPS_PROXY = cfg.proxy;
|
||||
}
|
||||
const launchArgs = ['--no-sandbox', '--disable-dev-shm-usage', '--ignore-certificate-errors'];
|
||||
if (cfg.proxy) launchArgs.push(`--proxy-server=${cfg.proxy}`);
|
||||
const browser = await puppeteer.launch({
|
||||
executablePath: cfg.chromium, headless: cfg.headless ? 'new' : false,
|
||||
args: launchArgs
|
||||
});
|
||||
const page = await browser.newPage();
|
||||
|
||||
// ---- WebSocket frame capture via CDP (fetch/XHR interception can't see WS) ----
|
||||
if (cfg.recordWs) {
|
||||
try {
|
||||
const cdp = await page.target().createCDPSession();
|
||||
await cdp.send('Network.enable');
|
||||
const wsUrl = {}; // requestId -> url
|
||||
cdp.on('Network.webSocketCreated', e => { wsUrl[e.requestId] = e.url; });
|
||||
const onFrame = dir => e => {
|
||||
const p = e.response && e.response.payloadData;
|
||||
if (p != null) ws.push({ url: (wsUrl[e.requestId] || '').replace(origin, ''), dir, data: String(p).slice(0, cfg.respMax) });
|
||||
};
|
||||
cdp.on('Network.webSocketFrameSent', onFrame('send'));
|
||||
cdp.on('Network.webSocketFrameReceived', onFrame('recv'));
|
||||
} catch (e) { console.log('[ws] CDP capture unavailable:', e.message); }
|
||||
}
|
||||
|
||||
// inject localStorage on every document
|
||||
if (cfg.localStorage) {
|
||||
await page.evaluateOnNewDocument((kv) => {
|
||||
try { for (const k in kv) localStorage.setItem(k, kv[k]); } catch (e) {}
|
||||
}, cfg.localStorage);
|
||||
}
|
||||
// fallback: block full-page navigations to the login url
|
||||
await page.evaluateOnNewDocument((pat) => {
|
||||
const bad = u => { try { return String(u).indexOf(pat) >= 0; } catch (e) { return false; } };
|
||||
try {
|
||||
const d = Object.getOwnPropertyDescriptor(Location.prototype, 'href');
|
||||
Object.defineProperty(Location.prototype, 'href', { configurable: true,
|
||||
get() { return d.get.call(this); },
|
||||
set(v) { if (bad(v)) return; return d.set.call(this, v); } });
|
||||
const a = Location.prototype.assign, r = Location.prototype.replace;
|
||||
Location.prototype.assign = function (v) { if (bad(v)) return; return a.call(this, v); };
|
||||
Location.prototype.replace = function (v) { if (bad(v)) return; return r.call(this, v); };
|
||||
} catch (e) {}
|
||||
}, cfg.loginUrlPattern);
|
||||
|
||||
await page.setRequestInterception(true);
|
||||
page.on('request', async (req) => {
|
||||
const u = req.url(), m = req.method();
|
||||
if (/\.js(\?|$)/.test(u) && /\/assets\/|\/static\/|\/js\//.test(u)) chunks.add(u.split('/').pop());
|
||||
|
||||
// suppress fallback login navigations (after the app has bootstrapped)
|
||||
if (req.isNavigationRequest() && req.frame() === page.mainFrame()
|
||||
&& u.includes(cfg.loginUrlPattern) && rec.length > 3) {
|
||||
return req.respond({ status: 204, body: '' });
|
||||
}
|
||||
if (!cfg.apiPattern.test(u)) return req.continue();
|
||||
|
||||
const entry = { m, u: u.replace(origin, '').split('?')[0], full: u, b: req.postData() ? req.postData().slice(0, 400) : null };
|
||||
rec.push(entry);
|
||||
|
||||
// explicit stubs (menu / permission forgery) win
|
||||
const stub = (cfg.stubs || []).find(s => s._re.test(u));
|
||||
if (stub) return req.respond({ status: 200, contentType: 'application/json', body: JSON.stringify(stub.body) });
|
||||
|
||||
// SSE: forwarding a text/event-stream would hang the handler — record + short-circuit
|
||||
const accept = (req.headers().accept || '');
|
||||
if (/text\/event-stream/.test(accept)) { sse.add(entry.u); return req.respond({ status: 200, contentType: 'application/json', body: '{}' }); }
|
||||
|
||||
if (!cfg.forward) {
|
||||
return req.respond({ status: 200, contentType: 'application/json',
|
||||
body: neutralize('{"data":{},"list":[],"total":0}', cfg.neutralize) });
|
||||
}
|
||||
// forward real request, then rewrite the unauthorized code field
|
||||
try {
|
||||
const headers = Object.assign({}, req.headers());
|
||||
if (cfg.cookies) headers.cookie = cfg.cookies.map(c => `${c.name}=${cookieValue(c.value)}`).join('; ');
|
||||
const r = await fetch(u, { method: m, headers, body: (m !== 'GET' && m !== 'HEAD') ? req.postData() : undefined });
|
||||
const t = await r.text();
|
||||
if (cfg.captureResponses) { entry.resp = t.slice(0, cfg.respMax); entry.ct = r.headers.get('content-type') || ''; }
|
||||
req.respond({ status: 200, contentType: 'application/json', body: neutralize(t, cfg.neutralize) });
|
||||
} catch (e) {
|
||||
req.respond({ status: 200, contentType: 'application/json', body: '{"response_code":0,"code":0,"data":{}}' });
|
||||
}
|
||||
});
|
||||
|
||||
// set auth cookies
|
||||
if (cfg.cookies) {
|
||||
for (const c of cfg.cookies)
|
||||
await page.setCookie({ name: c.name, value: cookieValue(c.value), domain: host, path: c.path || '/' });
|
||||
}
|
||||
|
||||
// boot
|
||||
await page.goto(cfg.baseUrl, { waitUntil: cfg.waitUntil, timeout: 45000 }).catch(e => console.log('[goto]', e.message));
|
||||
await new Promise(r => setTimeout(r, cfg.waitMs));
|
||||
const shell = await page.evaluate(() => ({
|
||||
url: location.href,
|
||||
loginForm: !!document.querySelector('input[type=password]'),
|
||||
links: [...document.querySelectorAll('a[href^="/"]')].map(a => a.getAttribute('href'))
|
||||
}));
|
||||
console.log(`[*] after boot: url=${shell.url} loginForm=${shell.loginForm}`);
|
||||
if (shell.loginForm) console.log('[!] still on login — recheck render-gate facts (cookies/localStorage) in config');
|
||||
|
||||
// discovered menu links extend the route list
|
||||
const routes = [...new Set([...cfg.routes, ...shell.links.filter(h => h && h.length > 1)])];
|
||||
const perRoute = {};
|
||||
for (const rt of routes) {
|
||||
const before = rec.length;
|
||||
try { await page.goto(origin + rt, { waitUntil: cfg.waitUntil, timeout: cfg.routeTimeout }); } catch (e) {}
|
||||
await new Promise(r => setTimeout(r, cfg.perRouteMs));
|
||||
const fresh = [...new Set(rec.slice(before).map(r => r.m + ' ' + r.u))];
|
||||
perRoute[rt] = fresh;
|
||||
}
|
||||
|
||||
const uniq = [...new Set(rec.map(r => r.m + ' ' + r.u))].sort();
|
||||
const wsUniq = [...new Set(ws.map(f => f.url))].filter(Boolean).sort();
|
||||
const outdir = path.dirname(process.argv[2] || '.');
|
||||
fs.writeFileSync(path.join(outdir, 'runtime_api.json'),
|
||||
JSON.stringify({ shell, uniq, perRoute, rec, ws, wsEndpoints: wsUniq, sse: [...sse] }, null, 2));
|
||||
|
||||
// merge with static
|
||||
let merged = new Set(uniq.map(x => x.split(' ')[1]));
|
||||
const staticFile = path.join(outdir, 'api_static.txt');
|
||||
if (fs.existsSync(staticFile)) fs.readFileSync(staticFile, 'utf8').split('\n').filter(Boolean).forEach(p => merged.add(p));
|
||||
fs.writeFileSync(path.join(outdir, 'api_merged.txt'), [...merged].sort().join('\n'));
|
||||
|
||||
console.log(`[+] runtime endpoints (with method): ${uniq.length}`);
|
||||
console.log(`[+] chunks seen at runtime: ${chunks.size}`);
|
||||
if (cfg.recordWs) console.log(`[+] websocket endpoints: ${wsUniq.length} (${ws.length} frames) | SSE endpoints: ${sse.size}`);
|
||||
if (cfg.captureResponses) console.log(`[+] response bodies captured for ${rec.filter(r => r.resp != null).length}/${rec.length} calls`);
|
||||
console.log(`[+] merged unique paths: ${merged.size} -> api_merged.txt`);
|
||||
console.log(`[+] full detail -> runtime_api.json (per-route + bodies + ws frames + sse)`);
|
||||
await browser.close();
|
||||
})();
|
||||
@@ -0,0 +1,123 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4]
|
||||
|
||||
参考模板 — 非通用成品。执行前须按目标调整 --exclude、cookie、depth/max 等同域策略。
|
||||
|
||||
Fallback for NON-SPA targets (traditional server-rendered MPAs: Django/Rails/PHP/
|
||||
JSP, classic admin panels). When there is no JS endpoint bundle, the API surface
|
||||
lives in HTML <form action>, <a href>, and inline-JS ajax urls. This BFS-crawls
|
||||
the same origin and extracts:
|
||||
- forms.txt : METHOD action [param1, param2, ...] (the real "endpoints")
|
||||
- links.txt : every same-origin URL reached
|
||||
- api_inline.txt : url-ish strings found in inline <script> / onclick (fetch/ajax)
|
||||
|
||||
Stdlib only. Same-origin, bounded, polite. Provide --cookie for an authed crawl.
|
||||
"""
|
||||
import sys, os, re, ssl, argparse, urllib.request, urllib.parse, time
|
||||
from html.parser import HTMLParser
|
||||
from collections import deque
|
||||
|
||||
CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
|
||||
UA = "Mozilla/5.0 (spa-api-recon spider)"
|
||||
SKIP_EXT = re.compile(r'\.(css|png|jpe?g|gif|svg|ico|woff2?|ttf|pdf|zip|mp4|webp|js)(\?|$)', re.I)
|
||||
API_HINT = re.compile(r'/(?:api|rest|service|ajax|action|do|rpc|graphql|v\d|admin|backend)\b', re.I)
|
||||
|
||||
def fetch(url, cookie):
|
||||
h = {"User-Agent": UA}
|
||||
if cookie: h["Cookie"] = cookie
|
||||
try:
|
||||
with urllib.request.urlopen(urllib.request.Request(url, headers=h), context=CTX, timeout=20) as r:
|
||||
ct = r.headers.get("Content-Type", "")
|
||||
if "html" not in ct and "xml" not in ct: return "", ct
|
||||
return r.read().decode("utf-8", "ignore"), ct
|
||||
except Exception:
|
||||
return "", ""
|
||||
|
||||
class Page(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.links, self.scripts, self.inline = [], [], []
|
||||
self.forms, self._cur = [], None
|
||||
self._in_script = False
|
||||
def handle_starttag(self, tag, attrs):
|
||||
a = dict(attrs)
|
||||
if tag == "a" and a.get("href"): self.links.append(a["href"])
|
||||
elif tag == "script":
|
||||
self._in_script = True
|
||||
if a.get("src"): self.scripts.append(a["src"])
|
||||
elif tag == "form":
|
||||
self._cur = {"action": a.get("action", ""), "method": (a.get("method") or "GET").upper(), "params": []}
|
||||
elif tag in ("input", "select", "textarea", "button") and self._cur is not None:
|
||||
n = a.get("name")
|
||||
if n: self._cur["params"].append(n)
|
||||
# ajax-ish handlers
|
||||
for v in a.values():
|
||||
for m in re.findall(r'''["'](/[^"']{2,120})["']''', v or ""):
|
||||
if API_HINT.search(m): self.inline.append(m)
|
||||
def handle_endtag(self, tag):
|
||||
if tag == "script": self._in_script = False
|
||||
elif tag == "form" and self._cur is not None:
|
||||
self.forms.append(self._cur); self._cur = None
|
||||
def handle_data(self, data):
|
||||
if self._in_script and data:
|
||||
for m in re.findall(r'''["'`](/[A-Za-z0-9_\-./{}$:?=&]{2,120})["'`]''', data):
|
||||
if API_HINT.search(m): self.inline.append(m)
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("base"); ap.add_argument("outdir")
|
||||
ap.add_argument("--cookie", default=""); ap.add_argument("--max", type=int, default=200)
|
||||
ap.add_argument("--depth", type=int, default=4); ap.add_argument("--delay", type=float, default=0.1)
|
||||
ap.add_argument("--include", default=""); ap.add_argument("--exclude", default="logout|signout|delete|remove|destroy")
|
||||
args = ap.parse_args()
|
||||
base = args.base if args.base.startswith("http") else "https://" + args.base
|
||||
os.makedirs(args.outdir, exist_ok=True)
|
||||
origin = "{0.scheme}://{0.netloc}".format(urllib.parse.urlparse(base))
|
||||
inc = re.compile(args.include) if args.include else None
|
||||
exc = re.compile(args.exclude, re.I) if args.exclude else None
|
||||
|
||||
seen, forms, inline, links = set(), {}, set(), set()
|
||||
q = deque([(base, 0)]); seen.add(base.split("#")[0])
|
||||
n = 0
|
||||
while q and n < args.max:
|
||||
url, d = q.popleft(); n += 1
|
||||
html, ct = fetch(url, args.cookie)
|
||||
if args.delay: time.sleep(args.delay)
|
||||
if not html: continue
|
||||
p = Page()
|
||||
try: p.feed(html)
|
||||
except Exception: pass
|
||||
links.add(url.replace(origin, "") or "/")
|
||||
for f in p.forms:
|
||||
act = urllib.parse.urljoin(url, f["action"] or url)
|
||||
key = f["method"] + " " + act.replace(origin, "")
|
||||
forms.setdefault(key, set()).update(f["params"])
|
||||
for s in p.inline: inline.add(s)
|
||||
if d < args.depth:
|
||||
for href in p.links:
|
||||
if href.startswith(("mailto:", "tel:", "javascript:", "#")): continue
|
||||
nxt = urllib.parse.urljoin(url, href).split("#")[0]
|
||||
if not nxt.startswith(origin): continue
|
||||
if SKIP_EXT.search(nxt): continue
|
||||
if exc and exc.search(nxt): continue
|
||||
if inc and not inc.search(nxt): continue
|
||||
if nxt not in seen:
|
||||
seen.add(nxt); q.append((nxt, d + 1))
|
||||
|
||||
with open(os.path.join(args.outdir, "forms.txt"), "w") as fh:
|
||||
for k in sorted(forms):
|
||||
ps = ", ".join(sorted(forms[k]))
|
||||
fh.write(f"{k} [{ps}]\n")
|
||||
open(os.path.join(args.outdir, "links.txt"), "w").write("\n".join(sorted(links)))
|
||||
open(os.path.join(args.outdir, "api_inline.txt"), "w").write("\n".join(sorted(inline)))
|
||||
|
||||
print(f"[+] crawled {n} pages (cap {args.max}, depth {args.depth})")
|
||||
print(f"[+] forms (endpoints): {len(forms)} -> forms.txt")
|
||||
print(f"[+] inline ajax urls: {len(inline)} -> api_inline.txt")
|
||||
print(f"[+] pages reached: {len(links)} -> links.txt")
|
||||
if not args.cookie:
|
||||
print("[i] no --cookie: only public pages crawled. Pass an authed session cookie for the full surface.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user