Compare commits
6
Commits
c633789a92
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5fdc3df22d | ||
|
|
7ffab24025 | ||
|
|
25aee35030 | ||
|
|
7636456d4e | ||
|
|
937e688cfa | ||
|
|
3e7719f200 |
+17
-2
@@ -45,8 +45,23 @@ __pycache__/
|
|||||||
# 本地凭据与一次性状态(测试账号密码等),只留在本机,不进版本库。
|
# 本地凭据与一次性状态(测试账号密码等),只留在本机,不进版本库。
|
||||||
# 目录说明见 .workbuddy/local/test-accounts.md 的抬头注释。
|
# 目录说明见 .workbuddy/local/test-accounts.md 的抬头注释。
|
||||||
/.workbuddy/local/
|
/.workbuddy/local/
|
||||||
|
/.codeartsdoer/
|
||||||
|
|
||||||
# WSL 测试环境的 app.ini 模板:含数据库密码与对象存储密钥,绝不能进版本库。
|
# 本地运行配置与 WSL 测试环境模板:含数据库密码与对象存储密钥,绝不能进版本库。
|
||||||
# 用法见 dev/sync-wsl.sh(同步时必须排除 conf/app.ini,否则会被 rsync --delete 删掉)。
|
# dev/wsl-app.ini 的用法见 dev/sync-wsl.sh;同步时必须排除 conf/app.ini,避免被 rsync --delete 删掉。
|
||||||
|
/conf/app.ini
|
||||||
/dev/wsl-app.ini
|
/dev/wsl-app.ini
|
||||||
/dev/_*.py
|
/dev/_*.py
|
||||||
|
|
||||||
|
# 含真实测试凭据的历史记录与巡检脚本:保留本地文件,但不再纳入版本控制。
|
||||||
|
/.workbuddy/memory/MEMORY.md
|
||||||
|
/.workbuddy/memory/topics/build.md
|
||||||
|
/.workbuddy/memory/topics/sec.md
|
||||||
|
/.workbuddy/memory/topics/misc.md
|
||||||
|
/.workbuddy/memory/2026-09-15.md
|
||||||
|
/.workbuddy/memory/2026-10-19.md
|
||||||
|
/.workbuddy/memory/2026-10-22.md
|
||||||
|
/dev/devcheck-e2e.py
|
||||||
|
/dev/devcheck-avatar.py
|
||||||
|
/dev/devcheck-image.py
|
||||||
|
/dev/devcheck-upload.py
|
||||||
|
|||||||
@@ -1,421 +0,0 @@
|
|||||||
# 2026-09-15 工作日志
|
|
||||||
|
|
||||||
## 落地 P2 推荐批次 #1–3(审核链路一致性 / 文档)
|
|
||||||
|
|
||||||
承接 09-11 的全库审阅(无 P0/P1,8 项 P2)。上次征询修复批次未获选择,本次按「继续 +
|
|
||||||
我标注的建议先做」自主推进**最低风险的一致性/文档类 #1–3**,未动 #4–8(需另行确认)。
|
|
||||||
|
|
||||||
### 已改(`internal/security/censor/`)
|
|
||||||
|
|
||||||
1. **#1 缓存/送审归一化统一**(censor.go:66-68,76,122,131)
|
|
||||||
- 原先:缓存按**原文** `text` 查 `GetByText`,实际送审 `RemoveTrustedURL(text)`,落库 `Input:text`。
|
|
||||||
→ 审计记录与实际送审内容不符;归一化后相同的原文会各占一条缓存、重复调 API。
|
|
||||||
- 现在:循环外算一次 `normalized := RemoveTrustedURL(text)`,缓存查找 / 送审 / `Input` 三者统一。
|
|
||||||
- 安全性论证:归一化是纯函数确定性映射;`TrustedHosts` 增减只会让旧条目 miss(重新送审),
|
|
||||||
不会让「已通过」的旧缓存被错配到新文本上 → 无漏放风险。
|
|
||||||
2. **#2 阿里云缓存回放分支与七牛对齐**(censor.go:99-110)
|
|
||||||
- 原先:aliyun 命中缓存**不看 Pass 直接 return**,依赖「aliyun 恒为链尾」这一隐式前提。
|
|
||||||
- 现在:与 qiniu 一致 —— 仅 `Pass` 才 return,否则 append + continue 交给下一通道复核。
|
|
||||||
- 等价性:aliyun 现为链尾,continue 后循环结束仍由 `responses[len-1]` 返回**同一份响应**,行为不变;
|
|
||||||
但消除了顺序耦合,将来若前置 aliyun 也不会短路后续通道。
|
|
||||||
3. **#3 修正失真注释**(text.go:66)
|
|
||||||
- 原注释称 `Details`「回放缓存结果时为 nil」**是错的**:
|
|
||||||
`AliyunTextCensorParser`(aliyun.go:433-461) 与 `QiniuTextCensorParser`(qiniu.go:133-170)
|
|
||||||
都会从 `RawResponse` 重建 `Details` → 缓存回放同样受严格度策略约束,改档位对历史缓存也生效。
|
|
||||||
- 已改为准确描述。**结论:审核缓存回放无 fail-open**(09-11 的怀疑已排除)。
|
|
||||||
|
|
||||||
### 验证
|
|
||||||
|
|
||||||
- `gofmt_keep_crlf.py` 两个文件 exit=0,CRLF 保留(censor 147 / text 141,loneLF=0)。
|
|
||||||
- `go build ./internal/security/censor/...` **EXIT=0**(定向构建,~秒级;用 goproxy.cn + overlay)。
|
|
||||||
- `go test ./internal/security/censor/` **ok 2.467s** —— 确认严格度策略与回放行为未被破坏。
|
|
||||||
|
|
||||||
### 顺带已完成 #6
|
|
||||||
|
|
||||||
- **#6 `User.Email` 改 `json:"-"`**(users.go:89)。
|
|
||||||
查证:`route/` 与 `internal/` 中**无任何非测试代码读取序列化后的 `.Email`**(模板里的 email 均为表单字段名 /
|
|
||||||
邮件模板,属 Go template 服务端渲染,不受 json tag 影响;输入走 `CreateUserOptions`/`UpdateUserOptions`,
|
|
||||||
不经 `User` 模型反序列化)→ 无消费方受损。
|
|
||||||
验证:gofmt exit=0,`go build ./internal/db/` **EXIT=0**。
|
|
||||||
|
|
||||||
### 待确认(P2 #4/#5/#7/#8)
|
|
||||||
|
|
||||||
| # | 项 | 我的建议 |
|
|
||||||
|---|---|---|
|
|
||||||
| 4 | 投票配额「插入后计数」非原子(route/vote/vote.go:814-822) | 暂缓:需 DB 层条件插入,超额有界且自动结束自愈 |
|
|
||||||
| 5 | ~~profile Name/Intro/link 无长度校验~~ | **误报,已关闭**:表单层已有校验,见下方更正 |
|
|
||||||
| 7 | `conf.Database.Schema` 拼 CREATE SEQUENCE(db.go:215-222) | 可选:配置来源非远端输入,纵深防御 |
|
|
||||||
| 8 | `RemoveTrustedURL` 24 次整串重建(text.go:114-141) | 可跳过:短文本开销可忽略(#1 已顺带把调用从循环内降到 1 次) |
|
|
||||||
|
|
||||||
#### ✅ #5 更正:是**误报**,已按「报错式校验」实现后**全部回滚**
|
|
||||||
|
|
||||||
- 先按用户选择实现了一版(Name 50 / Intro 500 / link 500 rune,4 语 i18n 词条),随后核对
|
|
||||||
`internal/form/user.go` 时发现**表单层早已有校验**:
|
|
||||||
- `Name` → `valid:"required;maxlen:20"`
|
|
||||||
- `Intro` → `valid:"required;maxlen:100"`
|
|
||||||
- 各 link → `valid:"maxlen:30"`
|
|
||||||
- 且确实生效:`internal/form/form.go:16` 引入 `govalid`,`:131` 在 `Bind()` 中 `govalid.Check()`;
|
|
||||||
handler 顶部 `ctx.HasError()` 承接;报错文案走标签 `msg:` 对应的 i18n 键
|
|
||||||
(`form.profile.name`="不超过 20 个字符"、`form.profile.intro`=100、`form.profile.linkMax`=30)。
|
|
||||||
- 结论:我那版是**永不触发的死代码**(超限在 Bind 阶段就被拒),且上限与真实值冲突 → 已完整回滚
|
|
||||||
(profile.go 的 utf8 导入 / 常量 / 校验块 + 4 份字典词条),`git status` 确认 5 个文件零 diff。
|
|
||||||
- 教训已写入 MEMORY.md「表单校验(勿重复造轮子)」:查某字段有无校验,**先看 `internal/form/*.go` 的 struct 标签**。
|
|
||||||
- 附带修正:Postgres 下 `string` 映射为 `text`(无 DB 层长度上限)这一点本身**没错**,
|
|
||||||
但不构成漏洞 —— 因为表单层已经挡住了。
|
|
||||||
|
|
||||||
### 环境
|
|
||||||
|
|
||||||
- ~~重包 `./route/user/ ./internal/route/ ./internal/db/` 仍会 30 分钟超时 → 判定为 1C1G 沙箱编译整图慢~~
|
|
||||||
**❌ 该结论已被推翻,见文末「编译卡死」误判更正。全库 `go build ./...` 实测 EXIT=0。**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 「编译卡死」误判更正(用户指出:1C1G 是正式环境,不是沙箱)
|
|
||||||
|
|
||||||
用户明确纠正:**1C1G 是正式生产环境**,我不该拿「沙箱慢」当挡箭牌解释构建超时。重新取证后确认:
|
|
||||||
**根本没有编译问题,是我把代理挂起误判成了编译慢。**
|
|
||||||
|
|
||||||
### 取证过程
|
|
||||||
|
|
||||||
| 步骤 | 命令 | 结果 | 结论 |
|
|
||||||
|---|---|---|---|
|
|
||||||
| 隔离模块解析 | `go list -deps ./route/user/` | **2.3s EXIT=0** | 模块全在本地缓存,网络不是瓶颈 |
|
|
||||||
| 编译目标包 | `go build -overlay=... ./route/user/` | **7s EXIT=0** | 之前 240s 超时是伪命题 |
|
|
||||||
| 关键包逐个 | route/ internal/route/ question/ auth/ security/... db/ mail/ | **全 OK** | |
|
|
||||||
| 全量 | `go build -overlay=... ./...` | **EXIT=0** | |
|
|
||||||
| 静态检查 | `go vet -overlay=...` 7 个包 | **EXIT=0** | |
|
|
||||||
| 测试 | censor/autoban/ipinfo/route/conf/form | **全部 ok**,EXIT=0 | |
|
|
||||||
|
|
||||||
**真因**:前几次超时是 `GOPROXY` 未导出 → 卡在 `proxy.golang.org` 连接超时(HTTP=000)。
|
|
||||||
与机器性能、与 1C1G 都无关。**只要每条 go 命令前 `export GOPROXY=https://goproxy.cn,direct
|
|
||||||
GOSUMDB=off GOFLAGS=-mod=mod` 就秒级完成。**
|
|
||||||
|
|
||||||
⚠️ 方法论教训:**超时 ≠ 慢**。要用 `go list -deps`(纯元数据、不编译)先把「模块解析」和
|
|
||||||
「编译」两层拆开,否则会把网络挂起误判成性能问题,进而拿「环境」当借口。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 复核:追问邮件 token + 其他严重错误
|
|
||||||
|
|
||||||
### 1. 追问邮件确实没带 token(用户怀疑成立)
|
|
||||||
|
|
||||||
- `internal/mail/mail.go:58-65` `SendNewFollowUpMail` 链接 `%s/_/%s/%d`,**无 `?t=`**。
|
|
||||||
- 收件人是**箱主** `pageUser.Email`(follow_up.go:131),箱主靠登录态查看 → **不是权限漏洞**。
|
|
||||||
- 但有两处真实问题:
|
|
||||||
1. **不一致**:同样发给箱主的 `SendNewQuestionMail`(mail.go:25)**带** `?t=`。
|
|
||||||
2. **具体失效场景**:若该提问被箱主设为私密(`IsPrivate=true`),箱主在**未登录设备**点邮件链接
|
|
||||||
→ `Questioner` 中间件(question.go:61-66)判定 `IsPrivate && !IsLogged && token 不匹配`
|
|
||||||
→ **302 回首页,看不到追问**。新提问邮件因带 token 则不会。
|
|
||||||
- 建议:给 `SendNewFollowUpMail` 补 token 参数,与 `SendNewQuestionMail` 对齐。
|
|
||||||
- 关键的那封(**发给匿名提问者的回答通知**)`question.ReceiveReplyEmail` 是**带 token 的**:
|
|
||||||
follow_up.go:185、question.go:214 → `SendNewAnswerMail(..., question.Token)` ✓ 无问题。
|
|
||||||
|
|
||||||
### 2. 新发现 P2:提问 token 仅 6 位且爆破无限流
|
|
||||||
|
|
||||||
- `internal/db/questions.go:95` `Token: randstr.String(6)`;`randstr` 用 `crypto/rand` +
|
|
||||||
base62 → 62⁶ ≈ 5.68×10¹⁰,**仅 35.7 bit**(随机源没问题,是长度短)。
|
|
||||||
- `GET /_/{domain}/{questionID}?t=` **无 rateLimitMiddleware**;猜错由 `Questioner` 返回
|
|
||||||
**302 → /**,而 `autoban` 只统计「敏感路径」与「404」(404 默认关闭)→ **完全不计入封禁**。
|
|
||||||
- token 仅在「未回答 或 私密」且非箱主时才需要,攻破即=读私密/未公开提问。
|
|
||||||
- 可行性:单条定向 ~2.8×10¹⁰ 次不可行;但 questionID 可枚举、只需命中**任意一条**,
|
|
||||||
受保护提问 N 条时期望 5.68×10¹⁰/N 次 → 量大时理论可行。
|
|
||||||
- 低成本修复:`randstr.String(6)` → `randstr.String(12)`(≈71 bit)。存量 6 位 token 不受影响
|
|
||||||
(列是 string,无长度校验)。或对该 GET 路径加限流。
|
|
||||||
|
|
||||||
### 3. 已复核无问题(勿再重复查)
|
|
||||||
|
|
||||||
- **密码**(`internal/db/password.go`):bcrypt cost 10;旧 HMAC-SHA1 用 `subtle.ConstantTimeCompare`
|
|
||||||
常量时间比较并透明升级;哈希失败**不降级**为弱哈希(返回空串=锁定);bcrypt 72 字节截断 ✓
|
|
||||||
- **越权**:`Delete/SetPrivate/SetPublic/TogglePin/SetRead/SetUnread` 全部由 `canDelete`
|
|
||||||
(question.go:72 `ctx.IsLogged && ctx.User.ID == pageUser.ID`)把关;
|
|
||||||
`AnswerFollowUp`(follow_up.go:143) 与 `PublishAnswer`(question.go:154,161) 双重校验 ✓
|
|
||||||
(`ctx.Redirect` 会写响应从而终止后续 handler,早退有效)
|
|
||||||
- **CSRF**:全局挂载,SessionKey=uid 正确,有回归测试 ✓
|
|
||||||
- **会话**:登录 RegenerateID、pwd_ver 失效、HTTPOnly + SameSite=Lax ✓
|
|
||||||
- **密码找回**:验证码 24h TTL + 用后即删 ✓
|
|
||||||
- **SQL 注入**:全参数化 ✓
|
|
||||||
- **token/邮箱泄露**:`Question.Token` 与 `User.Email` 均为 `json:"-"` ✓
|
|
||||||
- **审核**:提问/回答/追问/投票/私聊统一走 `moderation.Check` ✓
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 落地 A+B(追问邮件补 token + 提问 token 6→12)
|
|
||||||
|
|
||||||
按「继续未完成的任务」指示,执行上一条报告的 A+B 两项变更。已用 gofmt_keep_crlf 保 CRLF。
|
|
||||||
|
|
||||||
### A. 追问邮件(发给箱主)补 token,与 `SendNewQuestionMail` 对齐
|
|
||||||
|
|
||||||
- `internal/mail/mail.go:58` `SendNewFollowUpMail` 签名加 `token string`,链接改为 `%s/_/%s/%d?t=%s`。
|
|
||||||
- `route/question/follow_up.go:131` 调用处补传 `question.Token`。
|
|
||||||
- 目的:箱主在**未登录设备**点开**私密**提问的追问邮件时也能凭 token 查看(否则被 Questioner 重定向首页)。
|
|
||||||
- 仅一个调用方,构建已验证签名一致。
|
|
||||||
|
|
||||||
### B. 提问 token 长度 6 → 12
|
|
||||||
|
|
||||||
- `internal/db/questions.go:95` `randstr.String(6)` → `randstr.String(12)`(≈35.7→71 bit)。
|
|
||||||
- 存量 6 位 token 不受影响(列是 string,无长度校验,旧 token 仍有效)。
|
|
||||||
|
|
||||||
### 验证
|
|
||||||
|
|
||||||
- `go build ./route/question/ ./internal/mail/ ./internal/db/` → EXIT=0
|
|
||||||
- `go build ./...`(overlay)→ EXIT=0
|
|
||||||
- `go vet` 三包 → EXIT=0
|
|
||||||
- `internal/db` 单元测试 FAIL 为已知环境限制(需真实 PostgreSQL/MySQL,报 `Unknown database type: ""`),
|
|
||||||
与本次改动无关,可跳过;security/autoban/censor/ipinfo、route 测试均 ok。
|
|
||||||
|
|
||||||
### 待定(未做,按需)
|
|
||||||
|
|
||||||
- CHANGELOG.json 是否补一条 entry(未做:需确认版本号与条目格式)。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 用本地 Postgres 跑通 db 测试并修真实缺陷(用户提供 127.0.0.1:5432)
|
|
||||||
|
|
||||||
### 测试库环境变量(`internal/dbutil/testing.go:NewTestDB`)
|
|
||||||
- `DB_TYPE=postgres DB_USER=postgres DB_PASSWORD=<pwd> DB_HOST=127.0.0.1 DB_PORT=5432`
|
|
||||||
DSN = `postgres://$DB_USER:$DB_PASSWORD@$DB_HOST:$DB_PORT?sslmode=disable`(不带库名,连上后
|
|
||||||
自动 `CREATE DATABASE tamabox-test-<rand>`,测试结束(未失败时)DROP)。
|
|
||||||
- 仍需 `GOPROXY=https://goproxy.cn,direct GOSUMDB=off GOFLAGS=-mod=mod` + webp overlay。
|
|
||||||
|
|
||||||
### 修的真实缺陷(生产代码,2 处)
|
|
||||||
|
|
||||||
- `internal/db/questions.go:DeleteByID`:级联删追问/上传图 + 回退提问配额时**直接调包级全局单例**
|
|
||||||
`FollowUps`/`UploadImages`/`FeatureControls`。这些单例只在 `db.Init` 装配 → 测试/部分初始化场景 nil → **panic**。
|
|
||||||
改为 `NewFollowUpsStore(db.DB)` / `NewUploadImagesStore(db.DB)` / `NewFeatureControlsStore(db.DB)`,
|
|
||||||
用自身 store 的 `*gorm.DB`。生产行为不变(`Init` 里所有 store 共用同一 `*gorm.DB`),但去掉了隐藏全局依赖。
|
|
||||||
- `internal/db/conversation.go:MarkDeletedByOwner`:同类问题(`FeatureControls.IncrementQuota`),同样修。
|
|
||||||
- 同包 `feature_control.go:355` 的 `FeatureControls.Get` **已有** `if FeatureControls == nil` 守卫,故未动。
|
|
||||||
|
|
||||||
### 修的测试缺陷(全在 internal/db,非生产 bug)
|
|
||||||
|
|
||||||
1. `users_test.go`:5 处 `want.EncodePassword(); require.Equal(t, want, got)` —— bcrypt 随机盐,
|
|
||||||
哈希**永不相等**。新建 helper `requireUserEqual`:先 `got.Authenticate(明文)` 校验密码,再对齐
|
|
||||||
`Password` 与 `ChatMaxMessages`(= `conf.Limits.ChatMaxMessages`,默认 200)后比较其余字段。
|
|
||||||
2. `testUsersDeactivate`:复用已被 `testUsersCreate` 占用的 `[email protected]`/`e99` → 改唯一值。
|
|
||||||
3. `users_test.go`/`upload_images_test.go` 去掉顶层 `t.Parallel()`:子测试共用一张表、cleanup 会
|
|
||||||
`TRUNCATE ... RESTART IDENTITY`,并行时多个子测试拿到相同 id/邮箱 → 撞唯一键。
|
|
||||||
4. `upload_images_test.go`:cleanup 只 truncate `upload_images`,而链接表 `upload_image_questions`
|
|
||||||
**无 FK 级联**(复合主键 `upload_image_id+question_id`),残留行让下一子测试撞 `_pkey`
|
|
||||||
→ 补 `cleanup("upload_images", "upload_image_questions")`。
|
|
||||||
5. `UpdateBlockWords/normal`:期望 `HarassmentSettingNone`("none")但新用户实际 `""`
|
|
||||||
(模型字段无 `gorm:"default:"` 标签)。`""` 与 `"none"` 生产行为等价(只有 `==register_only` 才限制;
|
|
||||||
模板 profile.html:851 也只在 `register_only` 时勾选)→ **非生产 bug**。改成先显式
|
|
||||||
`UpdateHarassmentType(RegisterOnly)` 再断言未被误改(更强的测试,且不把 `""` 奉为期望)。
|
|
||||||
|
|
||||||
### 验证
|
|
||||||
|
|
||||||
- `go build ./...` **EXIT=0**、`go vet ./...` **EXIT=0**
|
|
||||||
- `go test ./...`(含 DB)**全绿 EXIT=0**:internal/db 57s、route/vote、internal/route、templates、
|
|
||||||
i18n、conf、form、cache、imagecheck、mail、shared、template、security/* 均 ok。
|
|
||||||
- 行尾:`internal/db/*` 与 `internal/mail/mail.go` 原本就是 LF,`route/question/*`、`route/vote/vote.go`、
|
|
||||||
`internal/route/route.go` 是 CRLF —— **仓库本来就是混合行尾**;本次编辑未改变任何文件原行尾(无 diff 污染)。
|
|
||||||
|
|
||||||
### 结论
|
|
||||||
|
|
||||||
- 除上述「全局单例 nil 依赖」(健壮性隐患:测试与部分初始化场景必崩)外,**未发现新的生产逻辑 bug**。
|
|
||||||
- db 测试套件此前的一堆失败**全部是测试自身缺陷**:bcrypt 直接比哈希、测试数据撞车、并行共享表、漏清链接表。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 修复 HarassmentSetting 潜在隐患(schema 默认值 + 启动期归一)
|
|
||||||
|
|
||||||
用户确认「影响不大就做」。改动面很小(3 处源码 + 1 处测试期望 + 1 个新回归测试)。
|
|
||||||
|
|
||||||
### 线索
|
|
||||||
- 模型 `HarassmentSetting HarassmentSettingType` **无 `gorm:"default:"` 标签** → 新用户该列为空串;
|
|
||||||
而规范取值是 none / register_only。全仓库**没有任何代码比较 `== HarassmentSettingNone`**
|
|
||||||
(production 只有 `== register_only`,模板 profile.html:851 也只看 register_only)→ **运行期无 bug**,
|
|
||||||
但将来若有人写 `== None` 判断就会漏判。
|
|
||||||
|
|
||||||
### 改动
|
|
||||||
1. `internal/db/users.go` 模型:`HarassmentSetting` 补 `gorm:"default:'none'"`。
|
|
||||||
⚠️ **字符串默认值必须带单引号**(`default:'none'`),参考既有 `ModerationStrictness` 的 `default:''`;
|
|
||||||
裸写 `default:none` 会生成非法 SQL。
|
|
||||||
2. `internal/db/users.go:Create`:显式 `HarassmentSetting: HarassmentSettingNone`
|
|
||||||
(确定写库,不依赖 DB 默认值回读)。
|
|
||||||
3. `internal/db/users.go`:新增 `normalizeHarassmentSetting(db)`,把历史空值(''/NULL)归一为 none,幂等。
|
|
||||||
4. `internal/db/db.go:Init`:在 `normalizeChatQuota` 之后调用,失败仅告警不阻断启动
|
|
||||||
(与 `backfillFeatureControls`/`reconcileLegacyChatEnabled` 等既有惯例一致)。
|
|
||||||
|
|
||||||
### 测试
|
|
||||||
- `users_test.go` helper `requireUserEqual` 补 `want.HarassmentSetting = HarassmentSettingNone`。
|
|
||||||
- 新增 `TestHarassmentSettingNormalize`:新建用户即 none + 历史空值经归一后为 none。
|
|
||||||
|
|
||||||
### 实测证据
|
|
||||||
- 临时探针查 `information_schema`:`harassment_setting` 的 `column_default = "'none'::text"` ✅
|
|
||||||
(AutoMigrate 确实把列默认值改成了 none;探针文件跑完即删)。
|
|
||||||
- `go build ./...` / `go vet ./...` EXIT=0;`go test ./...`(含 DB)**全绿**;gofmt 无 diff;
|
|
||||||
行尾未变(users.go CRLF、db.go/users_test.go LF)。
|
|
||||||
|
|
||||||
### ⚠️ 新踩到的 gofmt 陷阱
|
|
||||||
在**文档注释**里写 `''` 会被 gofmt 重排成 `”`(Go 1.19+ 的 doc comment 规范化:`` `` ``→`“`、`''`→`”`),
|
|
||||||
导致 `gofmt -l` 反复报未格式化。**注释里避免使用 `''`**(改写为「空字符串」等文字)。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 账号停用体验修复:独立成功页(/deactivated)
|
|
||||||
|
|
||||||
### 问题
|
|
||||||
停用账号后 `DeactivateProfileAction` 把用户重定向到 `/login`,只在登录页顶部弹一条 flash。
|
|
||||||
用户会觉得「被丢回登录页」,下意识以为操作没生效。
|
|
||||||
|
|
||||||
诊断结论:flash **确实渲染**(Flush 清 session 数据但保留 session,flash 能带到下一页),
|
|
||||||
真正的体验错配是「停用后落点仍是登录页」——而账号已删/停,用户根本不可能再登录。
|
|
||||||
|
|
||||||
### 方案(用户选「独立成功页(推荐)」)
|
|
||||||
1. `route/user/profile.go`
|
|
||||||
- 新增公开 handler `DeactivatedProfile`:`ctx.SetTitle(i18n.T(ctx.Lang(), "deactivated.title"))` + `ctx.Success("user/deactivated")`。
|
|
||||||
- `DeactivateProfileAction` 末尾 `ctx.Redirect("/login")` → `ctx.Redirect("/deactivated")`,**并移除原来的 `SetSuccessFlash`**(页面自身即提示)。
|
|
||||||
- 顺手修正 `UpdateProfile` 里一处陈旧注释(原写「顺序与 DeactivateProfileAction 保持一致…随后跳登录页」,停用路径已不再如此)。
|
|
||||||
2. `internal/route/route.go`:在公开区(`/explore` 旁)加 `f.Get("/deactivated", user.DeactivatedProfile)`。
|
|
||||||
**必须是公开路由**:停用后会话已 Flush、`reqUserSignIn` 会把人弹回登录页,正是要避免的现象。
|
|
||||||
3. 新建 `templates/user/deactivated.html`:复用 `question/success.html` 的范式(大 ✓ 图标 + 卡片 + 单按钮返回首页)。
|
|
||||||
`templates/embed.go` 是整目录 `//go:embed ... user ...`,新模板**自动被嵌入**,无需改 embed 指令。
|
|
||||||
4. i18n:四份字典都加 `deactivated.title` / `deactivated.desc` / `deactivated.backHome`
|
|
||||||
(`TestDictionaryComplete` 要求 zh_cn/zh_tw/en/ja **键集合完全一致**,漏一份必挂)。
|
|
||||||
同时**删除已无引用的 `admin.flash.deactivated`**(4 份字典同步删;删它不影响对齐组,因为该组最长键是 `admin.flash.blockWordsTooMany`)。
|
|
||||||
|
|
||||||
### 验证
|
|
||||||
- `go build ./...` / `go vet ./...` EXIT=0;`go test ./...`(含本地 PG)**全绿**,
|
|
||||||
其中 `internal/i18n`(键一致性)、`internal/route`、`templates`、`internal/template`(模板解析)均 ok → 新模板可被解析、键齐备。
|
|
||||||
- gofmt 无 diff;行尾未变(4 份 dict + route.go 保持 CRLF,`route/user/profile.go` 实为 **LF**)。
|
|
||||||
|
|
||||||
### 备注
|
|
||||||
- `route/user/profile.go` 的行尾是 **LF**(与「route/** 皆 CRLF」的旧印象不符,按文件单独确认即可)。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔴 P0 修复:账号停用后同一邮箱/域名永远无法重新注册
|
|
||||||
|
|
||||||
用户在测试环境 `https://box-dev-api.337845818.best` 注册 `1223383917@qq.com` / `tamakyi` 报 500:
|
|
||||||
```
|
|
||||||
SELECT ... WHERE email = '...' AND deleted_at IS NULL -> rows:0
|
|
||||||
INSERT INTO users ... -> ERROR: duplicate key value violates unique constraint "idx_users_email_unique" (SQLSTATE 23505)
|
|
||||||
ERRO Failed to create new user error="create user: ERROR: duplicate key ..."
|
|
||||||
```
|
|
||||||
|
|
||||||
### 根因(本地已实测复现,日志形态与线上一致)
|
|
||||||
`internal/db/users.go` 的 `Deactivate` 用的是 **GORM 默认的软删除**:
|
|
||||||
`db.WithContext(ctx).Model(&User{}).Delete("id = ?", u.ID)` → 只写 `deleted_at`,**行仍在库里**。
|
|
||||||
而 `User` 的 `email` / `domain` 是 `uniqueIndex`(**索引不含 deleted_at**),于是:
|
|
||||||
- 查重 `validate()` 走 gorm 软删除作用域 → 看不见这行 → 放行;
|
|
||||||
- INSERT 被数据库唯一索引拦下 → `Create` 的 TOCTOU 兜底再查一次仍看不见 → 抛底层 23505 → handler `default` 分支 → 500。
|
|
||||||
|
|
||||||
复现证据(临时测试,已删):停用后 `Unscoped` 查该邮箱 → 行仍在且 `deleted_at.Valid=true`;同邮箱重新注册 → 与线上同样的 23505。
|
|
||||||
**旁证**:`internal/cmd/ban.go:64` 早有注释「用 Unscoped 查找,支持封禁已软删除(自助停用)的账号」——
|
|
||||||
说明「自助停用会留软删除行」是既有事实,但没人把「唯一索引占位」这层后果接上。
|
|
||||||
同类先例:`internal/db/vote_submissions.go:234` 已因同一原因改用 Unscoped 物理删除。
|
|
||||||
|
|
||||||
顺带发现:`BanUserAndCleanup` 的「连带清理」也只是软删除子表(Question/FollowUp/Conversation/Message… 都嵌 `dbutil.Model`),
|
|
||||||
数据实际仍在库;也与停用页「所有数据将被删除」的承诺不符。
|
|
||||||
|
|
||||||
### 改动(`internal/db/users.go`)
|
|
||||||
1. `Deactivate` → 调 `db.purgeUser(ctx, u.ID)`;`BanUserAndCleanup` 同样委托给它。
|
|
||||||
2. 抽私有 `purgeUser`:级联删除**全部改为 `Unscoped()` 物理删除**(追问/消息/消息图片/会话/提问-图片关联/提问/拉黑/功能开关/用户行)。
|
|
||||||
3. `Create` 兜底加固:唯一键冲突且 `validate` 仍查不到时,用新增的 `duplicateFieldErr()`(Unscoped)判定是 email 还是 domain 被占,
|
|
||||||
返回 `ErrDuplicateEmail`/`ErrDuplicateDomain` 友好提示,而不是把 23505 抛成 500。(`isDuplicateKeyError` 复用 vote_submissions.go 里的)
|
|
||||||
4. 回归测试 `TestDeactivatePurgesUser`:停用后 Unscoped 也查不到用户行、提问被清理、同一邮箱可重新注册。
|
|
||||||
|
|
||||||
### ⚠️ 过程中自己踩的 GORM 大坑
|
|
||||||
把级联删除改成「先 `tx = tx.WithContext(ctx).Unscoped()` 再逐条 `tx.Where(...)`」——
|
|
||||||
**事务句柄 clone==0,复用同一个句柄会让上一条语句的 `Where` 条件串到下一条上**,
|
|
||||||
SQL 变成 `DELETE FROM follow_ups WHERE <follow_ups条件> AND <messages条件>` → `字段 conversation_id 不存在 (42703)`。
|
|
||||||
✅ 正确写法:**每条语句都重新取句柄** `tx.WithContext(ctx).Unscoped().Where(...)`(原代码本来就是每条 `tx.WithContext(ctx)`,是我改坏的)。
|
|
||||||
这条已写进 MEMORY.md。
|
|
||||||
|
|
||||||
### 验证
|
|
||||||
- `go build ./...` / `go vet ./...` EXIT=0;`go test ./...`(含本地 PG)**全绿**;gofmt clean;行尾未变(users.go CRLF、users_test.go LF)。
|
|
||||||
- 测试环境实测(脚本 `dev/devcheck-e2e.py`):
|
|
||||||
- ✅ 注册 / 登录 / `/user/profile` / `/user/questions` / `/user/votes`(302→`/user/questions#votes`) / 提问(返回提问成功页)/ 新建投票页 / 停用页 / **停用后跳 `/deactivated` 成功页** 全部正常;
|
|
||||||
- ✅ `/deactivated` 四语言标题渲染正确(账号已停用 / 帳號已停用 / Account deactivated / アカウントを停止しました);
|
|
||||||
- ❌ **停用后同邮箱重新注册仍 500** —— 测试环境跑的是**旧构建**(`/deactivated` 已在,但没有 purgeUser 修复),需重新部署。
|
|
||||||
- ⚠️ 测试环境 DB 里已留下本次测试产生的孤儿行,需清理:
|
|
||||||
`tbe2e0915122816@example.com`(仍在线)、`tbe2e0915122934@example.com`、`tbcheck0915122657@example.com`(均已停用=软删除占位);
|
|
||||||
用户自己的 `tamakyi` / `1223383917@qq.com` 同样是软删除占位(这就是他报错的直接原因)。
|
|
||||||
清理方式:`tamabox ban --email <邮箱> --yes`(CLI 支持查软删除账号),或直接 SQL 删 `users.deleted_at IS NOT NULL` 的行。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ⚠️ 方案变更(覆盖上一节):停用改为「保留身份占位 + 清空内容」
|
|
||||||
|
|
||||||
用户表示**尚未部署**,并提出:「曾经注册过的账户停用后,再次注册时应提示『该邮箱已被占用但账户已停用,如需重新使用请联系站长』」。
|
|
||||||
经询问确认两点:
|
|
||||||
1. 停用形态 = **保留身份占位**(用户行软删除)+ **物理删除内容数据**(停用页「所有数据将被删除」的承诺仍然成立);
|
|
||||||
2. 「联系站长」**复用页脚已有的「找站长」入口**(`site.footer.admin` → `https://box.shiroko.one/_/tamakyi`),不新增站长邮箱配置项。
|
|
||||||
|
|
||||||
→ **上面「Deactivate 改成物理删除」的做法已被取代**,但其中「级联内容必须 Unscoped 物理删除」的结论仍然有效。
|
|
||||||
|
|
||||||
### 最终实现
|
|
||||||
- `internal/db/users.go`
|
|
||||||
- `purgeUserContent(ctx, tx, userID)`(包级函数):把内容级联删除抽出来,全部 `Unscoped()` 物理删除。
|
|
||||||
- `Deactivate` = `purgeUserContent` + **软删除用户行**(保留 email/domain 占位)。
|
|
||||||
- `BanUserAndCleanup` = `purgeUserContent` + **物理删除用户行**(唯一差别;彻底释放邮箱/域名)。
|
|
||||||
- `validate` 重写为两轮:先查活跃账号(ErrDuplicateEmail/Domain),再 **Unscoped** 查已停用占位行
|
|
||||||
→ 新增 `ErrDeactivatedEmail` / `ErrDeactivatedDomain`;辅助方法 `fieldTaken(ctx, column, value, includeDeleted)`。
|
|
||||||
- 新增 `Reactivate(ctx, id)`(清 `deleted_at`,未停用时返回 ErrUserNotExists),已加进 `UsersStore` 接口。
|
|
||||||
- 上一轮加的 `duplicateFieldErr` 已删除:`validate` 现在本身就覆盖了软删除行,无需在 Create 兜底里重复判定。
|
|
||||||
- `route/auth/register.go`:把两个新错误映射到 `auth.errDeactivatedEmail` / `auth.errDeactivatedDomain`。
|
|
||||||
- i18n ×4:新增 `auth.errDeactivatedEmail`、`auth.errDeactivatedDomain`、`deactivate.note`(键集合保持一致)。
|
|
||||||
- `templates/user/deactivate.html`:desc 下方加一行 `deactivate.note`(uk-text-small uk-text-muted),
|
|
||||||
提前告知「邮箱/域名会作为身份占位保留,无法用同一邮箱重新注册,如需恢复请找站长」。
|
|
||||||
- **新增 CLI `internal/cmd/restore.go`(`TamaBox restore`)**:`--email|--domain [--password] [--yes]`,
|
|
||||||
清掉 deleted_at 完成恢复;`--password` 会重置密码(必须先恢复再改密码——`UpdatePassword` 走普通作用域)。
|
|
||||||
|
|
||||||
### 🐛 顺带修掉的隐藏问题
|
|
||||||
**`cmd/tamabox.go` 里根本没注册 `cmd.Ban`** —— `internal/cmd/ban.go` 写了但命令不可达(`TamaBox ban` 不存在)。
|
|
||||||
现已把 `cmd.Ban` 与 `cmd.Restore` 一起注册进 `app.Commands`。
|
|
||||||
|
|
||||||
### 验证
|
|
||||||
- `go build ./...` / `go vet ./...` EXIT=0;`go test ./...`(含本地 PG)**全绿**;gofmt 全部 clean;行尾未变。
|
|
||||||
- 新增/改写测试:`TestDeactivatePurgesUser`(占位行保留 + 内容清空 + 同邮箱得 ErrDeactivatedEmail + 同域名得
|
|
||||||
ErrDeactivatedDomain + Reactivate 后可正常注册 + 重复 Reactivate 报 ErrUserNotExists)、
|
|
||||||
`TestBanUserAndCleanupRemovesRow`(封禁物理删除且可重新注册)。
|
|
||||||
- **CLI 端到端实测**(临时库 `tamabox-clitest`,用 `NEKOBOX_CONFIG_PATH` 指向临时 ini,跑完已 DROP + 删除产物):
|
|
||||||
- `restore --email clitest@example.com --yes` → deleted_at 变 NULL ✅;重复恢复 → 正确拒绝「未被停用,无需恢复」✅
|
|
||||||
- `restore --password newpass12345` → 密码重哈希 + password_version=1 ✅;密码过短/邮箱不存在 → 明确报错 ✅
|
|
||||||
- `ban --domain banme --yes` → users 与 questions 计数归零(物理删除)✅
|
|
||||||
- ⚠️ `conf.Init` 还需要 `server.salt` / `server.xsrf_key` / `app.external_url` 非空,配置路径可用
|
|
||||||
`NEKOBOX_CONFIG_PATH` 覆盖 —— 这是本地跑 CLI 冒烟的关键(仓库里只有 `conf/app.sample.ini`)。
|
|
||||||
- 测试环境 `devcheck-e2e.py` 的核心回归项已改为断言「重注册被明确拒绝并出现『已停用』提示、且不是 500 兜底」;
|
|
||||||
在**未重新部署的旧构建**上会 FAIL 并打印「サーバー内部エラー」,正是需要重新部署的信号。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 部署后巡检扩容:投票全链路 + 登录态资产(28/28 全绿)
|
|
||||||
|
|
||||||
**背景**:用户重新部署后要求「所有功能都试一遍」。此前 `devcheck-e2e.py` 只验证了「新建投票页 GET」,
|
|
||||||
**投票创建(POST)这条主链路从未端到端跑过**,是覆盖盲区。
|
|
||||||
|
|
||||||
**改动**:`dev/devcheck-e2e.py` 扩到 **28 项**,新增两组:
|
|
||||||
|
|
||||||
1. `check_vote_flow(owner, base)` —— 投票全链路
|
|
||||||
- POST `/user/votes` 建单选投票并 `action=publish` → 断言 302 → `/user/questions#votes`
|
|
||||||
- GET `/user/questions` 断言标题可见,并从 `copyShareURL('...')` 取 **LinkToken**、从 `/user/votes/(\d+)/edit` 取 **ID**
|
|
||||||
- 匿名 GET `/vote/<token>` → 200 且含标题;从 `name="option_id" value="..."` 取选项 ID
|
|
||||||
- 匿名 POST `/vote/<token>/submit`(`option_id` + 数学验证码 + `_csrf`)→ 302
|
|
||||||
- GET `/user/votes/<id>/export` 断言 **`总参与,N` 这行的 N == 1**
|
|
||||||
- 草稿:`action=save` → 302 到 `/user/votes/<id>/edit?saved=1`
|
|
||||||
- 表单型:`type=form` + `form_fields` JSON(field_type/label/required/max_length)→ 发布 302
|
|
||||||
2. `check_user_assets(owner)` —— 登录态接口/导出
|
|
||||||
- `/api/v1/user` 返回 JSON、`/explore` 200、`POST /user/profile/export-json`(JSON)、
|
|
||||||
`POST /user/profile/export`(xlsx,校验 content-type)、停用页 `deactivate.note` 结构断言
|
|
||||||
|
|
||||||
**踩到的坑(都已在脚本里修掉)**:
|
|
||||||
- 单选投票导出 CSV 的**末行是 `总参与,N`**,与各选项票数行同为 `,数字` 结尾 →
|
|
||||||
用 `,\d+$` 一把捞会把总参与也累加进去(实测 `[1,0,0,1]` 求和成了 2)。
|
|
||||||
**必须单独匹配 `总参与,(\d+)`**。
|
|
||||||
- `urllib` 读 **HTTPError 的响应体本身也会超时**(`c.get("/i18n/xx?...")` 这类 302 最易触发),
|
|
||||||
异常从 except 块里抛出会把整轮巡检打断。已把 body 读取包在 try/except 里,并在 `__main__` 外层加兜底
|
|
||||||
——中断时仍打印已完成项,而不是只留一段 traceback。
|
|
||||||
- ⚠️ 复盘我自己的一个假警报:用 `T=/c/Users/Tamakyi\ Gekka/...` 做 gofmt 比对循环时,
|
|
||||||
变量未加引号导致路径断裂 → 全部文件假报 DIFF。**脚本里带空格的路径变量一定要整体加引号**,
|
|
||||||
单文件手跑同一个文件即为 OK,说明是脚本问题而非代码问题。
|
|
||||||
|
|
||||||
**结果**:测试环境 `https://box-dev-api.337845818.best` 跑 `python dev/devcheck-e2e.py` → **28 项,失败 0 项**。
|
|
||||||
10 个改动过的 Go 文件 gofmt 比对全部 OK(无重排),行尾未变。
|
|
||||||
|
|
||||||
**遗留**:巡检每轮会在测试环境留下一个**已停用的身份占位**(邮箱/域名被占用)。
|
|
||||||
本轮新增 `tbe2e*` 若干;用户自己的 `1223383917@qq.com` / `tamakyi` 也仍是占位状态。
|
|
||||||
清理:`TamaBox ban --email <邮箱> --yes`(连行物理删,彻底释放)或 `TamaBox restore --email <邮箱> --yes`(恢复)。
|
|
||||||
@@ -1,614 +0,0 @@
|
|||||||
# 2026-10-19
|
|
||||||
|
|
||||||
## 部署后复测(强制严格档那批)
|
|
||||||
|
|
||||||
- 规则页返回首页已居中:线上 `<div class="uk-margin-large-top uk-text-center">` ✔
|
|
||||||
- 验证码弹窗按钮已缩短:线上 `>送信<` ✔(旧的「認証して送信」已消失)
|
|
||||||
- 箱主面板 `/user/questions`:限制期条幅 + 「查看社区规则」链接都在 ✔
|
|
||||||
- 严格度锁定(admin6 当前 L2):
|
|
||||||
- `/user/profile` 出现「处置期间审核严格度已锁定为最高档,解除后自动恢复你的原设置」✔
|
|
||||||
- `select[name=strictness]` 与「保存审核严格度」按钮均带 `disabled` ✔
|
|
||||||
- 直接 POST `/user/moderation-strictness/update?tab=security` + `strictness=lax` → 302 回 profile,
|
|
||||||
flash 仍是「已锁定」,且选中档仍为 `strict`(说明未写入)✔
|
|
||||||
- 踩坑:表单 action 是 `/user/moderation-strictness/update`(**没有** `/profile` 段);用错路径返回 302 到 `/`,
|
|
||||||
容易误判成「被中间件拦」。另外每次 POST 都要重新取 `_csrf`,用上一页(已跳首页)的 token 会 400。
|
|
||||||
|
|
||||||
## 记忆维护
|
|
||||||
|
|
||||||
MEMORY.md 一度 19KB 被注入截断,已重写压缩到约 4.9k 字符(8.1KB),保留构建/行尾/embed/审核/处置/CLI 等硬约束,
|
|
||||||
删去已在日志里的过程细节。
|
|
||||||
|
|
||||||
## CLI 文档 + 全量代码审阅(同日)
|
|
||||||
|
|
||||||
新增 `docs/cli.md`:命令总览、reports 三种用法与口径、restrict 级别语义/副作用、unrestrict、
|
|
||||||
hide-public、ban/restore、典型处置流程、排错表、明确不做的事。
|
|
||||||
|
|
||||||
审阅中发现并修复(全部本地临时库实测):
|
|
||||||
|
|
||||||
1. **`hide-public` 举报闭环静默失效**(真 bug):`build()` 带 `is_private=false`,在 Update 之后拿它当子查询恒为空集
|
|
||||||
→ 隐藏成功但 `content_reports` 仍 pending。改为 Update 前 `Pluck("id")` 快照,再按 ID 更新/标记。
|
|
||||||
实测:3 条 pending → accepted,未波及其他箱主。
|
|
||||||
2. **整箱分享卡未拦截**:`ShareCard` 直接走缓存,L1–L3 仍能生成整箱图。已在缓存查询前加拦截(fail-closed)。
|
|
||||||
3. **L2 通知邮件误导**:强制私密后仍给提问者发「你的提问有回答」,链接点开 302。改为 `!forcedPrivate` 才发。
|
|
||||||
4. **L2 写入顺序**:`SetPrivate` + `AnswerByID` 两条语句有「公开+已作答」窗口,第二条失败还会留下「私密+无回答」。
|
|
||||||
新增 `Questions.AnswerByIDPrivate`(单条 UPDATE 同时写 answer/is_read/is_private),并补测试 `TestQuestions/AnswerByIDPrivate`。
|
|
||||||
5. **`ListAPI` 绕过 Pager**:L4 停用与页面访问密码只挡 HTML,`/api/v1/user/{domain}/questions` 仍能整页拉。
|
|
||||||
已补两道校验(均返回「用户不存在」,不暴露原因)。**页面密码可被 API 绕过是升级前就存在的问题。**
|
|
||||||
6. **L1 曝光残留**:`GetShowInHome` 的 Count 未过滤(分页总数虚高、末页空白)、热门标签查询未过滤。已补同一子查询。
|
|
||||||
7. **性能**:`ActiveLevel` 由两次查询改一次(取最高 rank);新增 `moderation.EffectiveStrictnessFor(level, raw)`,
|
|
||||||
发布路径上不再重复查库(原先一次发布最多 4 次处置查询)。
|
|
||||||
8. **证据包口径**:分母是「累计公开回答」不是窗口产出 → 文案加「累计」;新增 `待处理 N` / `已处理` 标记,
|
|
||||||
避免把已闭合的举报误读成还挂着;补 `q.deleted_at IS NULL`;`--stats` 说明最长回溯 30 天;CLI 对 `active` 做 nil 守卫。
|
|
||||||
|
|
||||||
验证:`go build ./...` EXIT=0;db/i18n/templates 测试全绿;CLI 全流程(restrict L3 → 证据包 → unrestrict)实测通过;
|
|
||||||
临时库已 DROP。
|
|
||||||
|
|
||||||
遗留(未改,已向用户说明):L4 不覆盖 `/vote/{token}`;`restrict` 没有 `--days` 上限;处置不发邮件给箱主(只有登录后条幅);
|
|
||||||
`abuseAlertCooldowns` 进程内 map 无清理(重启即失效)。
|
|
||||||
|
|
||||||
## 第二轮:处置通知邮件 + 超长确认 + 冷却表清理
|
|
||||||
|
|
||||||
- 新增 `templates/mail/restriction-notice.html` + `mail.SendRestrictionNoticeMail(RestrictionNotice)`:
|
|
||||||
文案由调用方按**箱主自己的语言**(`users.language`,空则 `conf.App.DefaultLang`)预翻译后传入,
|
|
||||||
模板本身单语言。**不含 `--reason` 原文**,只有级别名/到期/规则链接/申诉渠道。
|
|
||||||
i18n 新增 10 个键 ×4 语言:`mail.restriction.*`(6)+ `restriction.level.L1..L4`。
|
|
||||||
- `restrict`:新增 `--force`(`--days > 90` 必填,否则直接报错)与 `--no-notify`;交互模式下超长处置
|
|
||||||
还要再输一次级别名(`--force --yes` 视为两次确认)。发信失败只 warn + 终端提示,**不回滚处置**;
|
|
||||||
发信加了 20s 超时(gomail 无 Timeout 字段,裸 Dial 会等系统 TCP 超时)。
|
|
||||||
- `abuseAlertCooldowns` 加惰性清理(>1024 条时删除已过 24h 的项)。
|
|
||||||
- 顺带:`/user/questions` 限制期条幅改用 `restriction.level.*` 的名称(「收窄公开」)而不是裸 `L2`。
|
|
||||||
|
|
||||||
验证:真实 SMTP sink 实收两封(ja 与 zh-CN),标题/级别名/到期/规则链接均正确、`--reason` 未出现、
|
|
||||||
链接未被 `ZgotmplZ` 转义;`--days 120` 无 `--force` 被拒;黑洞 SMTP 下 23s 内返回(未阻塞两分钟)。
|
|
||||||
`go build ./...`、`go vet`、db/i18n/templates 测试、checktpl 全绿。
|
|
||||||
|
|
||||||
遗留:`unrestrict` 不发通知(只解禁);`restrict` 对 `--days` 无上限(设计如此)。
|
|
||||||
|
|
||||||
## 第三轮:解除处置也发通知
|
|
||||||
|
|
||||||
- 复用同一封模板 `mail/restriction-notice.html`:`appeal` 段改为 `{{if .appeal}}` 可选,
|
|
||||||
解除通知不传该字段就整段省略。
|
|
||||||
- 新增 i18n 键 ×4:`mail.unrestrict.heading/body/note`(body 带域与级别名;
|
|
||||||
note 明确写「处置记录保留用于审计、私密内容不会自动恢复」)。
|
|
||||||
- `unrestrict` 新增 `--no-notify`;撤销前先 `ActiveLevel` 取级别(撤销后读不到),
|
|
||||||
撤销后按箱主语言发解除通知。`affected==0` 直接返回、不发信。
|
|
||||||
- 抽出公共件:`sendNotice`(20s 超时)、`reportNoticeResult`(统一成功/失败输出)。
|
|
||||||
- 文档补充:`unrestrict` 的 `--no-notify`、解除通知内容、以及**到期自动解除不会发信**(无调度器)。
|
|
||||||
|
|
||||||
实测(本地 SMTP sink,2 封):处置信含级别名/到期/申诉段;解除信含「已解除」+ 私密内容提示、
|
|
||||||
**无申诉段**;两封都不含 `--reason`。`--no-notify` 两处都不发信;无生效处置时不发信。
|
|
||||||
`go build ./...`、db/i18n/templates 测试、checktpl 全绿。
|
|
||||||
|
|
||||||
## 上线前最终验收(部署后实测)
|
|
||||||
|
|
||||||
- 公开页:`/` `/rules` `/restricted` `/deactivated` 全部 200;`/rules` 四语言标题正确。
|
|
||||||
- 箱主面板(admin6,L2):条幅显示「收窄公开」+ 规则链接;严格度面板锁定提示 1 处、下拉 disabled。
|
|
||||||
- **L2 写入路径实测**(关键,因为改成了 `AnswerByIDPrivate` 单条 UPDATE):
|
|
||||||
对未回答题 q95 作答 → 302 回题目页 → flash「已保存为私密:当前处于平台处置期,回答不会公开」→ 匿名访问 302。
|
|
||||||
- 公开列表 API `/api/v1/user/{domain}/questions` 正常返回 `{code,data:{data,next_cursor,has_more}}`(新增的 L4/密码校验没有打断正常路径)。
|
|
||||||
- 整箱分享卡:受处置箱主 302(拦截生效);未处置箱主 500,原因是**测试站未配置字体**(`分享图生成失败:字体文件未配置或无法加载`),与本次改动无关。
|
|
||||||
这一点反而证明拦截位置在生成之前。
|
|
||||||
- 未能自测:CLI 发信(restrict/unrestrict 通知)需要在服务器上跑命令,只能由用户执行。
|
|
||||||
|
|
||||||
注意:admin6 现有测试题 73/88/89/90/94/95;其中 95 已被我回答并存为私密(内容带 `restrict self-check`)。
|
|
||||||
admin6 的 L2 于 2026-10-19 21:29 自动到期。
|
|
||||||
|
|
||||||
## 邮件模板统一(用户反馈:页脚与其它邮件不一致)
|
|
||||||
|
|
||||||
- 原因:`restriction-notice.html` 是我新写的,页脚用了自造的 `© 2026 TamaBox`,与项目其它 11 封邮件的
|
|
||||||
「此邮件由系统自动发送,请勿直接回复。<br/>{{ .year }} 狼的提问箱」不一致。
|
|
||||||
- 处理:整封模板按 `question-reported.html` 的结构重写(24px 标题 + `{{.time}}` 副行、同样的卡片/页脚/外间距),
|
|
||||||
`time` 由 `mail.SendRestrictionNoticeMail` 统一注入(调用方不用各自拼时间)。
|
|
||||||
- 实测(本地 SMTP sink):页脚文本与 question-reported 完全一致,`© ... TamaBox` 已消失,头部时间行存在。
|
|
||||||
- 注意:站点名在所有邮件模板里都是硬编码「狼的提问箱」,不是配置项;改站名要一起改 12 个模板。
|
|
||||||
|
|
||||||
## 页脚精简(移动端一行放不下)
|
|
||||||
|
|
||||||
- 去掉「盘」链接(`https://share.shiroko.one`,zh 文案就叫「盘」),并删除已无引用的
|
|
||||||
`site.footer.drive` 键(四语言同步删,保证 `TestDictionaryComplete` 词条数一致)。
|
|
||||||
- `site.footer.rules`:zh-CN「社区规则」→「规则」,zh-TW「社群規則」→「規則」(en/ja 本就短)。
|
|
||||||
- 现在页脚一行:规则 | 找站长 | 公告 | 聊天 | 状态 | 主页。
|
|
||||||
- 注意:`templates/base/footer.html` 用 `|` 分隔,行首行尾各带一个,删整行时不会留下悬空分隔符(已核对)。
|
|
||||||
|
|
||||||
## 页脚改两行(用户改主意:盘恢复)
|
|
||||||
|
|
||||||
- 恢复 `https://share.shiroko.one`「盘」链接与 `site.footer.drive` 键(zh-CN 盘 / zh-TW 盤 / en Drive / ja ドライブ)。
|
|
||||||
- `site.footer.rules` 文案恢复:社区规则 / 社群規則(en/ja 不变)。
|
|
||||||
- 布局两行(靠 `<br>` 断行,不用额外 CSS):
|
|
||||||
第一行 `社区规则 | 找站长`,第二行 `盘 | 公告板 | 聊天室 | 监控板 | 导航页`。
|
|
||||||
- 键插在 `site.footer.rules` 之后(组末),空格数按同长度键复制,gofmt 对齐不变;dict 仍 CRLF。
|
|
||||||
- 验证:build / checktpl / i18n / templates 测试全绿。
|
|
||||||
|
|
||||||
## 页脚改为「从少到多」四行
|
|
||||||
|
|
||||||
版权行与存活计时移到最后两行(原来在最上面),整块按文字量递增:
|
|
||||||
`社区规则 | 找站长` → `盘 | 公告板 | 聊天室 | 监控板 | 导航页` → `Powered by NekoBox | ICP` → `提问箱已成功存活:…`。
|
|
||||||
实现只调整了 `<br>` 的位置,仍共 4 个 `<br>`,所以块高与顶部空行跟原来一致(未新增占位)。
|
|
||||||
注意:`box_runtime` 的脚本按 id 取元素,移动位置不影响;但 `<br>` 数量别随意增删,否则页脚高度会变。
|
|
||||||
|
|
||||||
## 前端样式体检(只出结论,未改代码)
|
|
||||||
|
|
||||||
量化事实(脚本统计 templates/**/*.html):
|
|
||||||
|
|
||||||
- 全站 **0 个 .css 文件**,样式全部内联:`base/header.html` 内含 **17.5KB CSS + 10.6KB JS**(模板总 42.5KB),
|
|
||||||
footer 3.0KB JS;全模板共 **24 个 `<style>` 块**,`user/profile.html` 自身还有 11.2KB CSS + 22.8KB JS(总 62KB)。
|
|
||||||
- `--tb-*` 设计变量 **只在 `html[data-theme="dark"]` 里定义**,light 主题没有 `:root` 一套;
|
|
||||||
调用处全靠 fallback(`#e9e9e9` / `#e5e5e5` / `#fafafa` / `#333` / `#666` 混用),explore 等页面还写死色值。
|
|
||||||
- **深色模式 FOUC**:`applyTheme(getTheme())` 在 body 内、`<nav>` 之后执行(header.html:812),
|
|
||||||
深色用户刷新时顶栏会先白闪一下。修法:把 theme 初始化内联脚本提到 `<head>`。
|
|
||||||
- 结构:`uk-container-xsmall`(750px)由 header 统一输出,仅 explore/vote/chat 页面自套 small;
|
|
||||||
`question/list.html` 仍用 `<center>`、内联 `style="min-height:260px"`,用户名/简介颜色由用户自选压在背景图上,无 scrim → 可读性无保障。
|
|
||||||
- 首页 `home.html` 是纯 UIkit 卡片 + `<hr>` + `style="color: red"` + `<pre>` 统计 + 首页挂 Waline(250 行 locale)。
|
|
||||||
|
|
||||||
建议分档(未实施):P0 = light/dark 双套 token + 圆角/阴影/中文字体栈 + 首页重排 + 背景图 scrim;
|
|
||||||
P1 = header CSS 抽成 `static/css/*.css`(注意 `embed.go`/`static/fs.go` 都是显式清单)+ FOUC 修复;
|
|
||||||
P2 = 图标统一(UIkit icons 取代 emoji/位图 png)、标签减量、viewpoint/meta 去重。
|
|
||||||
|
|
||||||
## P0 前端美化(已实施)
|
|
||||||
|
|
||||||
用户拍板:卡片**结构不变,只加圆角和柔阴影**;先做 P0,不抽 CSS 文件。
|
|
||||||
|
|
||||||
改动文件:
|
|
||||||
- `templates/base/header.html`
|
|
||||||
- head 顶部新增**主题初始化内联脚本**(深色用户不再白闪;原 `applyTheme` 在 body 里、nav 之后执行)。
|
|
||||||
顺手删掉 body 脚本里的 `applyTheme(getTheme())`,只保留切换逻辑。
|
|
||||||
- 新增**设计令牌块**:`:root` 亮色 + `html[data-theme=dark]` 暗色,同名 14 个(bg/surface/surface-2/surface-hover/
|
|
||||||
border/text/text-secondary/text-muted/primary/primary-weak/radius/radius-sm/radius-xs/shadow/shadow-hover/font)。
|
|
||||||
原暗色块里的 7 个变量定义已删除(现在统一在令牌块)。
|
|
||||||
- 新增**基础质感层**:html+body 底色与字体栈(system-ui + 苹方/冬青黑/雅黑)、卡片 12px 圆角 + 双层柔投影、
|
|
||||||
按钮/输入 8px、标签 6px、alert 10px、modal 14px、导航栏白底、`:focus-visible` 描边、`tabular-nums`、
|
|
||||||
`prefers-reduced-motion`。
|
|
||||||
- `.tb-cover`:封面遮罩;默认浅色(配 `.uk-light` 的黑字兜底),`.tb-cover-dark` 为深色。
|
|
||||||
- `.tb-img-uploader` 上收到 header(原来 item / new-question 各写一份**内联 style**,写死 #f8f8f8 → 深色下露白)。
|
|
||||||
- `templates/home.html`:整页重排为 hero(标题+一句话+探索/规则按钮)+ 3 张功能卡 + 域名迁移公告
|
|
||||||
+ 提醒列表 + `<details>` 折叠更新日志 + 3 个数字统计卡(`.SiteStats` 数值字段,30 分钟 JS 刷新改为按字段更新)。
|
|
||||||
全部原有信息保留;页内 `style=` 归零。
|
|
||||||
- `templates/question/list.html`:封面加 `.tb-cover`(仅当有背景图),新增 12 行 JS 按用户名颜色亮度
|
|
||||||
决定遮罩方向;`min-height:260px` 内联改为 `.tb-profile-cover`。
|
|
||||||
- 6 个页面样式块里的硬编码色值收敛到令牌(explore/question-list/profile/list/item/vote-page,共 ~84 行),
|
|
||||||
CRLF 文件用字节级替换(未改动行尾)。
|
|
||||||
|
|
||||||
验证:`go test ./templates/... ./internal/i18n/...` 全绿;`go build`(overlay)通过。
|
|
||||||
新增 `templates/zz_preview_test.go`:`TEST_PREVIEW=1 go test ./templates -run TestGenPreview` 产出
|
|
||||||
`outputs/preview-{home,box,explore}-{light,dark}.html`,用无头 Chrome 截图核对过亮暗两套。
|
|
||||||
⚠️ 本机 **Chrome 新版 headless 会挂住**,必须 `--headless=old`,且页面外部资源要本地化
|
|
||||||
(`outputs/preview-assets/`,从 cdnjs.snrat.com 取 UIkit 3.3.3;cloudflare/jsdelivr 在本机不通)。
|
|
||||||
|
|
||||||
## 封面遮罩 P0 缺陷修复(重要)
|
|
||||||
|
|
||||||
**症状**:`.tb-cover::after` 遮罩不渲染,且**完全不报错**——截图对比像素只差 ±1(只看着像"没效果")。
|
|
||||||
|
|
||||||
**根因**:UIkit 3.3.3 自带
|
|
||||||
`.uk-flex::after,.uk-flex::before{display:none}`(优先级 0,2,0),
|
|
||||||
而封面元素是 `uk-flex`,我写的 `.tb-cover::after`(0,1,1)优先级不够 → 被压成 display:none。
|
|
||||||
定位方法:`getComputedStyle(el,'::after').display` 得到 `none`;再逐个 `classList.remove` 试 →
|
|
||||||
去掉 `uk-flex` 立刻变 `block`。最后在 uikit.min.css 里 grep 到那条规则。
|
|
||||||
**修法**:选择器写成 `.tb-cover::after, .tb-cover.uk-flex::after` 并显式 `display:block`。
|
|
||||||
|
|
||||||
**实测效果**(无头 Chrome 截图 + 像素测量,非目测):
|
|
||||||
- 深色渐变背景 + 白字:封面底部亮度 0.229→0.102,白字对比度 **3.77→6.89**(过 WCAG AA 4.5)
|
|
||||||
- 线上默认背景(`box_background.png` 2048×1152,接近全白)+ 黑字:0.912→0.971,对比度 19.25→20.42,
|
|
||||||
即**对默认背景几乎没有可见影响**
|
|
||||||
- 亮度判定 JS 已验证:`#fff` 会加 `tb-cover-dark`,`#000` 不加。
|
|
||||||
|
|
||||||
**教训**:CSS 改动不能只看"规则写上了/类加上了",必须量像素。伪元素 + UIkit 工具类组合要在计算样式层验证。
|
|
||||||
|
|
||||||
## 账号背景图的真实链路(回答用户提问时查证)
|
|
||||||
|
|
||||||
- 注册时把 `[upload] default_background` **一次性复制**进 `users.background`(`route/auth/register.go:51`),
|
|
||||||
不是每次渲染兜底 → 之后改配置**不影响存量用户**。
|
|
||||||
- 页面只用 `.PageUser.Background` 渲染 `data-src`;为空则无背景图、无遮罩。
|
|
||||||
- 分享卡是另一条路:`route/question/share.go:345` 用 `user.Background`,为空时回退 `conf.App.ShareBackground`。
|
|
||||||
- 线上 `box.shiroko.one/_/tamakyi` 实际值 = `https://r2.337845818.best/tamabox/box_background.png`。
|
|
||||||
- 预览里那张"渐变图"是 `zz_preview_test.go` 造的测试数据(data: URI 渐变),**不是产品行为**;
|
|
||||||
顺带发现 Go 模板会把 `data:` URI 在属性上下文里替换成 `#ZgotmplZ`(真实 http URL 不受影响)。
|
|
||||||
|
|
||||||
## 设置页「防护设置」缺白底(用户报)
|
|
||||||
|
|
||||||
根因:其它面板都包在 `<section class="tb-pcard">`,只有内容防护面板(profile.html:842)是裸 `div.profile-panel`,
|
|
||||||
直接露出页面灰底 `--tb-bg`。修法:包一层 `section.tb-pcard`,并把 `.tb-pcard` 的 `background:#fff` 改成
|
|
||||||
`var(--tb-surface)`(删掉随之冗余的 dark 覆盖)。
|
|
||||||
实测(计算样式):account 与 security 面板 bg 亮色均 `rgb(255,255,255)`、暗色均 `rgb(36,40,48)`,圆角 10px,完全一致。
|
|
||||||
顺手把 `security` 加入 `render_test.go` 的面板渲染测试列表(原来只测 account/question/chat/vote)。
|
|
||||||
|
|
||||||
⚠️ 教训:用脚本批量缩进/hash 插入时先 dry-run。这次第一版脚本把「空行」也当成需要写 `<section>` 的行,
|
|
||||||
一次性插了 5 个多余标签;靠立刻 `sed` 复查发现并回滚。
|
|
||||||
|
|
||||||
## P1 第一步:CSS 抽离(已完成)
|
|
||||||
|
|
||||||
- 新增 `static/css/base.css`(23KB,含注释;gzip 后约 6KB),内容 = 原 header 里的
|
|
||||||
设计令牌 / UIkit 补丁 / 深色覆盖 / 质感层 / 顶栏响应式 / 标签 / Markdown / 进度条 / 移动端底栏。
|
|
||||||
- `header.html`:内联 `<style>` 从 17.5KB 降到只剩 1 个(算数验证码,跟着 `{{ if CaptchaType }}` 条件渲染);
|
|
||||||
改为 `<link href="/static/css/base.css?v=20261019a">`,位置在 UIkit 样式之后(保证覆盖顺序)。
|
|
||||||
单页 HTML 42.5KB → 30.9KB。
|
|
||||||
- `static/fs.go` 的 `//go:embed` 显式清单加 `css/base.css`(漏了就是运行期 404 + 全站无样式,编译期不报错)。
|
|
||||||
- 静态缓存沿用既有 `CacheControl: public, max-age=86400`;靠 `?v=` 版本号破缓存。
|
|
||||||
|
|
||||||
**验证方式**:改动前后各截一次同样尺寸的截图做像素比对。
|
|
||||||
home-light / home-dark / explore-light 三页差异均 **0.024%**,bbox 只有 (11,56)-(29,74) 一个 18×18 方块
|
|
||||||
= 导航栏随机动物 emoji(每次加载都变),不是样式回归。go build + 模板/i18n 测试全绿。
|
|
||||||
|
|
||||||
## 测试环境复测(P1 CSS 抽离 + 防护面板白底,已上线)
|
|
||||||
|
|
||||||
- `GET /static/css/base.css?v=20261019a` → 200 / `text/css` / **23088 字节,与本地 `static/css/base.css` 逐字节一致** /
|
|
||||||
`Cache-Control: public, max-age=86400` / gzip 后 7109 字节。
|
|
||||||
- 首页 32747 字节(改动前 52747),gzip 10534(改动前 13642);页面内联 `<style>` 只剩验证码 + 首页自己的样式块。
|
|
||||||
- 设置页(登录 admin6):`data-profile-panel="security"` 下已有 `<section class="tb-pcard">`;
|
|
||||||
用线上 HTML + 本地 CSS 复算样式 → account / security 两块 `bg=rgb(255,255,255)`、`border-radius:10px`,页面底 `rgb(247,248,250)`。
|
|
||||||
- `/_/admin6` 公开页:`class="... uk-light tb-cover"`,`:after` 计算值 **display=block, width=689px**
|
|
||||||
→ 遮罩在真实部署上确实渲染了(UIkit `.uk-flex::after{display:none}` 的坑已解决)。
|
|
||||||
该用户用户名为 `rgb(0,0,0)`,走浅色遮罩(符合预期)。
|
|
||||||
- 各页 200:`/`、`/explore`、`/rules`、`/user/questions`;`/user/vote-form` 无参 302 属正常。
|
|
||||||
- 首页首次访问总字节略增(10.5+7.1 vs 13.6KB gzip),但 CSS 跨页缓存后每页少 3.1KB,且服务端少渲染 17.5KB 模板。
|
|
||||||
|
|
||||||
## 专属测试账号(用户指定)
|
|
||||||
|
|
||||||
- `https://box-dev-api.337845818.best/_/admin6`,邮箱 `[email protected]`,密码 `adminadmin`。
|
|
||||||
- 登录需算数验证码,curl 流程:GET /login 取 `_csrf`(同时存 cookie)→ `GET /api/captcha/generate` 拿 `{id, question}`
|
|
||||||
→ 算答案 → `POST /api/captcha/verify`(头 `X-CSRF-Token`,体 `{id, answer}`)→
|
|
||||||
`POST /login`(email/password/_csrf/`g-recaptcha-response=id:answer`)→ 302 到 `/`,cookie 已带登录态。
|
|
||||||
|
|
||||||
## P2 一致性收尾(已实施,未部署)
|
|
||||||
|
|
||||||
1. **导航图标**:🌐🌙☀️ emoji → 内联单色 SVG(`stroke: currentColor`,跟随主题/悬停色)。
|
|
||||||
⚠️ 原计划用 UIkit icons,实测 **UIkit 3.3.3 的图标集里没有 moon / sun**(只有 world),
|
|
||||||
照写会渲染成空白按钮——所以三个图标全部改成内联 SVG,也不依赖 JS。
|
|
||||||
2. **标签**:投票类型 20 处 `uk-label uk-label-primary`(实心蓝)→ `.tb-chip` 轻量胶囊;
|
|
||||||
探索页热门标签原来的 5 色轮换(success/warning/danger/primary/secondary)也统一成 `.tb-chip`。
|
|
||||||
箱主面板 `question-list.html` 另 6 处字节级替换(CRLF 保留)。
|
|
||||||
保留:置顶/私密/被举报/已读 这些**语义状态**仍是彩色 uk-label(有含义,不该降级)。
|
|
||||||
3. **遗留写法**:`<center>` ×2 → `div.uk-text-center`;占比条从
|
|
||||||
`div.uk-progress > div.uk-progressbar(style:width)` 改成 UIkit 3 原生
|
|
||||||
`<progress class="uk-progress" value=".." max="100">`,**删掉 25 行 UIkit2 兼容补丁**,
|
|
||||||
只保留 10px 高/6px 圆角的外观收窄。测试 `percent_check_test.go` 的断言同步改成 `value="66" max="100"`。
|
|
||||||
4. **meta**:删掉重复的 `msvalidate.01`(保留 head 顶部一份)、viewport 补 `initial-scale=1`。
|
|
||||||
|
|
||||||
**踩坑**:暗色主题图标原来写 `display:inline`,span 里放块级 `<svg>` 会让导航栏高度变化 → 整页下移 36px。
|
|
||||||
用行亮度曲线做互相关定位到 dy=36,改成 `inline-flex` 后复测差异回到 0.059%(只剩随机 emoji)。
|
|
||||||
教训:**改图标尺寸/显示模式一定要量整页位移**,不能只看局部截图。
|
|
||||||
|
|
||||||
## P2 线上复测(2026-10-19 12:2x)
|
|
||||||
|
|
||||||
- `GET /static/css/base.css?v=20261019c` → 200 / text/css / **23918 字节,与本地逐字节一致**,页面已引用该版本。
|
|
||||||
- meta:viewport 已带 `initial-scale=1`;`msvalidate.01` 只剩 1 份。
|
|
||||||
- 图标:首页 emoji 🌐🌙☀️ 归零,`<svg ... stroke="currentColor">` ×5(语言 1 + 桌面主题 2 + 移动主题 2);
|
|
||||||
计算样式暗色下 `#theme-toggle .tb-icon-dark` = `display:flex`、亮色图标 `none`;
|
|
||||||
**导航高度亮/暗均 80px** → 上次 36px 位移问题确认修好。
|
|
||||||
- 探索页:`.tb-chip` ×25(全部有底色 `rgb(244,246,248)`、文字 `rgb(92,99,112)`、圆角 6px、字号 11.52px);
|
|
||||||
残留彩色 `uk-label` 4 个,均为置顶/私密等语义状态。
|
|
||||||
- 结果页:`/user/votes/{id}/results` 原生 `<progress class="uk-progress" value max>`,计算高度 **10px**、圆角 6px;
|
|
||||||
遍历 10 个结果页 `uk-progressbar` 全部为 0。
|
|
||||||
- 巡检 `/ /explore /rules /login /register /_/admin6 /_/admin3` 均 200。
|
|
||||||
|
|
||||||
## P2 补漏(已改,待部署)
|
|
||||||
|
|
||||||
复测时发现 **结果页与参与页的题型标签仍是实心彩色** —— 那两处的 class 由 Go 模板函数
|
|
||||||
`fieldTypeLabelClass` 生成,我上一轮只按字面 `class="uk-label uk-label-primary"` 搜索,漏了它们。
|
|
||||||
|
|
||||||
- `templates/user/vote-results.html`(2 处)、`templates/vote/page.html`(2 处):序号 + 题型 → `.tb-chip`;
|
|
||||||
状态标签(未开始/已结束/进行中/已参与)保留彩色,属语义。
|
|
||||||
- 删除 `internal/template/template.go` 的 `fieldTypeLabelClass`(19 行,已无引用)。
|
|
||||||
顺带查明:该函数 `text/textarea` 走 `return ""`,注释说"默认灰",**实际渲染成 UIkit 默认蓝**,
|
|
||||||
所以文本框标签一直是蓝的——注释与实现不符。
|
|
||||||
- `.uk-progress::-webkit-progress-value` 圆角改成两端都圆(`-moz-progress-bar` 同步)。
|
|
||||||
⚠️ 起因是误判:我先前测"右端方角"时像素扫描循环上界写成 `range(110,830)`,恰好等于条宽,
|
|
||||||
于是右端永远"恒定不变"→ 假象。放宽到整幅宽度后两版端点曲线完全一致(830/832/833/834/833/832/830),
|
|
||||||
即**没有视觉 bug**。保留改动只为 Firefox 的 `-moz-progress-bar` 保险。
|
|
||||||
- `base.css?v=` → `20261019d`。`go build` + `./templates ./internal/i18n ./internal/template` 全绿。
|
|
||||||
|
|
||||||
**教训(重要)**:像素测量脚本的扫描范围必须覆盖整幅图,循环上界不能与被测对象边界重合,
|
|
||||||
否则会把"边界截断"读成"图形特征"。之前 3.77→6.89 那组对比不知是否也受此影响,需复查时再确认。
|
|
||||||
|
|
||||||
## P2 补漏部署后复测(12:48)
|
|
||||||
|
|
||||||
- `base.css?v=20261019d` 上线,24023 字节与本地逐字节一致。
|
|
||||||
- 删掉 `fieldTypeLabelClass` 后**无运行期报错**:公开投票页 3 个(200,tb-chip 8/16/4,uk-label 各 1 = 状态)、
|
|
||||||
结果页 4 个(200,tb-chip 8/2/0/16,彩色 label 全 0,progress 7/0/10/19)、
|
|
||||||
草稿预览页 2 个(200,tb-chip 2/6)。
|
|
||||||
- 巡检 `/ /explore /rules /user/questions /user/profile?tab=security /_/admin6 /_/admin3` 全 200 无模板错误。
|
|
||||||
- `/user/votes/{id}/preview` 对**已发布**投票 302 到 `/vote/{token}` 是设计如此(vote.go:1093),不是回归。
|
|
||||||
- 截图核对:结果页「1 単選 / 2 多選」已为浅色胶囊,进度条 10px 圆角 6px;参与页字段头同样统一。
|
|
||||||
|
|
||||||
## "移动端语言/主题按钮不见了"排查(结论:非缺陷)
|
|
||||||
|
|
||||||
实测(线上测试环境,`--force-device-scale-factor=1`):
|
|
||||||
- 宽 360/390/414/640 → 顶栏 `.tb-nav-utils` = `display:none`,底栏 `#tb-mobile-utils` = `flex`,
|
|
||||||
rect `0,639,W,46`,语言 `block`、主题按钮 `block`(48×30,SVG 16×16),命中测试落在 `mobile-lang-select`。
|
|
||||||
- 宽 700/768/980 → 底栏 `none`,顶栏 `flex`,右上角地球 + 月亮都可见(截图 `outputs/t-nav-right.png`)。
|
|
||||||
- 页面级差异:`/_/admin6` 底栏**只有月亮没有语言下拉** —— `IsUserPage=true` 让语言块整体不渲染(桌面同样没有)。
|
|
||||||
- 生产 `box.shiroko.one` 仍是旧构建(页面无 `base.css` 引用,底栏是 emoji 月亮),移动端同样正常。
|
|
||||||
|
|
||||||
用户确认是忘了「移动端移到固定底栏」这个设计,不是 bug。
|
|
||||||
留下的真实缺口:**箱主页(最常见的外部入口)完全没有语言切换**,见 MEMORY.md 前端样式条目。
|
|
||||||
|
|
||||||
## 修正:箱主页无语言切换是设计,不是缺口
|
|
||||||
|
|
||||||
上一节把它记成"真实缺口"是错的。用户明确:**箱主页属箱主个性化,以箱主配置为主**,
|
|
||||||
语言入口不出现是有意为之(`IsUserPage` 门控)。
|
|
||||||
MEMORY.md 对应条目已改为"有意设计、别当 bug 修、要动先问"。
|
|
||||||
|
|
||||||
## 备份(用户要求)
|
|
||||||
|
|
||||||
`../TamaBox-v4-mainline-20261019-1445.tar.gz`(940K,302 条目,排除 .git/.workbuddy/outputs)。
|
|
||||||
P1 剩余两项(首屏跟随系统深色、首页 Waline 懒加载)用户明确**先不做**。
|
|
||||||
|
|
||||||
## 主题系统方案(只出设计,未实现)
|
|
||||||
|
|
||||||
用户诉求:箱主在后台选主题、仅对箱主相关页面生效;站点级默认主题在配置里;要一份可复用的主题底稿。
|
|
||||||
|
|
||||||
已确认的现状事实(设计依据):
|
|
||||||
- 设计令牌只有 16 个,集中在 `static/css/base.css` 的 `:root`(亮) + `html[data-theme="dark"]`(暗)。
|
|
||||||
- `<html lang>` 在 `templates/base/header.html:2`,服务端渲染;主题属性可放在这里(无 FOUC)。
|
|
||||||
- `question.Pager`(`route/question/page.go:45`)是 `/_/{domain}` 与 `/_/{domain}/{id}` 的共同中间件
|
|
||||||
→ 打主题标记的唯一合适位置。
|
|
||||||
- `/user/*` 与 `/_/*` 共用 `IsUserPage`(context.go:241 按 URL 前缀判定),所以「箱主相关页面」
|
|
||||||
≠ IsUserPage,必须单独区分,否则设置页会被主题化。
|
|
||||||
- `static/fs.go` 与 `templates/embed.go` 都是显式 embed 清单(新增主题 css 必须手加)。
|
|
||||||
- 用户模型已有大量个性化字段(usernamecolor/introcolor/background…),主题作为 `users.theme` 同族最自然。
|
|
||||||
- 分享卡是服务端绘制图片(`route/question/share.go`),不经过 CSS → 主题不覆盖它。
|
|
||||||
|
|
||||||
关键结论(写进方案的核心):令牌覆盖必须用
|
|
||||||
`html[data-theme-style="x"]`(0,1,1,压过 `:root` 的 0,1,0)+
|
|
||||||
`html[data-theme="dark"][data-theme-style="x"]`(0,2,1,压过 base 的 dark)。
|
|
||||||
只写浅色块会让深色模式被浅色令牌污染(同特异性 + 主题文件后加载)。
|
|
||||||
|
|
||||||
## 主题系统初版(已实现,未部署)
|
|
||||||
|
|
||||||
用户拍板:①作用范围跟随箱主配置(即 /_/ 箱主相关页面)②主题只做配色/图案/整页背景,**不涉及布局**
|
|
||||||
③主题名可本地化 ④分享卡不参与。
|
|
||||||
|
|
||||||
实现:
|
|
||||||
- `internal/theme`:注册表 + 三级回退。`[theme] default/available` → 白名单(slug 正则 + css 是否存在),
|
|
||||||
`Resolve(用户值)`:`default`→空串(不加载)/命中→该 slug/其余→站点默认。"" 与 "default" 语义不同:
|
|
||||||
""=跟随站点默认,"default"=箱主明确选基础外观。
|
|
||||||
设置页色块从主题 css 浅色块里抓 `--tb-bg`/`--tb-primary` 拼渐变(值必须过色值正则,防注入)。
|
|
||||||
- `users.theme varchar(32) default ''`(AutoMigrate 加列)+ `UpdateTheme` 单列更新。
|
|
||||||
- `question.Pager` 注入 `Data["ThemeSlug"]`(`/_/{domain}`、单题、密码页、分享卡路由共用这一个中间件);
|
|
||||||
header.html 在 `<html>` 输出 `data-theme-style` 并条件加载 `themes/<slug>.css`。
|
|
||||||
- 设置页账号面板末尾新增「主题外观」卡片(独立表单 `POST /user/profile/theme`,不塞进 UpdateProfile,
|
|
||||||
避免与该面板其它字段互相覆盖);`{{ if .Themes }}` 包裹,未配置主题时不显示。
|
|
||||||
- 静态版本号统一 `conf.StaticVersion = "20261019e"`(base.css 与主题 css 共用,改样式只改一处)。
|
|
||||||
- `static/fs.go` 改嵌目录 `css/themes`(以后加主题文件不用再改 embed 清单)。
|
|
||||||
- 主题文件:`_scaffold.css`(底稿,含 19 令牌说明 + 素材规格 + 特异性规范)、`default.css`(恒等,仅色块预览)、
|
|
||||||
`sakura.css`(示例,纯 CSS 网点图案 + 粉调)。
|
|
||||||
- base.css:body 接 `--tb-page-image` / `--tb-page-pattern` / `--tb-page-pattern-size` 两层背景;
|
|
||||||
**UIkit 主色是编译期常量 #1e87f0**,新增映射块把 a/uk-link、uk-button-primary、checkbox/radio、input:focus、
|
|
||||||
uk-tab active 接到 `--tb-primary`(基础主题值相同,默认外观零变化)。
|
|
||||||
|
|
||||||
实测(无头 Chrome,box 页 data-theme-style=sakura):
|
|
||||||
- 浅色 primary `#e5678f`/bg `#fff6f8`/pattern 16px 网点命中;深色 `#ff8ab0`/`#241a1f`,卡片表面各自跟随。
|
|
||||||
- 未选主题的 box 页:无 data 属性、无 themes link、主按钮仍 `rgb(30,135,240)` → 零回归。
|
|
||||||
|
|
||||||
踩坑:
|
|
||||||
1. 第一版 sakura 页「发送提问」还是蓝色 —— UIkit 组件色写死,令牌覆盖不到,必须按组件映射。
|
|
||||||
2. 预览里 `{{ .StaticVersion }}` 为空(预览夹具没这个键),导致按固定串替换主题 css 失败;改用正则。
|
|
||||||
3. `writePreview` 用全文搜 `data-theme="dark"` 判断是否已加属性 → 命中内联主题 css 里的选择器,深色预览没生效;
|
|
||||||
改为只看 `<html ...>` 开标签。
|
|
||||||
4. 脚本改 form/user.go 时把插入块写在结构体中间,拼出 `} Language ...`;靠 Read 复查发现并重写。
|
|
||||||
|
|
||||||
## 主题 follow-up(部署后反馈两处)
|
|
||||||
|
|
||||||
1. **/user/* 不生效主题**:根因是 `ctx.Data["ThemeSlug"]` 只在 `question.Pager`(`/_/{domain}`)里注入,
|
|
||||||
`/user/*` 完全没走这条链路。修法:`internal/context/context.go` 的 Contexter 里按
|
|
||||||
`isThemeScope(path) && c.IsLogged` 注入 `theme.Resolve(c.User.Theme)`;
|
|
||||||
`isThemeScope` 只认 `/user` 与 `/user/`(不能用裸 `HasPrefix("/user")`,会误吞 `/username`)。
|
|
||||||
新增 `internal/context/context_test.go` 守住路径白名单;page.go 的注释同步改。
|
|
||||||
公开页(`/`、`/explore`、`/vote/{token}`)**仍不跟随**登录用户主题,保持站点默认。
|
|
||||||
|
|
||||||
2. **sakura 显示奇怪**(用户选择:顶栏白与粉底割裂 / 卡片纯白太突 / 底色分布不均):
|
|
||||||
- base.css 新增令牌 `--tb-navbar`(`:root` 里 `var(--tb-surface)`,亮暗两条 navbar 规则都改成
|
|
||||||
`var(--tb-navbar, var(--tb-surface))`),主题可单独给顶栏上色。
|
|
||||||
- sakura v2:底色改**纯色** `#fceef3`(不用渐变——渐变会被读成"不均匀"),
|
|
||||||
顶栏 `#fff6f9`、卡片 `#fffafc`(暖白,不再是纯白硬切),网点从 16px/0.14 放宽到 24px/0.05。
|
|
||||||
- 静态版本 `20261019f`;`_scaffold.css` 令牌说明 19→20。
|
|
||||||
- `profile.theme.hint` 四语言文案改成「提问箱页面 + 本人后台生效,公开页保持默认」。
|
|
||||||
- `zz_preview_test.go` 抽出 `inlineThemeCSS()`,并新增 `preview-profile-sakura-*`(后台页面也要在主题下检查)。
|
|
||||||
|
|
||||||
验证:`go build ./...`(须带 webp overlay)、i18n/theme/context/templates 四个包测试全绿。
|
|
||||||
渲染法:抓线上 `/_/admin6` HTML → 把 base.css 指到本地新文件、主题 css 内联 → 无头 Chrome `--headless=old` 截图。
|
|
||||||
|
|
||||||
## 投票页 + 标签主色(部署复核后第二轮)
|
|
||||||
|
|
||||||
- `/vote/{token}` 主题:`route/vote/vote.go` 的 `Page` 里本来就会 `db.Users.GetByID(vote.UserID)` 取 owner,
|
|
||||||
直接在同一分支补 `ctx.Data["ThemeSlug"] = theme.Resolve(owner.Theme)`;box 页 / /user 后台 / 投票页三处
|
|
||||||
全部由发布者配置决定。GetByID 失败(箱主已删)保持站点默认外观。
|
|
||||||
- **`uk-label` 默认色就是 UIkit 主色 #1e87f0**:主题下提问管理的「未回答 / 未读」两枚胶囊还是蓝的。
|
|
||||||
base.css 追加
|
|
||||||
`.uk-label:not(.uk-label-danger):not(.uk-label-warning):not(.uk-label-success):not(.uk-label-secondary)`
|
|
||||||
→ `--tb-primary`。**必须用 :not() 排除语义色**:同特异性且 base.css 在 UIkit 之后,不排除会把
|
|
||||||
danger/warning/success 一起刷成主色。默认主题实测仍是 (30,135,240),零回归。
|
|
||||||
- StaticVersion → `20261019g`。
|
|
||||||
- 线上复核(admin6 专属账号,curl 走完整登录:GET /login 取 _csrf → /api/captcha/generate →
|
|
||||||
/api/captcha/verify(**answer 必须是数字,传字符串会 400 请求格式错误**)→ POST /login):
|
|
||||||
`/user/profile`、`/user/questions` 均已输出 `data-theme-style="sakura"` 与 themes link(v=20261019f)。
|
|
||||||
投票页在 f 版本仍无主题 → 需再部署一次。
|
|
||||||
|
|
||||||
## 投票页语言跟随发布者(第三轮)
|
|
||||||
|
|
||||||
- 抽出 `applyOwnerLocale(data map[string]interface{}, owner *db.User)`(route/vote/vote.go 顶部):
|
|
||||||
同时写 `Owner` / `Lang` / `LangCode` / `ThemeSlug`。
|
|
||||||
签名故意收 Data 而不是 `context.Context`:`Context` 内部字段不可在包外构造,收 map 才能单测
|
|
||||||
(`route/vote/locale_test.go`,纯内存,不碰 DB)。
|
|
||||||
- `Page`:owner 查询提到 `FeatureStateOf` **之前**——否则「投票功能已关闭」这类提示先按访客语言生成。
|
|
||||||
- `Submit`:同样先取 owner 再取 `lang`,保证提交后回跳页的 flash 与页面语言一致;
|
|
||||||
顺带删掉尾部重复的 `GetByID(owner)`(表单类型通知邮件复用同一个 owner),少一次查询。
|
|
||||||
- 四语言 `profile.theme.hint` 与 changelog 范围说明补上「你发布的投票页」。
|
|
||||||
- 验证:vote/i18n/context/theme/templates 全绿;线上 `lang="ja"` + 中文内容的不一致待部署后复测。
|
|
||||||
|
|
||||||
### 部署后复测(v=20261019g)
|
|
||||||
|
|
||||||
- `/vote/7kcP5kR8YnJC`(admin6):`lang="zh-CN" data-theme-style="sakura"`,base/theme css 均 v=g,
|
|
||||||
日文串(投票する / 匿名での参加 / ログイン)全部消失 → 语言与主题同步跟随发布者。
|
|
||||||
- `/ _/admin6`、`/user/profile`:仍为 `zh-CN` + sakura,未回归。
|
|
||||||
- 隔离性:另取三个 token 抽查 —— 4G3i79OCYa2E / 7l0MwF7cLz4T 为 `lang="ja"`(无语言偏好、无主题),
|
|
||||||
9J1zgXwQ05Ai 为 `lang="zh-CN"` 且**无** data-theme-style → 主题不会跨箱主泄漏。
|
|
||||||
- 遗留观感问题(未改,等用户决定):投票页的「允许匿名参与」提示条是 `uk-alert-primary`,
|
|
||||||
仍是 UIkit 浅蓝,在粉底上偏冷。改它需要「浅底 + 深主色文字」的对比度,现有 `--tb-primary-weak`
|
|
||||||
叠白后对比度只有 ~2.6:1,得先加一个 primary-strong 令牌,故未动。
|
|
||||||
|
|
||||||
### 顶栏配色再修(v=20261019h)
|
|
||||||
|
|
||||||
用户反馈「探索/注册链接底色粉色,但整条顶栏是灰色,很突兀」。实测部署版:顶栏 `#fff6f9`、
|
|
||||||
页面 `#fceef3` —— 近白横带压在饱和粉底上,肉眼就读成「白色/灰色的一条」。无头 Chrome 截图 +
|
|
||||||
像素采样确认不是 UIkit 默认 #f8f8f8 回退(那说明变量没生效,现象会连带整页失效)。
|
|
||||||
- sakura 浅色 `--tb-navbar` 直接取页面底色 `#fceef3`(深色同样 `#241a1f`),横带消失;
|
|
||||||
- 顶栏链接 `#999` 在粉底上偏淡 → 主题内微调为 `--tb-text-secondary`,hover 用 `--tb-primary`
|
|
||||||
(深色被 base.css 的 `!important` 规则接管,不受影响);
|
|
||||||
- 顺带 `uk-alert-primary`(匿名参与提示/空列表)的 UIkit 浅蓝是页面上唯一的冷色块,
|
|
||||||
改为 `--tb-primary-weak` 底 + `#a63a5e` 文字(≈5.3:1)。
|
|
||||||
- 以上三条都写在 sakura.css 的「组件微调」段,只对 sakura 生效,默认主题零影响。
|
|
||||||
|
|
||||||
## 主题「砂狼白子 / shiroko」(v=20261019i)
|
|
||||||
|
|
||||||
- `static/css/themes/shiroko.css`:灰蓝银白令牌(浅 #edf2f8 / 主色 #4a7fc1),
|
|
||||||
组件微调只有两条 —— `.uk-card-default` 右下角水印立绘(`background-size: auto 176px`,纯背景层不改布局)、
|
|
||||||
顶栏链接色提到 `--tb-text-secondary`;深色不铺立绘(浅色构图在深色下会糊)。
|
|
||||||
- 素材 `static/images/themes/shiroko/`(68KB):`bg-desktop.webp`(2400×1350) / `bg-mobile.webp`(900×1600) /
|
|
||||||
`hero-fade.webp`(275×620,16% alpha) / `pattern.svg` / `pattern-dark.svg`。
|
|
||||||
- **`static/fs.go` 的 embed 清单加了 `images/themes`**(目录项,子目录自动覆盖)——
|
|
||||||
这是「新增顶层静态目录必须手工登记」的又一次踩点。
|
|
||||||
- 背景是 PIL 合成的:1280 行的竖向渐变 + 白色径向光晕 + 立绘 30% 贴右下 + 狼 10% / 光环 20% / 纸飞机 18%。
|
|
||||||
实测 mean≈244、std≈9;右侧留白区 std 13(立绘可见),中间 std 1.9(正文区干净)。
|
|
||||||
- ⚠️ **`cover` 会按视口裁剪**:竖屏/超宽屏会把贴在图片最右的立绘裁掉,所以立绘不能太靠边 ——
|
|
||||||
位置要留在图片横向 24%~76% 之外就得配 `@media` 换图(本次桌面 16:9 + 移动 9:16 两张)。
|
|
||||||
- 生成素材(ImageGen)8 张里只有 3 张可用:线稿图案(狼/光环/纸飞机)抠白底干净;
|
|
||||||
2 张人像抠底失败(背景不是纯白,`background: transparent` 实际没生效),2 张场景背景几乎没有内容(std<5)。
|
|
||||||
教训:**没有视觉能力时,生成图要用 `mean/std/edge` 先验内容量**,别直接拿去合成。
|
|
||||||
- 素材来源与版权写在 `docs/theme-shiroko.md`(角色版权 NEXON / Yostar,非商用同人)。
|
|
||||||
- 启用需要在 `conf/app.ini` 的 `[theme] available` 里加 `shiroko`(示例配置已更新)。
|
|
||||||
|
|
||||||
## shiroko 插画重做:动物线稿 → Q 版人物(v=20261019k)
|
|
||||||
|
|
||||||
用户反馈「简笔画不是画动物,是人物形象」,插画层整体重做。
|
|
||||||
|
|
||||||
- 新增 `dev/gen_shiroko_chibi.py`:手写 SVG 生成 6 个 Q 版白子(wave/cheer/think/run/doze/peek),
|
|
||||||
3.8KB/张,造型逐项对齐用户立绘(灰发蘑菇头 + 圆钝狼耳 + 倾斜光环 + 青围巾 + 深蓝外套 + 灰裙 + 青黑球鞋)。
|
|
||||||
**生成模型这条路走不通**:`background: transparent` 不生效、无视觉反馈时无法判断好坏;
|
|
||||||
代码画路径反而可控、体积小、任意尺寸不糊。
|
|
||||||
- ⚠️ **cover 必然裁掉贴边插画**:1280×1400 视口下,2400×1350 的背景图只露出中部 1234px,
|
|
||||||
右下立绘被裁成 38px 一条。→ base.css 新增装饰层令牌 `--tb-deco / --tb-deco-size / --tb-deco-position`
|
|
||||||
(固定像素 + 四角定位 + fixed,不随视口比例变化)。
|
|
||||||
- ⚠️ **多背景层的值个数必须恒定**:deco 先写成 4 个值,background-image 变 6 层而 size/position 只有 3 个,
|
|
||||||
CSS 循环取值 → 整页重复小人。改成固定 6 个槽位(1 图案 + 6 装饰 + 1 背景 = 8 层),
|
|
||||||
主题必须写满 6 个(缺的填 none),这条写进了 `_scaffold.css`。
|
|
||||||
- 删掉 `bg-desktop.webp` / `bg-mobile.webp` / `hero.webp` / `dev/gen_shiroko_bg.py`;
|
|
||||||
整页底色改成纯 CSS 渐变(不用图片,也就没有 cover 缩放差异)。
|
|
||||||
- 深色关掉 deco(浅色线稿在深色底上糊成灰块);pattern 里的爪印换成 Q 版头像,
|
|
||||||
opacity 0.42→0.2、tile 240→320(原来铺得太密,像壁纸)。
|
|
||||||
- 验证:默认主题背景仍 `#f7f8fa`、顶栏纯白、卡片纯白(零回归);箱主页与 `/user` 后台四角装饰都不压内容;
|
|
||||||
移动端只留立绘 + peek。
|
|
||||||
|
|
||||||
## shiroko 插画二次返工:手绘 SVG → 参考立绘图生图(v=20261019l)
|
|
||||||
|
|
||||||
用户反馈上次的简笔画"太抽象"。根因判断:用户的立绘是**平涂无描边色块风**,
|
|
||||||
我用手绘线稿去配它,风格本身就冲突,而且没有真人手感,形准也做不到。
|
|
||||||
|
|
||||||
- 新方案:ImageGen **图生图**(`image1` = `static/images/shiroko-cutout.png`,`input_fidelity: high`),
|
|
||||||
提示词统一为「same character, same art style... flat vector illustration, no outlines」+ 换动作句,
|
|
||||||
生成 4 个姿态(wave / cheer / think / doze),1024×1024、纯白背景。
|
|
||||||
**四张角色特征与画风一致**,这一步是手绘做不到的。
|
|
||||||
- 抠图:四角 + 四边中点 8 点 floodfill(容差 16)+ alpha 1px MinFilter 收边,裁 bbox 后统一高 620、
|
|
||||||
WebP q90。实测无白边,光环中间的孔也正确透出。
|
|
||||||
- 删除 `dev/gen_shiroko_chibi.py` 与 6 个手绘 SVG;`chibi/*.webp` 4 张共 106KB。
|
|
||||||
图案里的手绘小头像也删了(和位图人物并排显得最抽象),改成雪花 + 光环 + 圆环 + 点簇。
|
|
||||||
- ⚠️ **deco 的 `center` 定位在矮视口会撞车**:右下立绘 520px 高,1366×768 下从 y=248 一直到底,
|
|
||||||
托腮放 `right center` 会直接站在立绘头顶。百分比是「(视口-图高)×pct」,改用 `top 46%` 并挪到左侧中部。
|
|
||||||
验证视口:1366×768 / 1440×900 / 1280×1400 三档,人物互不重叠。
|
|
||||||
- docs 补了完整提示词骨架、动作句与抠图参数,说明"必须平涂无描边"这条经验。
|
|
||||||
|
|
||||||
## shiroko 布局重排 + 新坐姿(v=20261019m)
|
|
||||||
|
|
||||||
用户反馈:左右不协调、最上边的头被顶栏挡住。
|
|
||||||
- 布局改为**左右各两个**(左上 wave / 左下 sit,右上 cheer / 右下立绘),
|
|
||||||
`≥920px` 真实视口才补中间两个(think / doze),矮屏不放(否则压到立绘)。
|
|
||||||
- 顶部定位从百分比改成**固定 `top 96px`**:顶栏实际高 60px(不是 80),
|
|
||||||
百分比是「(视口-图高)×pct」,小视口下算出来的值会落进顶栏里。
|
|
||||||
`bottom 0` 也改成 `bottom 6px`,避免脚贴页底。
|
|
||||||
- 新增坐姿(盘腿),ImageGen 图生图第 2 次才可用:第 1 张脚被画到画布底边(底部 78 像素贴边),
|
|
||||||
提示词补 `centered composition, whole body fully visible with generous empty margin` 后正常。
|
|
||||||
- ⚠️ **Chrome headless 的 `--window-size` ≠ CSS 视口**:实测 1080 → innerHeight 985、768 → 673
|
|
||||||
(差约 95px 的窗口边框)。所以 `@media (min-height: 1050px)` 在 1080p 下永远不触发。
|
|
||||||
判断媒体查询门槛要按「真实视口」写,截图验证必须加 `--force-device-scale-factor=1`,
|
|
||||||
否则系统 DPI 缩放会让截图尺寸和 window-size 也对不上(曾出现 1920 请求出 1088 宽)。
|
|
||||||
- git:仓库可用(origin = https://git.shiroko.one/tamakyi/tamabox,main),
|
|
||||||
已本地提交 `0c39aff`,但 `credential.helper=store` 里的凭据过期 → `Failed to authenticate user`。
|
|
||||||
打包 `../TamaBox-v4-mainline-20261019-2037.tar.gz`(1.5M,含图片素材)供直接部署。
|
|
||||||
|
|
||||||
### git push 补推
|
|
||||||
|
|
||||||
上次的 `Failed to authenticate user` 是临时态 —— 用户本机 Git for Windows 认证过之后 store 凭据恢复,
|
|
||||||
本轮 `git push origin main` 直接成功(`38cd41a..0c39aff`),`git rev-list --left-right --count origin/main...HEAD` = 0 0。
|
|
||||||
远端 8 个 shiroko 素材文件齐全。教训:**不要用一次 push 失败就断言无凭据**,先看一眼提交是否已存在、再试一次。
|
|
||||||
|
|
||||||
### 部署后「设置页找不到 shiroko」
|
|
||||||
|
|
||||||
线上实测:`/static/css/themes/shiroko.css?v=20261019m` 200 / 5038B 且与本地逐字节一致,
|
|
||||||
说明二进制确实带了新主题;但 `/user/profile` 渲染出来的主题单选只有 `""`、`default`、`sakura`
|
|
||||||
→ **根因是 `[theme] available` 白名单没登记 shiroko**(列表在 `theme.load()` 里按配置构建,启动时读一次)。
|
|
||||||
- `internal/theme/theme.go` 新增 `reportRegistry()` / `discover()`:启动打 info「主题注册表已加载 enabled=...」,
|
|
||||||
并把「css 在、但没登记」的主题以 warning 打出建议的 available 值。
|
|
||||||
没有这段时,这个故障在日志里完全没有痕迹,只能靠猜。
|
|
||||||
`discover()` 用 `fs.Glob(static.FS, "css/themes/*.css")`,用 slugRe 过滤掉 `_scaffold.css`(下划线开头)。
|
|
||||||
- `conf/app.sample.ini`:`available = default,sakura,shiroko` + 补充「没登记就不显示」的注释。
|
|
||||||
- `docs/theme-shiroko.md` 补「部署后看不到主题」两段排查(先 curl css 看 200/404,再看 available)。
|
|
||||||
- 排查口径:**先分「文件没发布」与「没登记」两类**,别一上来怀疑缓存;`StaticVersion` 只影响缓存,不影响列表。
|
|
||||||
|
|
||||||
### shiroko 装饰返工:散射 → 左下角一组(v=20261019n)
|
|
||||||
|
|
||||||
用户反馈「主题正常了,但展示方式怪怪的」。实测线上 1512×982 截图确认三处:
|
|
||||||
1. 四角各一个人 = 贴纸墙纸(人物之间、人物与内容都没有关系);
|
|
||||||
2. 半透明立绘(16%)叠在实心人物旁边 → 虚实混搭,像重影;
|
|
||||||
3. 右下角人物和右对齐的页脚文案(社区规则 / 备案号 / 存活时长)挤在一起。
|
|
||||||
改法:只留左下角一组(sit 330 + wave 190,`left 0/bottom 0`),删掉 hero-fade,
|
|
||||||
纹样减到雪花 + 光环 + 两点(opacity .16、tile 340),卡片补 1px 冷调描边(box-shadow 不占布局)。
|
|
||||||
990px 以下缩小左移,900px 以下(卡片占满宽)不出人物。
|
|
||||||
|
|
||||||
⚠️ **`background-position` 的四值语法必须以边关键字开头**:写成 `max(8px, calc(50% - 555px)) bottom 0`
|
|
||||||
整条声明作废并回落到左上角(表现为人物跑到顶上、背景出现错位色带)。
|
|
||||||
要贴卡片边只能 `left max(8px, calc(50% - 555px)) bottom 0`。
|
|
||||||
|
|
||||||
### 纹样横向条纹(v=20261019o)
|
|
||||||
|
|
||||||
图案元素原先全在 tile 上半部(雪花 y=52 / 光环 62 / 点 150、176),铺开后在整页形成
|
|
||||||
**横向条纹带**(左栏 60px 带宽均值 245→237→249,跳变 12),这是"看着怪"的第二个来源。
|
|
||||||
改成 3 行均布(52 / 132 / 152-212)后收敛为单调 245→236。
|
|
||||||
⚠️ 图片不走 `StaticVersion` 缓存键(`Cache-Control: max-age=86400`),换图案要自己给 url 加 `?v=`。
|
|
||||||
|
|
||||||
### 定稿:半隐在卡片后的单人(v=20261019p)
|
|
||||||
|
|
||||||
用户从 A/B/C/D 四版里选了 D(半隐在卡片后),并追加两条:人物离开底边一点、纹样再淡一点。
|
|
||||||
- `sit` 单人,`left max(6px, calc(50% - 515px)) bottom 26px`;宽度 177px,右缘压进卡片 40px。
|
|
||||||
- **离底边后必须配接触阴影**:一层 `radial-gradient(ellipse, rgba(63,96,138,.22), transparent 72%)`
|
|
||||||
128×15px,垫在脚下一格(deco 第 2 个槽位,绘制顺序在人物之下)。没有它,抬高 26px 会读成悬空。
|
|
||||||
- 纹样 opacity 0.13→0.085(深色 0.09→0.055);`?v=20261019p` 同步改。
|
|
||||||
- 剩余素材(wave/cheer/think/doze)保留但不上页面,换姿态只改 `--tb-deco` 的 url。
|
|
||||||
|
|
||||||
### shiroko 按二次元官网重做:整页场景 + 悬浮磨砂 UI(v=20261019q)
|
|
||||||
|
|
||||||
用户反馈「还是很怪,去网上学一下布局」。实际去看站:
|
|
||||||
- 蔚蓝档案 JP 首页(bluearchive.jp,截图存 outputs/ref/bluearchive_jp_.png):**整屏一张场景插画,
|
|
||||||
logo/按钮/横幅浮在画面上**,没有"白卡片"这层东西;
|
|
||||||
- marshmallow-qa / peing 这类真正的提问箱服务被 Cloudflare 拦,拿不到参照。
|
|
||||||
结论:前几版都卡在"白底 + 留白摆人物"这个结构上,人物没有环境,怎么调位置都像贴纸。
|
|
||||||
|
|
||||||
改法:生成教室场景(ImageGen,1536×1024)→ 裁掉底部 160px 水印 → 混白 38% → 模糊 1.4px →
|
|
||||||
WebP 1920×1080(22KB)/ 720×1003 竖版(6KB);卡片与顶栏改 `rgba(255,255,255,.86)` +
|
|
||||||
`backdrop-filter: blur(16px)`;人物 340px 站在左侧地面、离底 36px + 椭圆接触阴影;浅色不再叠纹样。
|
|
||||||
深色仍用渐变 + pattern-dark(白天教室压在深底上会脏)。
|
|
||||||
|
|
||||||
⚠️ **`ruleBody` 必须剥 CSS 注释**:令牌前写一行注释时,`tokenValue` 按 ";" 切完再按 ":" 取键,
|
|
||||||
会拿到 `/* ... */ --tb-surface`,色块预览静默变空、`TestThemeCSSComplete` 报"缺少令牌"。
|
|
||||||
已加 `commentRe` + `TestRuleBodyIgnoresComments`。
|
|
||||||
⚠️ ImageGen 出的图右下角有 `AI生成 WORKBUDDY` 水印,用作整页背景必须裁掉。
|
|
||||||
|
|
||||||
## 23:40 人像换成「窗边背影」
|
|
||||||
|
|
||||||
- 用户否掉了盘腿坐的正面人像("和背景没那么突兀的")。根因:正面直视 + 无环境 + 饱和度高,
|
|
||||||
在压白 38% 的空教室里就是一块贴纸。
|
|
||||||
- 新素材 `chibi/window.webp`(135×460, 13KB):文生图出侧后方 3/4 视角(双手背在身后)→ 再
|
|
||||||
image-to-image 转"纯色块无描边 + 去窗框"→ 抠白底 → **RGB 压白 16%** → 高 460 / q86。
|
|
||||||
第一步文生图会把窗框画进背景,抠不干净,必须靠第二步重绘。
|
|
||||||
- CSS:`--tb-deco` 首槽换 window.webp,高 380→400px,接触阴影 96×13 复位到脚底;`StaticVersion` → `20261019r`
|
|
||||||
(图片另带 `?v=20261019r`)。删除 `sit.webp`;`docs/theme-shiroko.md` 补两步做法与摆位说明。
|
|
||||||
- 提交 `e95671f` 已推送。教训:**人像与场景的"雾度"要对齐**——场景压白多少,人像也压相当比例。
|
|
||||||
|
|
||||||
@@ -1,427 +0,0 @@
|
|||||||
# 2026-10-22
|
|
||||||
|
|
||||||
## new-sakura 主题:纯 CSS 樱花
|
|
||||||
|
|
||||||
用户要求「用编辑器生成 new-sakura,CSS 做樱花、每页位置不同、上传到已配好的 S3」。
|
|
||||||
|
|
||||||
- **结论:生成器做不了这个** —— 它的装饰层只支持图片素材(导出打 zip / 传对象存储),
|
|
||||||
没有内联 SVG 图层。所以改成手写主题 CSS(结构完全遵循生成器输出:令牌 + 10 槽 deco +
|
|
||||||
scope 覆盖),生成器只用来做视觉验证。
|
|
||||||
- 实现:`dev/theme-tools/gen_new_sakura.py` 生成 `static/css/themes/new-sakura.css`。
|
|
||||||
花/瓣/枝/底纹都是内联 SVG 的 data URI(`quote(svq, safe="")` 全编码,避免 URI 里的
|
|
||||||
逗号被 `TestDecoSlotCount` 当成分层符)。CSS 50KB(gzip ≈12KB)。
|
|
||||||
- 六个 scope 各一套 `--tb-deco` 位置;窄屏(≤640px)装饰全关、底纹缩到 200px;
|
|
||||||
深色换亮粉 + 关底纹(浅粉压深底=噪点)。
|
|
||||||
- 调参踩到的:①花枝 `auto 390px` 放右上角时可见部分只剩一团花 → 缩到 300px 并内移;
|
|
||||||
②枝上 5 朵 scale≈1 的花是一串糖葫芦 → 改成 8 朵 0.4~0.62;
|
|
||||||
③底纹 260px 双花双瓣读成「包装纸」→ 改成 300px 一朵花 + 两片瓣、各自错位。
|
|
||||||
- 对比度:primary 候选 `#c14a72` 只有 4.39:1,压深到 `#b8436a`(4.87);muted `#77626a`(5.27);
|
|
||||||
深色 `#ff93b4`(8.17)/`#ab9099`(5.82)。
|
|
||||||
- i18n 四份词典加 `theme.name.new-sakura`(插在 shiroko 之后,gofmt_keep_crlf 后每份只 +1 行);
|
|
||||||
`conf/app.sample.ini` 的 available 登记。
|
|
||||||
- ⚠️ 本机 Chrome/Edge 的 localStorage 里**没有**生成器的 S3 配置记录,项目也没有
|
|
||||||
`conf/app.ini` → 「已预先配置好的 S3」在本机取不到,上传这一步需要用户给凭据。
|
|
||||||
纯 CSS 主题本身零素材,严格来说 S3 无事可做(已向用户确认)。
|
|
||||||
|
|
||||||
**R2 自定义域可用 + 素材包一键上传(接上一条)**
|
|
||||||
- `r2.337845818.best` 实测公开可读(7 个对象全部 200、Content-Type 正确);桶的 S3 endpoint 仍 403。
|
|
||||||
- 樱花位图从 `gen_new_sakura.py` 的同一份 SVG 导出(`export_assets.py` 直接 import 它,不另画),
|
|
||||||
`--upload` 一次传 CSS + 5 张 webp + 生成的 INSTALL.txt(域名/体积写实数)。
|
|
||||||
- `serve.upload_files` 支持逐文件 `cacheControl`:原来硬编码 immutable,而
|
|
||||||
`INSTALL.txt` / 镜像 CSS 的键不带版本 → 改了看不到。现在位图 1 天、CSS 5 分钟、README 60 秒。
|
|
||||||
- ⚠️ **CF 边缘缓存不会因为重传而失效**:第一版以 immutable 传过,`cf-cache-status: HIT`、Age>800,
|
|
||||||
重传后拿到的仍是旧副本(带 `?cb=` 才是新内容)。清缓存只能去控制台按 URL 清。
|
|
||||||
- ⚠️ **gzip 纠错**:服务端 `internal/route/gzip.go` 会压 CSS,内联版 51KB 实际传输 **3.9KB**,
|
|
||||||
位图版 2.7KB + 5 张图 48.9KB ≈ 51.6KB。所以"换位图省 CSS 体积"是错的 ——
|
|
||||||
位图唯一的理由是素材要给非 CSS 场景复用。文档与 INSTALL.txt 都已改。
|
|
||||||
- `image_bucket_cdn_host` 沿用 app.ini 同名键(只写主机名),`export_assets.py` 据此拼公开地址。
|
|
||||||
|
|
||||||
## 线上 521 排障(box.shiroko.one 打不开)
|
|
||||||
|
|
||||||
- 现象:`https://box.shiroko.one` 全路径 **521**,同机 IP+端口正常。
|
|
||||||
- 实测链路:`box.shiroko.one` → CNAME `cloudflare.182682.xyz`(第三方 CF SaaS 接入/优选,
|
|
||||||
上游 `source.cfnode.eu.org`),**不是** `<UUID>.cfargotunnel.com`。DNS/TLS 均正常(tls=0),
|
|
||||||
响应体 `error code: 521`,5 次请求全 LAX,0.6s 快速失败(RST 特征,非超时)。
|
|
||||||
- 对照实验有价值:`box.tama.guru` 走**同一个** CNAME 目标却正常 301 → 中转 zone 边缘健康;
|
|
||||||
它不回源,而 box.shiroko.one 要回源才 521 ⇒ 断点在「CF → 源站」这一段。
|
|
||||||
- `shiroko.one` / `www` / `tama.guru` apex 都 CNAME `navi-ew2.pages.dev`(CF Pages 导航站),与本站无关,
|
|
||||||
别拿它们的 200 当证据。
|
|
||||||
- 候选根因(待服务器侧确认):① cloudflared 断开(严格说 tunnel 无 connector 多为 502/1033);
|
|
||||||
② `scripts/edgeone-origin-whitelist.sh` 的 `REJECT tcp-reset` 拦掉 CF 直连回源——脚本注释里
|
|
||||||
「521」本就是白名单漏放的既有症状;③ 回源 IP 过期(服务器 IP 变动 / 第三方面板未更新)。
|
|
||||||
- 注意矛盾点:用户本地能连 IP+端口 ⇒ 白名单大概率没生效或不覆盖该端口 ⇒ 更偏向 ③/tunnel 侧。
|
|
||||||
|
|
||||||
**根因已确认(服务器实测)**
|
|
||||||
- `systemctl is-active cloudflared` = **inactive**,`pgrep` 无进程 → Tunnel 断。
|
|
||||||
journald 最后一条 2026-10-21T21:42:22Z(CST 10-22 05:42),此后无日志 ⇒ **停了约 7 小时**。
|
|
||||||
- 死前日志模式:`accept stream listener encountered a failure while serving` +
|
|
||||||
`failed to accept QUIC stream: timeout: no recent network activity` + `datagram handler
|
|
||||||
error="context canceled"` + `Retrying connection in up to 1s`,connIndex 2/3 反复 →
|
|
||||||
UDP 7844 QUIC 长时间不稳(美西 23.94.211.138)。**inactive 而非 failed ⇒ 是被正常 stop 的,
|
|
||||||
不是崩溃**,与用户「昨晚升级 nginx 到 1.30.x」的时间点吻合(升级流程连带停服)。
|
|
||||||
- 应用侧全好:`*:5877` 监听(tamabox pid),loopback 200。
|
|
||||||
- `iptables -S INPUT` 只有 `-i eth0 --dport 8118/5877 -j DROP`,**EO_IN / ipset EO_ORIGIN 都不存在**
|
|
||||||
⇒ EdgeOne 白名单脚本当前未生效。且 DROP 限定 `-i eth0`,网卡名若不叫 eth0 则规则空转。
|
|
||||||
- 关键推论:CF 一定走 tunnel 而非直连 5877(否则 DROP 空转应 200 而非 521)⇒ 521 完全由 tunnel 断解释。
|
|
||||||
|
|
||||||
**续:start 之后仍 521**
|
|
||||||
- 用户确认:**5877 只允许 Tailscale(100.84.113.84:5877)访问**,公网靠 iptables DROP 封掉。
|
|
||||||
⇒ 排除「CF 直连公网 5877」这条路径,CF 只能经 tunnel 回源。
|
|
||||||
- `systemctl start cloudflared` → `active`,`sleep 5` 后 curl 域名仍 **521**。
|
|
||||||
- 待分辨:① connector 尚未完成边缘注册(sleep 5 偏短);② 本机 tunnel 与 CF/SaaS 侧
|
|
||||||
`box.shiroko.one` 绑定的 origin(应为 `<uuid>.cfargotunnel.com`)不是同一个;
|
|
||||||
③ 该 hostname 的 Public Hostname / ingress 未指向 `http://127.0.0.1:5877`。
|
|
||||||
- 决定性隔离实验:临时 quick tunnel(`cloudflared tunnel --url http://127.0.0.1:5877`)
|
|
||||||
拿 trycloudflare 域名直测 —— 200 则应用+出站正常,问题在 CF 侧绑定。
|
|
||||||
|
|
||||||
**journald 取证(关键)**
|
|
||||||
- `protocol=quic`,`location=la01`。从 10-21 13:53:30 起就是死循环:
|
|
||||||
`INF Registered tunnel connection` → 约 3s 后 `ERR failed to accept QUIC stream:
|
|
||||||
timeout: no recent network activity` + `datagram handler error="context canceled"` +
|
|
||||||
`WRN failed to serve tunnel connection error="accept stream listener encountered a failure
|
|
||||||
while serving"` → `INF Retrying connection in up to 1s` → 再注册。connIndex 0/1/2 全部如此,
|
|
||||||
每 ~3 分钟一轮,一直持续到 10-21 21:42:40。
|
|
||||||
- ⇒ **UDP 7844 的 QUIC 在这台机器上基本不可用**(RackNerd 美西 → CF la01),这是历史顽疾,
|
|
||||||
不是本次新故障。根治手段 = `TUNNEL_TRANSPORT_PROTOCOL=http2`(走 TCP 443)。
|
|
||||||
- 🔴 但 12:56 `systemctl start` 之后 journal 里**一条新日志都没有**(最后仍是 10-21 21:42:40,
|
|
||||||
PID 626)⇒ 新进程疑似启动即退出/token 无效,`is-active=active` 可能是重启窗口里的瞬时值。
|
|
||||||
需 `systemctl status` 看 Restart 计数与 ExecMainStatus。
|
|
||||||
- 旁证:EdgeOne 那条备用通道同样不通(18080 无监听、EO_IN/ipset 均不存在)。
|
|
||||||
|
|
||||||
**结论定案**
|
|
||||||
- 重大纠正:cloudflared **从未死过**。`systemctl status` 显示 `active (running) since 2026-10-20 05:32:44`,
|
|
||||||
Main PID 626 存活 2 天+。前面「21:42 后无日志」不是进程退出,而是 QUIC 连接最后注册成功后没再报错
|
|
||||||
(只有 24h 一次的版本提醒)。所以「tunnel 断导致 521」这个假设是错的。
|
|
||||||
- 改 `TUNNEL_TRANSPORT_PROTOCOL=http2` 后:4 个 connector 全部 `Registered tunnel connection
|
|
||||||
... protocol=http2`,`SUMMARY: Environment is healthy. Cloudflared will use 'http2' as primary
|
|
||||||
protocol`,零错误 ⇒ **tunnel 侧完全健康**。QUIC 抖动问题也顺手解决了(该 drop-in 值得长期保留)。
|
|
||||||
- 但站点仍 521 ⇒ **问题 100% 在 CF 侧 origin/路由,与本机 tunnel 无关**。
|
|
||||||
- 公网实测 `23.94.211.138`:5877 / 18080 / 80 / 443 **全部 timeout**(不止 5877)。
|
|
||||||
⇒ 若第三方 SaaS 的 origin 填的是「公网 IP:5877」,CF 同样连不上 → 521。
|
|
||||||
这解释了「突然」:用户近期把 5877 收敛为仅 Tailscale(100.84.113.84)可达。
|
|
||||||
- 本机 tunnel 身份(从 token 解出,注意 token 已明文暴露需轮换):
|
|
||||||
tunnel id `ba34e35d-e517-498a-b2cd-7275c22e354e`,account tag `0c99c40b475a65b393abaca3ca91731b`。
|
|
||||||
- 修复方向:①(推荐)第三方面板把 origin 改成 `ba34e35d-...cfargotunnel.com`,并在
|
|
||||||
Zero Trust → Tunnels → 该 tunnel 的 Public Hostnames 补 `box.shiroko.one → http://127.0.0.1:5877`;
|
|
||||||
②(不推荐)放行 CF 回源网段到 5877,等于重新暴露源站。
|
|
||||||
- ⚠️ 安全 P0:tunnel token 在对话截图里明文出现,等同凭据泄露 → 必须 Refresh token 后重装服务。
|
|
||||||
|
|
||||||
**真凶:SaaS origin 指向被封的公网**
|
|
||||||
- SaaS 自定义主机名 `box.shiroko.one` 的「源服务器」= `box-direct.337845818.best`。
|
|
||||||
- 实测该 origin 域名 **解析到 Cloudflare 自己的 IP**(104.21.47.52 / 172.67.144.141,
|
|
||||||
`Server: cloudflare`,CF-RAY)⇒ 它是**橙云(proxied)记录**,不是灰云 A 记录。
|
|
||||||
`http://` 301(Always Use HTTPS)、`https://` **521** —— 它自己也回源失败。
|
|
||||||
- 源站 `23.94.211.138`(colocrossing,RackNerd 上游):22 / 80 / 443 / 5877 / 18080
|
|
||||||
**全部 timeout**。不只应用端口,连 SSH 都不通 ⇒ 该机对公网几乎无入站(仅白名单/Tailscale)。
|
|
||||||
- 结论链:CF 回源 `box-direct` → 该记录橙云 → 真实源站端口被封 → **521**。
|
|
||||||
所有共用 `-direct` origin 的站点同时 521 ⇒ 用户说的「cloudflared 构建的网站全部 521」
|
|
||||||
与 cloudflared 无关,是**公网入站收敛**这一个动作把整批站点打掉的。nginx 升级纯属时间巧合。
|
|
||||||
- 修复(保持「只允许组网」的安全目标):① Zero Trust → 该 tunnel → Public Hostnames 加
|
|
||||||
`box.shiroko.one → http://127.0.0.1:5877`;② SaaS 面板把每个自定义主机名的「源服务器」
|
|
||||||
从 `xxx-direct.337845818.best` 改为 `ba34e35d-e517-498a-b2cd-7275c22e354e.cfargotunnel.com`。
|
|
||||||
注意不能把 `-direct` 改成「灰云 CNAME 到 cfargotunnel」——tunnel 路由要求橙云,SaaS origin 又
|
|
||||||
要求非橙云,两者冲突,必须直接在 origin 字段填 cfargotunnel。
|
|
||||||
|
|
||||||
**⚠️ 纠正(官方文档)**
|
|
||||||
- 上一条的后半段判断错了。CF for SaaS 文档明确:custom origin **必须**是该账号下
|
|
||||||
**proxied(橙云)的 A / AAAA / CNAME** 记录,不能用 IP。
|
|
||||||
⇒ `box-direct.337845818.best` 是橙云,**格式上完全正确**,不是配置错误。
|
|
||||||
用 tunnel 的正确形态就是「橙云 CNAME → `<uuid>.cfargotunnel.com`」,所以那条路径依然成立,
|
|
||||||
只是理由不是「橙云错了」。
|
|
||||||
- 真正断点仍是:`23.94.211.138` 的 **443 从公网不可达**(实测 timeout)。
|
|
||||||
但用户贴的 `iptables -S INPUT` 里**只有 8118 / 5877 两条 DROP,没有 443**
|
|
||||||
⇒ 对 443 的封锁来自 iptables 之外(云厂商安全组 / 上游 / 其他链),或者 nginx 压根没监听 443。
|
|
||||||
- 我上一轮的探测命令 `grep ':(80|5877|18080)'` **漏了 443**,这是取证缺口,需要补
|
|
||||||
`ss -tlnp | grep -E ':(443|80|5877)'` + `systemctl is-active nginx`。
|
|
||||||
- 注意:SaaS 回源默认 Host = 原始自定义主机名(box.shiroko.one),而 tunnel 的 Public Hostname
|
|
||||||
只能从账号内 zone 选(shiroko.one 不在 CF)⇒ 走 tunnel 时必须用 Origin Rule / 源站主机头覆盖,
|
|
||||||
否则 tunnel 会 404。
|
|
||||||
|
|
||||||
**最终根因(用户自查确认):nginx 停了 11 小时**
|
|
||||||
- 时间线:`10-22 00:12:43 Started`(升级后)→ `02:31:10 Stopping...` →
|
|
||||||
`02:31:15 Stopping timed out. Terminating.` → `Failed with result 'timeout'` → `Stopped`
|
|
||||||
→ **中间 11 小时无任何记录** → `13:34:42` 用户手动 `systemctl start` 才起来。
|
|
||||||
- ⇒ 全站 521 的直接原因就是 nginx 不在 443 监听;CF 回源 `box-direct`(橙云 A → 23.94.211.138:443)
|
|
||||||
拿不到东西。cloudflared/tunnel 全程健康且**从未参与这条链路**。
|
|
||||||
- 重启后实测:`box.shiroko.one` 连续 3 次 200,`box-direct` 200,源站 443 OPEN。
|
|
||||||
- 教训(我的误判链,值得记住):排障时先确认「443 上跑的是什么、进程在不在」(`ss -tlnp` +
|
|
||||||
`systemctl is-active nginx`),再往 DNS/CDN/tunnel 方向挖。这次因为 CNAME 指向第三方 SaaS
|
|
||||||
+ cloudflared 恰好健康,被带了很远。
|
|
||||||
- 遗留技术债:
|
|
||||||
① `listen 443 ssl http2` 在 1.25.1+ 已废弃 → 改 `listen 443 ssl;` + `http2 on;`;
|
|
||||||
② `Failed to parse PID from file /run/nginx.pid: Invalid argument` —— 大概率是上次 SIGKILL
|
|
||||||
残留的 pid 文件,配合 `TimeoutStopSec` 过短(5s 就 Terminating)导致 stop 卡死;
|
|
||||||
③ 全站无可用性监控(11 小时空窗没人知道)→ 建议外部 uptime + cron watchdog
|
|
||||||
`*/5 * * * * systemctl is-active --quiet nginx || systemctl start nginx`。
|
|
||||||
|
|
||||||
**6) 窄屏覆盖被作用域块压死(真 bug,已修)**
|
|
||||||
- 现象:`@media (max-width:640px)` 里把 `--tb-deco` 全写 none,手机上却照样是桌面装饰层。
|
|
||||||
- 根因:媒体查询**不提升特异性**。作用域块 `html[data-theme-style][data-page-scope="box"]`
|
|
||||||
是 (0,2,1),媒体里写的 `html[data-theme-style]` 只有 (0,1,1) → 压不住;深色作用域块 (0,3,1) 更难。
|
|
||||||
- 修法:媒体查询里按「作用域通配 `[data-page-scope]`(0,2,1)」+「深色基础(0,2,1)」+
|
|
||||||
「深色+作用域(0,3,1)」各写一条,靠文件更晚取胜。shiroko 的 900px 块只对浅色有效,
|
|
||||||
但它的深色块本来就是 none,所以不算坏。
|
|
||||||
- 守门:新增 `internal/theme/narrow_test.go::TestNarrowMediaOverridesScopedDeco`
|
|
||||||
——外层每处非 none 的 `--tb-deco`,媒体查询里必须有「同属性名、属性数不少于它」的选择器;
|
|
||||||
变异验证过(删掉深色基础那条立刻 FAIL)。
|
|
||||||
- 手机端最终决定**不放装饰层**:试过顶栏与卡片之间那条 35px 带子,box 页 OK、question/admin
|
|
||||||
页卡片从 y=110 起就切半个 → 变成黏在卡片边的粉色碎块。改用加强版底纹(透明度 ×1.9、格子 180px)。
|
|
||||||
- 埋点:`gofmt -l internal/theme` 会报 `slots_test.go`(它是 CRLF),新增文件是 LF,别顺手格式化它。
|
|
||||||
|
|
||||||
## 站内公告功能(配置驱动)
|
|
||||||
|
|
||||||
- 新增 `[announcement]` 段(content/start/end),改完重启生效;判定全在服务端按 time.Now()。
|
|
||||||
- 改动:conf/announcement.go(解析+Active+单测)、conf.Init MapTo、keynorm sectionKeySets 登记(漏了会误报未知段)、
|
|
||||||
template.FuncMap 新增 `announcement`(返回安全 HTML,空串 falsy 配 `{{ with }}`)、
|
|
||||||
internal/template/announcement.go(切段转义→URL 包链接,不能「先转义再正则」否则 " 被吞进 URL)、
|
|
||||||
header.html 导航下方插 `uk-container-xsmall` 容器包 `.tb-announce`、base.css 样式(StaticVersion → 20261022a)。
|
|
||||||
- 时间格式:`2006-01-02 15:04` / 带秒 / 纯日期 / RFC3339;裸时间按服务器本地时区(容器是 UTC),启动 INFO 打印解析结果。
|
|
||||||
- 时间写错只 WARN 关公告不阻断启动(与未知键口径一致);窗口左闭右开。
|
|
||||||
- 预览验证:TEST_PREVIEW=1 + 临时 test 设 conf.Announcement 渲染 question/list,chrome --headless=old 截图 OK(亮/暗都看过)。
|
|
||||||
|
|
||||||
- 公告间距修复:`.tb-announce` 自身的 margin-bottom 会被 UIkit 的
|
|
||||||
`.uk-container > :last-child { margin-bottom: 0 }` 清零 → 间距必须挂在容器类
|
|
||||||
`.tb-announce-wrap { margin-bottom: 20px }` 上。这是容器内末元素贴边的通用坑,
|
|
||||||
以后在 `.uk-container` 里加末位区块都绕这条。
|
|
||||||
- 环境坑:沙箱 go 是 1.25.5,go.mod 要 1.27.3,GOTOOLCHAIN=auto 需下载工具链模块,
|
|
||||||
而 GOSUMDB=off 会卡住校验 → 下载工具链那次要放开 GOSUMDB(goproxy.cn 会代理 sumdb)。
|
|
||||||
|
|
||||||
**7) 顶栏:实色 → 半透明磨砂(用户:不能全透明,以后别再来一次)**
|
|
||||||
- 线上 `https://box.shiroko.one/_/tamakyi` 实测:`--tb-navbar=#fdf6f8` 实色,顶栏带(y=20..80)
|
|
||||||
把右上花枝齐刷刷切一段;`--tb-navbar` 默认是 `--tb-surface`,彩色主题不覆盖就成一条近白横带。
|
|
||||||
- 第一版改成 `transparent` 被用户否掉:花枝完整了但顶栏"消失",导航没依托、文字压在花纹上。
|
|
||||||
- 最终:`rgba(253,246,248,0.72)` / `rgba(34,26,30,0.72)` + `backdrop-filter: blur(12px)`
|
|
||||||
(模糊只能写在规则里,令牌只是颜色)。0.72 有依据:文字 `#6d555d` 纯底 6.34:1,
|
|
||||||
最坏(深花瓣压顶栏后面)压平后 5.08:1。
|
|
||||||
- 花枝 `top` 从 -16px 下移到 26px:顶栏透明后 700~760px 宽度下枝头会压到「登录」(截图确认)。
|
|
||||||
- 守门:`internal/theme/navbar_test.go::TestNavbarNotFullyTransparent` 拦
|
|
||||||
`transparent` / `rgba(...,0)` / `#rrggbb00`,变异验证过。约定同步写进
|
|
||||||
`static/css/themes/_scaffold.css` 的令牌注释与 `topics/theme.md`。
|
|
||||||
- 手机底栏 `#tb-mobile-utils` 在 base.css 写死 `rgba(255,255,255,.92)`,彩色主题要自己调(本例改同色系 0.92)。
|
|
||||||
- ⚠️ `StaticVersion` → `20261022c`:主题 CSS 改了必须提版本号,否则浏览器拿着旧 css,
|
|
||||||
现象就是「改了但线上没变」。
|
|
||||||
- ⚠️ 跑 go 命令时**别加 `GOSUMDB=off`**:本机 go 1.25.5,go.mod 要 1.27.3,
|
|
||||||
`GOTOOLCHAIN=auto` 要校验 toolchain 模块,关掉 sumdb 会直接报 "verifying module"。
|
|
||||||
只设 `GOPROXY=https://goproxy.cn,direct GOFLAGS=-mod=mod` 即可(toolchain 已缓存)。
|
|
||||||
|
|
||||||
**8) 生成器新增「顶栏」面板:实色 / 半透明(磨砂)/ 全透明**
|
|
||||||
- 状态 `S.navbar = {mode, alpha, blur}`(默认 glass 0.72 / 12px),基色仍取「配色 → 顶栏」,
|
|
||||||
`navbarValue(p)` 统一算:solid → `rgb()`(先 flatten 压平旧工程里的 rgba)、glass → `rgba()`、
|
|
||||||
clear → `transparent`。模糊单独写 `.uk-navbar-container:not(.uk-navbar-transparent)` 规则,
|
|
||||||
一条同时管深浅(选择器不带 data-theme)。
|
|
||||||
- **全透明只给预览**:`validate()` 直接拦(导出报错),与 `TestNavbarNotFullyTransparent` 同一条规矩。
|
|
||||||
面板上写明原因 —— 不然下一轮又会有人问「为什么不能导出」。
|
|
||||||
- 对比度读数按「压平后的顶栏」算(半透明时是顶栏压在底色上的结果),低于 4.5 变橙。
|
|
||||||
- 旧工程迁移:配色里塞过 `rgba(...,0.52)` 的,按 alpha 反推档位(1→solid,0→clear,其余→glass)。
|
|
||||||
- 自检 `?dump=navbar` 10 项;Playwright 端到端 17 项(含滑杆跟手、刷新后保持、迁移三例)。
|
|
||||||
- ⚠️ `?fresh=1` 每次加载都会清草稿 —— 测「刷新后配置还在」必须导航到不带 fresh 的地址。
|
|
||||||
- ⚠️ Playwright 的 `fill()` 对 `input[type=range]` 报 Malformed value,要 evaluate 设 value + dispatch input。
|
|
||||||
- ⚠️ 滑杆步长别用 0.02:0.55 会被吸附成 0.54/0.56,用户以为坏了。改 0.01。
|
|
||||||
|
|
||||||
## 单题页「下一条未读」(箱主收件箱式导航)
|
|
||||||
|
|
||||||
- 新增 `db.Questions.NextUnread(ctx, userID, excludeID)`:未读 = `is_read=false AND answer=''`,
|
|
||||||
`created_at ASC` 取第一条,无则返回 `(nil, nil)`。正序=先来先处理,反复取即可清空积压。
|
|
||||||
- `POST /_/{domain}/{qid}/next-unread`(`question.NextUnread`):**先 SetRead 再跳**。
|
|
||||||
只跳不标记会「打转」—— 当前这条仍是未读,下一次取最早未读又把它选回来(2 条未读时最明显)。
|
|
||||||
- `PublishAnswer` 支持同表单第二个提交按钮 `name="next" value="1"`(「回答并下一条未读」)。
|
|
||||||
- 按钮位置:回答行 `[回答][回答并下一条未读]`(箱主+有下一条时);页脚与「返回提问箱」同一行
|
|
||||||
放 `下一条未读 ›`(POST 表单 + `uk-button-link`,因为要改状态)。
|
|
||||||
- `adminBackQuery(ctx)` 抽出(原 BackTo 逻辑),`NextQuery` 进模板 → 答案表单/next-unread 表单的
|
|
||||||
action 都带上 `?from=admin&sort=...`,跳转后续「返回」仍回带筛选的后台列表。
|
|
||||||
flamego `ctx.Query` 只读 URL query,所以 POST 必须把参数写在 action 里(不是隐藏域)。
|
|
||||||
- 预览:`templates/zz_preview_test.go` 新增 `writeQuestionOwnerPreviews`(未回答/已回答 × 深浅)。
|
|
||||||
⚠️ 预览 HTML 的 `base.css` 是 `/static/...` 绝对路径,file:// 打开**不含主题变量**(暗色看着像亮色);
|
|
||||||
要验暗色必须 `python -m http.server` 起本地服务走 http。移动端预览本身有横向溢出(waline 表情行所致),非本次改动。
|
|
||||||
- 移动端排版实测(关键数据):390px 视口下 `.tb-ans-row` 只有 **300px**(360px 视口 270px),
|
|
||||||
而「回答 90 + 回答并下一条未读 174 + 标记为已读 132 + 间距 20」= 416px,**收窄内边距也不够**(320px)。
|
|
||||||
→ 手机上一行放不下三个按钮,改成:第 1 行保留原「回答 … 标记为已读」,第 2 行「回答并下一条未读」整行宽。
|
|
||||||
实现:`@media(max-width:640px){ .tb-ans-actions{display:contents} .tb-ans-state{order:2;margin-left:auto}
|
|
||||||
.tb-ans-actions .uk-button{order:1} .tb-ans-actions .uk-button.tb-ans-next{order:3;flex:1 1 100%} }`
|
|
||||||
⚠️ `.tb-ans-actions .uk-button`(0,2,0)特异性高于 `.tb-ans-next`(0,1,0),必须再写一条同级选择器盖 order。
|
|
||||||
- 页脚两个次级操作之间加 `.tb-foot-sep`(`·`,浅色 #b0b6bd / 暗色 #5c6674),原来两个蓝色链接几乎连成一片。
|
|
||||||
- Chrome headless 在 Windows 上 `--window-size` < 500 无效(innerWidth 恒 500)——要测真·移动端布局,
|
|
||||||
用「父页面 + 宽 390 的 iframe」再取 `contentDocument` 量尺寸/截图,媒体查询才会按 iframe 宽度生效。
|
|
||||||
- 文案定为「回答并进入下一条未读」(en: Answer & open next unread / ja: 回答して次の未読を開く)。
|
|
||||||
实测宽度:该按钮 202px;661px 视口(桌面态最窄)行宽 526 ≥ 90+202+132+20=444,仍单行;
|
|
||||||
640px 断点两侧切换干净,无需再调。
|
|
||||||
|
|
||||||
## 近期新增功能审阅(10-18~10-22)
|
|
||||||
|
|
||||||
- 编译(s3 overlay 桩)+ conf/template/theme/route/security/templates 全测试通过,无 P0。
|
|
||||||
- 🔴 **P1 实测复现**:`NextQuery` 不含 `from=admin`,而 `adminBackQuery` 首行要求它
|
|
||||||
→ 「回答并下一条未读」/「下一条未读」POST 后筛选上下文全丢,「返回」掉回箱主页。
|
|
||||||
实测:`POST /answer?sort=new&status=unread` → `from="" backQuery=""`;
|
|
||||||
带 `?from=admin` 才返回 `sort=..`。修法:`adminBackQuery` 里 `backQuery.Set("from","admin")`,
|
|
||||||
并把 `PublishAnswer` 普通分支(question.go:279)也改用 `questionPath`。
|
|
||||||
- P2:公告单语言(四语站点);裸时间按 time.Local(容器 UTC)易偏 8h;
|
|
||||||
NextUnread 无 `(user_id,is_read,created_at)` 复合索引且忽略后台筛选;公告出现在 pixel 画板页
|
|
||||||
(xsmall 容器 vs 全宽画布);new-sakura 磨砂顶栏 backdrop-filter 低端机掉帧风险;
|
|
||||||
`conf/app.sample.ini:296` 注释行尾残留 `,shiroko`。
|
|
||||||
- 蜜罐仅做代码审阅(用户要求不进用户说明):无正常用户误封路径,前提是 `trust_proxy` 已开。
|
|
||||||
- 交付:`outputs/2026-10-新增功能审阅报告.md`、`outputs/2026-10-功能说明.md`(面向一般用户)。
|
|
||||||
|
|
||||||
## 修 P1:后台上下文丢失(已落地)
|
|
||||||
|
|
||||||
- `adminBackQuery` 改为 `url.Values{"from": {"admin"}}` 起手:返回串同时用于模板 action
|
|
||||||
与 `questionPath`,POST 回来才能重新认出来源(Flamego `ctx.Query` 只读 URL query)。
|
|
||||||
- `PublishAnswer` 末尾裸重定向 → `questionPath`,让「回答」这条也保留筛选。
|
|
||||||
- 新增 `route/question/admin_backquery_test.go` 两条用例;build + route/templates/conf/template 测试全绿。
|
|
||||||
- 副作用(可接受):`BackTo` 拼成 `/user/questions?from=admin&sort=..`,列表页不读 from。
|
|
||||||
- `question.go` 是 CRLF:改完跑 `python dev/gofmt_keep_crlf.py`,diff 只有 6 增 2 删。
|
|
||||||
|
|
||||||
## 修 P2:公告分语言 / 东八区 / 未读索引(已落地)
|
|
||||||
|
|
||||||
- **公告分语言**:新增 `content_zh_cn/_zh_tw/_en/_ja`,`content` 降级为默认回退;
|
|
||||||
`hasContent()` 改成「任一语言非空即开启」(只看 content 会把「只写英文」误判成未配置)。
|
|
||||||
模板函数签名变为 `announcement .LangCode`(header.html 同步),判定仍在服务端。
|
|
||||||
- **时区**:新增 `[announcement] timezone`,默认 `Asia/Shanghai`;`import _ "time/tzdata"`
|
|
||||||
保证容器无 zoneinfo 也能 LoadLocation。RFC3339 仍以自带偏移为准。
|
|
||||||
启动日志扩成 `zone/languages/start/end/active_now`,解析结果直接带 `+08:00`。
|
|
||||||
- **索引**:`idx_questions_user_unread_created (user_id, is_read, created_at)` 加进
|
|
||||||
`ensureFrequentlyUsedIndexes`;本机测试库 psql 实跑两次验证语法 + 幂等。
|
|
||||||
- 测试:conf 侧 7 组(语言回退/归一化/东八区/时区覆盖/无效时区)、
|
|
||||||
templates 侧 2 条渲染断言(zh-CN 与 en 只出现各自文案、窗口外不渲染 `tb-announce`)。
|
|
||||||
- 未改:`NextUnread` 仍不看后台筛选(产品取舍);画板页公告问题按用户确认忽略(画板已下线)。
|
|
||||||
|
|
||||||
## 测试站部署验证(box-dev-api)
|
|
||||||
|
|
||||||
- 结论:部署的确实是最新代码。判据:`base.css?v=20261022c`、CSS 含 `.tb-announce-wrap`、
|
|
||||||
`--tb-primary:#0e6fd0`、`themes/new-sakura.css` 含 `backdrop-filter`。
|
|
||||||
- **公告分语言在线上生效**:同一页面切 `/i18n/{zh-CN,zh-TW,en,ja}` 拿到四套不同文案,
|
|
||||||
链接渲染成 `<a target="_blank" rel="noopener noreferrer nofollow">`。
|
|
||||||
- 移动端(CDP `Emulation.setDeviceMetricsOverride` 390×844, mobile:true):
|
|
||||||
`scrollWidth == innerWidth == 390`,无横向溢出,公告块 right=390 贴容器边 —— 正常。
|
|
||||||
⚠️ 纯 `chrome --headless --screenshot --window-size=390` 的截图会让人误判「被切」,
|
|
||||||
真要量布局用 CDP + mobile:true,别只看图。
|
|
||||||
- 外部验证不到的三项:时区解析(看启动日志 `zone=Asia/Shanghai` + `+08:00`)、
|
|
||||||
复合索引(`SELECT indexname FROM pg_indexes WHERE tablename='questions'`)、
|
|
||||||
「下一条未读」(需箱主登录,且不能靠 POST 探测 —— 路由缺失会落到蜜罐返回 200 假登录页)。
|
|
||||||
- 偶发一次 CF 520(同 URL 复测 3 次均 200),未复现,非部署问题。
|
|
||||||
- 本机 curl 走 `http_proxy=127.0.0.1:58589`,node 直连本地调试端口要 `NO_PROXY=127.0.0.1`。
|
|
||||||
|
|
||||||
## 登录态实测(自建测试账号)→ 又抓到一个 P1 残留
|
|
||||||
|
|
||||||
- 账号:`tbunread<ts>@example.com` / TbUnread!2026(脚本 `Temp/tb_unread_check.py`,注册→发提问→登录→走流程→删题)。
|
|
||||||
- 🔴 **模板 URL 上下文会把 `?{{ .NextQuery }}` 整段 percent-encode**:
|
|
||||||
实测渲染出 `action=".../answer?from%3dadmin%26read%3dunread%26sort%3dnew"`,
|
|
||||||
`ctx.Query("from")` 永远为空 ⇒ 浏览器提交时 P1 修复失效。之前只测了 handler 层,没测渲染结果。
|
|
||||||
本地 html/template 实验确认:`action="{{ .A }}"`(值是完整 URL)→ 只把 `&` 转 `&`;
|
|
||||||
`action="...?{{ .B }}"`(值是查询片段)→ `%3d`/`%26`。**以后 query 一律在 handler 拼完整 URL。**
|
|
||||||
- 修法:Questioner 里生成 `AnswerAction` / `NextUnreadAction`(完整 URL),模板直接引用;
|
|
||||||
守门 `templates/question_action_test.go`(断言无 `%3d`/`%26` 且有 `&sort=new`)。
|
|
||||||
- 顺带确认(线上):单题页「回答并进入下一条未读」跳转、页脚「下一条未读」跳转、
|
|
||||||
全部读完的「没有未读提问了」提示 —— 三条行为都正确,`Location` 带 `from=admin&read=unread&sort=new`。
|
|
||||||
- 断言坑:页面上有 `/user/questions`(导航)和 `/user/questions?from=admin&…`(页脚返回)两个链接,
|
|
||||||
用 `re.search` 取第一个会误判「返回丢参数」。要按 href 全量匹配。
|
|
||||||
|
|
||||||
## 部署后复验:14/14 通过(含「日志重复」排障)
|
|
||||||
|
|
||||||
- 复验结果:`answer?from=admin&read=unread&sort=new`(未被编码)、
|
|
||||||
回答并下一条 → 次早、页脚下一条 → 再下一条、读完后「没有未读提问了」、
|
|
||||||
下一条页「返回」= `/user/questions?from=admin&read=unread&sort=new`。数据已删净(0 条)。
|
|
||||||
- 🔴 **日志「一直重复同一个操作」的真相:是自检脚本,不是站点。**
|
|
||||||
两个原因叠加:① 补数据循环 `while len(ids) < 3` 没有上限、不校验 POST 结果;
|
|
||||||
② 每轮发的**内容完全相同**,被 `shared.DuplicateGuard`(同 IP + 相同内容防刷屏,
|
|
||||||
`route/question/page.go:587`)拦下 —— 返回 200 + 错误提示但不落库,脚本没读提示于是无限重发。
|
|
||||||
→ 泛化教训:跑端到端脚本时,注入的文本必须逐条不同;循环必须有 attempts 上限 + 「未增长即停」。
|
|
||||||
- `urllib` 偶发 `TimeoutError` 的根因是代理复用 stale keep-alive 连接:请求头加
|
|
||||||
`Connection: close` 后不再出现,比加重试更有效。
|
|
||||||
- 测试账号 `tbunread1022181904`(自建,密码不入库)。提问已全部删除,账号可留作后续自检。
|
|
||||||
|
|
||||||
## 测试账号存档(凭据不入库)
|
|
||||||
|
|
||||||
- 用户要求「记住别再忘」→ 凭据写在 `.workbuddy/local/test-accounts.md`,
|
|
||||||
`.gitignore` 已加 `/.workbuddy/local/`(明文密码不随仓库走)。
|
|
||||||
- 内容:`tbunread1022181904@example.com` / 域名 `tbunread1022181904` / 密码 + 登录四步
|
|
||||||
(`/login` 取 csrf → `/api/captcha/generate` 算数学题 → POST `g-recaptcha-response="<id>:<答案>"`)。
|
|
||||||
- 以后要跑登录态自检**先读这个文件复用账号**,不要再注册新的;`admin6` 仍只有域名、无密码。
|
|
||||||
# 2026-10-22
|
|
||||||
|
|
||||||
## shiroko 平涂极简肖像(参考用户给的图)
|
|
||||||
|
|
||||||
用户给了一张参考图:平涂零描边、纯黑圆角矩形眼、无嘴、腮红、头占满画面被裁边,
|
|
||||||
要求做同风格的白子,位置他自己放。
|
|
||||||
|
|
||||||
- 产出 3 张透明 WebP:`static/images/themes/shiroko/portraits/portrait-{front,tilt,close}.webp`
|
|
||||||
(907–1024 × 921,共 197KB),未接进 `--tb-deco`,留给用户自己摆。
|
|
||||||
- ⚠️ **ImageGen 的 `background: "transparent"` 依然不生效**:返回的是 RGB 无 alpha 通道(实测 mode=RGB)。
|
|
||||||
- ⚠️ **不能按白底抠**:白衬衫会一起被吃掉,闭合光环内部的白色 floodfill 也进不去(留下白斑)。
|
|
||||||
解法:提示词里把背景写成一个**具体纯色** `#E6EEF8`,抠图时按该色值算 alpha:
|
|
||||||
`alpha = clip((maxChannelDiff - 6) / 12)`,再 `MinFilter(3)` 收 1px。实测头发 alpha 全 255,
|
|
||||||
中间过渡只占 2.5–6%(全是边缘),不会被误抠。
|
|
||||||
- 水印在右下角("AI生成 WORKBUDDY"),统一裁掉底部 10%,切口平直,正好可以贴页面下沿。
|
|
||||||
- 提示词关键:no outlines / no gradients、眼睛写死 `solid black vertically elongated rounded rectangles`
|
|
||||||
(否则出白眼球+高光+睫毛)、无嘴要重复强调。3/4 侧脸那版还是画出了嘴唇,弃用。
|
|
||||||
|
|
||||||
## 分享卡「二维码下链接不对齐 + 长链接乱掉」(已定位,待批准落地)
|
|
||||||
|
|
||||||
- 根因在 `route/question/share.go` 的**字宽估算**:`charPixelWidth` 把 `unicode.Punct/Symbol`
|
|
||||||
一律按 1.0em 计,而 URL 里的 `: / . _ -` 在 Noto Sans SC 里只有 0.25~0.35em。
|
|
||||||
- 实测(NotoSansSC + 本机原型 `dev/tmpcard`):`https://box.shiroko.one/_/tamakyi` 15px 真实
|
|
||||||
227.9px、估算 352.5px(+55%);卡片可用宽仅 225px → 触发缩字号到 **9.57px**,实际只画 145px 宽,
|
|
||||||
且起点按估算算 → 整段左偏 **40px**。dev 域名 7.24px / 偏 39px;长域名 5.26px / 偏 34px。
|
|
||||||
- 纯中文不受影响(CJK 就是 1em,偏差 0);底部水印 `Powered by 狼的提问箱` 偏 9.8px。
|
|
||||||
- 正确测法:`Font.HMetric(scale).AdvanceWidth` + `Font.Kern` 累加,scale=size*64(DPI 72)。
|
|
||||||
实测与 `freetype.Context.DrawString` 返回笔位完全一致(227.88 vs 227.88)—— 以后居中文字都用它,
|
|
||||||
别再靠字符宽度系数猜。
|
|
||||||
- 只修测量还不够:等比缩字号没有下限也没有断行/省略兜底,长链接必然掉到 5px 以下。
|
|
||||||
- 另记:QR 用 `qr.Image(256)` 再 NearestNeighbor 到 165,41 模块 → 每模块 4.02px,模块宽窄不一;
|
|
||||||
`qr.Image(-k)` 按模块整数倍直接出图更锐(41*4=164)。
|
|
||||||
- 交付:`outputs/sharecard-link-compare-2026-10-22.png`(现状/修复后对照)。
|
|
||||||
|
|
||||||
**已落地(用户选方案 A:单行优先,放不下才拆 host/path)**
|
|
||||||
- `measureTextWidth` 改为 `HMetric.AdvanceWidth + Kern` 累加,字体不可用时才回退估算;
|
|
||||||
所有 `drawTextCentered` 自动吃到修正(链接、水印、提示语)。
|
|
||||||
- 新增 `displayURL`(去 `https://`)/ `layoutLink`(单行 15→13 → host+path 两行 13→11 →
|
|
||||||
host 中间省略 @11)/ `ellipsizeMiddle`;`LinkMaxWidth = QRCardW-40 = 230`。
|
|
||||||
- 卡片内容高度改为按行数算,两行时 startY 自动下移(281 ≤ 315,顶距 17)。
|
|
||||||
- 守门测试 `route/question/share_layout_test.go`:测量=笔位(0.01px 级)、限宽、
|
|
||||||
居中(按着墨边界 ±3px)、`TEST_PREVIEW=1` 出图。**变异验证过**:把 measure 换回估算,
|
|
||||||
三条用例全 FAIL(偏移 39/59px,且文字真的溢出卡片 615 < 660)。
|
|
||||||
- 删掉了临时原型 `dev/tmpcard`(结论已进测试与 outputs 图)。
|
|
||||||
- P1 未做(二维码模块整数倍、间距 4px 栅格、水印相对左侧内容区居中)—— 等用户看完效果再定。
|
|
||||||
- 已推送 main:`0054e50`(修复 + 测试 + go.mod 依赖转正)、`80e042d`(记忆/忽略规则),等测试环境部署复验。
|
|
||||||
|
|
||||||
**dev 部署后复验(box-dev-api)**
|
|
||||||
- `/_/tbunread1022181904/share-card` → **500「分享图生成失败:字体文件未配置或无法加载」**。
|
|
||||||
- 定位:不是代码回退。`static/fonts/` **从未入库**(`git log -- static/fonts` 空、`git ls-files` 无、
|
|
||||||
也没被 ignore),`static/fs.go` 的 embed 清单里也没有字体;`/static/fonts/*.otf` 走 HTTP 是 302 回首页。
|
|
||||||
生产能出图(200 jpeg)纯粹因为老环境手工放过字体。
|
|
||||||
- 代码侧补强(`d51629e`):`getFont` 去掉 `sync.Once`(失败不再永久缓存,补文件即生效)、
|
|
||||||
错误信息带「已尝试路径」、候选路径 = font_path → `static/fonts/*.{otf,ttf,ttc}` → 系统 CJK 字体;
|
|
||||||
`app.sample.ini` 注明必须自备。
|
|
||||||
- 顺带做了 P1 第一项:`generateQR` 按模块整数倍出图(41 模块 @165 → 每模块 4px / 符号 164px 居中),
|
|
||||||
头像内嵌改为按目标尺寸等比 + 2px 白边。新增 `TestGenerateQRIsModuleAligned`(扫描线游程必须是
|
|
||||||
模块宽度整数倍,变异验证过)与 `TestGenerateQRWithLogo`(中心头像、四角静默区纯白)。
|
|
||||||
- 待用户:在 dev 部署目录放字体(生产上有同一份)后我复验;Dockerfile 仍是旧模板
|
|
||||||
(module path 写成 NekoWheel/NekoBox、没 COPY static、golang:1.20 低于 go.mod 要求的 1.27.3),
|
|
||||||
需确认 dev 是否走容器部署。
|
|
||||||
|
|
||||||
**用户放好字体后复验(200 OK)**
|
|
||||||
- 实测着墨居中偏差:链接两行 −0.5 / 0.0px、提示语 −1.0px、底部水印 480.0(原偏 9.8px)——字宽测量修复确认生效。
|
|
||||||
- 单题卡要「已回答 + 非私密」才出图:提交的提问必须先回答,否则 403/302。
|
|
||||||
探针脚本 `.workbuddy/local/qcard_probe.py`(提问 → 登录 → 回答 → 拉图 → 删除,不入库)。
|
|
||||||
- **抓到上一版自己引入的回归**:`generateQR` 把 4 模块静默区画进了图里,49 模块挤进 165px
|
|
||||||
→ 符号只有 111px(3px/模块),比改造前的 135px 还小。960 卡缩到 480 宽时模块 1.5px,
|
|
||||||
gozxing 直接解不出。
|
|
||||||
修法(`b35c2a8`):`qr.DisableBorder = true`,图像只出符号,静默区交给白卡;
|
|
||||||
调用方按实际符号尺寸居中排版;单题卡 `QQRSize` 128 → 150(37 模块拿到 4px);卡内间距收到
|
|
||||||
24/20/20/16。profile 与单题卡二维码均 111 → 148px,gozxing 在 960 原图与 75% 缩放图上都能解出正确 URL。
|
|
||||||
- 结论性数字:**960×540 下 4px/模块是当前上限(profile 卡 37 模块 @148px,用满 90%)**;
|
|
||||||
缩到 480 宽仍解不出 → 真要抗微信二次压缩只能上 2x 渲染(1920×1080),1C1G 需实测 CPU。
|
|
||||||
- `dev/overlay.json` 原来指向 /tmp 的桩文件(早已失效),改成指向仓库内 `dev/s3stub/s3_stub.go`。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
**第二轮复验(用户重新部署后)+ 换 2x 的替代方案**
|
|
||||||
- 部署确认:箱子卡 200,二维码 148px、左右留白 61/61、三处居中偏差 −0.5/0.0/−0.5px。
|
|
||||||
- 延迟别拿 HTTP 测:这台机器 **静态 CSS 的 TTFB 就 2.7s**(TLS+代理),分享卡 TTFB 2.4s
|
|
||||||
比静态资源还快,测不出服务端渲染耗时。要测就在本地跑基准。
|
|
||||||
- 本地基准(1C1G 上大约 ×3~5):1x 像素工作 ≈ 40ms(JPEG 12 + 背景缩放 25 + 画布 3),
|
|
||||||
2x ≈ 103ms;**二维码生成只有 0.8ms**,与尺寸无关 → 放大二维码不花 CPU,2x 贵在编码与缩放。
|
|
||||||
- 实验(37 模块 / 真实 55 字符链接 / 含头像,gozxing 逐宽度解码):4px/模块 480 宽就挂,
|
|
||||||
5px 能撑到 320。降纠错 Q→M 只省 11% 模块且容错 25%→15%,不划算;PNG 救不了缩放。
|
|
||||||
- 落地 `72f08cd`:`QRCodeSize` 165→190(→185px / 5px 模块)、`QRCardH` 315→335,
|
|
||||||
新增 `TestGenerateQRModuleSizeAtLeast5`(改回 165 立刻 FAIL)。单题卡仍是 150(4px),未动。
|
|
||||||
- 比 2x 更值得做的:进程内缓存只有 16 条 + TTL 1h,箱子多时冷渲染频繁 → 结果落盘/R2。
|
|
||||||
@@ -1,145 +0,0 @@
|
|||||||
# TamaBox-v4 长期笔记(索引版)
|
|
||||||
|
|
||||||
明细已拆分到 `topics/*.md`(按需 Read 全文):
|
|
||||||
`coop` 协作 / `build` 构建测试 / `embed` 行尾·模板·embed / `review` 审核处置举报 /
|
|
||||||
`gorm` GORM / `style` 前端样式 / `theme` 主题 / `builder` 主题生成器 / `share` 分享卡 /
|
|
||||||
`honeypot` 蜜罐多家族伪装 / `stats` 全站数据页 / `sec` 安全评估·客户端 IP 信任 /
|
|
||||||
`quality` 业务正确性·运维·前端(2026-10-29) / `misc` 其它约定。
|
|
||||||
|
|
||||||
新增(2026-10-29 数据面板批次):`insights` 箱主数据面板(box_views 表 + 异步记录器 +
|
|
||||||
`/user/insights` 页面 + 图片前端压缩)—— 详见 topics 可暂并入 quality,或另起 topic。
|
|
||||||
|
|
||||||
新增(2026-10-30 安全汇总批次):`honeypot report` 周报/月报(`geoip` 定位层 + `report`
|
|
||||||
聚合绘图 + `ip_geo_cache`/`security_report_logs` + 邮件)—— 详见 `2026-10-30.md`。
|
|
||||||
|
|
||||||
## 安全汇总(要点,明细见 2026-10-30.md)
|
|
||||||
|
|
||||||
- **ipinfo ≠ 地理位置**:ipinfo 是 cdncheck 离线指纹,答「是不是云厂商/CDN/WAF」,
|
|
||||||
肉鸡必然显示「未识别(普通网络)」;「在哪儿」走 `internal/security/geoip`。
|
|
||||||
- `geoip.Result{Geo, Unresolved, Requests, Failures}` 三态:只有「服务明确说没数据」
|
|
||||||
才可写失败缓存(3 天),网络失败**绝不写** —— 否则一次抖动毒化整批 IP 三天。
|
|
||||||
- ip-api 免费版慢(22 个 IP 实测 4–10s),超时给 45s + 重试一次;`addrs` 必须按
|
|
||||||
调用方顺序(命中数降序)构建,否则腾讯精修配额花在长尾上。
|
|
||||||
- 自绘地图**零国界**(合规),真实底图走腾讯;代码不预置 key。报告里港澳台统一
|
|
||||||
写「中国香港/中国澳门/中国台湾」。地图标签必须防重叠(叠出过假的「AU 305」)。
|
|
||||||
- 周期窗口一律 `[start, end)` 左闭右开;报告幂等靠 `security_report_logs.period_key`
|
|
||||||
唯一索引,失败重试 5 次后停。
|
|
||||||
|
|
||||||
## 协作
|
|
||||||
|
|
||||||
- 结论分级 P0/P1/P2;先根因取证 + 实测复现,别只静态推断。
|
|
||||||
- 改代码前先出变更清单;重大问题显式指出。用户常授权自主推进。
|
|
||||||
- 用户是运营者也是圈内人:处置必须可解释、可留痕、可申诉,不做自动级联封号。
|
|
||||||
|
|
||||||
## 构建与测试(详见 topics/build.md)
|
|
||||||
|
|
||||||
- 无 libwebp/GCC:`internal/storage/s3.go` 挡编译 → overlay 换桩 `dev/s3stub/s3_stub.go`。
|
|
||||||
- 「编译卡死」= `proxy.golang.org` 不可达。每条 go 命令前
|
|
||||||
`export GOPROXY=https://goproxy.cn,direct GOSUMDB=off GOFLAGS=-mod=mod`。
|
|
||||||
- DB 测试:`DB_TYPE=postgres DB_USER=postgres DB_PASSWORD=15207786347hXPm DB_HOST=127.0.0.1 DB_PORT=5432`(DSN 不带库名)。
|
|
||||||
- store 方法内用 `NewXStore(db.DB)`,别引用包级单例。
|
|
||||||
- ~~本沙箱起不了 `TamaBox web`~~ **2026-10-27 起可在 WSL2 Debian 上真跑**(PG+Redis+systemd,端口 5877):
|
|
||||||
详见 topics/build.md「本地部署」小节。要点:`env -u GOROOT`(bashrc 指着 1.26.3)、
|
|
||||||
`[recaptcha] captcha_type = math`、Windows 侧 127.0.0.1:5877 被占要用 WSL eth0 IP。
|
|
||||||
HTTP 端到端与 P1-3 的双路径 A/B 都能在这里做(回环=可信、跨主机=不可信)。
|
|
||||||
- python urllib 不稳定(用 curl);python 别用 `/tmp`;PIL 在 `~/.workbuddy/binaries/python/envs/default`。
|
|
||||||
- **Windows 上 `Path.write_text()` 默认把 `\n` 写成 `\r\n`** —— 改 LF 文件必须传 `newline="\n"`
|
|
||||||
或走 bytes,否则整个文件行尾翻车。症状:`gofmt -l` 突然把整文件标为未格式化。
|
|
||||||
**排查看 `git diff --stat`**:真实改动应是几十行,整文件重写就是行尾变了;
|
|
||||||
恢复用 `git show HEAD:<file>` 取其行尾再统一替换。
|
|
||||||
另:本项目 `core.autocrlf=true`,仓库存 LF、工作区 checkout 成 CRLF —— 判断基准要看 `git show`。
|
|
||||||
|
|
||||||
## 行尾 / 模板 / embed(topics/embed.md)
|
|
||||||
|
|
||||||
- 行尾混合不可推断,改前用 Python 数 `b'\r\n'`;gofmt 走 `dev/gofmt_keep_crlf.py`。
|
|
||||||
- `templates/embed.go` 是显式清单:新增**顶层**模板必须加 `//go:embed`。
|
|
||||||
- `//go:embed` 的 `*` 不匹配 `_`/`.` 开头文件。
|
|
||||||
|
|
||||||
## 审核 / 处置 / 举报(topics/review.md)
|
|
||||||
|
|
||||||
- **图片存储 = Cloudflare R2**(S3 兼容协议,非七牛):`conf.Upload.ImageEndpoint`/`ImageBucket`/
|
|
||||||
`ImageBucketCDNHost`,客户端见 `internal/storage/client.go`(R2 的 endpoint resolver 写法,
|
|
||||||
`HostnameImmutable` + `region=auto`);key = `picture/YYYY/MM/DD/<15hex>`。代码里还留了一套
|
|
||||||
阿里 OSS 客户端(`GetOSSBucket`),是历史遗留,实际在用的是 S3/R2 这条。
|
|
||||||
- **文本审核通道 = 七牛/阿里云**(`[security] text_censor_source = qiniu|aliyun|auto`,七牛优先)——
|
|
||||||
这是内容审核 API,与图片存储是两码事,别搞混。线上通道 = 七牛;裁决在 `censor/policy.go` +
|
|
||||||
`moderation.Check`,新增调用点必须走 `moderation.Check`,不可用时 fail-closed。
|
|
||||||
- 严格度存 `users.moderation_strictness`;受处置用户强制 strict(`moderation.EffectiveStrictness`),查询失败回退用户设置。
|
|
||||||
- 同请求渲染只能用 `ctx.SetError`/`SetInternalError`,不能 `*Flash`。
|
|
||||||
- 写路径查功能开关用 `db.FeatureStateOfStrict`(失败即拒);`FeatureStateOf` 的
|
|
||||||
fail-open 只给读展示用 —— 它的默认值对提问/投票是「开启」,等于让一次配置读失败
|
|
||||||
绕过箱主的关闭与定量上限。
|
|
||||||
- 处置四级 L1–L4;到期靠 `expires_at` 无 cron;`restrict` 是替换语义;比较用 `RestrictionAtLeast`。
|
|
||||||
- L2 强制私密须在审核通过后、与写回答同一条 UPDATE(`AnswerByIDPrivate`)。
|
|
||||||
- `ListAPI` 不经 Pager 中间件,L4 与页面密码须自校验;分享卡拦截须在缓存查询之前。
|
|
||||||
- 举报:免登录+验证码+限流;同 IP 同题唯一索引 → 幂等;箱主只看次数不给 IP。
|
|
||||||
- 告警 24h≥8 或 7d≥20 → 邮件;`[security] security_alert_email` 为空则静默跳过。
|
|
||||||
- **处置覆盖面:新增任何「用户发起」的写入口都要问一句「这里要不要查 `ActiveLevel`」**。
|
|
||||||
提问链路 7 处齐全;投票链路曾 0 处(L4 后 `/vote/{token}` 照常可投、可发新投票、探索页照常曝光),
|
|
||||||
2026-10-29 已补齐:`vote.Page`/`vote.Submit` 拦 L4、`VotePublish` 拦 L2、探索页按 `ActiveLevels` 批量过滤。
|
|
||||||
|
|
||||||
## GORM(topics/gorm.md)
|
|
||||||
|
|
||||||
- 默认值必须带引号;事务里每句 `tx.WithContext(ctx)`;链式对象带状态,统计/更新各自重建 query。
|
|
||||||
- 软删除+唯一索引 = 槽位永久占用;级联删除一律 `Unscoped()`。
|
|
||||||
- **「用户主动删除」= 真删**:模型全带 `dbutil.Model` 的 `DeletedAt`,默认 Delete 是软删 ——
|
|
||||||
提交记录里的 `voter_ip`/文本、提问里的 `FromIP` 会永久留在库里。删投票/删提问已改
|
|
||||||
`Unscoped` 并收进一个事务(`Votes.DeleteWithCascade`、`Questions.DeleteByID`)。
|
|
||||||
- **配额必须与内容同事务**:`incrementQuotaTx(tx,...)` 是唯一实现;别在 handler 里
|
|
||||||
再补一次 `IncrementQuota`(那是独立写入,失败只 Warn → `quota_count` 永久偏低)。
|
|
||||||
注意口径按功能不同:提问/私聊是「条数」,投票是「累计收到的票数」。
|
|
||||||
|
|
||||||
## 前端样式(topics/style.md)
|
|
||||||
|
|
||||||
- `static/css/base.css`(`?v=` 取 `conf.StaticVersion`,改完 +1)。令牌在 `:root` + `html[data-theme="dark"]`,新样式一律 `var(--tb-*)`。
|
|
||||||
- 移动端(≤640px)语言/主题按钮在固定底栏 `#tb-mobile-utils`;箱主页**有意**无语言切换。
|
|
||||||
- **错误页不套模板**:`f.NotFound` 不经过组级中间件(`template.Templater`/`Contexter`
|
|
||||||
连同 session、CSRF 都在 `f.Group` 里),在里面注入 `template.Data` 会 panic。
|
|
||||||
404/500 统一走 `internal/route/errorpage.go` 的内联 HTML + `i18n.T`,
|
|
||||||
语言从 `Accept-Language` 猜(`pickErrorPageLang`)。Recovery 也是自写的。
|
|
||||||
- **提问者找回自己的提问**:`templates/question/{success,list}.html` + `POST /api/v1/ask-status`。
|
|
||||||
记录只在本机 localStorage;接口必须同时给对 token 才返回信息,三种失败情形响应完全一致。
|
|
||||||
- UIkit 3.3.3:`.uk-flex::before/::after{display:none}`;主色是编译期常量,需在 base.css 末尾映射。
|
|
||||||
- 视觉自检:`TEST_PREVIEW=1 go test ./templates -run TestGenPreview`;截图用 `chrome --headless=old`。
|
|
||||||
|
|
||||||
## 主题(topics/theme.md、topics/builder.md)
|
|
||||||
|
|
||||||
- 三个注入点按箱主/发布者的 `Theme`;投票页语言也跟发布者(`applyOwnerLocale`)。
|
|
||||||
- 一套主题 = `static/css/themes/<slug>.css`,只覆盖令牌;显示名用 `{{ themeName ... }}`,别直接 `printf "theme.name.%s"`。
|
|
||||||
- `[theme] available` 是硬白名单(启动时读一次);css 必须成对写亮/暗两个选择器。
|
|
||||||
- 装饰层固定 10 槽,三组 `--tb-deco/-size/-position` 必须等长;位置用两值 `calc(0%|50%|100% ± N)`。
|
|
||||||
- 主题生成器 `dev/theme-builder/`:非破坏性图片处理、`layerSize`/`unitGeom` 必须共用同一函数、
|
|
||||||
`hideDeco` 会把 @media 10 槽全写 none(窄屏素材消失的头号来源)。
|
|
||||||
|
|
||||||
## 安全(topics/sec.md)
|
|
||||||
|
|
||||||
- 权威头 `CF-Connecting-IP` / `EO-Client-IP` 只在 TCP 对端可信(回环 或 `trusted_proxies` 命中)
|
|
||||||
才采信,公网对端一律忽略;唯一读取点 `internal/shared/ip.go`,别在别处另读 header。
|
|
||||||
- 生产回源 = cloudflared Tunnel 回本机(对端 127.0.0.1)→ 零配置;`trust_proxy` 两套环境都 true。
|
|
||||||
- **EdgeOne 已取消**(境内速度差)。重新启用前必须把 EO 回源网段写进 `trusted_proxies`,
|
|
||||||
否则 EO 线退化成按边缘 IP 聚合(限流互挤、自动封禁误伤整片)。`app.sample.ini` 已注明。
|
|
||||||
- 跳转参数只走 `shared.SafeRedirectTarget`;`url.Parse` 会解 `%0d%0a`,必须逐 rune 查 `u.Path`。
|
|
||||||
- **同一 handler 里 GET 有所有权校验 ≠ POST 有**(私聊 `SendMessage` 曾漏,可跨箱骚扰)。逐分支核对。
|
|
||||||
- CSP 默认 **enforce**(逃生口 `csp = report|off` 后重启);`script-src` 必须含 `mami.shiroko.one`
|
|
||||||
(umami 动态注入,grep 抓不到)与按 `sentry_dsn` 条件放行的 `browser.sentry-cdn.com`。
|
|
||||||
改 CSP 白名单后必须跑 `csp_sweep.py`(真浏览器巡检),静态检查不算数。
|
|
||||||
- 注册防枚举:邮箱重复/停用与正常注册共用同一个成功出口;代价是忘密只能走找回。
|
|
||||||
- **`flamego.Classic()` 自带 `Logger()` 会记 `RequestURI`(含 query)** —— `?t=` 私密题钥匙、
|
|
||||||
`?code=` 改密凭证都会明文进日志(已实测)。要自写访问日志,只记 `URL.Path`。
|
|
||||||
- 报告 `outputs/security-audit-2026-10-25.md` + 复现脚本 `outputs/sec-audit-2026-10-25/`;
|
|
||||||
非安全向的复盘见 `outputs/review-2026-10-29.md`。
|
|
||||||
|
|
||||||
## 其它约定(topics/misc.md)
|
|
||||||
|
|
||||||
- 健康检查:`/healthz` 永远 200(不碰任何依赖,只看进程活着)+ `/readyz` 查 DB(失败 503)。
|
|
||||||
别把 DB 检查并进 healthz —— 数据库抖动会让进程被反复重启,对恢复毫无帮助。
|
|
||||||
- 校验用 `valid:"required;maxlen:N"` 标签。
|
|
||||||
- 只有箱主 `/user` 面板,**无管理员角色、无站方后台**(用户要求不要加):站方动作只走 CLI / 邮件 / 改配置重启。
|
|
||||||
- CLI 有两个构建入口必须同步:根 `go build .` 与 Dockerfile 的 `./cmd/`。
|
|
||||||
- 现有 CLI:`web censor uid ban restore reports restrict unrestrict hide-public`。
|
|
||||||
- git `https://git.shiroko.one/tamakyi/tamabox`(main),`git push` 可直推;`outputs/` 已 gitignore。
|
|
||||||
- **ImageGen 的 `background: "transparent"` 至今不生效**(返回 RGB、无 alpha 通道)。要透明素材:
|
|
||||||
提示词把背景写成一个**具体纯色**(不要写 white,白衬衫/光环会被一起吃),
|
|
||||||
再按该色值算 alpha:`alpha = clip((maxChannelDiff - 6) / 12)`,最后 `MinFilter(3)` 收 1px。
|
|
||||||
闭合图形(如光环圆环)内部的白/底色用 floodfill 填不到,按色值抠才干净。
|
|
||||||
- 生成图的右下角有 "AI生成 WORKBUDDY" 水印,做素材时统一裁掉底部约 10%(切口平直,正好贴页面下沿)。
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# 构建与测试(长版明细)
|
|
||||||
|
|
||||||
## 构建与测试
|
|
||||||
|
|
||||||
- **无 libwebp/GCC**:cgo 的 `internal/storage/s3.go` 挡编译 → overlay 换桩
|
|
||||||
`dev/s3stub/s3_stub.go`(`-overlay="C:/Users/Tamakyi Gekka/AppData/Local/Temp/tb-webpcheck/overlay.json"`)。
|
|
||||||
要检查 s3.go 本身得 tar 复制项目 + `go mod edit -replace`(overlay 换不了 GOMODCACHE)。
|
|
||||||
- **「编译卡死」= `proxy.golang.org` 不可达**。每条 go 命令前
|
|
||||||
`export GOPROXY=https://goproxy.cn,direct GOSUMDB=off GOFLAGS=-mod=mod`(`go env -w` 沙箱失败)。
|
|
||||||
1C1G 是正式生产环境,别用「沙箱慢」解释超时。
|
|
||||||
- DB 测试:`DB_TYPE=postgres DB_USER=postgres DB_PASSWORD=15207786347hXPm DB_HOST=127.0.0.1 DB_PORT=5432`
|
|
||||||
(DSN 不带库名,`NewTestDB` 自建临时库后 DROP)。带 DB 的 `go test ./...` 全绿。
|
|
||||||
- store 方法内用 `NewXStore(db.DB)`,别引用包级单例(测试下 nil panic;`FeatureStateOf` 有守卫)。
|
|
||||||
- ⚠️ **本沙箱起不了 `TamaBox web`**(session 固定 redis,6379 未开,启动 panic)→ HTTP 端到端只能部署后做。
|
|
||||||
探测端口 200 可能是别的进程,必须看日志确认 `Starting web server` 且无 panic。
|
|
||||||
- ⚠️ **本机 python urllib 不稳定**(代理下偶发读超时,curl 正常);python 别用 `/tmp`
|
|
||||||
(Windows 解析成 `C:\tmp`,与 MSYS 不通)。PIL 在 `~/.workbuddy/binaries/python/envs/default`。
|
|
||||||
- ⚠️ **验证 Go 二进制里有没有某个中文字符串,不能用 `strings`**:GNU strings 默认只输出
|
|
||||||
可打印 ASCII(0x20–0x7E),UTF-8 中文(0xE4+ 开头)整段被丢弃,`strings x | grep 中文`
|
|
||||||
恒为 0 —— 曾据此误判「用户没重新编译」,实际二进制是新的。
|
|
||||||
正确做法:`grep -ac "中文字符串" 二进制文件`(实测同一个二进制:strings=0、grep -a=1)。
|
|
||||||
- ⚠️ 替换二进制后**必须重启进程**:`go build -o X` 只改磁盘文件,正在跑的进程仍是旧映像。
|
|
||||||
若服务由 systemd/supervisor 管理,再手动 `./X web` 会因端口被占直接退出。
|
|
||||||
- ⚠️ **分享卡字体不在仓库里**:`static/fonts/` 从未入库也没 embed(`static/fs.go` 的清单里没有),
|
|
||||||
`/static/fonts/...` 走 HTTP 是 302 回首页。新环境不手工放字体 → `share-card` 恒 500
|
|
||||||
「字体文件未配置或无法加载」。老环境(如生产)能出图只是因为当年手工补过文件。
|
|
||||||
排查口诀:响应体是这句中文 → 一定是部署侧缺文件,不是代码回退。
|
|
||||||
`getFont` 现已支持 candidates(`font_path` → `static/fonts/*` → 系统 CJK 字体)且不缓存失败,
|
|
||||||
补文件后无需重启。
|
|
||||||
- ⚠️ **`GOSUMDB=off` 会让 toolchain 自动下载失败**:go.mod 要 1.27.3 而 PATH 上的 `go` 是旧版时,
|
|
||||||
会报 `golang.org/toolchain@...: verifying module: checksum database disabled by GOSUMDB=off`
|
|
||||||
(`go version` 显示的还是 shell 包装器的版本,容易误判成「版本没变」)。
|
|
||||||
解法:直接用缓存里的 toolchain 并禁用自动切换——
|
|
||||||
`PATH="$HOME/go/pkg/mod/golang.org/[email protected]/bin:$PATH" GOTOOLCHAIN=local go test ./...`
|
|
||||||
|
|
||||||
## 本地部署(WSL2 Debian,2026-10-27 打通)
|
|
||||||
|
|
||||||
用户本机有 WSL2 `Debian`(可 `wsl.exe -d Debian -u root` 免密进 root)。在这上面跑通了完整服务,
|
|
||||||
从此 HTTP 端到端不必「部署到测试站才能做」。要点:
|
|
||||||
|
|
||||||
- **GOROOT 是个坑**:`~/.bashrc` 里 `export GOROOT=/usr/local/go`(1.26.3)。用 `/opt/go/bin/go`
|
|
||||||
(1.27.3,从 dl.google.com 下的 tar 解到 /opt)时必须 `env -u GOROOT`,否则
|
|
||||||
`compile: version "go1.26.3" does not match go tool version "go1.27.3"`。
|
|
||||||
另:GOCACHE 被旧版本编译过也会报同样的错 → 换独立 `GOCACHE=~/.cache/go-build-tb`。
|
|
||||||
- **依赖**:`apt-get install redis-server python3 python3-requests gcc libwebp-dev pkg-config`
|
|
||||||
(有 libwebp 后 `internal/storage/s3.go` 正常编译,不再需要 s3stub overlay)。
|
|
||||||
session 与 cache 都硬依赖 redis,6379 不通就起不来;`systemctl start redis-server` 即可(WSL 有 systemd)。
|
|
||||||
- **库**:DSN 不带库名 → 连的是与用户名同名的库。测试还要 `CREATEDB` 权限(`NewTestDB` 自建临时库)。
|
|
||||||
`CREATE ROLE tamabox LOGIN PASSWORD '...' CREATEDB; CREATE DATABASE tamabox OWNER tamabox;
|
|
||||||
CREATE DATABASE nekobox OWNER tamabox; CREATE SCHEMA nekobox AUTHORIZATION tamabox;`
|
|
||||||
- **配置坑**:`data/data.txt` 里的 `[app] title`、`[server] xsrf_expire`、`[captcha] provider`、
|
|
||||||
`[rate_limit] *` 全是当前版本不认识的键(启动 WARN 后忽略)。验证码要关外网校验得写
|
|
||||||
`[recaptcha] captcha_type = math`。`external_url` 用 http → session cookie 不带 Secure,本地能登录。
|
|
||||||
- **进程托管**:`/etc/systemd/system/tamabox-test.service`(User=tamakyi、WorkingDirectory=源码目录、
|
|
||||||
ExecStart=<二进制> web、日志 append 到文件)。`systemctl restart` 必须 root,普通用户报 Access denied。
|
|
||||||
- **访问路径**:Windows 侧 `http://127.0.0.1:5877` 被本机别的进程占用(返回 502),要用
|
|
||||||
`http://<wsl eth0 ip>:5877`。这个差异正好白送一套 P1-3 的 A/B:
|
|
||||||
WSL 内 `curl 127.0.0.1` = 回环对端(模拟 cloudflared 回源),Windows 侧访问 = 非回环对端(模拟公网直连)。
|
|
||||||
- **抓 IP 伪造**:临时配 `ip_blacklist = 1.2.3.4` + `ip_blacklist_redirect_url`,带头 curl 看有没有 302,
|
|
||||||
比读日志直观;配合日志里的 `remote_addr` / `header_ip` 双重取证。
|
|
||||||
- **限流会绊脚本**:`/api/captcha/generate` 按 IP 限流,批量跑用例会 429,
|
|
||||||
`audit.get_captcha` 已加退避重试。SMTP 密码不对只会打日志(`526 Authentication failure`),不影响用例。
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
# 其它约定(长版明细)
|
|
||||||
|
|
||||||
## 其它约定
|
|
||||||
|
|
||||||
- 校验靠 `valid:"required;maxlen:N"` 标签(`internal/form/*.go`),别在 handler 手写。
|
|
||||||
- 终止性操作(停用)落独立公开页 `/deactivated`(注册在 `reqUserSignIn` 外);改密码后重登才用 flash + `/login`。
|
|
||||||
- 只有箱主 `/user` 面板,**无管理员角色、无站方后台,用户要求不要加**:站方动作只走 CLI / 邮件 / 改配置重启。
|
|
||||||
- git `https://git.shiroko.one/tamakyi/tamabox`(`main`)。**`git push` 能直接推**(本机 Git for Windows 已认证);
|
|
||||||
「Failed to authenticate user」是临时态。推前 `git log --oneline -3`。`outputs/` 已 gitignore,别整目录 add。
|
|
||||||
- 打包交付用 tar.gz(排除 `.git`/`.workbuddy`/`outputs`),放项目同级。
|
|
||||||
|
|
||||||
## 测试环境 box-dev-api.337845818.best
|
|
||||||
|
|
||||||
- 账号 `[email protected]` / `adminadmin`,公开页 `/_/admin6`。
|
|
||||||
- 登录要过算数验证码:`GET /login` 取 `_csrf`+cookie → `GET /api/captcha/generate` → 算 `a op b` →
|
|
||||||
`POST /login` 带 `g-recaptcha-response="<id>:<答案>"`。默认语言 **ja**,四语言文案要一起加。
|
|
||||||
- **部署有延迟**;模板是编译期 embed,**只重启不重编不会变**。用户说「已部署但没变化」→ 先抓线上 HTML 比对标记。
|
|
||||||
- 表单字段:提问 `content`,回答 `answer`。`GET /user/votes` 故意 302;匿名无邮箱提问成功直接 200 渲染。
|
|
||||||
- 自检:`dev/devcheck-e2e.py`(28 项)、`dev/devcheck-restrict.sh`(基线非 200 就别往下走;`RUN=1` 才真跑)。
|
|
||||||
- 图片编码三档(头像 512/q78、背景 1920/q80、通用 1600/q80),**EXIF 方向必须先归一**(转 WebP 会丢)。
|
|
||||||
|
|
||||||
## CLI(`TamaBox <cmd>`)
|
|
||||||
|
|
||||||
- ⚠️ **两个构建入口必须同步**:根 `go build .`(`tamabox.go`)与 Dockerfile 的 `./cmd/`(`cmd/tamabox.go`),
|
|
||||||
否则 `No help topic for '<cmd>'`。
|
|
||||||
- 现有:`web` `censor` `uid` `ban` `restore` `reports` `restrict` `unrestrict` `hide-public`。
|
|
||||||
- `restrict --domain|--email --level L1..L4 --days N --reason` / `unrestrict`;
|
|
||||||
`reports --domain|--email [--days]`(证据包)、`--pending`、`--stats --days N`;`hide-public [--only-reported] [--since N]`。
|
|
||||||
- 本地冒烟:`NEKOBOX_CONFIG_PATH=<ini>`;`conf.Init` 强制 `app.external_url`、`server.salt`、`server.xsrf_key` 非空。
|
|
||||||
|
|
||||||
- 蜜罐 CLI:`TamaBox honeypot hits|stats|prune`(`--since 24h` / `--show-password` / `--yes`),
|
|
||||||
注册须在 `cmd/tamabox.go` + 根 `tamabox.go` 两处同步。
|
|
||||||
- 蜜罐配置:`[security] enable_honeypot` / `honeypot_paths` / `honeypot_retain_days`(默认全关)。
|
|
||||||
|
|
||||||
## 行尾与测试坑(2026-10-21 蜜罐轮)
|
|
||||||
- ⚠️ `internal/route/route.go`、`internal/conf/static.go`、`conf/app.sample.ini` **实际是全 CRLF**;
|
|
||||||
`grep | cat -A` 不显示 `^M` 极易误判 → Python 替换用 `\r\n` 锚点,写后统计 `b'\r\n'` 复核。
|
|
||||||
- ⚠️ Go 里 `time.Duration(60)` = 60 **纳秒**,测试写裸数字会让封禁/冷却瞬间过期 → 一律 `60 * time.Minute`。
|
|
||||||
- ⚠️ 测 `IsBanned` → 403 必须注册中间件(`f.Use(autobanMiddleware(ab, hp))`),只注册路由+NotFound 会被绕过。
|
|
||||||
|
|
||||||
## 配置(ini)相关约定
|
|
||||||
|
|
||||||
- ⚠️ **ini 对未知键静默忽略** → 键名写错 = 配置不生效且无任何现场线索。
|
|
||||||
已加两道防线(`internal/conf/keynorm.go` + `unknown.go`):
|
|
||||||
键名归一化(自动纠正常见误拼)+ 未知键启动告警(附最接近的合法键名)。
|
|
||||||
- ⚠️ **键名必须与代码逐字一致**,包括警告文案里提到的键名 —— 写错等于误导运维。
|
|
||||||
- ⚠️ `MapTo` 在**键缺失/值非法时不覆盖**目标字段 → 可安全预置缺省值。
|
|
||||||
- ⚠️ **bool 的 ini 解析是「有键才赋值」** → 「未配置」与「显式 false」无法用零值区分,
|
|
||||||
必须用 `HasKey` 判定。缺省为 true 的开关尤其要小心(否则未配置 = 关闭)。
|
|
||||||
- 归一化必须在 `MapTo` **之前**执行,否则错拼的键已被丢弃。
|
|
||||||
- 合法键名白名单由**反射从结构体 ini tag 得出**,是单一事实来源;新增配置项自动纳入。
|
|
||||||
- `TestSampleIniHasNoUnknownKeys` 回归检测:官方样例配置零误报。
|
|
||||||
改结构体字段名时必须同步 `conf/app.sample.ini`,否则这个测试会失败。
|
|
||||||
- app.sample.ini 里不要留「代码中从未读取」的死配置(`title` / `xsrf_expire` 已清理)。
|
|
||||||
@@ -1,135 +0,0 @@
|
|||||||
# 安全评估与客户端 IP 信任
|
|
||||||
|
|
||||||
报告:`outputs/security-audit-2026-10-25.md`;复现脚本 `outputs/sec-audit-2026-10-25/`(可重跑)。
|
|
||||||
|
|
||||||
## 客户端 IP 采信(P1-3,2026-10-26 已修)
|
|
||||||
|
|
||||||
- 唯一读取点:`internal/shared/ip.go` 的 `ResolveClientIP`,23 处调用共用。
|
|
||||||
**不要在别处另读 `CF-Connecting-IP` / `EO-Client-IP`**,否则绕过校验。
|
|
||||||
- 规则:权威头只在 `headerSourceTrusted(remote)` 为真时采信 ——
|
|
||||||
对端是回环地址,或命中 `trusted_proxies`。公网对端带来的权威头一律忽略,回退 remote。
|
|
||||||
- XFF / X-Real-IP 仍只认 `trusted_proxies`;回环**不**隐含信任这两个头。
|
|
||||||
- 公网对端却带权威头 → 进程内一条 WARN(`remote_addr` + `header_ip`)。
|
|
||||||
出现这条日志 = 要么有人在伪造,要么 CDN 直连回源但没登记网段。
|
|
||||||
- 不能靠「哪个头存在」推断请求走了哪家 CDN:客户端自带任意头都能反向利用。
|
|
||||||
|
|
||||||
## 部署形态
|
|
||||||
|
|
||||||
- 生产/测试都是 `trust_proxy = true`,回源 = **cloudflared Tunnel 回本机**(对端 127.0.0.1),
|
|
||||||
因此修复后零配置变更。
|
|
||||||
- **EdgeOne 已取消**(境内访问速度不理想),当前只有 Cloudflare 一条线。
|
|
||||||
重新启用前必须把 EO 回源网段写进 `[security] trusted_proxies`,否则 EO 线退化成按边缘 IP 聚合
|
|
||||||
(限流互挤、自动封禁误伤整片)。注释已写进 `conf/app.sample.ini`。
|
|
||||||
- `box.shiroko.one` CNAME 到第三方 `cloudflare.182682.xyz`(CF SaaS 优选),该 zone 的
|
|
||||||
Transform/WAF 不归我们管 —— 别把安全前提建立在那边的规则上。
|
|
||||||
- 源站隐藏:外网尽力排查(A 记录 / certspotter 51 个子域 / MX / 候选 IP)未找到直连入口;
|
|
||||||
仍须服务器侧 `dev/security-selfcheck.sh local` 确认 `listen = 127.0.0.1:5877`。
|
|
||||||
|
|
||||||
## 实测确认「防护到位」(别重复怀疑)
|
|
||||||
|
|
||||||
CSRF 绑 uid(无 token POST → 400);18 条 XSS 载荷全灭(含实体编码、`java\tscript:`、
|
|
||||||
协议相对 `//`、反斜杠);跨箱题目/投票越权全 302;页面密码对主页/整箱分享图/单题分享图/API
|
|
||||||
四个出口都生效;pixel/service/ShareCard 无 SSRF;`exec cwebp` 不接用户输入;SMTP 默认校验证书;
|
|
||||||
`RegenerateID` + `pwd_ver`;TLS1.0/1.1 拒绝;无 CORS 头;测试站蜜罐已开。
|
|
||||||
|
|
||||||
## 已修(2026-10-26)
|
|
||||||
|
|
||||||
- **P1-1** 外链 `<img>`:`markdown.go` 的 `safeImageSrc` 只放行站内相对路径与自有图片 CDN 主机
|
|
||||||
(配置来源 `image_bucket_cdn_host` / `aliyun_bucket_cdn_host` / `<bucket>.<endpoint>` / `external_url`);
|
|
||||||
`a[href]` 仍允许外站。无合法 src 的 img 整段丢弃。
|
|
||||||
- **P1-2** 密码:page/ask 密码 `minlen:6`(`form.PasswordMinLen`)+ handler 二次校验;
|
|
||||||
提问密码比对挪到验证码之后(`askPasswordDenied`,私聊与普通提问各调一次);
|
|
||||||
页面密码失败锁定 `shared.AttemptLimiter(5, 10min, 10min)`,键 = 箱主ID + LimitKey(IP)。
|
|
||||||
- **P1-3** 权威头:见上「客户端 IP 采信」。
|
|
||||||
- 部署后验收:`python outputs/sec-audit-2026-10-25/verify_p1_fixes.py`,四项应全 PASS
|
|
||||||
(旧版本上实测四项全 FAIL)。跑完会自动还原严格度/密码/测试提问。
|
|
||||||
|
|
||||||
## 第二轮修复(2026-10-27,P2/P3 全清)
|
|
||||||
|
|
||||||
- **跳转白名单收敛为唯一实现 `shared.SafeRedirectTarget`**(`internal/shared/redirect.go`),
|
|
||||||
登录 `to` 与 `/_/<domain>/password` 的 `next` 共用。规则:拒绝空/含 `\`、控制字符、空格;
|
|
||||||
必须单斜杠开头、非 `//`;`url.Parse` 后不得有 Scheme/Host/User/Opaque;遍历 `u.Path` 的 rune
|
|
||||||
拒 `\`、`<0x20`、`0x7f`(**关键**:`url.Parse` 会把 `%0d%0a` 解码成 CRLF,只看原始串抓不住);
|
|
||||||
最后 `path.Clean` 再验一次前缀。测试在 `redirect_test.go`(25 条定向 + 全量不变式)。
|
|
||||||
- **注册防枚举**:邮箱重复/已停用与全新注册走**同一个** `registerSucceeded(ctx)` 出口
|
|
||||||
(302 → `/login`、无 alert),日志里才留痕。域名占用仍明文提示(不然用户没法选)。
|
|
||||||
代价:忘密用户只能走「找回密码」。
|
|
||||||
- **CSP**:`internal/route/csp.go` + `[security] csp = off|report|enforce`(默认 report)、
|
|
||||||
`csp_report_uri`。`script-src` 白名单必须含 **`https://mami.shiroko.one`**——
|
|
||||||
`templates/base/header.html` 用 `document.createElement("script")` 动态注 umami,
|
|
||||||
静态 grep `<script src=` 抓不到,切 enforce 后会在无头 Chrome 里报违规。
|
|
||||||
存储 API endpoint(`r2.cloudflarestorage.com` 等)**不要**写进 CSP,只登记对外发资源的 CDN 域。
|
|
||||||
- **`no-store`**:`/user`、`/_*` 命中 `isNoStorePath`,`/share-card` 例外(要能被爬)。
|
|
||||||
- **`securityHeaders` 注册顺序**:原先排在静态资源中间件之后,静态文件拿不到 nosniff/HSTS,已前移。
|
|
||||||
- **封禁落库** `internal/db/ban_record.go`(`ban_records`:ban_key/reason/duration_minutes/expires_at),
|
|
||||||
`newBanner()` 的 `OnBan` 异步写(3s 独立 ctx,失败只告警,store 为 nil 直接跳过)。
|
|
||||||
- 其它:token 比较改 `subtle.ConstantTimeCompare`;`/explore` 页码上限 500;
|
|
||||||
429 统一 `tooManyRequests()`(带 `Retry-After`);`ctx.JSON/JSONError` 显式补 Content-Type。
|
|
||||||
|
|
||||||
## 全局复查新发现(2026-10-27)
|
|
||||||
|
|
||||||
- 🔴 **私聊 `SendMessage` 缺所有权校验**(高危):`internal/route/chat/chat_handler.go` 只有 GET 分支
|
|
||||||
校验 `conv.OwnerID != pageUser.ID`,POST 没有 → 被拉黑的人换个 domain 就能继续往他人会话发消息、
|
|
||||||
触发邮件、绕过屏蔽词与日限额。已补校验 + `verify_chat_owner.py` 端到端 3/3 + **变异验证**
|
|
||||||
(去掉校验后「换 domain 不能操作他人的会话」立刻变红)。**教训:同一 handler 里 GET 有校验
|
|
||||||
不代表 POST 有,逐分支核对。**
|
|
||||||
- 🟡 页面密码 `next` 与 P2-3 同源缺陷(`/\evil` 外跳),随白名单收敛一并修好。
|
|
||||||
- 🟡 **分享卡字体**:无可用字体时每个请求要烧 0.7–4s CPU,而 `/share-card` 是公开端点。
|
|
||||||
现在成功永久缓存、失败缓存 1 分钟(`fontFailRetry`);实测连发 3.44s → 0.53s。
|
|
||||||
|
|
||||||
### 分享卡字体的硬限制
|
|
||||||
|
|
||||||
`github.com/golang/freetype` **只支持 TrueType(glyf)轮廓**,CFF/OTTO 一律报 `bad TTF version`。
|
|
||||||
`NotoSansCJK-Regular.ttc` 是 OTTO → 就算 `ttcFirstFont()` 正确提取出子字体也仍不可用。
|
|
||||||
`.ttc` 是字体集合,需先按 `ttcf` 魔数 → `numFonts` → 子字体偏移 → 重排 sfnt header + 表目录
|
|
||||||
(4 字节对齐)才能交给 freetype(`route/question/ttc.go`,测试 `ttc_test.go` 自造最小 TTC)。
|
|
||||||
|
|
||||||
**2026-10-27 已收敛**:`fontCandidates()` 补 glyf 系统候选(wqy-zenhei / wqy-microhei /
|
|
||||||
DroidSansFallbackFull / uming)且排在 Noto(CFF) 之前;`static/fonts` 同时按 CWD 与可执行文件目录找;
|
|
||||||
`CheckShareCardFont()` 在 `runWeb` 启动时探一次(Info 打路径 / WARN 给 `apt install fonts-wqy-zenhei`)。
|
|
||||||
本地实测该 apt 包即可零配置出图(`/share-card` 500 → 200 image/jpeg)。生产同理,别用 Noto CJK。
|
|
||||||
|
|
||||||
## 仍待决策/未做
|
|
||||||
|
|
||||||
- 限流/失败计数迁 Redis(单进程内存态,多副本会各算各的)—— 明确不做,配置注释已写明单副本前提。
|
|
||||||
- 匿名投票清 cookie 即换身份:只加了结果页匿名说明,未做 IP 唯一索引(会被 NAT/移动网误伤)。
|
|
||||||
- 颜色字段只校验长度(有 Go 上下文转义兜底,非安全问题)。
|
|
||||||
- P1-1 残余:存量内容里的外链 img 仍在库里,仅渲染层不再输出;将来若加「原样导出」入口会复发。
|
|
||||||
- CSP 默认 **enforce**(2026-10-27 定):不给 `csp_report_uri` 的 report-only 等于没装,
|
|
||||||
留个不生效的开关更容易让人误以为已防护。白名单由 `csp_sweep.py` 真浏览器巡检背书
|
|
||||||
(18 页面 + 2 图片端点,零违规);误伤时的逃生口是 `csp = report|off` + 重启,不用改代码。
|
|
||||||
- **Sentry CDN 是条件白名单**:`header.html` 的 Sentry 在 `{{ if ne SentryDSN "" }}` 里,
|
|
||||||
静态 grep 和本地巡检都碰不到 → 按 `conf.App.SentryDSN` 条件放行,单测双向断言。
|
|
||||||
|
|
||||||
## 复现脚本怎么跑(踩过的两次假红,别再误判成回归)
|
|
||||||
|
|
||||||
- 四套脚本默认目标是**远端** `box-dev-api`(`audit.py` 的 fallback),在本地实例上验证
|
|
||||||
必须显式给环境变量,否则 4/4 全 FAIL —— 那是「远端没部署修复」,不是回归:
|
|
||||||
`PYTHONDONTWRITEBYTECODE=1 TB_BASE=http://172.27.49.117:5877 [email protected]
|
|
||||||
TB_PWD='TbLocal!2026' TB_DOMAIN=tblocal <脚本>`(Windows 侧 127.0.0.1:5877 被占,需用 WSL eth0 IP)。
|
|
||||||
- **`verify_chat_owner.py` 必须在 WSL 里跑**:它直接调 `psql` 查库,Windows 侧 `psql` 会连到
|
|
||||||
宿主自己的 PostgreSQL 18(`/c/OtherFiles/PostgreSQL/18/bin/psql`),账号/库都不对;且 psql 的中文
|
|
||||||
报错是 GBK,Python 按 UTF-8 解会先抛 `UnicodeDecodeError` 把真实错误吃掉。WSL 侧:
|
|
||||||
`cd /mnt/c/.../sec-audit-2026-10-25 && TB_BASE=http://127.0.0.1:5877 ... python3 verify_chat_owner.py`。
|
|
||||||
- 页面密码入口限流 **10 次/分钟/IP**,`verify_page_next.py` 一次要 POST 6 次 → **一分钟内连跑两次**
|
|
||||||
会从第 5 项开始 `status=429`;`verify_p1_fixes.py` 的 p1-2c 还会把「本机 IP + 测试箱」锁 10 分钟。
|
|
||||||
两者都靠 `systemctl restart tamabox-test` 清内存态。顺序:先 `verify_page_next.py`,再 `verify_p1_fixes.py`。
|
|
||||||
- 判据别写「Location 里不含 evil」这种弱断言:一旦被限流/锁定停在密码页也会假绿。
|
|
||||||
`verify_page_next.py` 已改成精确比对 `Location == /_/<domain>`、429 单独报,且**每条载荷开新会话**
|
|
||||||
(验证通过会给会话发通行标记,复用会话的话后面几条根本不走密码校验)。
|
|
||||||
- 残留:`verify_round2.py` / `verify_chat_owner.py` 会注册临时账号(`verify*@example.com`);
|
|
||||||
跑完 `DELETE FROM nekobox.users WHERE email LIKE 'verify%@example.com'`(无 FK,先确认
|
|
||||||
conversations/questions/votes 无关联行)。
|
|
||||||
|
|
||||||
## 本地部署实测(2026-10-27,WSL)
|
|
||||||
|
|
||||||
- 三项 P1 在真实实例上端到端复验通过:P1-1 正/负与变体、P1-2a/b/c、P1-3 的可信/不可信对端 A/B
|
|
||||||
(探针用临时 `ip_blacklist = 1.2.3.4`:回环对端 + `CF-Connecting-IP: 1.2.3.4` → 302 命中;
|
|
||||||
跨主机对端同样头 → 200 忽略,日志出 `remote_addr` / `header_ip` 的 WARN)。
|
|
||||||
- ⚠️ **P1-1 白名单的坑(已修)**:配置里 `image_bucket_cdn_host` 实际写成 `host/path`
|
|
||||||
(本项目就是 `r2.337845818.best/tamabox`),整串当主机名比 → 自家图被剥。
|
|
||||||
现在统一走 `normalizeHostLike()`(补协议 → url.Parse → Hostname),并把 S3 endpoint 主机自身也入白名单
|
|
||||||
(R2 path-style 链接是 `<endpoint>/<bucket>/…`)。**白名单类单测必须照抄生产配置写法。**
|
|
||||||
- **P2-3 实测确认可利用**:`/login?to=/\evil.example` → `Location: /\evil.example`,
|
|
||||||
WHATWG 解析为 `http://evil.example/`;`\\evil.example` 同样外跳;`//`、`///` 已被规整。
|
|
||||||
正文渲染侧同类载荷是安全的(`<a href="/\example.org">` → `<a>`),缺口只在跳转参数。
|
|
||||||
@@ -30,6 +30,8 @@ func main() {
|
|||||||
cmd.HidePublic,
|
cmd.HidePublic,
|
||||||
cmd.Honeypot,
|
cmd.Honeypot,
|
||||||
cmd.Backup,
|
cmd.Backup,
|
||||||
|
cmd.RestoreImages,
|
||||||
|
cmd.R2Outbox,
|
||||||
}
|
}
|
||||||
if err := app.Run(os.Args); err != nil {
|
if err := app.Run(os.Args); err != nil {
|
||||||
logrus.WithError(err).Fatal("Failed to start application")
|
logrus.WithError(err).Fatal("Failed to start application")
|
||||||
|
|||||||
Binary file not shown.
@@ -1,346 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""头像上传链路端到端验证(批次 A 分档 + B1 EXIF 归一)。
|
|
||||||
|
|
||||||
为什么走头像而不是提问图:
|
|
||||||
- 提问图那条路要经过内容审核/提问入库才能拿到 key,链路长且易受其它因素干扰;
|
|
||||||
- 头像强制 1:1 且 POST /user/profile/update 直接返回页面,便于立即取回 CDN URL。
|
|
||||||
|
|
||||||
验证两件事:
|
|
||||||
A 分档生效:上传 1600x1600 的大图,头像档 maxSide=512 -> 存盘应为 512x512
|
|
||||||
(改前是 1920,实测体积差 ~7 倍)
|
|
||||||
B1 EXIF Orientation=6 归一:上传「像素 512x256 + EXIF 顺时针 90」的方形图,
|
|
||||||
归一后仍应是 512x512(宽高互换但比例不变,故改用体积+尺寸双断言,
|
|
||||||
并额外验证竖长条图 256x512 + Orientation=6 -> 应变成 512x256)
|
|
||||||
|
|
||||||
用法:
|
|
||||||
python dev/devcheck-avatar.py [--base https://box-dev-api.337845818.best]
|
|
||||||
依赖:Pillow(仅用于本地生成测试图片)
|
|
||||||
清理:TamaBox ban --email <输出中的测试邮箱> --yes
|
|
||||||
"""
|
|
||||||
import argparse
|
|
||||||
import http.cookiejar
|
|
||||||
import io
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import struct
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
import uuid
|
|
||||||
|
|
||||||
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
|
||||||
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
|
|
||||||
RESULTS = []
|
|
||||||
|
|
||||||
|
|
||||||
class NoRedirect(urllib.request.HTTPRedirectHandler):
|
|
||||||
def redirect_request(self, req, fp, code, msg, headers, newurl): # noqa: D102
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
class Resp:
|
|
||||||
def __init__(self, status, headers, body):
|
|
||||||
self.status = status
|
|
||||||
self.h = {k.lower(): v for k, v in headers.items()}
|
|
||||||
self.body = body
|
|
||||||
|
|
||||||
@property
|
|
||||||
def loc(self):
|
|
||||||
return self.h.get("location", "")
|
|
||||||
|
|
||||||
@property
|
|
||||||
def text(self):
|
|
||||||
return self.body.decode("utf-8", "replace")
|
|
||||||
|
|
||||||
|
|
||||||
class Client:
|
|
||||||
def __init__(self, base):
|
|
||||||
self.base = base
|
|
||||||
self.cj = http.cookiejar.CookieJar()
|
|
||||||
self.op = urllib.request.build_opener(
|
|
||||||
urllib.request.HTTPCookieProcessor(self.cj), NoRedirect())
|
|
||||||
|
|
||||||
def _send(self, req, timeout=120):
|
|
||||||
try:
|
|
||||||
with self.op.open(req, timeout=timeout) as r:
|
|
||||||
return Resp(r.status, dict(r.headers), r.read())
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
try:
|
|
||||||
body = e.read()
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
body = b""
|
|
||||||
return Resp(e.code, dict(e.headers), body)
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
return Resp(-1, {}, ("TRANSPORT_ERROR: %s" % e).encode())
|
|
||||||
|
|
||||||
def req(self, method, path, data=None, content_type=None, timeout=120):
|
|
||||||
url = path if path.startswith("http") else self.base + path
|
|
||||||
headers = {"User-Agent": UA, "Referer": self.base + path.split("?")[0]}
|
|
||||||
body = None
|
|
||||||
if data is not None:
|
|
||||||
body = data if isinstance(data, bytes) else urllib.parse.urlencode(data).encode()
|
|
||||||
headers["Content-Type"] = content_type or "application/x-www-form-urlencoded"
|
|
||||||
return self._send(urllib.request.Request(url, data=body, headers=headers, method=method),
|
|
||||||
timeout)
|
|
||||||
|
|
||||||
def get(self, p, **kw):
|
|
||||||
r = self.req("GET", p, **kw)
|
|
||||||
return r if r.status != -1 else self.req("GET", p, **kw)
|
|
||||||
|
|
||||||
def post(self, p, data=None, **kw):
|
|
||||||
return self.req("POST", p, data=data, **kw)
|
|
||||||
|
|
||||||
def get_binary(self, url):
|
|
||||||
return self._send(urllib.request.Request(
|
|
||||||
url, headers={"User-Agent": UA}, method="GET"), timeout=60)
|
|
||||||
|
|
||||||
|
|
||||||
def step(desc, ok, extra=""):
|
|
||||||
RESULTS.append((desc, ok, extra))
|
|
||||||
print(" [%s] %s%s" % ("PASS" if ok else "FAIL", desc, (" (%s)" % extra) if extra else ""))
|
|
||||||
return ok
|
|
||||||
|
|
||||||
|
|
||||||
def csrf_of(html):
|
|
||||||
m = re.search(r'name="_csrf"\s+value="([^"]+)"', html)
|
|
||||||
return m.group(1) if m else None
|
|
||||||
|
|
||||||
|
|
||||||
def captcha(c):
|
|
||||||
j = json.loads(c.get("/api/captcha/generate").text)
|
|
||||||
pid, q = j["data"]["id"], j["data"]["question"]
|
|
||||||
m = re.match(r"\s*(\d+)\s*([+\-×])\s*(\d+)\s*=", q)
|
|
||||||
a, op, b = int(m.group(1)), m.group(2), int(m.group(3))
|
|
||||||
return "%s:%d" % (pid, {"+": a + b, "-": a - b, "×": a * b}[op])
|
|
||||||
|
|
||||||
|
|
||||||
def multipart(fields, files):
|
|
||||||
boundary = "----tbav%s" % uuid.uuid4().hex
|
|
||||||
buf = io.BytesIO()
|
|
||||||
for k, v in fields.items():
|
|
||||||
buf.write(("--%s\r\n" % boundary).encode())
|
|
||||||
buf.write(('Content-Disposition: form-data; name="%s"\r\n\r\n' % k).encode())
|
|
||||||
buf.write(("%s\r\n" % v).encode())
|
|
||||||
for name, fn, ct, data in files:
|
|
||||||
buf.write(("--%s\r\n" % boundary).encode())
|
|
||||||
buf.write(('Content-Disposition: form-data; name="%s"; filename="%s"\r\n' % (name, fn)).encode())
|
|
||||||
buf.write(("Content-Type: %s\r\n\r\n" % ct).encode())
|
|
||||||
buf.write(data)
|
|
||||||
buf.write(b"\r\n")
|
|
||||||
buf.write(("--%s--\r\n" % boundary).encode())
|
|
||||||
return buf.getvalue(), "multipart/form-data; boundary=%s" % boundary
|
|
||||||
|
|
||||||
|
|
||||||
def sniff(d):
|
|
||||||
if len(d) >= 12 and d[:4] == b"RIFF" and d[8:12] == b"WEBP":
|
|
||||||
return "webp", webp_dims(d)
|
|
||||||
if d[:3] == b"\xff\xd8\xff":
|
|
||||||
return "jpeg", jpeg_dims(d)
|
|
||||||
if d[:8] == b"\x89PNG\r\n\x1a\n":
|
|
||||||
return "png", (struct.unpack(">II", d[16:24]) if len(d) >= 24 else None)
|
|
||||||
return "unknown", None
|
|
||||||
|
|
||||||
|
|
||||||
def webp_dims(d):
|
|
||||||
c = d[12:16]
|
|
||||||
if c == b"VP8 " and len(d) >= 30:
|
|
||||||
return (struct.unpack("<H", d[26:28])[0] & 0x3FFF,
|
|
||||||
struct.unpack("<H", d[28:30])[0] & 0x3FFF)
|
|
||||||
if c == b"VP8L" and len(d) >= 25:
|
|
||||||
b = struct.unpack("<I", d[21:25])[0]
|
|
||||||
return (b & 0x3FFF) + 1, ((b >> 14) & 0x3FFF) + 1
|
|
||||||
if c == b"VP8X" and len(d) >= 30:
|
|
||||||
return ((d[24] | d[25] << 8 | d[26] << 16) + 1,
|
|
||||||
(d[27] | d[28] << 8 | d[29] << 16) + 1)
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def jpeg_dims(d):
|
|
||||||
i = 2
|
|
||||||
while i + 9 < len(d):
|
|
||||||
if d[i] != 0xFF:
|
|
||||||
i += 1
|
|
||||||
continue
|
|
||||||
m = d[i + 1]
|
|
||||||
if m in (0xC0, 0xC1, 0xC2, 0xC3):
|
|
||||||
h, w = struct.unpack(">HH", d[i + 5:i + 9])
|
|
||||||
return (w, h)
|
|
||||||
if m in (0xD8, 0xD9) or 0xD0 <= m <= 0xD7:
|
|
||||||
i += 2
|
|
||||||
continue
|
|
||||||
i += 2 + struct.unpack(">H", d[i + 2:i + 4])[0]
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def make_avatar(w, h, orientation=None):
|
|
||||||
"""生成一张带明显方位特征的方形头像。"""
|
|
||||||
from PIL import Image, ImageDraw
|
|
||||||
im = Image.new("RGB", (w, h), (30, 120, 200))
|
|
||||||
d = ImageDraw.Draw(im)
|
|
||||||
# 顶部一条粗红条:旋转 90 度后它会跑到侧面,方向一眼可辨
|
|
||||||
d.rectangle([0, 0, w, max(2, h // 12)], fill=(230, 30, 30))
|
|
||||||
for i in range(0, w, max(8, w // 20)):
|
|
||||||
d.line([(i, h // 4), (i, h)], fill=(255, 255, 255), width=2)
|
|
||||||
buf = io.BytesIO()
|
|
||||||
if orientation:
|
|
||||||
exif = im.getexif()
|
|
||||||
exif[0x0112] = orientation
|
|
||||||
im.save(buf, "JPEG", quality=90, exif=exif)
|
|
||||||
else:
|
|
||||||
im.save(buf, "JPEG", quality=90)
|
|
||||||
return buf.getvalue()
|
|
||||||
|
|
||||||
|
|
||||||
def scrape_profile_fields(html):
|
|
||||||
"""从 profile 页抓取所有 input 的 name/value,保证回填必填项。
|
|
||||||
|
|
||||||
UpdateProfile 的必填字段有十来个(昵称/介绍/5 个二维码颜色/点大小/透明度…),
|
|
||||||
漏一个就会因 `valid:"required"` 被 Bind 挡下、头像根本不会走到上传分支。
|
|
||||||
"""
|
|
||||||
fields = {}
|
|
||||||
for m in re.finditer(r'<input\b[^>]*>', html, re.I):
|
|
||||||
tag = m.group(0)
|
|
||||||
nm = re.search(r'name="([^"]+)"', tag, re.I)
|
|
||||||
if not nm:
|
|
||||||
continue
|
|
||||||
name = nm.group(1)
|
|
||||||
if name in ("avatar", "background", "backgroundimage", "_csrf", "submit"):
|
|
||||||
continue
|
|
||||||
typ = re.search(r'type="([^"]+)"', tag, re.I)
|
|
||||||
if typ and typ.group(1).lower() in ("checkbox", "radio"):
|
|
||||||
# 只保留勾选态的,避免把「本想取消」的项又勾上
|
|
||||||
if not re.search(r'\bchecked\b', tag, re.I):
|
|
||||||
continue
|
|
||||||
val = re.search(r'value="([^"]*)"', tag, re.I)
|
|
||||||
fields[name] = val.group(1) if val else ""
|
|
||||||
# select 走第一个非空 option
|
|
||||||
for m in re.finditer(r'<select\b[^>]*name="([^"]+)"[^>]*>(.*?)</select>', html, re.I | re.S):
|
|
||||||
name, body = m.group(1), m.group(2)
|
|
||||||
opts = re.findall(r'<option\b([^>]*)>', body, re.I)
|
|
||||||
chosen = ""
|
|
||||||
for o in opts:
|
|
||||||
vm = re.search(r'value="([^"]*)"', o)
|
|
||||||
v = vm.group(1) if vm else ""
|
|
||||||
if re.search(r'\bselected\b', o, re.I):
|
|
||||||
chosen = v
|
|
||||||
break
|
|
||||||
if not chosen:
|
|
||||||
chosen = v
|
|
||||||
fields[name] = chosen
|
|
||||||
# textarea
|
|
||||||
for m in re.finditer(r'<textarea\b[^>]*name="([^"]+)"[^>]*>(.*?)</textarea>', html, re.I | re.S):
|
|
||||||
fields[m.group(1)] = re.sub(r"<[^>]+>", "", m.group(2)).strip()
|
|
||||||
return fields
|
|
||||||
|
|
||||||
|
|
||||||
def upload_avatar(c, data, fname="avatar.jpg"):
|
|
||||||
page = c.get("/user/profile")
|
|
||||||
if page.status != 200:
|
|
||||||
return None, "profile HTTP %s" % page.status
|
|
||||||
fields = scrape_profile_fields(page.text)
|
|
||||||
fields["_csrf"] = csrf_of(page.text)
|
|
||||||
body, ct = multipart(fields, [("avatar", fname, "image/jpeg", data)])
|
|
||||||
res = c.post("/user/profile/update", data=body, content_type=ct)
|
|
||||||
# 成功时是 302 跳回 /user/profile,头像 URL 在跳转后的页面里
|
|
||||||
if res.status == 302 and res.loc:
|
|
||||||
res = c.get(res.loc if res.loc.startswith("/") else "/user/profile")
|
|
||||||
if res.status != 200:
|
|
||||||
return None, "update 后页面 HTTP %s" % res.status
|
|
||||||
# 从页面里取头像 URL:<img src="https://r2.../picture/..."> 或 og:image
|
|
||||||
urls = re.findall(r'https://r2\.[^"\'\s<>]+/picture/[0-9]+/\d{2}/\d{2}/[0-9a-f]+', res.text)
|
|
||||||
if not urls:
|
|
||||||
m = re.search(r'(https://r2\.[^"\'\s<>]+/picture/[^"\'\s<>]+)', res.text)
|
|
||||||
urls = [m.group(1)] if m else []
|
|
||||||
if urls:
|
|
||||||
return urls[-1], None
|
|
||||||
# 没匹配到:落盘返回页并打印可疑片段,便于定位是「上传被拒」还是「正则没覆盖」
|
|
||||||
dump = "C:/Users/Tamakyi Gekka/AppData/Local/Temp/tbav_profile.html"
|
|
||||||
with open(dump, "wb") as f:
|
|
||||||
f.write(res.body)
|
|
||||||
alerts = re.findall(r'class="uk-alert[^"]*"[^>]*>(.*?)</div>', res.text, re.S)
|
|
||||||
hint = " | ".join(re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", a)).strip()[:80]
|
|
||||||
for a in alerts[:3])
|
|
||||||
return None, "未匹配到 picture URL(已存 %s)%s%s" % (dump, (" 提示:" + hint) if hint else "",
|
|
||||||
(" HTTP %s" % res.status))
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--base", default="https://box-dev-api.337845818.best")
|
|
||||||
args = ap.parse_args()
|
|
||||||
base = args.base.rstrip("/")
|
|
||||||
pwd = "TbAv!2026"
|
|
||||||
ts = time.strftime("%m%d%H%M%S")
|
|
||||||
email, domain = "tbav%s@example.com" % ts, "tbav%s" % ts
|
|
||||||
print("== 账号:%s / %s ==" % (email, domain))
|
|
||||||
|
|
||||||
# 方形 1600x1600:验证 A 分档(应存成 512x512)
|
|
||||||
square = make_avatar(1600, 1600)
|
|
||||||
# EXIF Orientation=6(顺时针 90):像素存成横的 1536x512,归一后显示为竖的 512x1536,
|
|
||||||
# 再经头像档限边 512 -> 512x512。
|
|
||||||
# 注意必须用「像素横躺」的样本:头像有 1:1 比例校验,若像素本身就是竖的会被先拒掉,
|
|
||||||
# 那样就测不到 EXIF 分支了。
|
|
||||||
exif_shot = make_avatar(1536, 512, orientation=6)
|
|
||||||
print(" 本地样本 square 1600x1600%7d bytes" % len(square))
|
|
||||||
print(" 本地样本 exifshot 1536x512 (EXIF=6)%d bytes" % len(exif_shot))
|
|
||||||
|
|
||||||
c = Client(base)
|
|
||||||
r = c.get("/register")
|
|
||||||
reg = c.post("/register", {
|
|
||||||
"email": email, "domain": domain, "name": "TBAv%s" % ts,
|
|
||||||
"password": pwd, "repeat_password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(c), "language": "zh-CN",
|
|
||||||
"_csrf": csrf_of(r.text),
|
|
||||||
})
|
|
||||||
if not step("注册", reg.status == 302, "HTTP %s -> %s" % (reg.status, reg.loc)):
|
|
||||||
return 1
|
|
||||||
|
|
||||||
c = Client(base)
|
|
||||||
r = c.get("/login")
|
|
||||||
lg = c.post("/login", {"email": email, "password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(c), "_csrf": csrf_of(r.text)})
|
|
||||||
if not step("登录", lg.status == 302, "HTTP %s -> %s" % (lg.status, lg.loc)):
|
|
||||||
return 1
|
|
||||||
|
|
||||||
# ---- A 分档 ----
|
|
||||||
url, err = upload_avatar(c, square)
|
|
||||||
if not step("上传方形大头像(1600x1600)", url is not None, err or (url or "")):
|
|
||||||
return 1
|
|
||||||
img = c.get_binary(url)
|
|
||||||
st, data = img.status, img.body
|
|
||||||
if not step("拉回头像", st == 200 and bool(data), "HTTP %s" % st):
|
|
||||||
return 1
|
|
||||||
kind, dims = sniff(data)
|
|
||||||
print(" 存储结果:%s %s %d bytes" % (kind, dims, len(data)))
|
|
||||||
step("A 分档生效:大图被缩到头像档 512(改前是 1920)",
|
|
||||||
dims == (512, 512), "实际 %s" % (dims,))
|
|
||||||
|
|
||||||
# ---- B1 EXIF ----
|
|
||||||
# 关键:两次上传的 URL 必须不同,否则说明第二次没生效(回退到了旧头像)。
|
|
||||||
url2, err2 = upload_avatar(c, exif_shot)
|
|
||||||
if not step("上传 EXIF 横躺头像(1536x512, Orientation=6)", bool(url2), err2 or (url2 or "")):
|
|
||||||
return 0
|
|
||||||
if url2 == url:
|
|
||||||
step("B1 EXIF 归一(第二次上传未生效,URL 未变)", False,
|
|
||||||
"两次上传返回同一 URL,无法区分;请检查是否被 1:1 校验拒绝")
|
|
||||||
return 0
|
|
||||||
img2 = c.get_binary(url2)
|
|
||||||
kind2, dims2 = sniff(img2.body)
|
|
||||||
print(" 存储结果:%s %s %d bytes" % (kind2, dims2, len(img2.body)))
|
|
||||||
# 归一后:像素 1536x512 -> 旋转成 512x1536 -> 再限边 512 -> 512x512
|
|
||||||
step("B1 EXIF Orientation=6 归一(1536x512 -> 512x1536 -> 512x512)",
|
|
||||||
dims2 == (512, 512), "实际 %s" % (dims2,))
|
|
||||||
# 体积对照:未归一时 1536x512 的 WebP 明显更大
|
|
||||||
step("B1 归一后体积已按 512 档压过(应与方形样本同量级)",
|
|
||||||
len(img2.body) < 6000, "%d bytes" % len(img2.body))
|
|
||||||
|
|
||||||
print("\n== 结果:%d/%d 通过 ==" % (sum(1 for _, ok, _ in RESULTS if ok), len(RESULTS)))
|
|
||||||
print("清理:TamaBox ban --email %s --yes" % email)
|
|
||||||
return 0 if all(ok for _, ok, _ in RESULTS) else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,360 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""TamaBox 部署后冒烟巡检(端到端)。
|
|
||||||
|
|
||||||
用途:对已部署环境跑一遍核心链路,用于发版/重新部署后的回归。
|
|
||||||
覆盖:注册 -> 登录 -> 个人页/提问列表/投票面板 -> 投票全链路(建票发布/公开页/匿名提交/
|
|
||||||
结果导出计数/草稿/表单型)-> 匿名提问 -> 停用
|
|
||||||
-> /deactivated 成功页 -> **同邮箱重注册必须被明确拒绝并提示联系站长**
|
|
||||||
-> 公开页只读巡检 -> /deactivated 四语言渲染。
|
|
||||||
|
|
||||||
语义约定(2026-09-15 起):停用 = 清空内容 + 保留用户行作「身份占位」,
|
|
||||||
故邮箱/域名继续被占用,重注册应得到「账号已停用,请找站长」的提示而不是 302 或 500。
|
|
||||||
|
|
||||||
用法:
|
|
||||||
python dev/devcheck-e2e.py [--base https://box-dev-api.337845818.best]
|
|
||||||
|
|
||||||
注意:
|
|
||||||
- 会在目标环境**真实创建账号并发送提问**(使用 tbe2e<时间戳>@example.com 之类的可识别测试邮箱),
|
|
||||||
仅在测试/开发环境运行。
|
|
||||||
- 每次完整运行都会**留下一个已停用的身份占位**(邮箱/域名被占用)。清理方式:
|
|
||||||
`TamaBox ban --email <该邮箱> --yes`(连行物理删除)或 `TamaBox restore --email <该邮箱> --yes`(恢复账号)。
|
|
||||||
- 依赖数学验证码(conf.Recaptcha.CaptchaType == "math");若站点改用 recaptcha/turnstile 则无法自动通过。
|
|
||||||
"""
|
|
||||||
import argparse
|
|
||||||
import http.cookiejar
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
|
||||||
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
|
|
||||||
|
|
||||||
RESULTS = []
|
|
||||||
|
|
||||||
|
|
||||||
class NoRedirect(urllib.request.HTTPRedirectHandler):
|
|
||||||
def redirect_request(self, req, fp, code, msg, headers, newurl): # noqa: D102
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
class Resp:
|
|
||||||
def __init__(self, status, headers, body):
|
|
||||||
self.status = status
|
|
||||||
self.h = {k.lower(): v for k, v in headers.items()}
|
|
||||||
self.body = body
|
|
||||||
|
|
||||||
@property
|
|
||||||
def loc(self):
|
|
||||||
return self.h.get("location", "")
|
|
||||||
|
|
||||||
def alert(self):
|
|
||||||
m = re.search(r'class="uk-alert[^"]*"[^>]*>(.*?)</div>', self.body, re.S)
|
|
||||||
return re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", m.group(0))).strip()[:200] if m else ""
|
|
||||||
|
|
||||||
|
|
||||||
class Client:
|
|
||||||
def __init__(self, base):
|
|
||||||
self.base = base
|
|
||||||
self.cj = http.cookiejar.CookieJar()
|
|
||||||
self.op = urllib.request.build_opener(
|
|
||||||
urllib.request.HTTPCookieProcessor(self.cj), NoRedirect())
|
|
||||||
|
|
||||||
def req(self, method, path, data=None, timeout=90):
|
|
||||||
url = path if path.startswith("http") else self.base + path
|
|
||||||
headers = {"User-Agent": UA, "Referer": self.base + path.split("?")[0]}
|
|
||||||
body = urllib.parse.urlencode(data).encode() if data is not None else None
|
|
||||||
if body:
|
|
||||||
headers["Content-Type"] = "application/x-www-form-urlencoded"
|
|
||||||
req = urllib.request.Request(url, data=body, headers=headers, method=method)
|
|
||||||
try:
|
|
||||||
with self.op.open(req, timeout=timeout) as r:
|
|
||||||
return Resp(r.status, dict(r.headers), r.read().decode("utf-8", "replace"))
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
# 读取错误响应体本身也可能超时(尤其 302 无体/远端慢),不能让它把巡检整轮打断
|
|
||||||
try:
|
|
||||||
body = e.read().decode("utf-8", "replace")
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
body = ""
|
|
||||||
return Resp(e.code, dict(e.headers), body)
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
return Resp(-1, {}, "TRANSPORT_ERROR: %s: %s" % (type(e).__name__, e))
|
|
||||||
|
|
||||||
def get(self, p, **kw):
|
|
||||||
# 只读 GET 重试一次:远端偶发慢请求会产生 TRANSPORT_ERROR,避免巡检出现假失败。
|
|
||||||
r = self.req("GET", p, **kw)
|
|
||||||
if r.status == -1:
|
|
||||||
r = self.req("GET", p, **kw)
|
|
||||||
return r
|
|
||||||
|
|
||||||
def post(self, p, data=None, **kw):
|
|
||||||
return self.req("POST", p, data=data, **kw)
|
|
||||||
|
|
||||||
|
|
||||||
def csrf_of(html):
|
|
||||||
m = re.search(r'name="_csrf"\s+value="([^"]+)"', html)
|
|
||||||
return m.group(1) if m else None
|
|
||||||
|
|
||||||
|
|
||||||
def captcha(c):
|
|
||||||
"""取一道数学验证码并算出答案,返回 <id>:<answer>。"""
|
|
||||||
j = json.loads(c.get("/api/captcha/generate").body)
|
|
||||||
pid, q = j["data"]["id"], j["data"]["question"]
|
|
||||||
m = re.match(r"\s*(\d+)\s*([+\-×])\s*(\d+)\s*=", q)
|
|
||||||
a, op, b = int(m.group(1)), m.group(2), int(m.group(3))
|
|
||||||
ans = {"+": a + b, "-": a - b, "×": a * b}[op]
|
|
||||||
return "%s:%d" % (pid, ans)
|
|
||||||
|
|
||||||
|
|
||||||
def step(desc, ok, extra=""):
|
|
||||||
RESULTS.append((desc, ok, extra))
|
|
||||||
print(" [%s] %s%s" % ("PASS" if ok else "FAIL", desc, (" (%s)" % extra) if extra else ""))
|
|
||||||
|
|
||||||
|
|
||||||
# 停用提示的跨语言特征串(站点默认语言可能是 ja,断言不能只看中文)
|
|
||||||
DEACTIVATED_HINTS = ("已停用", "has been deactivated", "は停止されています")
|
|
||||||
INTERNAL_ERROR_HINTS = ("サーバー内部エラー", "Internal Server Error", "服务器内部错误")
|
|
||||||
|
|
||||||
|
|
||||||
def is_deactivated_rejected(r):
|
|
||||||
"""停用后的重注册应被明确拒绝:HTTP 200 + 提示「账号已停用」,且不得是 500 兜底。"""
|
|
||||||
if r.status != 200:
|
|
||||||
return False
|
|
||||||
if any(h in r.body for h in INTERNAL_ERROR_HINTS):
|
|
||||||
return False
|
|
||||||
return any(h in r.body for h in DEACTIVATED_HINTS)
|
|
||||||
|
|
||||||
|
|
||||||
def check_public_pages(base):
|
|
||||||
"""公开页只读巡检:期望全部 200(/user/votes 之类的登录页除外)。"""
|
|
||||||
paths = ["/", "/explore", "/register", "/login", "/forgot-password",
|
|
||||||
"/sponsor", "/change-logs", "/about-me", "/robots.txt",
|
|
||||||
"/sitemap.xml", "/llms.txt", "/llms-full.txt", "/favicon.ico",
|
|
||||||
"/deactivated", "/api/captcha/generate", "/api/v1/site-stats"]
|
|
||||||
c = Client(base)
|
|
||||||
bad = []
|
|
||||||
for p in paths:
|
|
||||||
r = c.get(p, timeout=40)
|
|
||||||
if r.status != 200:
|
|
||||||
bad.append("%s=%s" % (p, r.status))
|
|
||||||
step("公开页只读巡检(%d 个端点)" % len(paths), not bad, ", ".join(bad))
|
|
||||||
|
|
||||||
|
|
||||||
def create_vote(c, title, vtype="single", options=None, fields=None, action="publish"):
|
|
||||||
"""提交一次投票创建(POST /user/votes)。
|
|
||||||
|
|
||||||
后端对创建动作一律按 draft=true 解析(草稿可随时暂存、跳过审核),
|
|
||||||
action=publish 时再走 validateForPublish + 内容审核;因此这里两种动作都可直接测。
|
|
||||||
"""
|
|
||||||
page = c.get("/user/votes/new")
|
|
||||||
data = {
|
|
||||||
"title": title, "description": "", "visibility": "public",
|
|
||||||
"voter_scope": "anyone", "show_result": "after_vote",
|
|
||||||
"type": vtype, "max_choices": "1", "allow_custom": "",
|
|
||||||
"start_at": "", "end_at": "", "quota_limit": "0",
|
|
||||||
"options": json.dumps(options or [], ensure_ascii=False),
|
|
||||||
"form_fields": json.dumps(fields or [], ensure_ascii=False),
|
|
||||||
"action": action, "_csrf": csrf_of(page.body),
|
|
||||||
}
|
|
||||||
return c.post("/user/votes", data)
|
|
||||||
|
|
||||||
|
|
||||||
def check_user_assets(owner):
|
|
||||||
"""登录态的接口/导出,以及停用页的「身份占位」说明。
|
|
||||||
|
|
||||||
停用页说明用**结构断言**(.uk-text-muted 段落非空),不写死文案 —— 站点语言随会话变化。
|
|
||||||
"""
|
|
||||||
r = owner.get("/api/v1/user")
|
|
||||||
step("登录态 /api/v1/user 返回 JSON", r.status == 200 and r.body.lstrip().startswith("{"),
|
|
||||||
"HTTP %s" % r.status)
|
|
||||||
step("登录态 /explore", owner.get("/explore").status == 200)
|
|
||||||
|
|
||||||
csrf = csrf_of(owner.get("/user/profile").body)
|
|
||||||
r = owner.post("/user/profile/export-json", {"_csrf": csrf})
|
|
||||||
step("导出 JSON", r.status == 200 and r.body.lstrip().startswith(("{", "[")),
|
|
||||||
"HTTP %s" % r.status)
|
|
||||||
r = owner.post("/user/profile/export", {"_csrf": csrf})
|
|
||||||
step("导出 Excel", r.status == 200, "HTTP %s %s" % (r.status, r.h.get("content-type", "")))
|
|
||||||
|
|
||||||
r = owner.get("/user/profile/deactivate")
|
|
||||||
note = re.search(r'class="uk-text-small uk-text-muted">(.*?)</span>', r.body, re.S)
|
|
||||||
step("停用页含身份占位说明", r.status == 200 and bool(note) and note.group(1).strip() != "")
|
|
||||||
|
|
||||||
|
|
||||||
def check_vote_flow(owner, base):
|
|
||||||
"""投票全链路:建票发布 -> 管理列表 -> 公开页 -> 匿名提交 -> 导出计数 -> 草稿 -> 表单型。"""
|
|
||||||
ts = time.strftime("%m%d%H%M%S")
|
|
||||||
|
|
||||||
title = "巡检单选投票%s" % ts
|
|
||||||
r = create_vote(owner, title, "single", options=["选项甲", "选项乙", "选项丙"])
|
|
||||||
step("创建单选投票并发布", r.status == 302 and "questions" in r.loc,
|
|
||||||
"HTTP %s -> %s %s" % (r.status, r.loc, r.alert()))
|
|
||||||
|
|
||||||
q = owner.get("/user/questions")
|
|
||||||
step("投票出现在管理列表", title in q.body)
|
|
||||||
mt = re.search(r"copyShareURL\('([^']+)'", q.body)
|
|
||||||
mv = re.search(r"/user/votes/(\d+)/edit", q.body)
|
|
||||||
token, vid = (mt.group(1) if mt else None), (mv.group(1) if mv else None)
|
|
||||||
step("取得分享 token / 投票 ID", bool(token and vid), "%s / %s" % (token, vid))
|
|
||||||
if not (token and vid):
|
|
||||||
return
|
|
||||||
|
|
||||||
anon = Client(base)
|
|
||||||
pv = anon.get("/vote/%s" % token)
|
|
||||||
step("公开投票页渲染", pv.status == 200 and title in pv.body, "HTTP %s" % pv.status)
|
|
||||||
mo = re.search(r'name="option_id" value="(\d+)"', pv.body)
|
|
||||||
step("公开页含可选项", bool(mo))
|
|
||||||
if not mo:
|
|
||||||
return
|
|
||||||
|
|
||||||
sub = anon.post("/vote/%s/submit" % token, {
|
|
||||||
"option_id": mo.group(1), "g-recaptcha-response": captcha(anon),
|
|
||||||
"custom_option": "", "_csrf": csrf_of(pv.body),
|
|
||||||
})
|
|
||||||
step("匿名提交投票", sub.status == 302, "HTTP %s -> %s %s" % (sub.status, sub.loc, sub.alert()))
|
|
||||||
|
|
||||||
# 导出仅作只读校验:单选导出的 CSV 每行是 <选项,票数>,末行为 <总参与,N>。
|
|
||||||
# 只认「总参与」这一行,避免把各选项票数一起求和导致 2 倍计数。
|
|
||||||
exp = owner.get("/user/votes/%s/export" % vid)
|
|
||||||
mtot = re.search(r"总参与,(\d+)", exp.body)
|
|
||||||
total = int(mtot.group(1)) if mtot else -1
|
|
||||||
step("导出显示总参与 = 1", exp.status == 200 and total == 1,
|
|
||||||
"HTTP %s 总参与=%s" % (exp.status, total))
|
|
||||||
|
|
||||||
dr = create_vote(owner, "巡检草稿%s" % ts, "single", options=["甲", "乙"], action="save")
|
|
||||||
step("保存草稿跳编辑页(?saved=1)",
|
|
||||||
dr.status == 302 and "/user/votes/" in dr.loc and "saved=1" in dr.loc,
|
|
||||||
"HTTP %s -> %s" % (dr.status, dr.loc))
|
|
||||||
|
|
||||||
fr = create_vote(owner, "巡检表单投票%s" % ts, "form", fields=[
|
|
||||||
{"field_type": "text", "label": "昵称", "required": True, "max_length": 50},
|
|
||||||
{"field_type": "textarea", "label": "建议", "required": False, "max_length": 200},
|
|
||||||
])
|
|
||||||
step("创建表单型投票并发布", fr.status == 302 and "questions" in fr.loc,
|
|
||||||
"HTTP %s -> %s %s" % (fr.status, fr.loc, fr.alert()))
|
|
||||||
|
|
||||||
|
|
||||||
def check_deactivated_i18n(base):
|
|
||||||
"""停用成功页四语言渲染(用语言无关的 ratio:3 标记 + 标题非空)。"""
|
|
||||||
bad = []
|
|
||||||
for lang in ["zh-CN", "zh-TW", "en", "ja"]:
|
|
||||||
c = Client(base)
|
|
||||||
c.get("/i18n/%s?to=/deactivated" % lang, timeout=40) # 302,语言写入 session
|
|
||||||
r = c.get("/deactivated", timeout=40) # 再请求一次才渲染新语言
|
|
||||||
title = re.search(r'uk-card-body uk-text-center".*?<h3[^>]*>([^<]*)</h3>',
|
|
||||||
r.body, re.S)
|
|
||||||
if not title or not title.group(1).strip():
|
|
||||||
bad.append(lang)
|
|
||||||
step("停用成功页四语言渲染", not bad, "缺失: " + ",".join(bad) if bad else "")
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--base", default="https://box-dev-api.337845818.best")
|
|
||||||
ap.add_argument("--reuse-email", default="",
|
|
||||||
help="只跑一项:用该既有邮箱重新注册(用于复测停用后能否重注册)")
|
|
||||||
args = ap.parse_args()
|
|
||||||
base = args.base.rstrip("/")
|
|
||||||
pwd = "TbCheck!2026"
|
|
||||||
|
|
||||||
if args.reuse_email:
|
|
||||||
domain = args.reuse_email.split("@")[0]
|
|
||||||
c = Client(base)
|
|
||||||
r = c.get("/register")
|
|
||||||
r = c.post("/register", {
|
|
||||||
"email": args.reuse_email, "domain": domain, "name": "TBCheck",
|
|
||||||
"password": pwd, "repeat_password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(c), "language": "zh-CN", "agree_rules": "1",
|
|
||||||
"_csrf": csrf_of(r.body),
|
|
||||||
})
|
|
||||||
step("以既有邮箱重新注册(已停用则应被明确拒绝并提示找站长)", is_deactivated_rejected(r),
|
|
||||||
"HTTP %s %s" % (r.status, r.alert()))
|
|
||||||
return 0 if all(x[1] for x in RESULTS) else 1
|
|
||||||
|
|
||||||
ts = time.strftime("%m%d%H%M%S")
|
|
||||||
email, domain = "tbe2e%s@example.com" % ts, "tbe2e%s" % ts
|
|
||||||
print("== 新账号:%s / %s ==" % (email, domain))
|
|
||||||
|
|
||||||
pub = Client(base)
|
|
||||||
r = pub.get("/register")
|
|
||||||
reg = pub.post("/register", {
|
|
||||||
"email": email, "domain": domain, "name": "TB%s" % ts,
|
|
||||||
"password": pwd, "repeat_password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(pub), "language": "zh-CN", "agree_rules": "1",
|
|
||||||
"_csrf": csrf_of(r.body),
|
|
||||||
})
|
|
||||||
step("注册", reg.status == 302, "HTTP %s -> %s %s" % (reg.status, reg.loc, reg.alert()))
|
|
||||||
|
|
||||||
owner = Client(base)
|
|
||||||
r = owner.get("/login")
|
|
||||||
r = owner.post("/login", {
|
|
||||||
"email": email, "password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(owner), "_csrf": csrf_of(r.body),
|
|
||||||
})
|
|
||||||
step("登录", r.status == 302, "HTTP %s -> %s" % (r.status, r.loc))
|
|
||||||
|
|
||||||
step("登录态 /user/profile", owner.get("/user/profile").status == 200)
|
|
||||||
step("登录态 /user/questions", owner.get("/user/questions").status == 200)
|
|
||||||
r = owner.get("/user/votes")
|
|
||||||
step("/user/votes 302 -> /user/questions#votes",
|
|
||||||
r.status == 302 and "questions" in r.loc, "HTTP %s -> %s" % (r.status, r.loc))
|
|
||||||
step("新建投票页", owner.get("/user/votes/new").status == 200)
|
|
||||||
check_user_assets(owner)
|
|
||||||
check_vote_flow(owner, base)
|
|
||||||
|
|
||||||
# 匿名提问:无邮箱时成功后直接 200 渲染提问成功页(不重定向),故只断言提问详情可达
|
|
||||||
r = pub.get("/_/%s" % domain)
|
|
||||||
step("打开提问箱", r.status == 200)
|
|
||||||
r = pub.post("/_/%s" % domain, {
|
|
||||||
"content": "自动化巡检测试提问(可忽略)",
|
|
||||||
"g-recaptcha-response": captcha(pub),
|
|
||||||
"receive_reply_email": "", "private": "", "ask_password": "",
|
|
||||||
"start_chat_val": "", "_csrf": csrf_of(r.body),
|
|
||||||
})
|
|
||||||
step("提交提问", r.status in (200, 302), "HTTP %s -> %s" % (r.status, r.loc))
|
|
||||||
|
|
||||||
r = owner.get("/user/profile/deactivate")
|
|
||||||
step("打开停用页", r.status == 200)
|
|
||||||
r = owner.post("/user/profile/deactivate", {"_csrf": csrf_of(r.body)})
|
|
||||||
step("停用后跳 /deactivated", r.status == 302 and "deactivated" in r.loc,
|
|
||||||
"HTTP %s -> %s" % (r.status, r.loc))
|
|
||||||
r = owner.get("/deactivated")
|
|
||||||
step("停用成功页渲染", r.status == 200 and "ratio: 3" in r.body, "HTTP %s" % r.status)
|
|
||||||
|
|
||||||
pub2 = Client(base)
|
|
||||||
r = pub2.get("/register")
|
|
||||||
r = pub2.post("/register", {
|
|
||||||
"email": email, "domain": domain, "name": "TB%sb" % ts,
|
|
||||||
"password": pwd, "repeat_password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(pub2), "language": "zh-CN", "agree_rules": "1",
|
|
||||||
"_csrf": csrf_of(r.body),
|
|
||||||
})
|
|
||||||
step("停用后同邮箱重注册被明确拒绝并提示找站长(核心回归)", is_deactivated_rejected(r),
|
|
||||||
"HTTP %s %s" % (r.status, r.alert()))
|
|
||||||
|
|
||||||
check_public_pages(base)
|
|
||||||
check_deactivated_i18n(base)
|
|
||||||
|
|
||||||
print("\n===== 汇总 =====")
|
|
||||||
for d, ok, ex in RESULTS:
|
|
||||||
print("%s %s%s" % ("PASS" if ok else "FAIL", d, (" (%s)" % ex) if ex else ""))
|
|
||||||
n = sum(1 for x in RESULTS if not x[1])
|
|
||||||
print("\n共 %d 项,失败 %d 项" % (len(RESULTS), n))
|
|
||||||
return 1 if n else 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
try:
|
|
||||||
sys.exit(main())
|
|
||||||
except KeyboardInterrupt:
|
|
||||||
raise
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
# 远端抖动等意外中断时,仍把已完成项的结论打出来,避免只剩一段 traceback
|
|
||||||
print("\n巡检意外中断:%s: %s" % (type(e).__name__, e))
|
|
||||||
for d, ok, ex in RESULTS:
|
|
||||||
print("%s %s%s" % ("PASS" if ok else "FAIL", d, (" (%s)" % ex) if ex else ""))
|
|
||||||
sys.exit(2)
|
|
||||||
@@ -1,370 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""图片上传链路端到端验证(批次 A / B1 / B2)。
|
|
||||||
|
|
||||||
针对已部署环境验证三件事:
|
|
||||||
B1 EXIF Orientation=6 的竖拍 JPEG 转换后必须变成竖图(核心验收点)
|
|
||||||
B2 动画 GIF 必须被转成 .webp(cwebp 可用性 + 提问路径与头像路径一致性)
|
|
||||||
A 转换后不比原图小时必须回退原图(用一张极小线框图验证)
|
|
||||||
|
|
||||||
做法:注册账号 -> 匿名提问(multipart 同时带三张图)-> 解析成功页里的 CDN URL
|
|
||||||
-> 逐张拉回字节,解析真实容器格式与尺寸。
|
|
||||||
|
|
||||||
用法:
|
|
||||||
python dev/devcheck-image.py [--base https://box-dev-api.337845818.best]
|
|
||||||
|
|
||||||
依赖:Pillow(仅用于在本地生成测试图片,不参与被测逻辑)。
|
|
||||||
注意:会在目标环境真实创建账号并发送带图提问(测试邮箱前缀 tbimg<ts>@)。
|
|
||||||
清理:TamaBox ban --email <该邮箱> --yes
|
|
||||||
"""
|
|
||||||
import argparse
|
|
||||||
import http.cookiejar
|
|
||||||
import io
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import tempfile
|
|
||||||
import re
|
|
||||||
import struct
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
import uuid
|
|
||||||
|
|
||||||
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
|
||||||
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
|
|
||||||
RESULTS = []
|
|
||||||
CREATED_EMAIL = ""
|
|
||||||
|
|
||||||
|
|
||||||
class NoRedirect(urllib.request.HTTPRedirectHandler):
|
|
||||||
def redirect_request(self, req, fp, code, msg, headers, newurl): # noqa: D102
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
class Resp:
|
|
||||||
def __init__(self, status, headers, body):
|
|
||||||
self.status = status
|
|
||||||
self.h = {k.lower(): v for k, v in headers.items()}
|
|
||||||
self.body = body
|
|
||||||
|
|
||||||
@property
|
|
||||||
def loc(self):
|
|
||||||
return self.h.get("location", "")
|
|
||||||
|
|
||||||
@property
|
|
||||||
def text(self):
|
|
||||||
return self.body.decode("utf-8", "replace")
|
|
||||||
|
|
||||||
|
|
||||||
class Client:
|
|
||||||
def __init__(self, base):
|
|
||||||
self.base = base
|
|
||||||
self.cj = http.cookiejar.CookieJar()
|
|
||||||
self.op = urllib.request.build_opener(
|
|
||||||
urllib.request.HTTPCookieProcessor(self.cj), NoRedirect())
|
|
||||||
|
|
||||||
def _send(self, req, timeout=120):
|
|
||||||
try:
|
|
||||||
with self.op.open(req, timeout=timeout) as r:
|
|
||||||
return Resp(r.status, dict(r.headers), r.read())
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
try:
|
|
||||||
body = e.read()
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
body = b""
|
|
||||||
return Resp(e.code, dict(e.headers), body)
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
return Resp(-1, {}, ("TRANSPORT_ERROR: %s" % e).encode())
|
|
||||||
|
|
||||||
def req(self, method, path, data=None, content_type=None, timeout=120):
|
|
||||||
url = path if path.startswith("http") else self.base + path
|
|
||||||
headers = {"User-Agent": UA, "Referer": self.base + path.split("?")[0]}
|
|
||||||
body = None
|
|
||||||
if data is not None:
|
|
||||||
body = data if isinstance(data, bytes) else urllib.parse.urlencode(data).encode()
|
|
||||||
headers["Content-Type"] = content_type or "application/x-www-form-urlencoded"
|
|
||||||
return self._send(urllib.request.Request(url, data=body, headers=headers, method=method),
|
|
||||||
timeout)
|
|
||||||
|
|
||||||
def get(self, p, **kw):
|
|
||||||
r = self.req("GET", p, **kw)
|
|
||||||
return r if r.status != -1 else self.req("GET", p, **kw)
|
|
||||||
|
|
||||||
def post(self, p, data=None, **kw):
|
|
||||||
return self.req("POST", p, data=data, **kw)
|
|
||||||
|
|
||||||
def get_binary(self, url):
|
|
||||||
return self._send(urllib.request.Request(
|
|
||||||
url, headers={"User-Agent": UA}, method="GET"), timeout=60)
|
|
||||||
|
|
||||||
|
|
||||||
def step(desc, ok, extra=""):
|
|
||||||
RESULTS.append((desc, ok, extra))
|
|
||||||
print(" [%s] %s%s" % ("PASS" if ok else "FAIL", desc, (" (%s)" % extra) if extra else ""))
|
|
||||||
return ok
|
|
||||||
|
|
||||||
|
|
||||||
def csrf_of(html):
|
|
||||||
m = re.search(r'name="_csrf"\s+value="([^"]+)"', html)
|
|
||||||
return m.group(1) if m else None
|
|
||||||
|
|
||||||
|
|
||||||
def captcha(c):
|
|
||||||
j = json.loads(c.get("/api/captcha/generate").text)
|
|
||||||
pid, q = j["data"]["id"], j["data"]["question"]
|
|
||||||
m = re.match(r"\s*(\d+)\s*([+\-×])\s*(\d+)\s*=", q)
|
|
||||||
a, op, b = int(m.group(1)), m.group(2), int(m.group(3))
|
|
||||||
return "%s:%d" % (pid, {"+": a + b, "-": a - b, "×": a * b}[op])
|
|
||||||
|
|
||||||
|
|
||||||
def multipart(fields, files):
|
|
||||||
boundary = "----tbimg%s" % uuid.uuid4().hex
|
|
||||||
buf = io.BytesIO()
|
|
||||||
for k, v in fields.items():
|
|
||||||
buf.write(("--%s\r\n" % boundary).encode())
|
|
||||||
buf.write(('Content-Disposition: form-data; name="%s"\r\n\r\n' % k).encode())
|
|
||||||
buf.write(("%s\r\n" % v).encode())
|
|
||||||
for name, fn, ct, data in files:
|
|
||||||
buf.write(("--%s\r\n" % boundary).encode())
|
|
||||||
buf.write(('Content-Disposition: form-data; name="%s"; filename="%s"\r\n' % (name, fn)).encode())
|
|
||||||
buf.write(("Content-Type: %s\r\n\r\n" % ct).encode())
|
|
||||||
buf.write(data)
|
|
||||||
buf.write(b"\r\n")
|
|
||||||
buf.write(("--%s--\r\n" % boundary).encode())
|
|
||||||
return buf.getvalue(), "multipart/form-data; boundary=%s" % boundary
|
|
||||||
|
|
||||||
|
|
||||||
# ---------- 容器格式解析(纯 stdlib) ----------
|
|
||||||
|
|
||||||
def sniff(data):
|
|
||||||
"""返回 (kind, (w,h))。kind in {webp, jpeg, gif, png, unknown}"""
|
|
||||||
if len(data) >= 12 and data[:4] == b"RIFF" and data[8:12] == b"WEBP":
|
|
||||||
return "webp", webp_dims(data)
|
|
||||||
if data[:3] == b"\xff\xd8\xff":
|
|
||||||
return "jpeg", jpeg_dims(data)
|
|
||||||
if data[:3] == b"GIF":
|
|
||||||
return "gif", gif_dims(data)
|
|
||||||
if data[:8] == b"\x89PNG\r\n\x1a\n":
|
|
||||||
return "png", png_dims(data)
|
|
||||||
return "unknown", None
|
|
||||||
|
|
||||||
|
|
||||||
def webp_dims(d):
|
|
||||||
c = d[12:16]
|
|
||||||
if c == b"VP8 " and len(d) >= 30:
|
|
||||||
return (struct.unpack("<H", d[26:28])[0] & 0x3FFF,
|
|
||||||
struct.unpack("<H", d[28:30])[0] & 0x3FFF)
|
|
||||||
if c == b"VP8L" and len(d) >= 25:
|
|
||||||
b = struct.unpack("<I", d[21:25])[0]
|
|
||||||
return (b & 0x3FFF) + 1, ((b >> 14) & 0x3FFF) + 1
|
|
||||||
if c == b"VP8X" and len(d) >= 30:
|
|
||||||
return ((d[24] | d[25] << 8 | d[26] << 16) + 1,
|
|
||||||
(d[27] | d[28] << 8 | d[29] << 16) + 1)
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def jpeg_dims(d):
|
|
||||||
i = 2
|
|
||||||
while i + 9 < len(d):
|
|
||||||
if d[i] != 0xFF:
|
|
||||||
i += 1
|
|
||||||
continue
|
|
||||||
m = d[i + 1]
|
|
||||||
if m in (0xC0, 0xC1, 0xC2, 0xC3):
|
|
||||||
h, w = struct.unpack(">HH", d[i + 5:i + 9])
|
|
||||||
return (w, h)
|
|
||||||
if m in (0xD8, 0xD9) or 0xD0 <= m <= 0xD7:
|
|
||||||
i += 2
|
|
||||||
continue
|
|
||||||
i += 2 + struct.unpack(">H", d[i + 2:i + 4])[0]
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def gif_dims(d):
|
|
||||||
return struct.unpack("<HH", d[6:10]) if len(d) >= 10 else None
|
|
||||||
|
|
||||||
|
|
||||||
def png_dims(d):
|
|
||||||
return struct.unpack(">II", d[16:24]) if len(d) >= 24 else None
|
|
||||||
|
|
||||||
|
|
||||||
# ---------- 测试图片生成(Pillow 仅此一处使用) ----------
|
|
||||||
|
|
||||||
def make_images():
|
|
||||||
from PIL import Image, ImageDraw
|
|
||||||
import io as _io
|
|
||||||
|
|
||||||
# B1:EXIF Orientation=6。像素存 240x120,正确归一后应显示 120x240。
|
|
||||||
base = Image.new("RGB", (240, 120))
|
|
||||||
dr = ImageDraw.Draw(base)
|
|
||||||
for i in range(0, 240, 20):
|
|
||||||
dr.line([(i, 0), (i, 120)], fill=(200, 30, 30), width=3)
|
|
||||||
dr.rectangle([0, 0, 239, 119], outline=(0, 0, 255), width=4)
|
|
||||||
exif = base.getexif()
|
|
||||||
exif[0x0112] = 6
|
|
||||||
buf = _io.BytesIO()
|
|
||||||
base.save(buf, "JPEG", quality=88, exif=exif)
|
|
||||||
portrait = buf.getvalue()
|
|
||||||
|
|
||||||
# B2:6 帧动画 GIF。加大到 640x480,确保转换后仍小于原图(否则会走回退)。
|
|
||||||
frames = []
|
|
||||||
pal_frames = []
|
|
||||||
for f in range(6):
|
|
||||||
im = Image.new("RGB", (640, 480), (240, 240, 240))
|
|
||||||
d = ImageDraw.Draw(im)
|
|
||||||
d.ellipse([40 + f * 60, 120, 200 + f * 60, 280], fill=(220, 40, 40))
|
|
||||||
d.rectangle([300, 60 + f * 20, 600, 200], fill=(30, 90, 220))
|
|
||||||
pal_frames.append(im.convert("P", palette=Image.ADAPTIVE, colors=64))
|
|
||||||
gbuf = _io.BytesIO()
|
|
||||||
pal_frames[0].save(gbuf, "GIF", save_all=True, append_images=pal_frames[1:],
|
|
||||||
duration=120, loop=0)
|
|
||||||
animated = gbuf.getvalue()
|
|
||||||
|
|
||||||
# A:极小线框图(200x200),WebP 有损编码后反而比 PNG 大 -> 应回退原图。
|
|
||||||
wire = Image.new("RGB", (200, 200), (255, 255, 255))
|
|
||||||
d = ImageDraw.Draw(wire)
|
|
||||||
for i in range(0, 200, 20):
|
|
||||||
d.line([(i, 0), (i, 199)], fill=(0, 0, 0))
|
|
||||||
d.line([(0, i), (199, i)], fill=(0, 0, 0))
|
|
||||||
wbuf = _io.BytesIO()
|
|
||||||
wire.save(wbuf, "PNG")
|
|
||||||
wireframe = wbuf.getvalue()
|
|
||||||
|
|
||||||
return {"portrait": portrait, "anim": animated, "wire": wireframe}
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
global CREATED_EMAIL
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--base", default="https://box-dev-api.337845818.best")
|
|
||||||
args = ap.parse_args()
|
|
||||||
base = args.base.rstrip("/")
|
|
||||||
pwd = "TbImg!2026"
|
|
||||||
ts = time.strftime("%m%d%H%M%S")
|
|
||||||
email, domain = "tbimg%s@example.com" % ts, "tbimg%s" % ts
|
|
||||||
CREATED_EMAIL = email
|
|
||||||
print("== 账号:%s / %s ==" % (email, domain))
|
|
||||||
|
|
||||||
imgs = make_images()
|
|
||||||
for k, v in imgs.items():
|
|
||||||
print(" 本地样本 %-9s %7d bytes" % (k, len(v)))
|
|
||||||
|
|
||||||
c = Client(base)
|
|
||||||
r = c.get("/register")
|
|
||||||
reg = c.post("/register", {
|
|
||||||
"email": email, "domain": domain, "name": "TBImg%s" % ts,
|
|
||||||
"password": pwd, "repeat_password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(c), "language": "zh-CN",
|
|
||||||
"_csrf": csrf_of(r.text),
|
|
||||||
})
|
|
||||||
if not step("注册", reg.status == 302, "HTTP %s -> %s" % (reg.status, reg.loc)):
|
|
||||||
return 1
|
|
||||||
|
|
||||||
c = Client(base)
|
|
||||||
r = c.get("/login")
|
|
||||||
lg = c.post("/login", {"email": email, "password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(c), "_csrf": csrf_of(r.text)})
|
|
||||||
if not step("登录", lg.status == 302, "HTTP %s -> %s" % (lg.status, lg.loc)):
|
|
||||||
return 1
|
|
||||||
|
|
||||||
page = c.get("/_/%s" % domain)
|
|
||||||
if not step("打开提问箱", page.status == 200, "HTTP %s" % page.status):
|
|
||||||
return 1
|
|
||||||
|
|
||||||
# route/question/page.go 只取 f.Images[0](单张上限),
|
|
||||||
# 故每张样本各发一条提问,逐张验证。
|
|
||||||
fetched = []
|
|
||||||
for key, (fname, ctype) in (("portrait", ("portrait.jpg", "image/jpeg")),
|
|
||||||
("anim", ("anim.gif", "image/gif")),
|
|
||||||
("wire", ("wire.png", "image/png"))):
|
|
||||||
box = c.get("/_/%s" % domain)
|
|
||||||
body, ct = multipart({
|
|
||||||
"content": "图片上传链路验证 %s(可忽略)" % key,
|
|
||||||
"g-recaptcha-response": captcha(c),
|
|
||||||
"receive_reply_email": "", "private": "", "ask_password": "",
|
|
||||||
"start_chat_val": "", "_csrf": csrf_of(box.text),
|
|
||||||
}, [("images", fname, ctype, imgs[key])])
|
|
||||||
|
|
||||||
res = c.post("/_/%s" % domain, data=body, content_type=ct)
|
|
||||||
if not step("提交带图提问 [%s]" % key, res.status in (200, 302),
|
|
||||||
"HTTP %s -> %s" % (res.status, res.loc)):
|
|
||||||
continue
|
|
||||||
|
|
||||||
# 成功页不渲染图片(templates/question/success.html 无图片区),
|
|
||||||
# 故从成功页取出 question ID,再访问详情页取图片 key。
|
|
||||||
qid = re.search(r'/_/%s/(\d+)' % re.escape(domain), res.text)
|
|
||||||
if not qid:
|
|
||||||
step(" └ 取提问 ID [%s]" % key, False, "成功页未给出 /_/<domain>/<id>")
|
|
||||||
continue
|
|
||||||
detail = c.get("/_/%s/%s" % (domain, qid.group(1)))
|
|
||||||
# 提问图的 key 不带 picture/ 前缀(是 "<年>/<月>/<纳秒>.<ext>"),
|
|
||||||
# 模板拼成 https://<CDN>/<key>,故按 CDN host + 数字开头路径来匹配。
|
|
||||||
urls = sorted(set(re.findall(
|
|
||||||
r'https://r2\.[^"\'\s<>]+/\d{4}/\d{1,2}/\d+\.\w+', detail.text)))
|
|
||||||
if not urls:
|
|
||||||
step(" └ 详情页回显图片 [%s]" % key, False,
|
|
||||||
"HTTP %s,未找到 picture URL" % detail.status)
|
|
||||||
continue
|
|
||||||
step(" └ 详情页回显图片 [%s]" % key, True, "%d 个" % len(urls))
|
|
||||||
|
|
||||||
for u in urls:
|
|
||||||
st, data = c.get_binary(u)
|
|
||||||
if st != 200 or not data:
|
|
||||||
step(" └ 拉回图片 [%s]" % key, False, "HTTP %s" % st)
|
|
||||||
continue
|
|
||||||
kind, dims = sniff(data)
|
|
||||||
fetched.append((key, kind, dims, len(data)))
|
|
||||||
print(" %-14s %-7s %-12s %7d bytes" % (u.rsplit("/", 1)[-1], kind, dims, len(data)))
|
|
||||||
|
|
||||||
|
|
||||||
print("\n== 断言 ==")
|
|
||||||
|
|
||||||
# B1
|
|
||||||
portrait = [f for f in fetched if f[0] == "portrait" and f[1] in ("webp", "jpeg")
|
|
||||||
and f[2] == (120, 240)]
|
|
||||||
land = [f for f in fetched if f[0] == "portrait" and f[1] in ("webp", "jpeg")
|
|
||||||
and f[2] == (240, 120)]
|
|
||||||
if portrait:
|
|
||||||
step("B1 EXIF Orientation=6 已归一(240x120 -> 120x240)", True,
|
|
||||||
"%s, %d bytes" % (portrait[0][1], portrait[0][3]))
|
|
||||||
elif land:
|
|
||||||
step("B1 EXIF Orientation=6 已归一", False, "仍是 240x120,说明未应用 EXIF 旋转")
|
|
||||||
else:
|
|
||||||
step("B1 EXIF 方向归一", False, "未匹配到 240x120 / 120x240 的图片,实际: %s" %
|
|
||||||
[(f[1], f[2]) for f in fetched])
|
|
||||||
|
|
||||||
# B2:GIF 样本必须变成 webp;关键是按 key 归属判定,而不是看整体有没有 webp
|
|
||||||
gif_res = [f for f in fetched if f[0] == "anim"]
|
|
||||||
if not gif_res:
|
|
||||||
step("B2 动画 GIF 转 WebP", False, "未取到 anim 样本结果")
|
|
||||||
elif gif_res[0][1] == "webp":
|
|
||||||
step("B2 动画 GIF 已转为 WebP", True,
|
|
||||||
"%dx%d, %d bytes(原 GIF %d bytes,-%.0f%%)" % (
|
|
||||||
gif_res[0][2][0], gif_res[0][2][1], gif_res[0][3], len(imgs["anim"]),
|
|
||||||
(1 - gif_res[0][3] / len(imgs["anim"])) * 100))
|
|
||||||
elif gif_res[0][1] == "gif":
|
|
||||||
step("B2 动画 GIF 已转为 WebP", False,
|
|
||||||
"仍是原始 GIF %d bytes -> cwebp 不可用或转换失败" % gif_res[0][3])
|
|
||||||
else:
|
|
||||||
step("B2 动画 GIF 转 WebP", False, "得到 %s(预期 webp)" % gif_res[0][1])
|
|
||||||
|
|
||||||
# A:线框图应回退原图(PNG),而非转成更大的 WebP
|
|
||||||
wire = [f for f in fetched if f[0] == "wire"]
|
|
||||||
if not wire:
|
|
||||||
step("A 线框图兜底回退", False, "未取到 wire 样本结果")
|
|
||||||
elif wire[0][1] == "png":
|
|
||||||
step("A 转换无收益时回退原图(线框图保持 PNG)", True,
|
|
||||||
"%d bytes(原 PNG %d bytes)" % (wire[0][3], len(imgs["wire"])))
|
|
||||||
else:
|
|
||||||
step("A 线框图兜底回退", False,
|
|
||||||
"被转成了 %s(%d bytes)——本应回退原图" % (wire[0][1], wire[0][3]))
|
|
||||||
|
|
||||||
print("\n== 结果:%d/%d 通过 ==" % (sum(1 for _, ok, _ in RESULTS if ok), len(RESULTS)))
|
|
||||||
print("清理:TamaBox ban --email %s --yes" % email)
|
|
||||||
return 0 if all(ok for _, ok, _ in RESULTS) else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,419 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""上传链路端到端验证(A 分档 / B1 EXIF 归一 + EXIF 比例校验 / B3 单次解码)。
|
|
||||||
|
|
||||||
为什么主用例走「背景」而不是头像
|
|
||||||
--------------------------------
|
|
||||||
头像要求 1:1,是**对称**比例:EXIF 归一(5/6/7/8 交换宽高)后比例不变。
|
|
||||||
所以无论修没修比例校验的 EXIF 缺口,头像路径的表现完全一样 —— **测不出缺口**。
|
|
||||||
背景要求 16:9,非对称,且用户真实场景就是「手机竖拍一张照当背景」:
|
|
||||||
像素 1080x1920(竖,比例 0.5625)+ EXIF Orientation=6(顺时针 90)
|
|
||||||
-> 用户肉眼看到的是 1920x1080 横图,比例 1.7778,符合 16:9
|
|
||||||
-> 修复前:只看原始像素 0.5625 vs 1.7778,偏差 1.215 > 0.01 → **被误拒**
|
|
||||||
-> 修复后:先交换宽高得 1920x1080,偏差 0 → 放行
|
|
||||||
|
|
||||||
头像用例保留,但只用来回归 A 分档(不再承担 EXIF 比例校验的验证职责)。
|
|
||||||
|
|
||||||
用法:
|
|
||||||
python dev/devcheck-upload.py [--base https://box-dev-api.337845818.best]
|
|
||||||
依赖:Pillow(仅用于本地生成测试图片)
|
|
||||||
清理:TamaBox ban --email <输出中的测试邮箱> --yes
|
|
||||||
"""
|
|
||||||
import argparse
|
|
||||||
import http.cookiejar
|
|
||||||
import io
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import struct
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
import uuid
|
|
||||||
|
|
||||||
from PIL import Image, ImageOps
|
|
||||||
|
|
||||||
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
|
||||||
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
|
|
||||||
RESULTS = []
|
|
||||||
|
|
||||||
|
|
||||||
class NoRedirect(urllib.request.HTTPRedirectHandler):
|
|
||||||
def redirect_request(self, req, fp, code, msg, headers, newurl): # noqa: D102
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
class Resp:
|
|
||||||
def __init__(self, status, headers, body):
|
|
||||||
self.status = status
|
|
||||||
self.h = {k.lower(): v for k, v in headers.items()}
|
|
||||||
self.body = body
|
|
||||||
|
|
||||||
@property
|
|
||||||
def loc(self):
|
|
||||||
return self.h.get("location", "")
|
|
||||||
|
|
||||||
@property
|
|
||||||
def text(self):
|
|
||||||
return self.body.decode("utf-8", "replace")
|
|
||||||
|
|
||||||
|
|
||||||
class Client:
|
|
||||||
def __init__(self, base):
|
|
||||||
self.base = base
|
|
||||||
self.cj = http.cookiejar.CookieJar()
|
|
||||||
self.op = urllib.request.build_opener(
|
|
||||||
urllib.request.HTTPCookieProcessor(self.cj), NoRedirect())
|
|
||||||
|
|
||||||
def _send(self, req, timeout=180):
|
|
||||||
try:
|
|
||||||
with self.op.open(req, timeout=timeout) as r:
|
|
||||||
return Resp(r.status, dict(r.headers), r.read())
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
try:
|
|
||||||
body = e.read()
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
body = b""
|
|
||||||
return Resp(e.code, dict(e.headers), body)
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
return Resp(-1, {}, ("TRANSPORT_ERROR: %s" % e).encode())
|
|
||||||
|
|
||||||
def req(self, method, path, data=None, content_type=None, timeout=180):
|
|
||||||
url = path if path.startswith("http") else self.base + path
|
|
||||||
headers = {"User-Agent": UA, "Referer": self.base + path.split("?")[0]}
|
|
||||||
body = None
|
|
||||||
if data is not None:
|
|
||||||
body = data if isinstance(data, bytes) else urllib.parse.urlencode(data).encode()
|
|
||||||
headers["Content-Type"] = content_type or "application/x-www-form-urlencoded"
|
|
||||||
return self._send(urllib.request.Request(url, data=body, headers=headers, method=method),
|
|
||||||
timeout)
|
|
||||||
|
|
||||||
def get(self, p, **kw):
|
|
||||||
r = self.req("GET", p, **kw)
|
|
||||||
return r if r.status != -1 else self.req("GET", p, **kw)
|
|
||||||
|
|
||||||
def post(self, p, data=None, **kw):
|
|
||||||
return self.req("POST", p, data=data, **kw)
|
|
||||||
|
|
||||||
def get_binary(self, url):
|
|
||||||
return self._send(urllib.request.Request(
|
|
||||||
url, headers={"User-Agent": UA}, method="GET"), timeout=60)
|
|
||||||
|
|
||||||
|
|
||||||
def step(desc, ok, extra=""):
|
|
||||||
RESULTS.append((desc, ok, extra))
|
|
||||||
print(" [%s] %s%s" % ("PASS" if ok else "FAIL", desc, (" (%s)" % extra) if extra else ""))
|
|
||||||
return ok
|
|
||||||
|
|
||||||
|
|
||||||
def redness_bbox(im):
|
|
||||||
"""返回红色标记条的包围盒 (x0, y0, x1, y1)。
|
|
||||||
|
|
||||||
用来做「方向是否真的被旋转」的判据:尺寸对不代表方向对,必须看特征位置。
|
|
||||||
有损 WebP 会让红色略有偏差,故阈值放宽并按 4px 步长采样。
|
|
||||||
"""
|
|
||||||
W, H = im.size
|
|
||||||
px = im.load()
|
|
||||||
xs, ys = [], []
|
|
||||||
for y in range(0, H, 4):
|
|
||||||
for x in range(0, W, 4):
|
|
||||||
r, g, b = px[x, y]
|
|
||||||
if r > 150 and g < 90 and b < 90:
|
|
||||||
xs.append(x)
|
|
||||||
ys.append(y)
|
|
||||||
return (min(xs), min(ys), max(xs), max(ys)) if xs else None
|
|
||||||
|
|
||||||
|
|
||||||
def csrf_of(html):
|
|
||||||
m = re.search(r'name="_csrf"\s+value="([^"]+)"', html)
|
|
||||||
return m.group(1) if m else None
|
|
||||||
|
|
||||||
|
|
||||||
def captcha(c):
|
|
||||||
j = json.loads(c.get("/api/captcha/generate").text)
|
|
||||||
pid, q = j["data"]["id"], j["data"]["question"]
|
|
||||||
m = re.match(r"\s*(\d+)\s*([+\-×])\s*(\d+)\s*=", q)
|
|
||||||
a, op, b = int(m.group(1)), m.group(2), int(m.group(3))
|
|
||||||
return "%s:%d" % (pid, {"+": a + b, "-": a - b, "×": a * b}[op])
|
|
||||||
|
|
||||||
|
|
||||||
def multipart(fields, files):
|
|
||||||
boundary = "----tbuv%s" % uuid.uuid4().hex
|
|
||||||
buf = io.BytesIO()
|
|
||||||
for k, v in fields.items():
|
|
||||||
buf.write(("--%s\r\n" % boundary).encode())
|
|
||||||
buf.write(('Content-Disposition: form-data; name="%s"\r\n\r\n' % k).encode())
|
|
||||||
buf.write(("%s\r\n" % v).encode())
|
|
||||||
for name, fn, ct, data in files:
|
|
||||||
buf.write(("--%s\r\n" % boundary).encode())
|
|
||||||
buf.write(('Content-Disposition: form-data; name="%s"; filename="%s"\r\n' % (name, fn)).encode())
|
|
||||||
buf.write(("Content-Type: %s\r\n\r\n" % ct).encode())
|
|
||||||
buf.write(data)
|
|
||||||
buf.write(b"\r\n")
|
|
||||||
buf.write(("--%s--\r\n" % boundary).encode())
|
|
||||||
return buf.getvalue(), "multipart/form-data; boundary=%s" % boundary
|
|
||||||
|
|
||||||
|
|
||||||
def sniff(d):
|
|
||||||
if len(d) >= 12 and d[:4] == b"RIFF" and d[8:12] == b"WEBP":
|
|
||||||
return "webp", webp_dims(d)
|
|
||||||
if d[:3] == b"\xff\xd8\xff":
|
|
||||||
return "jpeg", jpeg_dims(d)
|
|
||||||
if d[:8] == b"\x89PNG\r\n\x1a\n":
|
|
||||||
return "png", (struct.unpack(">II", d[16:24]) if len(d) >= 24 else None)
|
|
||||||
return "unknown", None
|
|
||||||
|
|
||||||
|
|
||||||
def webp_dims(d):
|
|
||||||
c = d[12:16]
|
|
||||||
if c == b"VP8 " and len(d) >= 30:
|
|
||||||
return (struct.unpack("<H", d[26:28])[0] & 0x3FFF,
|
|
||||||
struct.unpack("<H", d[28:30])[0] & 0x3FFF)
|
|
||||||
if c == b"VP8L" and len(d) >= 25:
|
|
||||||
b = struct.unpack("<I", d[21:25])[0]
|
|
||||||
return (b & 0x3FFF) + 1, ((b >> 14) & 0x3FFF) + 1
|
|
||||||
if c == b"VP8X" and len(d) >= 30:
|
|
||||||
return ((d[24] | d[25] << 8 | d[26] << 16) + 1,
|
|
||||||
(d[27] | d[28] << 8 | d[29] << 16) + 1)
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def jpeg_dims(d):
|
|
||||||
i = 2
|
|
||||||
while i + 9 < len(d):
|
|
||||||
if d[i] != 0xFF:
|
|
||||||
i += 1
|
|
||||||
continue
|
|
||||||
m = d[i + 1]
|
|
||||||
if m in (0xC0, 0xC1, 0xC2, 0xC3):
|
|
||||||
h, w = struct.unpack(">HH", d[i + 5:i + 9])
|
|
||||||
return (w, h)
|
|
||||||
if m in (0xD8, 0xD9) or 0xD0 <= m <= 0xD7:
|
|
||||||
i += 2
|
|
||||||
continue
|
|
||||||
i += 2 + struct.unpack(">H", d[i + 2:i + 4])[0]
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def make_img(w, h, orientation=None, portrait=False):
|
|
||||||
"""生成带方位特征的图片。
|
|
||||||
|
|
||||||
portrait=True 时在**左侧**画一条粗红色竖条:EXIF 顺时针 90 后它会跑到顶部,
|
|
||||||
用于像素级确认方向确实被归一了(而非仅尺寸数字对)。
|
|
||||||
"""
|
|
||||||
from PIL import ImageDraw
|
|
||||||
im = Image.new("RGB", (w, h), (30, 120, 200))
|
|
||||||
d = ImageDraw.Draw(im)
|
|
||||||
if portrait:
|
|
||||||
d.rectangle([0, 0, max(2, w // 12), h], fill=(230, 30, 30))
|
|
||||||
d.text((max(4, w // 12 + 6), h // 2), "TOP-IS-LEFT", fill=(255, 255, 255))
|
|
||||||
else:
|
|
||||||
d.rectangle([0, 0, w, max(2, h // 12)], fill=(230, 30, 30))
|
|
||||||
d.text((w // 4, max(4, h // 12 + 4)), "TOP-IS-UP", fill=(255, 255, 255))
|
|
||||||
for i in range(0, w, max(8, w // 20)):
|
|
||||||
d.line([(i, h // 4), (i, h)], fill=(255, 255, 255), width=2)
|
|
||||||
buf = io.BytesIO()
|
|
||||||
if orientation:
|
|
||||||
exif = im.getexif()
|
|
||||||
exif[0x0112] = orientation
|
|
||||||
im.save(buf, "JPEG", quality=90, exif=exif)
|
|
||||||
else:
|
|
||||||
im.save(buf, "JPEG", quality=90)
|
|
||||||
return buf.getvalue()
|
|
||||||
|
|
||||||
|
|
||||||
def scrape_profile_fields(html):
|
|
||||||
"""从 profile 页抓取所有 input 的 name/value,保证回填必填项。
|
|
||||||
|
|
||||||
UpdateProfile 的必填字段有十来个(昵称/介绍/5 个二维码颜色/点大小/透明度…),
|
|
||||||
漏一个就会因 `valid:"required"` 被Bind 挡下、图片根本不会走到上传分支。
|
|
||||||
"""
|
|
||||||
fields = {}
|
|
||||||
for m in re.finditer(r'<input\b[^>]*>', html, re.I):
|
|
||||||
tag = m.group(0)
|
|
||||||
nm = re.search(r'name="([^"]+)"', tag, re.I)
|
|
||||||
if not nm:
|
|
||||||
continue
|
|
||||||
name = nm.group(1)
|
|
||||||
if name in ("avatar", "background", "backgroundimage", "_csrf", "submit"):
|
|
||||||
continue
|
|
||||||
typ = re.search(r'type="([^"]+)"', tag, re.I)
|
|
||||||
if typ and typ.group(1).lower() in ("checkbox", "radio"):
|
|
||||||
if not re.search(r'\bchecked\b', tag, re.I):
|
|
||||||
continue
|
|
||||||
val = re.search(r'value="([^"]*)"', tag, re.I)
|
|
||||||
fields[name] = val.group(1) if val else ""
|
|
||||||
for m in re.finditer(r'<select\b[^>]*name="([^"]+)"[^>]*>(.*?)</select>', html, re.I | re.S):
|
|
||||||
name, body = m.group(1), m.group(2)
|
|
||||||
opts = re.findall(r'<option\b([^>]*)>', body, re.I)
|
|
||||||
chosen = ""
|
|
||||||
for o in opts:
|
|
||||||
vm = re.search(r'value="([^"]*)"', o, re.I)
|
|
||||||
v = vm.group(1) if vm else ""
|
|
||||||
if re.search(r'\bselected\b', o, re.I):
|
|
||||||
chosen = v
|
|
||||||
break
|
|
||||||
if not chosen:
|
|
||||||
chosen = v
|
|
||||||
fields[name] = chosen
|
|
||||||
for m in re.finditer(r'<textarea\b[^>]*name="([^"]+)"[^>]*>(.*?)</textarea>', html, re.I | re.S):
|
|
||||||
fields[m.group(1)] = re.sub(r"<[^>]+>", "", m.group(2)).strip()
|
|
||||||
return fields
|
|
||||||
|
|
||||||
|
|
||||||
def upload(c, field, data, fname="img.jpg"):
|
|
||||||
"""上传图片到 profile 表单的指定字段(avatar / background)。
|
|
||||||
|
|
||||||
返回 (图片CDN URL 或 None, 错误描述或 None)。
|
|
||||||
"""
|
|
||||||
page = c.get("/user/profile")
|
|
||||||
if page.status != 200:
|
|
||||||
return None, "profile HTTP %s" % page.status
|
|
||||||
fields = scrape_profile_fields(page.text)
|
|
||||||
fields["_csrf"] = csrf_of(page.text)
|
|
||||||
body, ct = multipart(fields, [(field, fname, "image/jpeg", data)])
|
|
||||||
res = c.post("/user/profile/update", data=body, content_type=ct)
|
|
||||||
if res.status == 302 and res.loc:
|
|
||||||
res = c.get(res.loc if res.loc.startswith("/") else "/user/profile")
|
|
||||||
if res.status != 200:
|
|
||||||
return None, "update 后页面 HTTP %s" % res.status
|
|
||||||
|
|
||||||
if field == "background":
|
|
||||||
m = re.search(r'id="bg-prev"[^>]*src="([^"]+)"', res.text) or \
|
|
||||||
re.search(r'src="([^"]+)"[^>]*id="bg-prev"', res.text)
|
|
||||||
url = m.group(1) if m else None
|
|
||||||
else:
|
|
||||||
urls = re.findall(r'https://r2\.[^"\'\s<>]+/picture/[0-9]+/\d{2}/\d{2}/[0-9a-f]+', res.text)
|
|
||||||
if not urls:
|
|
||||||
m = re.search(r'(https://r2\.[^"\'\s<>]+/picture/[^"\'\s<>]+)', res.text)
|
|
||||||
urls = [m.group(1)] if m else []
|
|
||||||
url = urls[-1] if urls else None
|
|
||||||
|
|
||||||
if url:
|
|
||||||
return url, None
|
|
||||||
dump = "C:/Users/Tamakyi Gekka/AppData/Local/Temp/tbuv_profile.html"
|
|
||||||
with open(dump, "wb") as f:
|
|
||||||
f.write(res.body)
|
|
||||||
alerts = re.findall(r'class="uk-alert[^"]*"[^>]*>(.*?)</div>', res.text, re.S)
|
|
||||||
hint = " | ".join(re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", a)).strip()[:80]
|
|
||||||
for a in alerts[:3])
|
|
||||||
return None, "未取到图片 URL(已存 %s)%s%s" % (dump, (" 提示:" + hint) if hint else "",
|
|
||||||
(" HTTP %s" % res.status))
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--base", default="https://box-dev-api.337845818.best")
|
|
||||||
args = ap.parse_args()
|
|
||||||
base = args.base.rstrip("/")
|
|
||||||
pwd = "TbUv!2026"
|
|
||||||
ts = time.strftime("%m%d%H%M%S")
|
|
||||||
email, domain = "tbuv%s@example.com" % ts, "tbuv%s" % ts
|
|
||||||
print("== 账号:%s / %s ==" % (email, domain))
|
|
||||||
|
|
||||||
# ---- 本地样本 ----
|
|
||||||
square = make_img(1600, 1600) # A 分档回归:头像
|
|
||||||
bg_ok = make_img(1920, 1080) # 背景 16:9,无 EXIF(正向对照)
|
|
||||||
bg_ori6 = make_img(1080, 1920, orientation=6, portrait=True) # 核心用例
|
|
||||||
bg_bad = make_img(1080, 1920) # 无 EXIF 的竖图:应被拒(16:9 校验仍生效)
|
|
||||||
print(" 样本 avatar 1600x1600 %7d B" % len(square))
|
|
||||||
print(" 样本 bg-ok 1920x1080 %7d B" % len(bg_ok))
|
|
||||||
print(" 样本 bg-ori6 1080x1920 (EXIF=6) %7d B" % len(bg_ori6))
|
|
||||||
print(" 样本 bg-bad 1080x1920 (无 EXIF) %7d B" % len(bg_bad))
|
|
||||||
|
|
||||||
c = Client(base)
|
|
||||||
r = c.get("/register")
|
|
||||||
reg = c.post("/register", {
|
|
||||||
"email": email, "domain": domain, "name": "TBUv%s" % ts,
|
|
||||||
"password": pwd, "repeat_password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(c), "language": "zh-CN",
|
|
||||||
"_csrf": csrf_of(r.text),
|
|
||||||
})
|
|
||||||
if not step("注册", reg.status == 302, "HTTP %s -> %s" % (reg.status, reg.loc)):
|
|
||||||
return 1
|
|
||||||
|
|
||||||
c = Client(base)
|
|
||||||
r = c.get("/login")
|
|
||||||
lg = c.post("/login", {"email": email, "password": pwd,
|
|
||||||
"g-recaptcha-response": captcha(c), "_csrf": csrf_of(r.text)})
|
|
||||||
if not step("登录", lg.status == 302, "HTTP %s -> %s" % (lg.status, lg.loc)):
|
|
||||||
return 1
|
|
||||||
|
|
||||||
# ================= A 分档(头像回归) =================
|
|
||||||
print("\n-- A 分档(头像 512/q78)--")
|
|
||||||
url, err = upload(c, "avatar", square)
|
|
||||||
if not step("上传方形大头像(1600x1600)", url is not None, err or (url or "")):
|
|
||||||
return 1
|
|
||||||
img = c.get_binary(url)
|
|
||||||
if not step("拉回头像", img.status == 200 and bool(img.body), "HTTP %s" % img.status):
|
|
||||||
return 1
|
|
||||||
kind, dims = sniff(img.body)
|
|
||||||
print(" 存储结果:%s %s %d bytes" % (kind, dims, len(img.body)))
|
|
||||||
step("A 分档生效:大图被缩到头像档 512", dims == (512, 512), "实际 %s" % (dims,))
|
|
||||||
|
|
||||||
# ================= B1 背景正向对照(无 EXIF) =================
|
|
||||||
print("\n-- 背景 16:9 正向对照(无 EXIF)--")
|
|
||||||
url_bg, err = upload(c, "background", bg_ok)
|
|
||||||
if not step("上传 16:9 背景(1920x1080,无 EXIF)", url_bg is not None, err or (url_bg or "")):
|
|
||||||
return 1
|
|
||||||
r1 = c.get_binary(url_bg)
|
|
||||||
k1, d1 = sniff(r1.body)
|
|
||||||
print(" 存储结果:%s %s %d bytes" % (k1, d1, len(r1.body)))
|
|
||||||
step("背景档 1920 且比例正确", k1 == "webp" and d1 == (1920, 1080), "实际 %s %s" % (k1, d1))
|
|
||||||
|
|
||||||
# ================= B1 核心:EXIF Orientation=6 的竖拍背景 =================
|
|
||||||
print("\n-- B1 核心:EXIF 比例校验缺口(手机竖拍当背景)--")
|
|
||||||
print(" 像素 1080x1920 + EXIF=6 ->肉眼看到 1920x1080 (16:9)")
|
|
||||||
print(" 修复前:按像素 0.5625 判比例 -> 偏差 1.215 >> 0.01 -> 被误拒")
|
|
||||||
url_o, err = upload(c, "background", bg_ori6)
|
|
||||||
if not step("上传 EXIF 竖拍背景(1080x1920, Orientation=6)", url_o is not None,
|
|
||||||
err or (url_o or "")):
|
|
||||||
print(" >>> 若此处 FAIL,说明 EXIF 比例校验缺口仍未修复")
|
|
||||||
return 1
|
|
||||||
if url_o == url_bg:
|
|
||||||
step("EXIF 背景确实换了新图", False, "两次上传返回同一 URL(第二次被拒、沿用旧值)")
|
|
||||||
return 1
|
|
||||||
r2 = c.get_binary(url_o)
|
|
||||||
k2, d2 = sniff(r2.body)
|
|
||||||
print(" 存储结果:%s %s %d bytes" % (k2, d2, len(r2.body)))
|
|
||||||
step("B1 EXIF 归一后按16:9 存盘(1920x1080)", k2 == "webp" and d2 == (1920, 1080),
|
|
||||||
"实际 %s %s" % (k2, d2))
|
|
||||||
step("B1 归一后限边生效(长边 1920)",
|
|
||||||
d2 is not None and max(d2) == 1920, "实际 %s" % (d2,))
|
|
||||||
|
|
||||||
# 尺寸对了不代表方向对了 —— 与 Pillow exif_transpose 作像素级黄金标准比对:
|
|
||||||
# 定位红色条的包围盒,两边必须一致(原图红条在左侧,顺时针 90 后应跑到顶部)。
|
|
||||||
ref = ImageOps.exif_transpose(Image.open(io.BytesIO(bg_ori6))).convert("RGB")
|
|
||||||
got = Image.open(io.BytesIO(r2.body)).convert("RGB")
|
|
||||||
if ref.size == got.size:
|
|
||||||
bb_ref, bb_got = redness_bbox(ref), redness_bbox(got)
|
|
||||||
step("B1 像素级方向与 Pillow exif_transpose 一致(红条包围盒)",
|
|
||||||
bb_ref is not None and bb_ref == bb_got,
|
|
||||||
"黄金标准 %s vs 存盘 %s" % (bb_ref, bb_got))
|
|
||||||
else:
|
|
||||||
step("B1 像素级方向与 Pillow exif_transpose 一致(红条包围盒)", False,
|
|
||||||
"尺寸不同:黄金标准 %s vs 存盘 %s" % (ref.size, got.size))
|
|
||||||
|
|
||||||
# ================= 反向用例:EXIF 修复不能放松比例校验 =================
|
|
||||||
print("\n-- 反向用例:无 EXIF 的竖图仍应被拒(确认没把校验放得过宽)--")
|
|
||||||
url_bad, err_bad = upload(c, "background", bg_bad)
|
|
||||||
# 期望:上传被拒 -> bg-prev 仍是上一步的 ori6 图(url_o)
|
|
||||||
if url_bad == url_o:
|
|
||||||
step("无 EXIF 的 9:16 竖图被 16:9 校验拒绝", True, "bg-prev 保持 ori6 那张")
|
|
||||||
elif url_bad is None:
|
|
||||||
step("无 EXIF 的 9:16 竖图被 16:9 校验拒绝", True, "上传被拒(%s)" % (err_bad or "")[:60])
|
|
||||||
else:
|
|
||||||
step("无 EXIF 的 9:16 竖图被 16:9 校验拒绝", False,
|
|
||||||
"竟被接受并存成 %s" % (url_bad,))
|
|
||||||
|
|
||||||
print("\n== 结果:%d/%d 通过 ==" % (sum(1 for _, ok, _ in RESULTS if ok), len(RESULTS)))
|
|
||||||
for d, ok, e in RESULTS:
|
|
||||||
if not ok:
|
|
||||||
print(" FAIL: %s %s" % (d, e))
|
|
||||||
print("清理:TamaBox ban --email %s --yes" % email)
|
|
||||||
return 0 if all(ok for _, ok, _ in RESULTS) else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
+56
-5
@@ -32,7 +32,7 @@ ps -eo pid,lstart,cmd | grep '[t]amabox' # 看进程启动时间,与二进
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
./tamabox --help | sed -n '/COMMANDS/,/GLOBAL/p'
|
./tamabox --help | sed -n '/COMMANDS/,/GLOBAL/p'
|
||||||
# web / censor / uid / ban / restore / reports / restrict / unrestrict / hide-public / honeypot
|
# web / censor / uid / ban / restore / reports / restrict / unrestrict / hide-public / honeypot / backup / restore-images
|
||||||
```
|
```
|
||||||
|
|
||||||
> 想确认某个字符串有没有编进二进制,**不要用 `strings`** —— 它默认只输出可打印 ASCII,
|
> 想确认某个字符串有没有编进二进制,**不要用 `strings`** —— 它默认只输出可打印 ASCII,
|
||||||
@@ -77,10 +77,13 @@ pg_dump -Fc <dbname> > before-$(date +%F-%H%M).dump
|
|||||||
| `censor` | 重新审核历史内容 | 是 | 审核规则变更后复检 |
|
| `censor` | 重新审核历史内容 | 是 | 审核规则变更后复检 |
|
||||||
| `uid` | 为用户生成/补齐 uid | 是 | 数据修复 |
|
| `uid` | 为用户生成/补齐 uid | 是 | 数据修复 |
|
||||||
| `honeypot` | 扫描器诱捕取证(假登录页) | 部分是(`prune` / `purge-creds`) | 看谁在扫、拿到了什么字典 |
|
| `honeypot` | 扫描器诱捕取证(假登录页) | 部分是(`prune` / `purge-creds`) | 看谁在扫、拿到了什么字典 |
|
||||||
|
| `backup` | 备份 PostgreSQL 与配置 bucket 的全部 R2 对象 | 写本地备份目录 | 灾难恢复前留档 |
|
||||||
|
| `restore-images` | 校验并恢复 R2 图片及元数据 | 是(除 `--dry-run`) | 图片灾难恢复、隔离恢复演练 |
|
||||||
| `web` | 启动 Web 服务 | — | 生产入口 |
|
| `web` | 启动 Web 服务 | — | 生产入口 |
|
||||||
|
|
||||||
除 `reports`、`honeypot hits/stats/paths/size` 外,所有写操作都要求交互式输入大写 `YES` 确认;
|
除 `reports`、`honeypot hits/stats/paths/size` 外,既有运营写操作要求交互式输入大写 `YES` 确认;
|
||||||
脚本化时加 `--yes`。注意 `honeypot prune` / `purge-creds` 的确认词是 **`yes`**(小写)。
|
脚本化时加 `--yes`。注意 `honeypot prune` / `purge-creds` 的确认词是 **`yes`**(小写)。
|
||||||
|
`restore-images` 不做交互确认:实际写入必须显式提供 `--yes`,`--dry-run` 除外。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -329,7 +332,55 @@ pg_dump -Fc <dbname> > before-$(date +%F-%H%M).dump
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. 典型处置流程
|
## 8. `backup` / `restore-images` —— 数据库与 R2 图片
|
||||||
|
|
||||||
|
### 8.1 备份
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 数据库 + 配置 bucket 的全部对象
|
||||||
|
./tamabox backup --output /var/backups/tamabox
|
||||||
|
|
||||||
|
# 只备份对象,或只备份数据库
|
||||||
|
./tamabox backup --images-only --output /var/backups/tamabox
|
||||||
|
./tamabox backup --db-only --output /var/backups/tamabox
|
||||||
|
```
|
||||||
|
|
||||||
|
图片备份使用**空前缀**分页列举配置 bucket 的全部对象,不再只查 `tamabox/`。根日期 key、
|
||||||
|
`picture/`、`tamabox/` 和静态资源都会按原 key 保存到 `<output>/images/`。
|
||||||
|
备份会先完成全部分页 List、key 与本地路径预检,再开始下载;非法路径、目录 key、文件/目录碰撞会中止备份。
|
||||||
|
|
||||||
|
`images-manifest.json` 记录实际下载字节的 `size`、`sha256`,以及 GET 返回的 `etag`、
|
||||||
|
`content_type`、`cache_control`。对象与 manifest 都采用临时文件原子落盘;下载流中断会整对象重试,
|
||||||
|
最终失败不会留下 `.part`。空 bucket 的 manifest 是 `[]`。
|
||||||
|
|
||||||
|
### 8.2 恢复与隔离演练
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 校验全部本地文件并检查默认防覆盖,不写远端
|
||||||
|
./tamabox restore-images --input /var/backups/tamabox --dry-run
|
||||||
|
|
||||||
|
# 实际恢复到原 key;必须显式 --yes
|
||||||
|
./tamabox restore-images --input /var/backups/tamabox --yes
|
||||||
|
|
||||||
|
# 明确允许替换已存在对象
|
||||||
|
./tamabox restore-images --input /var/backups/tamabox --yes --overwrite
|
||||||
|
|
||||||
|
# 在原完整 key 前加隔离前缀,不覆盖生产 key
|
||||||
|
./tamabox restore-images --input /var/backups/tamabox --yes \
|
||||||
|
--target-prefix tamabox-backup-test/manual-restore/
|
||||||
|
```
|
||||||
|
|
||||||
|
`--input` 可指向备份目录,也可直接指向 `images-manifest.json`。恢复会在任何远端操作前校验全部
|
||||||
|
manifest 条目和本地文件(普通文件、非 symlink、大小、实际 SHA-256)。默认先 Head 目标;任一目标已存在
|
||||||
|
即失败且零 Put,只有 `--overwrite` 才允许替换。`--dry-run` 不写对象,实际写入必须 `--yes`。
|
||||||
|
|
||||||
|
`--target-prefix` 会加在 manifest 的完整 key 前;非空前缀末尾没有 `/` 时会自动补一个。
|
||||||
|
Put 保留内容、原 key 路径(额外添加指定前缀)、`ContentType` 与 `CacheControl`,并使用单对象超时和有限重试。
|
||||||
|
**恢复不是事务**:中途失败会明确报告已成功数量,已上传对象不会自动回滚。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 典型处置流程
|
||||||
|
|
||||||
1. **发现**:收到告警邮件(同一箱主 24h ≥ 8 条 或 7d ≥ 20 条)→ 或箱主后台出现「被举报 N 次」。
|
1. **发现**:收到告警邮件(同一箱主 24h ≥ 8 条 或 7d ≥ 20 条)→ 或箱主后台出现「被举报 N 次」。
|
||||||
2. **取证**:`reports --domain <domain>` 看条目、占比、历史处置。
|
2. **取证**:`reports --domain <domain>` 看条目、占比、历史处置。
|
||||||
@@ -351,7 +402,7 @@ pg_dump -Fc <dbname> > before-$(date +%F-%H%M).dump
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 9. 排错
|
## 10. 排错
|
||||||
|
|
||||||
| 现象 | 原因 | 处理 |
|
| 现象 | 原因 | 处理 |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
@@ -374,7 +425,7 @@ pg_dump -Fc <dbname> > before-$(date +%F-%H%M).dump
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10. 明确不做的事
|
## 11. 明确不做的事
|
||||||
|
|
||||||
- **不做自动处置**:举报数达标只发邮件,不自动封禁。这个场景里举报本身就是攻击武器。
|
- **不做自动处置**:举报数达标只发邮件,不自动封禁。这个场景里举报本身就是攻击武器。
|
||||||
- **不做申诉后台**:申诉走邮件,处置结果由 CLI 写入。
|
- **不做申诉后台**:申诉走邮件,处置结果由 CLI 写入。
|
||||||
|
|||||||
+391
-122
@@ -6,40 +6,46 @@ package cmd
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/aws/aws-sdk-go-v2/aws"
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
"github.com/aws/aws-sdk-go-v2/config"
|
|
||||||
"github.com/aws/aws-sdk-go-v2/credentials"
|
|
||||||
"github.com/aws/aws-sdk-go-v2/service/s3"
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
"github.com/sirupsen/logrus"
|
"github.com/sirupsen/logrus"
|
||||||
"github.com/urfave/cli/v2"
|
"github.com/urfave/cli/v2"
|
||||||
|
|
||||||
"github.com/tamakyi/TamaBox/internal/conf"
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
imageObjectAttemptTimeout = 60 * time.Second
|
||||||
|
imageObjectMaxAttempts = 3
|
||||||
|
imageObjectRetryBackoff = 100 * time.Millisecond
|
||||||
)
|
)
|
||||||
|
|
||||||
// Backup 全量备份数据库与对象存储(Cloudflare R2)图片到本地目录。
|
// Backup 全量备份数据库与对象存储(Cloudflare R2)图片到本地目录。
|
||||||
//
|
//
|
||||||
// 为什么必须同时备份图片:站点的所有图片(头像 / 背景 / 二维码 logo / 提问图 / 回答图)
|
// 为什么必须同时备份图片:站点的所有图片(头像 / 背景 / 二维码 logo / 提问图 / 回答图)
|
||||||
// 都以 picture/YYYY/MM/DD/<hex>(新)或 tamabox/YYYY/MM/<hex>.<ext>(历史)的形式存在
|
// 都以根日期 key、picture/ 或 tamabox/ 等形式存在 R2 上,数据库里只存 key 或 CDN URL。
|
||||||
// R2 上,数据库里只存 key(upload_images.key)或 URL(users.avatar/background 里的完整
|
// 只备份数据库的话,恢复后这些引用指向的图片若已不在 R2,页面就全是裂图。
|
||||||
// CDN 地址)。只备份数据库的话,恢复后这些引用指向的图片若已不在 R2(bucket 误删 /
|
|
||||||
// 账号丢失 / 图片被清理),页面就全是裂图。因此备份必须把 R2 上的图片一并落盘。
|
|
||||||
//
|
//
|
||||||
// 输出目录结构:
|
// 输出目录结构:
|
||||||
//
|
//
|
||||||
// <output>/
|
// <output>/
|
||||||
// db-<20060102-150405>.sql # pg_dump 全量导出(schema + data,可用 psql 恢复)
|
// db-<20060102-150405>.sql # pg_dump 全量导出(schema + data,可用 psql 恢复)
|
||||||
// images/<key> # R2 对象,保持原 key 路径
|
// images/<key> # R2 对象,保持原 key 路径
|
||||||
// images-manifest.json # key → md5 / size 清单,用于恢复后完整性校验
|
// images-manifest.json # key → 实际大小 / SHA-256 / GET 元数据
|
||||||
//
|
//
|
||||||
// 用法:TamaBox backup --output /var/backups/tamabox
|
// 用法:TamaBox backup --output /var/backups/tamabox
|
||||||
//
|
//
|
||||||
@@ -47,7 +53,7 @@ import (
|
|||||||
// TamaBox backup --output . --images-only # 只备份图片
|
// TamaBox backup --output . --images-only # 只备份图片
|
||||||
var Backup = &cli.Command{
|
var Backup = &cli.Command{
|
||||||
Name: "backup",
|
Name: "backup",
|
||||||
Usage: "Back up the database and R2 images to a local directory",
|
Usage: "Back up the database and all R2 objects to a local directory",
|
||||||
Action: runBackup,
|
Action: runBackup,
|
||||||
Flags: []cli.Flag{
|
Flags: []cli.Flag{
|
||||||
&cli.StringFlag{
|
&cli.StringFlag{
|
||||||
@@ -67,6 +73,9 @@ var Backup = &cli.Command{
|
|||||||
}
|
}
|
||||||
|
|
||||||
func runBackup(ctx *cli.Context) error {
|
func runBackup(ctx *cli.Context) error {
|
||||||
|
if ctx.Bool("db-only") && ctx.Bool("images-only") {
|
||||||
|
return errors.New("--db-only and --images-only cannot be used together")
|
||||||
|
}
|
||||||
if err := conf.Init(); err != nil {
|
if err := conf.Init(); err != nil {
|
||||||
return errors.Wrap(err, "load configuration")
|
return errors.Wrap(err, "load configuration")
|
||||||
}
|
}
|
||||||
@@ -127,190 +136,450 @@ func backupDatabase(ctx context.Context, outDir string) error {
|
|||||||
)
|
)
|
||||||
cmd.Env = append(os.Environ(), "PGPASSWORD="+conf.Database.Password)
|
cmd.Env = append(os.Environ(), "PGPASSWORD="+conf.Database.Password)
|
||||||
|
|
||||||
f, err := os.OpenFile(outFile, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
|
|
||||||
if err != nil {
|
|
||||||
return errors.Wrapf(err, "create backup file %q", outFile)
|
|
||||||
}
|
|
||||||
defer f.Close()
|
|
||||||
|
|
||||||
var dumpErr strings.Builder
|
var dumpErr strings.Builder
|
||||||
cmd.Stdout = f
|
if err := writeFileAtomically(outFile, func(w io.Writer) error {
|
||||||
|
cmd.Stdout = w
|
||||||
cmd.Stderr = &dumpErr
|
cmd.Stderr = &dumpErr
|
||||||
if err := cmd.Run(); err != nil {
|
if err := cmd.Run(); err != nil {
|
||||||
return errors.Wrapf(err, "pg_dump failed: %s", strings.TrimSpace(dumpErr.String()))
|
return errors.Wrapf(err, "pg_dump failed: %s", strings.TrimSpace(dumpErr.String()))
|
||||||
}
|
}
|
||||||
|
return nil
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
logrus.WithField("file", outFile).Info("数据库已备份")
|
logrus.WithField("file", outFile).Info("数据库已备份")
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// imageManifestEntry 图片备份清单里的一条:key → 大小 → md5(用于恢复后校验完整性)。
|
// writeFileAtomically 保留数据库、manifest 与对象下载共用的原子写修复:
|
||||||
|
// 只有完整写入、Sync 并关闭成功后才替换目标,失败时清理临时 .part 文件。
|
||||||
|
func writeFileAtomically(outFile string, write func(io.Writer) error) error {
|
||||||
|
f, err := os.CreateTemp(filepath.Dir(outFile), "."+filepath.Base(outFile)+"-*.part")
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrapf(err, "create temporary file for %q", outFile)
|
||||||
|
}
|
||||||
|
tmpFile := f.Name()
|
||||||
|
defer os.Remove(tmpFile)
|
||||||
|
defer f.Close()
|
||||||
|
|
||||||
|
if err := write(f); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := f.Sync(); err != nil {
|
||||||
|
return errors.Wrapf(err, "sync temporary file for %q", outFile)
|
||||||
|
}
|
||||||
|
if err := f.Close(); err != nil {
|
||||||
|
return errors.Wrapf(err, "close temporary file for %q", outFile)
|
||||||
|
}
|
||||||
|
if err := os.Rename(tmpFile, outFile); err != nil {
|
||||||
|
return errors.Wrapf(err, "finalize %q", outFile)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// imageManifestEntry 记录实际下载内容的大小和 SHA-256,以及 GET 返回的原始元数据。
|
||||||
|
// ETag 只是服务商元数据,不能当作内容 MD5 或完整性校验的替代品。
|
||||||
type imageManifestEntry struct {
|
type imageManifestEntry struct {
|
||||||
Key string `json:"key"`
|
Key string `json:"key"`
|
||||||
Size int64 `json:"size"`
|
Size int64 `json:"size"`
|
||||||
MD5 string `json:"md5,omitempty"`
|
SHA256 string `json:"sha256"`
|
||||||
|
ETag string `json:"etag,omitempty"`
|
||||||
|
ContentType string `json:"content_type,omitempty"`
|
||||||
|
CacheControl string `json:"cache_control,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// backupImages 从 R2 下载 tamabox 站的全部图片对象到本地,并生成 manifest。
|
type backupS3API interface {
|
||||||
//
|
ListObjectsV2(context.Context, *s3.ListObjectsV2Input, ...func(*s3.Options)) (*s3.ListObjectsV2Output, error)
|
||||||
// R2 里 tamabox 站图片的 key 都带 `tamabox/` 前缀(如 tamabox/2025/12/<ts>.jpg),
|
GetObject(context.Context, *s3.GetObjectInput, ...func(*s3.Options)) (*s3.GetObjectOutput, error)
|
||||||
// 这是「bucket 用 path-style 访问时 List 返回的 key 带 bucket 名作前缀」这一 R2 行为。
|
}
|
||||||
// 因此只需列出 `tamabox/` 前缀即可覆盖全部图片。blog/ 是同一 R2 account 下另一个站点
|
|
||||||
// 的内容,不在 tamabox 的备份范围。
|
type backupDownloadPlan struct {
|
||||||
|
key string
|
||||||
|
destination string
|
||||||
|
}
|
||||||
|
|
||||||
|
type downloadedImageObject struct {
|
||||||
|
size int64
|
||||||
|
sha256 string
|
||||||
|
etag string
|
||||||
|
contentType string
|
||||||
|
cacheControl string
|
||||||
|
}
|
||||||
|
|
||||||
|
// backupImages 默认使用空前缀列举配置 bucket 的全部对象,而不是假定图片都在 tamabox/ 下。
|
||||||
func backupImages(ctx context.Context, outDir string) error {
|
func backupImages(ctx context.Context, outDir string) error {
|
||||||
client, err := backupS3Client()
|
client, err := backupS3Client(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Wrap(err, "get R2 client for backup")
|
return errors.Wrap(err, "get R2 client for backup")
|
||||||
}
|
}
|
||||||
|
return backupImagesWithClient(ctx, client, conf.Upload.ImageBucket, outDir, "")
|
||||||
imageDir := filepath.Join(outDir, "images")
|
|
||||||
if err := os.MkdirAll(imageDir, 0o700); err != nil {
|
|
||||||
return errors.Wrap(err, "create images directory")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var manifest []imageManifestEntry
|
// backupImagesWithClient 允许测试和隔离演练显式指定 prefix。
|
||||||
var count int64
|
// 所有分页 List、key 校验和本地路径预检都在第一次 GetObject 之前完成。
|
||||||
var totalBytes int64
|
func backupImagesWithClient(ctx context.Context, client backupS3API, bucket, outDir, prefix string) error {
|
||||||
|
keys, err := listAllBackupKeys(ctx, client, bucket, prefix)
|
||||||
// 见函数注释:tamabox/ 前缀即覆盖全部图片(含新旧 key 格式)。
|
if err != nil {
|
||||||
if err := backupPrefix(ctx, client, imageDir, "tamabox/", &manifest, &count, &totalBytes); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// 写 manifest:记录 key → 大小 → md5,恢复后可用于校验完整性。
|
imageDir := filepath.Join(outDir, "images")
|
||||||
mf, err := os.Create(filepath.Join(outDir, "images-manifest.json"))
|
plans, err := planBackupDownloads(imageDir, keys)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Wrap(err, "create manifest file")
|
return err
|
||||||
}
|
}
|
||||||
defer mf.Close()
|
|
||||||
enc := json.NewEncoder(mf)
|
manifest := make([]imageManifestEntry, 0, len(plans))
|
||||||
|
var totalBytes int64
|
||||||
|
for _, plan := range plans {
|
||||||
|
if err := ensureBackupDestinationDirectory(imageDir, plan.destination); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
downloaded, err := downloadObject(ctx, client, bucket, plan.key, plan.destination)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
manifest = append(manifest, imageManifestEntry{
|
||||||
|
Key: plan.key,
|
||||||
|
Size: downloaded.size,
|
||||||
|
SHA256: downloaded.sha256,
|
||||||
|
ETag: downloaded.etag,
|
||||||
|
ContentType: downloaded.contentType,
|
||||||
|
CacheControl: downloaded.cacheControl,
|
||||||
|
})
|
||||||
|
totalBytes += downloaded.size
|
||||||
|
}
|
||||||
|
|
||||||
|
manifestFile := filepath.Join(outDir, "images-manifest.json")
|
||||||
|
if err := writeFileAtomically(manifestFile, func(w io.Writer) error {
|
||||||
|
enc := json.NewEncoder(w)
|
||||||
enc.SetIndent("", " ")
|
enc.SetIndent("", " ")
|
||||||
if err := enc.Encode(manifest); err != nil {
|
if err := enc.Encode(manifest); err != nil {
|
||||||
return errors.Wrap(err, "write manifest")
|
return errors.Wrap(err, "write manifest")
|
||||||
}
|
}
|
||||||
|
return nil
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
logrus.WithFields(logrus.Fields{
|
logrus.WithFields(logrus.Fields{
|
||||||
"objects": count,
|
"objects": len(manifest),
|
||||||
"bytes": totalBytes,
|
"bytes": totalBytes,
|
||||||
"dir": imageDir,
|
"dir": imageDir,
|
||||||
|
"prefix": prefix,
|
||||||
}).Info("图片已备份")
|
}).Info("图片已备份")
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// backupPrefix 分页列出并下载指定前缀下的全部对象。
|
func listAllBackupKeys(ctx context.Context, client backupS3API, bucket, prefix string) ([]string, error) {
|
||||||
func backupPrefix(ctx context.Context, client *s3.Client, imageDir, prefix string, manifest *[]imageManifestEntry, count, totalBytes *int64) error {
|
keys := make([]string, 0)
|
||||||
paginator := s3.NewListObjectsV2Paginator(client, &s3.ListObjectsV2Input{
|
var continuationToken *string
|
||||||
Bucket: aws.String(conf.Upload.ImageBucket),
|
seenTokens := make(map[string]struct{})
|
||||||
|
|
||||||
|
for {
|
||||||
|
page, err := client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||||
|
Bucket: aws.String(bucket),
|
||||||
Prefix: aws.String(prefix),
|
Prefix: aws.String(prefix),
|
||||||
|
ContinuationToken: continuationToken,
|
||||||
})
|
})
|
||||||
for paginator.HasMorePages() {
|
|
||||||
page, err := paginator.NextPage(ctx)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Wrapf(err, "list R2 objects prefix %q", prefix)
|
return nil, errors.Wrapf(err, "list R2 objects prefix %q", prefix)
|
||||||
}
|
}
|
||||||
for _, obj := range page.Contents {
|
if page == nil {
|
||||||
if obj.Key == nil {
|
return nil, errors.New("list R2 objects returned a nil page")
|
||||||
continue
|
|
||||||
}
|
}
|
||||||
key := *obj.Key
|
for _, object := range page.Contents {
|
||||||
|
// nil key 与空 key 一样是无效输入;统一转成空串,由完整 List 后的预检拒绝。
|
||||||
// 落盘路径 = imageDir + "/" + key,保持对象 key 结构(含 tamabox/ 前缀),
|
keys = append(keys, aws.ToString(object.Key))
|
||||||
// 恢复时可直接用同一 key 回传 R2。
|
|
||||||
dst := filepath.Join(imageDir, filepath.FromSlash(key))
|
|
||||||
if err := os.MkdirAll(filepath.Dir(dst), 0o700); err != nil {
|
|
||||||
return errors.Wrapf(err, "create dir for %q", key)
|
|
||||||
}
|
|
||||||
if err := downloadObject(ctx, client, conf.Upload.ImageBucket, key, dst); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
}
|
||||||
|
|
||||||
size := int64(0)
|
if !aws.ToBool(page.IsTruncated) {
|
||||||
if obj.Size != nil {
|
return keys, nil
|
||||||
size = *obj.Size
|
}
|
||||||
|
next := aws.ToString(page.NextContinuationToken)
|
||||||
|
if next == "" {
|
||||||
|
return nil, errors.New("truncated R2 list response has no continuation token")
|
||||||
|
}
|
||||||
|
if _, exists := seenTokens[next]; exists {
|
||||||
|
return nil, errors.Errorf("R2 list continuation token %q repeated", next)
|
||||||
|
}
|
||||||
|
seenTokens[next] = struct{}{}
|
||||||
|
continuationToken = aws.String(next)
|
||||||
}
|
}
|
||||||
md5 := ""
|
|
||||||
if obj.ETag != nil {
|
|
||||||
md5 = strings.Trim(*obj.ETag, `"`)
|
|
||||||
}
|
}
|
||||||
*manifest = append(*manifest, imageManifestEntry{Key: key, Size: size, MD5: md5})
|
|
||||||
|
|
||||||
*count++
|
func planBackupDownloads(imageDir string, keys []string) ([]backupDownloadPlan, error) {
|
||||||
*totalBytes += size
|
plans := make([]backupDownloadPlan, 0, len(keys))
|
||||||
|
files := make(map[string]string, len(keys))
|
||||||
|
directories := make(map[string]string)
|
||||||
|
|
||||||
|
root, err := filepath.Abs(imageDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "resolve images directory")
|
||||||
|
}
|
||||||
|
directories[canonicalLocalPath(root)] = ""
|
||||||
|
|
||||||
|
for _, key := range keys {
|
||||||
|
destination, err := localImagePath(root, key)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
canonicalDestination := canonicalLocalPath(destination)
|
||||||
|
if previous, exists := files[canonicalDestination]; exists {
|
||||||
|
return nil, errors.Errorf("objects %q and %q map to the same local file", previous, key)
|
||||||
|
}
|
||||||
|
if previous, exists := directories[canonicalDestination]; exists {
|
||||||
|
return nil, errors.Errorf("file/directory collision between objects %q and %q", previous, key)
|
||||||
|
}
|
||||||
|
|
||||||
|
for parent := filepath.Dir(destination); pathWithinDirectory(root, parent); parent = filepath.Dir(parent) {
|
||||||
|
canonicalParent := canonicalLocalPath(parent)
|
||||||
|
if previous, exists := files[canonicalParent]; exists {
|
||||||
|
return nil, errors.Errorf("file/directory collision between objects %q and %q", previous, key)
|
||||||
|
}
|
||||||
|
if _, exists := directories[canonicalParent]; !exists {
|
||||||
|
directories[canonicalParent] = key
|
||||||
|
}
|
||||||
|
if parent == root {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
files[canonicalDestination] = key
|
||||||
|
plans = append(plans, backupDownloadPlan{key: key, destination: destination})
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := prepareBackupImageRoot(root); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
for _, plan := range plans {
|
||||||
|
if err := preflightBackupLocalPath(root, plan.destination); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return plans, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateObjectKey(key string) error {
|
||||||
|
if key == "" {
|
||||||
|
return errors.New("R2 object key must not be empty")
|
||||||
|
}
|
||||||
|
if strings.ContainsRune(key, 0) {
|
||||||
|
return errors.Errorf("R2 object key %q contains NUL", key)
|
||||||
|
}
|
||||||
|
if strings.Contains(key, `\`) {
|
||||||
|
return errors.Errorf("R2 object key %q contains a backslash", key)
|
||||||
|
}
|
||||||
|
if strings.HasSuffix(key, "/") {
|
||||||
|
return errors.Errorf("R2 object key %q is a directory key", key)
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(key, "/") || filepath.IsAbs(filepath.FromSlash(key)) || filepath.VolumeName(filepath.FromSlash(key)) != "" ||
|
||||||
|
(len(key) >= 2 && ((key[0] >= 'A' && key[0] <= 'Z') || (key[0] >= 'a' && key[0] <= 'z')) && key[1] == ':') {
|
||||||
|
return errors.Errorf("R2 object key %q is an absolute or volume-qualified path", key)
|
||||||
|
}
|
||||||
|
for _, segment := range strings.Split(key, "/") {
|
||||||
|
if segment == "" || segment == "." || segment == ".." {
|
||||||
|
return errors.Errorf("R2 object key %q contains an empty or dot path segment", key)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// backupS3Client 构造用于备份的 R2 客户端。
|
func localImagePath(imageDir, key string) (string, error) {
|
||||||
//
|
if err := validateObjectKey(key); err != nil {
|
||||||
// 不能直接复用 storage.GetS3Client 做 List:那个客户端把 bucket 路径拼进了 endpoint
|
return "", err
|
||||||
// (image_endpoint 形如 https://<account>.r2.cloudflarestorage.com/<bucket>),
|
}
|
||||||
// 它对 GetObject/PutObject 恰好可用(R2 容错),但 ListObjectsV2 会把 URL 拼成
|
root, err := filepath.Abs(imageDir)
|
||||||
// /<bucket>/<bucket>?list-type=2,R2 找不到这个 key 前缀而返回 404 NoSuchKey。
|
if err != nil {
|
||||||
//
|
return "", errors.Wrap(err, "resolve images directory")
|
||||||
// 这里剥离 endpoint 的 bucket 后缀,得到纯 account endpoint,并启用 path-style:
|
}
|
||||||
// List/Get 都用 Bucket 参数 + 完整 key(List 返回的 key 自带 bucket 名作前缀,
|
destination := filepath.Join(root, filepath.FromSlash(key))
|
||||||
// 如 tamabox/2025/12/<ts>.jpg),这是 R2 上唯一稳定可用的姿势。
|
if !pathWithinDirectory(root, destination) || destination == root {
|
||||||
func backupS3Client() (*s3.Client, error) {
|
return "", errors.Errorf("R2 object key %q escapes the images directory", key)
|
||||||
endpoint := strings.TrimRight(conf.Upload.ImageEndpoint, "/")
|
}
|
||||||
// 剥离最后一个路径段(bucket 名)。endpoint 形如
|
return destination, nil
|
||||||
// https://<account>.r2.cloudflarestorage.com/<bucket>,去掉 /<bucket>。
|
|
||||||
if i := strings.LastIndex(endpoint, "/"); i > len("https://") {
|
|
||||||
endpoint = endpoint[:i]
|
|
||||||
}
|
}
|
||||||
|
|
||||||
resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
|
func pathWithinDirectory(root, candidate string) bool {
|
||||||
return aws.Endpoint{
|
rel, err := filepath.Rel(root, candidate)
|
||||||
URL: endpoint,
|
if err != nil || filepath.IsAbs(rel) {
|
||||||
HostnameImmutable: true,
|
return false
|
||||||
Source: aws.EndpointSourceCustom,
|
}
|
||||||
}, nil
|
return rel == "." || rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator))
|
||||||
})
|
}
|
||||||
cfg, err := config.LoadDefaultConfig(context.Background(),
|
|
||||||
config.WithEndpointResolverWithOptions(resolver),
|
func canonicalLocalPath(path string) string {
|
||||||
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
|
cleaned := filepath.Clean(path)
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
return strings.ToLower(cleaned)
|
||||||
|
}
|
||||||
|
return cleaned
|
||||||
|
}
|
||||||
|
|
||||||
|
func prepareBackupImageRoot(imageDir string) error {
|
||||||
|
info, err := os.Lstat(imageDir)
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
if err := os.MkdirAll(imageDir, 0o700); err != nil {
|
||||||
|
return errors.Wrap(err, "create images directory")
|
||||||
|
}
|
||||||
|
info, err = os.Lstat(imageDir)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrap(err, "inspect images directory")
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() {
|
||||||
|
return errors.Errorf("images path %q must be a real directory, not a symlink or file", imageDir)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func preflightBackupLocalPath(imageDir, destination string) error {
|
||||||
|
info, err := os.Lstat(destination)
|
||||||
|
if err == nil && (info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular()) {
|
||||||
|
return errors.Errorf("backup destination %q is not a regular file", destination)
|
||||||
|
}
|
||||||
|
if err != nil && !os.IsNotExist(err) {
|
||||||
|
return errors.Wrapf(err, "inspect backup destination %q", destination)
|
||||||
|
}
|
||||||
|
|
||||||
|
for parent := filepath.Dir(destination); parent != imageDir; parent = filepath.Dir(parent) {
|
||||||
|
if !pathWithinDirectory(imageDir, parent) {
|
||||||
|
return errors.Errorf("backup destination %q escapes images directory", destination)
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(parent)
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrapf(err, "inspect backup parent %q", parent)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() {
|
||||||
|
return errors.Errorf("backup parent %q is not a real directory", parent)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ensureBackupDestinationDirectory(imageDir, destination string) error {
|
||||||
|
root, err := filepath.Abs(imageDir)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
parent := filepath.Dir(destination)
|
||||||
|
rel, err := filepath.Rel(root, parent)
|
||||||
|
if err != nil || !pathWithinDirectory(root, parent) {
|
||||||
|
return errors.Errorf("invalid backup destination %q", destination)
|
||||||
|
}
|
||||||
|
if rel == "." {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
current := root
|
||||||
|
for _, segment := range strings.Split(rel, string(filepath.Separator)) {
|
||||||
|
current = filepath.Join(current, segment)
|
||||||
|
info, err := os.Lstat(current)
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
if err := os.Mkdir(current, 0o700); err != nil && !os.IsExist(err) {
|
||||||
|
return errors.Wrapf(err, "create backup directory %q", current)
|
||||||
|
}
|
||||||
|
info, err = os.Lstat(current)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrapf(err, "inspect backup directory %q", current)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() {
|
||||||
|
return errors.Errorf("backup parent %q is not a real directory", current)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// backupS3Client 与上传、恢复共用 storage.NewS3Client 的 endpoint 语义:
|
||||||
|
// endpoint 可不带路径或只带 /<bucket>,请求始终以 Bucket 参数和原始完整 key 寻址。
|
||||||
|
// List 返回什么 key 就下载什么 key,不额外添加或剥离 bucket 前缀。
|
||||||
|
func backupS3Client(ctx context.Context) (*s3.Client, error) {
|
||||||
|
return storage.NewS3Client(
|
||||||
|
ctx,
|
||||||
|
conf.Upload.ImageEndpoint,
|
||||||
|
conf.Upload.ImageBucket,
|
||||||
conf.Upload.ImageAccessID,
|
conf.Upload.ImageAccessID,
|
||||||
conf.Upload.ImageAccessSecret,
|
conf.Upload.ImageAccessSecret,
|
||||||
"",
|
|
||||||
)),
|
|
||||||
config.WithRegion("auto"),
|
|
||||||
)
|
)
|
||||||
if err != nil {
|
|
||||||
return nil, errors.Wrap(err, "load R2 config")
|
|
||||||
}
|
|
||||||
return s3.NewFromConfig(cfg, func(o *s3.Options) {
|
|
||||||
o.UsePathStyle = true
|
|
||||||
}), nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// downloadObject 下载单个对象到本地文件,失败时删除半成品避免留下损坏文件。
|
// downloadObject 原子下载单个对象,并返回实际落盘内容的 SHA-256 和 GET 元数据。
|
||||||
func downloadObject(ctx context.Context, client *s3.Client, bucket, key, dst string) error {
|
// 每次尝试都有独立 60 秒 context;流中断会从头重试整个对象,最多 3 次。
|
||||||
|
func downloadObject(ctx context.Context, client backupS3API, bucket, key, destination string) (downloadedImageObject, error) {
|
||||||
|
var lastErr error
|
||||||
|
for attempt := 0; attempt < imageObjectMaxAttempts; attempt++ {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return downloadedImageObject{}, err
|
||||||
|
}
|
||||||
|
attemptCtx, cancel := context.WithTimeout(ctx, imageObjectAttemptTimeout)
|
||||||
|
downloaded, err := downloadObjectOnce(attemptCtx, client, bucket, key, destination)
|
||||||
|
cancel()
|
||||||
|
if err == nil {
|
||||||
|
return downloaded, nil
|
||||||
|
}
|
||||||
|
lastErr = err
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return downloadedImageObject{}, ctx.Err()
|
||||||
|
}
|
||||||
|
if attempt == imageObjectMaxAttempts-1 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err := waitForImageObjectRetry(ctx, imageObjectRetryBackoff<<attempt); err != nil {
|
||||||
|
return downloadedImageObject{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return downloadedImageObject{}, errors.Wrapf(lastErr, "download %q after %d attempts", key, imageObjectMaxAttempts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func downloadObjectOnce(ctx context.Context, client backupS3API, bucket, key, destination string) (downloadedImageObject, error) {
|
||||||
out, err := client.GetObject(ctx, &s3.GetObjectInput{
|
out, err := client.GetObject(ctx, &s3.GetObjectInput{
|
||||||
Bucket: aws.String(bucket),
|
Bucket: aws.String(bucket),
|
||||||
Key: aws.String(key),
|
Key: aws.String(key),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Wrapf(err, "get R2 object %q", key)
|
return downloadedImageObject{}, errors.Wrapf(err, "get R2 object %q", key)
|
||||||
|
}
|
||||||
|
if out == nil || out.Body == nil {
|
||||||
|
return downloadedImageObject{}, errors.Errorf("get R2 object %q returned an empty response body", key)
|
||||||
}
|
}
|
||||||
defer out.Body.Close()
|
|
||||||
|
|
||||||
tmp := dst + ".part"
|
hash := sha256.New()
|
||||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
|
var size int64
|
||||||
|
writeErr := writeFileAtomically(destination, func(w io.Writer) error {
|
||||||
|
var err error
|
||||||
|
size, err = io.Copy(io.MultiWriter(w, hash), out.Body)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Wrapf(err, "create %q", tmp)
|
|
||||||
}
|
|
||||||
if _, err := io.Copy(f, out.Body); err != nil {
|
|
||||||
f.Close()
|
|
||||||
os.Remove(tmp)
|
|
||||||
return errors.Wrapf(err, "download %q", key)
|
return errors.Wrapf(err, "download %q", key)
|
||||||
}
|
}
|
||||||
if err := f.Close(); err != nil {
|
if out.ContentLength != nil && size != aws.ToInt64(out.ContentLength) {
|
||||||
os.Remove(tmp)
|
return errors.Wrapf(io.ErrUnexpectedEOF, "download %q: GET content length is %d, received %d", key, aws.ToInt64(out.ContentLength), size)
|
||||||
return errors.Wrapf(err, "close %q", tmp)
|
|
||||||
}
|
|
||||||
if err := os.Rename(tmp, dst); err != nil {
|
|
||||||
os.Remove(tmp)
|
|
||||||
return errors.Wrapf(err, "finalize %q", dst)
|
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
})
|
||||||
|
_ = out.Body.Close()
|
||||||
|
if writeErr != nil {
|
||||||
|
return downloadedImageObject{}, writeErr
|
||||||
|
}
|
||||||
|
return downloadedImageObject{
|
||||||
|
size: size,
|
||||||
|
sha256: hex.EncodeToString(hash.Sum(nil)),
|
||||||
|
etag: aws.ToString(out.ETag),
|
||||||
|
contentType: aws.ToString(out.ContentType),
|
||||||
|
cacheControl: aws.ToString(out.CacheControl),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitForImageObjectRetry(ctx context.Context, delay time.Duration) error {
|
||||||
|
timer := time.NewTimer(delay)
|
||||||
|
defer timer.Stop()
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
case <-timer.C:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,437 @@
|
|||||||
|
// Copyright 2026 tamakyi. All rights reserved.
|
||||||
|
// Use of this source code is governed by a MIT-style
|
||||||
|
// license that can be found in the LICENSE file.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
"github.com/aws/smithy-go"
|
||||||
|
"github.com/pkg/errors"
|
||||||
|
"github.com/sirupsen/logrus"
|
||||||
|
"github.com/urfave/cli/v2"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
const maxImageManifestBytes = 64 << 20 // 64 MiB,足以容纳大量对象且避免恶意清单耗尽内存。
|
||||||
|
|
||||||
|
// RestoreImages 从 backup 生成的 images-manifest.json 和 images/ 恢复 R2 对象。
|
||||||
|
var RestoreImages = &cli.Command{
|
||||||
|
Name: "restore-images",
|
||||||
|
Usage: "Validate and restore backed-up images to R2",
|
||||||
|
Action: runRestoreImages,
|
||||||
|
Flags: []cli.Flag{
|
||||||
|
&cli.StringFlag{
|
||||||
|
Name: "input",
|
||||||
|
Required: true,
|
||||||
|
Usage: "Backup directory or images-manifest.json path",
|
||||||
|
},
|
||||||
|
&cli.BoolFlag{
|
||||||
|
Name: "dry-run",
|
||||||
|
Usage: "Validate local files and remote targets without writing objects",
|
||||||
|
},
|
||||||
|
&cli.BoolFlag{
|
||||||
|
Name: "yes",
|
||||||
|
Usage: "Confirm actual remote writes",
|
||||||
|
},
|
||||||
|
&cli.BoolFlag{
|
||||||
|
Name: "overwrite",
|
||||||
|
Usage: "Allow replacing objects that already exist",
|
||||||
|
},
|
||||||
|
&cli.StringFlag{
|
||||||
|
Name: "target-prefix",
|
||||||
|
Value: "",
|
||||||
|
Usage: "Prefix prepended to every restored key (for isolated drills)",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
type restoreImagesOptions struct {
|
||||||
|
Input string
|
||||||
|
DryRun bool
|
||||||
|
Yes bool
|
||||||
|
Overwrite bool
|
||||||
|
TargetPrefix string
|
||||||
|
}
|
||||||
|
|
||||||
|
// restoreObjectStore 的 PutObject 是带重试的高层操作,便于离线 fake 精确断言零写入。
|
||||||
|
type restoreObjectStore interface {
|
||||||
|
HeadObject(context.Context, *s3.HeadObjectInput, ...func(*s3.Options)) (*s3.HeadObjectOutput, error)
|
||||||
|
PutObject(context.Context, *s3.PutObjectInput) error
|
||||||
|
}
|
||||||
|
|
||||||
|
type r2RestoreObjectStore struct {
|
||||||
|
client *s3.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *r2RestoreObjectStore) HeadObject(ctx context.Context, input *s3.HeadObjectInput, options ...func(*s3.Options)) (*s3.HeadObjectOutput, error) {
|
||||||
|
return s.client.HeadObject(ctx, input, options...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *r2RestoreObjectStore) PutObject(ctx context.Context, input *s3.PutObjectInput) error {
|
||||||
|
// storage.PutObject 提供每次独立 60 秒超时、最多 3 次重试、Body 回绕和可取消退避。
|
||||||
|
return storage.PutObject(ctx, s.client, input)
|
||||||
|
}
|
||||||
|
|
||||||
|
type restorePlannedObject struct {
|
||||||
|
entry imageManifestEntry
|
||||||
|
sourcePath string
|
||||||
|
targetKey string
|
||||||
|
}
|
||||||
|
|
||||||
|
func runRestoreImages(ctx *cli.Context) error {
|
||||||
|
opts := restoreImagesOptions{
|
||||||
|
Input: ctx.String("input"),
|
||||||
|
DryRun: ctx.Bool("dry-run"),
|
||||||
|
Yes: ctx.Bool("yes"),
|
||||||
|
Overwrite: ctx.Bool("overwrite"),
|
||||||
|
TargetPrefix: ctx.String("target-prefix"),
|
||||||
|
}
|
||||||
|
if opts.Input == "" {
|
||||||
|
return errors.New("--input must not be empty")
|
||||||
|
}
|
||||||
|
if !opts.DryRun && !opts.Yes {
|
||||||
|
return errors.New("actual restore writes require --yes (or use --dry-run)")
|
||||||
|
}
|
||||||
|
if err := conf.Init(); err != nil {
|
||||||
|
return errors.Wrap(err, "load configuration")
|
||||||
|
}
|
||||||
|
|
||||||
|
client, err := backupS3Client(ctx.Context)
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrap(err, "get R2 client for image restore")
|
||||||
|
}
|
||||||
|
return restoreImagesWithStore(ctx.Context, &r2RestoreObjectStore{client: client}, conf.Upload.ImageBucket, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func restoreImagesWithStore(ctx context.Context, store restoreObjectStore, bucket string, opts restoreImagesOptions) error {
|
||||||
|
if opts.Input == "" {
|
||||||
|
return errors.New("restore input must not be empty")
|
||||||
|
}
|
||||||
|
if !opts.DryRun && !opts.Yes {
|
||||||
|
return errors.New("actual restore writes require --yes (or use --dry-run)")
|
||||||
|
}
|
||||||
|
|
||||||
|
manifestPath, imageDir, err := resolveRestoreInput(opts.Input)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
entries, err := readImageManifest(manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// 关键安全顺序:先验证所有 manifest 条目和全部本地文件,再做任何 Head/Put。
|
||||||
|
// 因此重复 key、非法 hash/size、缺文件、篡改或 symlink 都不会造成部分上传。
|
||||||
|
objects := make([]restorePlannedObject, 0, len(entries))
|
||||||
|
for _, entry := range entries {
|
||||||
|
targetKey, err := restoreTargetKey(opts.TargetPrefix, entry.Key)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
sourcePath, err := localImagePath(imageDir, entry.Key)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := validateRestoreLocalFile(imageDir, sourcePath, entry); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
objects = append(objects, restorePlannedObject{
|
||||||
|
entry: entry,
|
||||||
|
sourcePath: sourcePath,
|
||||||
|
targetKey: targetKey,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// 默认先 Head 所有目标。任何目标已存在都会在 Put 循环开始前失败,保证零 Put。
|
||||||
|
if !opts.Overwrite {
|
||||||
|
for _, object := range objects {
|
||||||
|
headCtx, cancel := context.WithTimeout(ctx, imageObjectAttemptTimeout)
|
||||||
|
_, err := store.HeadObject(headCtx, &s3.HeadObjectInput{
|
||||||
|
Bucket: aws.String(bucket),
|
||||||
|
Key: aws.String(object.targetKey),
|
||||||
|
})
|
||||||
|
cancel()
|
||||||
|
if err == nil {
|
||||||
|
return errors.Errorf("restore target %q already exists; no objects were uploaded (use --overwrite to replace)", object.targetKey)
|
||||||
|
}
|
||||||
|
if !isS3ObjectNotFound(err) {
|
||||||
|
return errors.Wrapf(err, "check restore target %q; no objects were uploaded", object.targetKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if opts.DryRun {
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"objects": len(objects),
|
||||||
|
"prefix": opts.TargetPrefix,
|
||||||
|
}).Info("图片恢复演练完成,未写入对象")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
succeeded := 0
|
||||||
|
for _, object := range objects {
|
||||||
|
file, err := openRestoreFileForPut(object.sourcePath, object.entry.Size)
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrapf(err, "open validated restore file for %q after %d successful uploads; restore is not transactional", object.entry.Key, succeeded)
|
||||||
|
}
|
||||||
|
input := &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String(bucket),
|
||||||
|
Key: aws.String(object.targetKey),
|
||||||
|
Body: file,
|
||||||
|
ContentLength: aws.Int64(object.entry.Size),
|
||||||
|
ContentType: nonEmptyStringPointer(object.entry.ContentType),
|
||||||
|
CacheControl: nonEmptyStringPointer(object.entry.CacheControl),
|
||||||
|
}
|
||||||
|
putErr := store.PutObject(ctx, input)
|
||||||
|
closeErr := file.Close()
|
||||||
|
if putErr != nil {
|
||||||
|
return errors.Wrapf(putErr, "restore object %q to %q failed after %d successful uploads; restore is not transactional", object.entry.Key, object.targetKey, succeeded)
|
||||||
|
}
|
||||||
|
if closeErr != nil {
|
||||||
|
return errors.Wrapf(closeErr, "close restored file %q (remote object was uploaded); restore is not transactional", object.sourcePath)
|
||||||
|
}
|
||||||
|
succeeded++
|
||||||
|
}
|
||||||
|
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"objects": succeeded,
|
||||||
|
"prefix": opts.TargetPrefix,
|
||||||
|
}).Info("图片恢复完成")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveRestoreInput(input string) (manifestPath, imageDir string, err error) {
|
||||||
|
info, err := os.Lstat(input)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", errors.Wrapf(err, "inspect restore input %q", input)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return "", "", errors.Errorf("restore input %q must not be a symlink", input)
|
||||||
|
}
|
||||||
|
|
||||||
|
manifestPath = input
|
||||||
|
if info.IsDir() {
|
||||||
|
manifestPath = filepath.Join(input, "images-manifest.json")
|
||||||
|
} else if !info.Mode().IsRegular() {
|
||||||
|
return "", "", errors.Errorf("restore input %q is neither a directory nor a regular manifest file", input)
|
||||||
|
}
|
||||||
|
return manifestPath, filepath.Join(filepath.Dir(manifestPath), "images"), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func readImageManifest(manifestPath string) ([]imageManifestEntry, error) {
|
||||||
|
info, err := os.Lstat(manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrapf(err, "inspect image manifest %q", manifestPath)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
|
||||||
|
return nil, errors.Errorf("image manifest %q must be a regular non-symlink file", manifestPath)
|
||||||
|
}
|
||||||
|
if info.Size() > maxImageManifestBytes {
|
||||||
|
return nil, errors.Errorf("image manifest %q exceeds the %d-byte limit", manifestPath, maxImageManifestBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
file, err := os.Open(manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrapf(err, "open image manifest %q", manifestPath)
|
||||||
|
}
|
||||||
|
data, readErr := io.ReadAll(io.LimitReader(file, maxImageManifestBytes+1))
|
||||||
|
closeErr := file.Close()
|
||||||
|
if readErr != nil {
|
||||||
|
return nil, errors.Wrap(readErr, "read image manifest")
|
||||||
|
}
|
||||||
|
if closeErr != nil {
|
||||||
|
return nil, errors.Wrap(closeErr, "close image manifest")
|
||||||
|
}
|
||||||
|
if len(data) > maxImageManifestBytes {
|
||||||
|
return nil, errors.Errorf("image manifest %q exceeds the %d-byte limit", manifestPath, maxImageManifestBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(data))
|
||||||
|
var entries []imageManifestEntry
|
||||||
|
if err := decoder.Decode(&entries); err != nil {
|
||||||
|
return nil, errors.Wrap(err, "decode image manifest")
|
||||||
|
}
|
||||||
|
if entries == nil {
|
||||||
|
return nil, errors.New("image manifest must be a JSON array, not null")
|
||||||
|
}
|
||||||
|
var trailing json.RawMessage
|
||||||
|
if err := decoder.Decode(&trailing); err != io.EOF {
|
||||||
|
if err == nil {
|
||||||
|
return nil, errors.New("image manifest contains trailing JSON")
|
||||||
|
}
|
||||||
|
return nil, errors.Wrap(err, "image manifest contains trailing data")
|
||||||
|
}
|
||||||
|
|
||||||
|
seen := make(map[string]struct{}, len(entries))
|
||||||
|
for index := range entries {
|
||||||
|
entry := &entries[index]
|
||||||
|
if err := validateObjectKey(entry.Key); err != nil {
|
||||||
|
return nil, errors.Wrapf(err, "invalid manifest entry %d", index)
|
||||||
|
}
|
||||||
|
if _, exists := seen[entry.Key]; exists {
|
||||||
|
return nil, errors.Errorf("image manifest contains duplicate key %q", entry.Key)
|
||||||
|
}
|
||||||
|
seen[entry.Key] = struct{}{}
|
||||||
|
if entry.Size < 0 {
|
||||||
|
return nil, errors.Errorf("manifest entry %q has negative size %d", entry.Key, entry.Size)
|
||||||
|
}
|
||||||
|
decodedHash, err := hex.DecodeString(entry.SHA256)
|
||||||
|
if err != nil || len(decodedHash) != sha256.Size {
|
||||||
|
return nil, errors.Errorf("manifest entry %q has invalid SHA-256 %q", entry.Key, entry.SHA256)
|
||||||
|
}
|
||||||
|
entry.SHA256 = strings.ToLower(entry.SHA256)
|
||||||
|
}
|
||||||
|
return entries, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func restoreTargetKey(prefix, key string) (string, error) {
|
||||||
|
if prefix != "" && !strings.HasSuffix(prefix, "/") {
|
||||||
|
prefix += "/"
|
||||||
|
}
|
||||||
|
target := prefix + key
|
||||||
|
if err := validateObjectKey(target); err != nil {
|
||||||
|
return "", errors.Wrapf(err, "invalid restore target for source key %q", key)
|
||||||
|
}
|
||||||
|
return target, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRestoreLocalFile(imageDir, sourcePath string, entry imageManifestEntry) error {
|
||||||
|
root, err := filepath.Abs(imageDir)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := validateRestoreDirectoryChain(root, filepath.Dir(sourcePath)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrapf(err, "inspect local image for %q", entry.Key)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
|
||||||
|
return errors.Errorf("local image for %q must be a regular non-symlink file", entry.Key)
|
||||||
|
}
|
||||||
|
if info.Size() != entry.Size {
|
||||||
|
return errors.Errorf("local image size mismatch for %q: manifest=%d actual=%d", entry.Key, entry.Size, info.Size())
|
||||||
|
}
|
||||||
|
|
||||||
|
file, err := os.Open(sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrapf(err, "open local image for %q", entry.Key)
|
||||||
|
}
|
||||||
|
openedInfo, statErr := file.Stat()
|
||||||
|
if statErr != nil {
|
||||||
|
_ = file.Close()
|
||||||
|
return errors.Wrapf(statErr, "stat opened local image for %q", entry.Key)
|
||||||
|
}
|
||||||
|
if !openedInfo.Mode().IsRegular() || openedInfo.Size() != entry.Size {
|
||||||
|
_ = file.Close()
|
||||||
|
return errors.Errorf("opened local image for %q is not the expected regular file", entry.Key)
|
||||||
|
}
|
||||||
|
hash := sha256.New()
|
||||||
|
_, hashErr := io.Copy(hash, file)
|
||||||
|
closeErr := file.Close()
|
||||||
|
if hashErr != nil {
|
||||||
|
return errors.Wrapf(hashErr, "hash local image for %q", entry.Key)
|
||||||
|
}
|
||||||
|
if closeErr != nil {
|
||||||
|
return errors.Wrapf(closeErr, "close local image for %q", entry.Key)
|
||||||
|
}
|
||||||
|
actualHash := hex.EncodeToString(hash.Sum(nil))
|
||||||
|
if actualHash != entry.SHA256 {
|
||||||
|
return errors.Errorf("local image SHA-256 mismatch for %q: manifest=%s actual=%s", entry.Key, entry.SHA256, actualHash)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRestoreDirectoryChain(root, parent string) error {
|
||||||
|
rootInfo, err := os.Lstat(root)
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrapf(err, "inspect restore images directory %q", root)
|
||||||
|
}
|
||||||
|
if rootInfo.Mode()&os.ModeSymlink != 0 || !rootInfo.IsDir() {
|
||||||
|
return errors.Errorf("restore images path %q must be a real directory", root)
|
||||||
|
}
|
||||||
|
rel, err := filepath.Rel(root, parent)
|
||||||
|
if err != nil || !pathWithinDirectory(root, parent) {
|
||||||
|
return errors.Errorf("restore file parent %q escapes images directory", parent)
|
||||||
|
}
|
||||||
|
if rel == "." {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
current := root
|
||||||
|
for _, segment := range strings.Split(rel, string(filepath.Separator)) {
|
||||||
|
current = filepath.Join(current, segment)
|
||||||
|
info, err := os.Lstat(current)
|
||||||
|
if err != nil {
|
||||||
|
return errors.Wrapf(err, "inspect restore parent directory %q", current)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() {
|
||||||
|
return errors.Errorf("restore parent %q must be a real directory", current)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func openRestoreFileForPut(sourcePath string, expectedSize int64) (*os.File, error) {
|
||||||
|
info, err := os.Lstat(sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() || info.Size() != expectedSize {
|
||||||
|
return nil, fmt.Errorf("local file %q changed after validation", sourcePath)
|
||||||
|
}
|
||||||
|
file, err := os.Open(sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
openedInfo, err := file.Stat()
|
||||||
|
if err != nil {
|
||||||
|
_ = file.Close()
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !openedInfo.Mode().IsRegular() || openedInfo.Size() != expectedSize {
|
||||||
|
_ = file.Close()
|
||||||
|
return nil, fmt.Errorf("local file %q changed after validation", sourcePath)
|
||||||
|
}
|
||||||
|
return file, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isS3ObjectNotFound(err error) bool {
|
||||||
|
if err == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
var statusErr interface{ HTTPStatusCode() int }
|
||||||
|
if errors.As(err, &statusErr) && statusErr.HTTPStatusCode() == 404 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
var apiErr smithy.APIError
|
||||||
|
if errors.As(err, &apiErr) {
|
||||||
|
switch apiErr.ErrorCode() {
|
||||||
|
case "NotFound", "NoSuchKey", "404":
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func nonEmptyStringPointer(value string) *string {
|
||||||
|
if value == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return aws.String(value)
|
||||||
|
}
|
||||||
@@ -0,0 +1,184 @@
|
|||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestR2BackupRestoreIntegration 默认跳过。只有显式设置 TB_R2_INTEGRATION=1,
|
||||||
|
// 并通过 NEKOBOX_CONFIG_PATH 提供真实 R2 配置时才运行。
|
||||||
|
// 测试只访问 tamabox-backup-test/<随机>/ 下的 source/ 与 target/ 两个专属前缀,
|
||||||
|
// 绝不执行全桶 List,也不会读取、覆盖或删除其他 key。
|
||||||
|
func TestR2BackupRestoreIntegration(t *testing.T) {
|
||||||
|
if os.Getenv("TB_R2_INTEGRATION") != "1" {
|
||||||
|
t.Skip("set TB_R2_INTEGRATION=1 to run the real R2 integration test")
|
||||||
|
}
|
||||||
|
if os.Getenv("NEKOBOX_CONFIG_PATH") == "" {
|
||||||
|
t.Fatal("NEKOBOX_CONFIG_PATH must point to the R2 integration configuration")
|
||||||
|
}
|
||||||
|
require.NoError(t, conf.Init())
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
client, err := storage.NewS3Client(
|
||||||
|
ctx,
|
||||||
|
conf.Upload.ImageEndpoint,
|
||||||
|
conf.Upload.ImageBucket,
|
||||||
|
conf.Upload.ImageAccessID,
|
||||||
|
conf.Upload.ImageAccessSecret,
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
random := make([]byte, 12)
|
||||||
|
_, err = rand.Read(random)
|
||||||
|
require.NoError(t, err)
|
||||||
|
basePrefix := fmt.Sprintf("tamabox-backup-test/%d-%s/", time.Now().UTC().UnixNano(), hex.EncodeToString(random))
|
||||||
|
sourcePrefix := basePrefix + "source/"
|
||||||
|
targetPrefix := basePrefix + "target/"
|
||||||
|
|
||||||
|
objects := []struct {
|
||||||
|
relativeKey string
|
||||||
|
body []byte
|
||||||
|
contentType string
|
||||||
|
cacheControl string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
relativeKey: "2026/11/04/root.webp",
|
||||||
|
body: []byte("tamabox-r2-integration-root"),
|
||||||
|
contentType: "image/webp",
|
||||||
|
cacheControl: "public, max-age=60",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
relativeKey: "picture/2026/11/04/picture.webp",
|
||||||
|
body: []byte("tamabox-r2-integration-picture"),
|
||||||
|
contentType: "image/webp",
|
||||||
|
cacheControl: "public, max-age=120",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
// created 只记录本测试成功创建的精确 key。任何失败清理也只删除这些 key。
|
||||||
|
created := make(map[string]bool)
|
||||||
|
t.Cleanup(func() {
|
||||||
|
cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||||
|
defer cleanupCancel()
|
||||||
|
for key := range created {
|
||||||
|
if err := storage.DeleteObject(cleanupCtx, client, conf.Upload.ImageBucket, key); err != nil {
|
||||||
|
t.Errorf("clean up own R2 key %q: %v", key, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, object := range objects {
|
||||||
|
key := sourcePrefix + object.relativeKey
|
||||||
|
err := storage.PutObject(ctx, client, &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String(conf.Upload.ImageBucket),
|
||||||
|
Key: aws.String(key),
|
||||||
|
Body: bytes.NewReader(object.body),
|
||||||
|
ContentLength: aws.Int64(int64(len(object.body))),
|
||||||
|
ContentType: aws.String(object.contentType),
|
||||||
|
CacheControl: aws.String(object.cacheControl),
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
created[key] = true
|
||||||
|
}
|
||||||
|
|
||||||
|
backupDir := t.TempDir()
|
||||||
|
require.NoError(t, backupImagesWithClient(ctx, client, conf.Upload.ImageBucket, backupDir, sourcePrefix))
|
||||||
|
manifest, err := readImageManifest(backupDir + string(os.PathSeparator) + "images-manifest.json")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, manifest, len(objects))
|
||||||
|
manifestKeys := make(map[string]bool, len(manifest))
|
||||||
|
for _, entry := range manifest {
|
||||||
|
manifestKeys[entry.Key] = true
|
||||||
|
}
|
||||||
|
for _, object := range objects {
|
||||||
|
require.True(t, manifestKeys[sourcePrefix+object.relativeKey])
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, object := range objects {
|
||||||
|
sourceKey := sourcePrefix + object.relativeKey
|
||||||
|
require.NoError(t, storage.DeleteObject(ctx, client, conf.Upload.ImageBucket, sourceKey))
|
||||||
|
delete(created, sourceKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
store := &integrationTrackingRestoreStore{
|
||||||
|
base: &r2RestoreObjectStore{client: client},
|
||||||
|
onUpload: func(key string) {
|
||||||
|
created[key] = true
|
||||||
|
},
|
||||||
|
}
|
||||||
|
require.NoError(t, restoreImagesWithStore(ctx, store, conf.Upload.ImageBucket, restoreImagesOptions{
|
||||||
|
Input: backupDir,
|
||||||
|
Yes: true,
|
||||||
|
TargetPrefix: targetPrefix,
|
||||||
|
}))
|
||||||
|
|
||||||
|
for _, object := range objects {
|
||||||
|
targetKey := targetPrefix + sourcePrefix + object.relativeKey
|
||||||
|
require.True(t, created[targetKey], "restore must report its own successful Put")
|
||||||
|
output, err := client.GetObject(ctx, &s3.GetObjectInput{
|
||||||
|
Bucket: aws.String(conf.Upload.ImageBucket),
|
||||||
|
Key: aws.String(targetKey),
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
body, readErr := io.ReadAll(output.Body)
|
||||||
|
closeErr := output.Body.Close()
|
||||||
|
require.NoError(t, readErr)
|
||||||
|
require.NoError(t, closeErr)
|
||||||
|
digest := sha256.Sum256(object.body)
|
||||||
|
require.Equal(t, fmt.Sprintf("%x", digest), fmt.Sprintf("%x", sha256.Sum256(body)))
|
||||||
|
require.Equal(t, object.contentType, aws.ToString(output.ContentType))
|
||||||
|
require.Equal(t, object.cacheControl, aws.ToString(output.CacheControl))
|
||||||
|
}
|
||||||
|
|
||||||
|
for key := range created {
|
||||||
|
require.NoError(t, storage.DeleteObject(ctx, client, conf.Upload.ImageBucket, key))
|
||||||
|
delete(created, key)
|
||||||
|
}
|
||||||
|
requireOwnPrefixEmpty(ctx, t, client, conf.Upload.ImageBucket, sourcePrefix)
|
||||||
|
requireOwnPrefixEmpty(ctx, t, client, conf.Upload.ImageBucket, targetPrefix)
|
||||||
|
}
|
||||||
|
|
||||||
|
type integrationTrackingRestoreStore struct {
|
||||||
|
base restoreObjectStore
|
||||||
|
onUpload func(string)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *integrationTrackingRestoreStore) HeadObject(ctx context.Context, input *s3.HeadObjectInput, options ...func(*s3.Options)) (*s3.HeadObjectOutput, error) {
|
||||||
|
return s.base.HeadObject(ctx, input, options...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *integrationTrackingRestoreStore) PutObject(ctx context.Context, input *s3.PutObjectInput) error {
|
||||||
|
if err := s.base.PutObject(ctx, input); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
s.onUpload(aws.ToString(input.Key))
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireOwnPrefixEmpty(ctx context.Context, t *testing.T, client *s3.Client, bucket, prefix string) {
|
||||||
|
t.Helper()
|
||||||
|
require.NotEmpty(t, prefix)
|
||||||
|
output, err := client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||||
|
Bucket: aws.String(bucket),
|
||||||
|
Prefix: aws.String(prefix),
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Empty(t, output.Contents, "专属 prefix 清理后必须为空:%q", prefix)
|
||||||
|
require.False(t, aws.ToBool(output.IsTruncated), "随机专属 prefix 下对象数不应触发分页")
|
||||||
|
}
|
||||||
@@ -0,0 +1,603 @@
|
|||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3/types"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeBackupObject struct {
|
||||||
|
body []byte
|
||||||
|
etag string
|
||||||
|
contentType string
|
||||||
|
cacheControl string
|
||||||
|
interruptedAttempts int
|
||||||
|
wrongLengthAttempts int
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeBackupS3 struct {
|
||||||
|
pages []*s3.ListObjectsV2Output
|
||||||
|
listInputs []*s3.ListObjectsV2Input
|
||||||
|
objects map[string]fakeBackupObject
|
||||||
|
getCalls map[string]int
|
||||||
|
getOrder []string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeBackupS3) ListObjectsV2(_ context.Context, input *s3.ListObjectsV2Input, _ ...func(*s3.Options)) (*s3.ListObjectsV2Output, error) {
|
||||||
|
f.listInputs = append(f.listInputs, input)
|
||||||
|
index := len(f.listInputs) - 1
|
||||||
|
if index >= len(f.pages) {
|
||||||
|
return nil, fmt.Errorf("unexpected ListObjectsV2 call %d", index+1)
|
||||||
|
}
|
||||||
|
return f.pages[index], nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeBackupS3) GetObject(_ context.Context, input *s3.GetObjectInput, _ ...func(*s3.Options)) (*s3.GetObjectOutput, error) {
|
||||||
|
key := aws.ToString(input.Key)
|
||||||
|
if f.getCalls == nil {
|
||||||
|
f.getCalls = make(map[string]int)
|
||||||
|
}
|
||||||
|
f.getCalls[key]++
|
||||||
|
f.getOrder = append(f.getOrder, key)
|
||||||
|
object, exists := f.objects[key]
|
||||||
|
if !exists {
|
||||||
|
return nil, fmt.Errorf("unexpected GetObject key %q", key)
|
||||||
|
}
|
||||||
|
|
||||||
|
var body io.ReadCloser
|
||||||
|
if f.getCalls[key] <= object.interruptedAttempts {
|
||||||
|
body = io.NopCloser(&interruptedReader{data: append([]byte(nil), object.body...)})
|
||||||
|
} else {
|
||||||
|
body = io.NopCloser(strings.NewReader(string(object.body)))
|
||||||
|
}
|
||||||
|
contentLength := int64(len(object.body))
|
||||||
|
if f.getCalls[key] <= object.wrongLengthAttempts {
|
||||||
|
contentLength++
|
||||||
|
}
|
||||||
|
return &s3.GetObjectOutput{
|
||||||
|
Body: body,
|
||||||
|
ContentLength: aws.Int64(contentLength),
|
||||||
|
ETag: aws.String(object.etag),
|
||||||
|
ContentType: aws.String(object.contentType),
|
||||||
|
CacheControl: aws.String(object.cacheControl),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type interruptedReader struct {
|
||||||
|
data []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *interruptedReader) Read(p []byte) (int, error) {
|
||||||
|
if len(r.data) == 0 {
|
||||||
|
return 0, io.ErrUnexpectedEOF
|
||||||
|
}
|
||||||
|
n := copy(p, r.data)
|
||||||
|
r.data = r.data[n:]
|
||||||
|
if len(r.data) == 0 {
|
||||||
|
return n, io.ErrUnexpectedEOF
|
||||||
|
}
|
||||||
|
return n, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func backupListPage(keys []string, truncated bool, nextToken string) *s3.ListObjectsV2Output {
|
||||||
|
objects := make([]types.Object, 0, len(keys))
|
||||||
|
for _, key := range keys {
|
||||||
|
objects = append(objects, types.Object{
|
||||||
|
Key: aws.String(key),
|
||||||
|
Size: aws.Int64(999999), // 故意不可信;manifest 必须记录实际下载字节数。
|
||||||
|
ETag: aws.String("list-etag-must-not-be-used"),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
output := &s3.ListObjectsV2Output{Contents: objects, IsTruncated: aws.Bool(truncated)}
|
||||||
|
if nextToken != "" {
|
||||||
|
output.NextContinuationToken = aws.String(nextToken)
|
||||||
|
}
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBackupImagesWithClientCoversAllKeysAndPages(t *testing.T) {
|
||||||
|
keys := []string{
|
||||||
|
"2026/11/04/root.webp",
|
||||||
|
"picture/2026/11/04/picture.webp",
|
||||||
|
"tamabox/2025/12/historical.jpg",
|
||||||
|
"static/site.css",
|
||||||
|
}
|
||||||
|
objects := make(map[string]fakeBackupObject, len(keys))
|
||||||
|
for index, key := range keys {
|
||||||
|
objects[key] = fakeBackupObject{
|
||||||
|
body: []byte(fmt.Sprintf("actual-body-%d", index)),
|
||||||
|
etag: fmt.Sprintf("get-etag-%d", index),
|
||||||
|
contentType: []string{"image/webp", "image/webp", "image/jpeg", "text/css"}[index],
|
||||||
|
cacheControl: fmt.Sprintf("public, max-age=%d", index+60),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
client := &fakeBackupS3{
|
||||||
|
pages: []*s3.ListObjectsV2Output{
|
||||||
|
backupListPage(keys[:2], true, "page-2"),
|
||||||
|
backupListPage(keys[2:], false, ""),
|
||||||
|
},
|
||||||
|
objects: objects,
|
||||||
|
}
|
||||||
|
outDir := t.TempDir()
|
||||||
|
require.NoError(t, backupImagesWithClient(context.Background(), client, "configured-bucket", outDir, ""))
|
||||||
|
|
||||||
|
require.Len(t, client.listInputs, 2)
|
||||||
|
for index, input := range client.listInputs {
|
||||||
|
require.Equal(t, "configured-bucket", aws.ToString(input.Bucket))
|
||||||
|
require.Equal(t, "", aws.ToString(input.Prefix), "生产默认必须列举整个 bucket")
|
||||||
|
if index == 0 {
|
||||||
|
require.Nil(t, input.ContinuationToken)
|
||||||
|
} else {
|
||||||
|
require.Equal(t, "page-2", aws.ToString(input.ContinuationToken))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
require.ElementsMatch(t, keys, client.getOrder)
|
||||||
|
|
||||||
|
manifestBytes, err := os.ReadFile(filepath.Join(outDir, "images-manifest.json"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
var manifest []imageManifestEntry
|
||||||
|
require.NoError(t, json.Unmarshal(manifestBytes, &manifest))
|
||||||
|
require.Len(t, manifest, len(keys))
|
||||||
|
for index, entry := range manifest {
|
||||||
|
require.Equal(t, keys[index], entry.Key)
|
||||||
|
object := objects[entry.Key]
|
||||||
|
require.Equal(t, int64(len(object.body)), entry.Size)
|
||||||
|
digest := sha256.Sum256(object.body)
|
||||||
|
require.Equal(t, fmt.Sprintf("%x", digest), entry.SHA256)
|
||||||
|
require.Equal(t, object.etag, entry.ETag)
|
||||||
|
require.Equal(t, object.contentType, entry.ContentType)
|
||||||
|
require.Equal(t, object.cacheControl, entry.CacheControl)
|
||||||
|
|
||||||
|
path, err := localImagePath(filepath.Join(outDir, "images"), entry.Key)
|
||||||
|
require.NoError(t, err)
|
||||||
|
content, err := os.ReadFile(path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, object.body, content)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBackupImagesWithClientSupportsIsolatedPrefixAndEmptyManifest(t *testing.T) {
|
||||||
|
t.Run("prefix", func(t *testing.T) {
|
||||||
|
key := "tamabox-backup-test/unit/source/root.webp"
|
||||||
|
client := &fakeBackupS3{
|
||||||
|
pages: []*s3.ListObjectsV2Output{backupListPage([]string{key}, false, "")},
|
||||||
|
objects: map[string]fakeBackupObject{
|
||||||
|
key: {body: []byte("isolated"), contentType: "image/webp"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
require.NoError(t, backupImagesWithClient(context.Background(), client, "bucket", t.TempDir(), "tamabox-backup-test/unit/source/"))
|
||||||
|
require.Len(t, client.listInputs, 1)
|
||||||
|
require.Equal(t, "tamabox-backup-test/unit/source/", aws.ToString(client.listInputs[0].Prefix))
|
||||||
|
require.Equal(t, []string{key}, client.getOrder)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("empty array", func(t *testing.T) {
|
||||||
|
client := &fakeBackupS3{pages: []*s3.ListObjectsV2Output{backupListPage(nil, false, "")}}
|
||||||
|
outDir := t.TempDir()
|
||||||
|
require.NoError(t, backupImagesWithClient(context.Background(), client, "bucket", outDir, ""))
|
||||||
|
content, err := os.ReadFile(filepath.Join(outDir, "images-manifest.json"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "[]", strings.TrimSpace(string(content)))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBackupRejectsPathAttacksBeforeAnyDownload(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
key *string
|
||||||
|
}{
|
||||||
|
{name: "nil key", key: nil},
|
||||||
|
{name: "empty key", key: aws.String("")},
|
||||||
|
{name: "absolute POSIX", key: aws.String("/outside")},
|
||||||
|
{name: "absolute Windows", key: aws.String("C:/outside")},
|
||||||
|
{name: "volume relative", key: aws.String("C:outside")},
|
||||||
|
{name: "parent", key: aws.String("../outside")},
|
||||||
|
{name: "nested parent", key: aws.String("picture/../outside")},
|
||||||
|
{name: "backslash", key: aws.String(`picture\outside`)},
|
||||||
|
{name: "NUL", key: aws.String("picture/\x00outside")},
|
||||||
|
{name: "empty segment", key: aws.String("picture//outside")},
|
||||||
|
{name: "leading dot segment", key: aws.String("./outside")},
|
||||||
|
{name: "inner dot segment", key: aws.String("picture/./outside")},
|
||||||
|
{name: "directory", key: aws.String("picture/")},
|
||||||
|
{name: "dot only", key: aws.String(".")},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
page := backupListPage(nil, false, "")
|
||||||
|
page.Contents = []types.Object{{Key: tt.key}}
|
||||||
|
client := &fakeBackupS3{pages: []*s3.ListObjectsV2Output{page}}
|
||||||
|
err := backupImagesWithClient(context.Background(), client, "bucket", t.TempDir(), "")
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Empty(t, client.getOrder, "非法 key 必须在任何下载前被拒绝")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBackupPreflightsAllPagesAndLocalCollisionsBeforeDownload(t *testing.T) {
|
||||||
|
t.Run("invalid key on later page", func(t *testing.T) {
|
||||||
|
client := &fakeBackupS3{
|
||||||
|
pages: []*s3.ListObjectsV2Output{
|
||||||
|
backupListPage([]string{"picture/valid.webp"}, true, "next"),
|
||||||
|
backupListPage([]string{"../escape"}, false, ""),
|
||||||
|
},
|
||||||
|
objects: map[string]fakeBackupObject{"picture/valid.webp": {body: []byte("valid")}},
|
||||||
|
}
|
||||||
|
require.Error(t, backupImagesWithClient(context.Background(), client, "bucket", t.TempDir(), ""))
|
||||||
|
require.Len(t, client.listInputs, 2)
|
||||||
|
require.Empty(t, client.getOrder)
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, keys := range [][]string{{"a", "a/b"}, {"a/b", "a"}, {"same", "same"}} {
|
||||||
|
t.Run(fmt.Sprintf("collision %q", keys), func(t *testing.T) {
|
||||||
|
client := &fakeBackupS3{pages: []*s3.ListObjectsV2Output{backupListPage(keys, false, "")}}
|
||||||
|
require.Error(t, backupImagesWithClient(context.Background(), client, "bucket", t.TempDir(), ""))
|
||||||
|
require.Empty(t, client.getOrder)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("existing file blocks planned directory", func(t *testing.T) {
|
||||||
|
outDir := t.TempDir()
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Join(outDir, "images"), 0o700))
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(outDir, "images", "a"), []byte("existing"), 0o600))
|
||||||
|
client := &fakeBackupS3{pages: []*s3.ListObjectsV2Output{backupListPage([]string{"a/b"}, false, "")}}
|
||||||
|
require.Error(t, backupImagesWithClient(context.Background(), client, "bucket", outDir, ""))
|
||||||
|
require.Empty(t, client.getOrder)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("existing directory blocks planned file", func(t *testing.T) {
|
||||||
|
outDir := t.TempDir()
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Join(outDir, "images", "a"), 0o700))
|
||||||
|
client := &fakeBackupS3{pages: []*s3.ListObjectsV2Output{backupListPage([]string{"a"}, false, "")}}
|
||||||
|
require.Error(t, backupImagesWithClient(context.Background(), client, "bucket", outDir, ""))
|
||||||
|
require.Empty(t, client.getOrder)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBackupRetriesInterruptedStreamsAndLeavesNoPart(t *testing.T) {
|
||||||
|
t.Run("eventual success", func(t *testing.T) {
|
||||||
|
key := "picture/retry.webp"
|
||||||
|
body := []byte("complete-object-after-retries")
|
||||||
|
client := &fakeBackupS3{
|
||||||
|
pages: []*s3.ListObjectsV2Output{backupListPage([]string{key}, false, "")},
|
||||||
|
objects: map[string]fakeBackupObject{
|
||||||
|
key: {body: body, interruptedAttempts: 2, etag: "successful-get-etag"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
outDir := t.TempDir()
|
||||||
|
require.NoError(t, backupImagesWithClient(context.Background(), client, "bucket", outDir, ""))
|
||||||
|
require.Equal(t, 3, client.getCalls[key])
|
||||||
|
content, err := os.ReadFile(filepath.Join(outDir, "images", "picture", "retry.webp"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, body, content)
|
||||||
|
requireNoPartFiles(t, filepath.Join(outDir, "images", "picture"))
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("short body relative to GET content length", func(t *testing.T) {
|
||||||
|
key := "short-body.webp"
|
||||||
|
client := &fakeBackupS3{
|
||||||
|
pages: []*s3.ListObjectsV2Output{backupListPage([]string{key}, false, "")},
|
||||||
|
objects: map[string]fakeBackupObject{
|
||||||
|
key: {body: []byte("complete-on-second-attempt"), wrongLengthAttempts: 1},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
outDir := t.TempDir()
|
||||||
|
require.NoError(t, backupImagesWithClient(context.Background(), client, "bucket", outDir, ""))
|
||||||
|
require.Equal(t, 2, client.getCalls[key])
|
||||||
|
requireNoPartFiles(t, filepath.Join(outDir, "images"))
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("final failure", func(t *testing.T) {
|
||||||
|
key := "retry.txt"
|
||||||
|
client := &fakeBackupS3{
|
||||||
|
pages: []*s3.ListObjectsV2Output{backupListPage([]string{key}, false, "")},
|
||||||
|
objects: map[string]fakeBackupObject{
|
||||||
|
key: {body: []byte("never-complete"), interruptedAttempts: 3},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
outDir := t.TempDir()
|
||||||
|
err := backupImagesWithClient(context.Background(), client, "bucket", outDir, "")
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Contains(t, err.Error(), "after 3 attempts")
|
||||||
|
require.Equal(t, 3, client.getCalls[key])
|
||||||
|
require.NoFileExists(t, filepath.Join(outDir, "images", key))
|
||||||
|
requireNoPartFiles(t, filepath.Join(outDir, "images"))
|
||||||
|
require.NoFileExists(t, filepath.Join(outDir, "images-manifest.json"))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
type capturedRestorePut struct {
|
||||||
|
key string
|
||||||
|
body []byte
|
||||||
|
contentType string
|
||||||
|
cacheControl string
|
||||||
|
size int64
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeRestoreStore struct {
|
||||||
|
existing map[string]bool
|
||||||
|
headErrors map[string]error
|
||||||
|
putFailures map[string]error
|
||||||
|
headCalls []string
|
||||||
|
puts []capturedRestorePut
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeRestoreStore) HeadObject(_ context.Context, input *s3.HeadObjectInput, _ ...func(*s3.Options)) (*s3.HeadObjectOutput, error) {
|
||||||
|
key := aws.ToString(input.Key)
|
||||||
|
f.headCalls = append(f.headCalls, key)
|
||||||
|
if err := f.headErrors[key]; err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if f.existing[key] {
|
||||||
|
return &s3.HeadObjectOutput{}, nil
|
||||||
|
}
|
||||||
|
return nil, &types.NotFound{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeRestoreStore) PutObject(_ context.Context, input *s3.PutObjectInput) error {
|
||||||
|
body, err := io.ReadAll(input.Body)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
key := aws.ToString(input.Key)
|
||||||
|
f.puts = append(f.puts, capturedRestorePut{
|
||||||
|
key: key,
|
||||||
|
body: body,
|
||||||
|
contentType: aws.ToString(input.ContentType),
|
||||||
|
cacheControl: aws.ToString(input.CacheControl),
|
||||||
|
size: aws.ToInt64(input.ContentLength),
|
||||||
|
})
|
||||||
|
return f.putFailures[key]
|
||||||
|
}
|
||||||
|
|
||||||
|
type restoreFixtureObject struct {
|
||||||
|
key string
|
||||||
|
body []byte
|
||||||
|
storedBody []byte
|
||||||
|
manifestSize *int64
|
||||||
|
manifestSHA string
|
||||||
|
contentType string
|
||||||
|
cacheControl string
|
||||||
|
}
|
||||||
|
|
||||||
|
func createRestoreFixture(t *testing.T, objects []restoreFixtureObject) string {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
manifest := make([]imageManifestEntry, 0, len(objects))
|
||||||
|
for _, object := range objects {
|
||||||
|
storedBody := object.storedBody
|
||||||
|
if storedBody == nil {
|
||||||
|
storedBody = object.body
|
||||||
|
}
|
||||||
|
path := filepath.Join(dir, "images", filepath.FromSlash(object.key))
|
||||||
|
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o700))
|
||||||
|
require.NoError(t, os.WriteFile(path, storedBody, 0o600))
|
||||||
|
|
||||||
|
size := int64(len(object.body))
|
||||||
|
if object.manifestSize != nil {
|
||||||
|
size = *object.manifestSize
|
||||||
|
}
|
||||||
|
digest := sha256.Sum256(object.body)
|
||||||
|
hash := fmt.Sprintf("%x", digest)
|
||||||
|
if object.manifestSHA != "" {
|
||||||
|
hash = object.manifestSHA
|
||||||
|
}
|
||||||
|
manifest = append(manifest, imageManifestEntry{
|
||||||
|
Key: object.key,
|
||||||
|
Size: size,
|
||||||
|
SHA256: hash,
|
||||||
|
ETag: "source-get-etag",
|
||||||
|
ContentType: object.contentType,
|
||||||
|
CacheControl: object.cacheControl,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
data, err := json.MarshalIndent(manifest, "", " ")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(dir, "images-manifest.json"), data, 0o600))
|
||||||
|
return dir
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestoreImagesPreservesKeysContentAndMetadataWithTargetPrefix(t *testing.T) {
|
||||||
|
objects := []restoreFixtureObject{
|
||||||
|
{key: "2026/11/04/root.webp", body: []byte("root-body"), contentType: "image/webp", cacheControl: "public, max-age=60"},
|
||||||
|
{key: "picture/2026/11/04/pic.webp", body: []byte("picture-body"), contentType: "image/webp", cacheControl: "public, max-age=120"},
|
||||||
|
{key: "static/site.css", body: []byte("body{}"), contentType: "text/css", cacheControl: "no-cache"},
|
||||||
|
}
|
||||||
|
input := createRestoreFixture(t, objects)
|
||||||
|
store := &fakeRestoreStore{}
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{
|
||||||
|
Input: input,
|
||||||
|
Yes: true,
|
||||||
|
TargetPrefix: "isolated-restore", // 不带斜杠也应规范化为恰好一个路径分隔符。
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, store.headCalls, len(objects))
|
||||||
|
require.Len(t, store.puts, len(objects))
|
||||||
|
|
||||||
|
byKey := make(map[string]capturedRestorePut, len(store.puts))
|
||||||
|
for _, put := range store.puts {
|
||||||
|
byKey[put.key] = put
|
||||||
|
}
|
||||||
|
for _, object := range objects {
|
||||||
|
target := "isolated-restore/" + object.key
|
||||||
|
put := byKey[target]
|
||||||
|
require.Equal(t, object.body, put.body)
|
||||||
|
require.Equal(t, object.contentType, put.contentType)
|
||||||
|
require.Equal(t, object.cacheControl, put.cacheControl)
|
||||||
|
require.Equal(t, int64(len(object.body)), put.size)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestoreImagesDefaultRefusesOverwriteWithZeroPut(t *testing.T) {
|
||||||
|
objects := []restoreFixtureObject{
|
||||||
|
{key: "first.webp", body: []byte("first")},
|
||||||
|
{key: "second.webp", body: []byte("second")},
|
||||||
|
}
|
||||||
|
store := &fakeRestoreStore{existing: map[string]bool{"second.webp": true}}
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{
|
||||||
|
Input: createRestoreFixture(t, objects),
|
||||||
|
Yes: true,
|
||||||
|
})
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Contains(t, err.Error(), "no objects were uploaded")
|
||||||
|
require.Empty(t, store.puts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestoreImagesOverwriteAndDryRun(t *testing.T) {
|
||||||
|
t.Run("overwrite", func(t *testing.T) {
|
||||||
|
store := &fakeRestoreStore{existing: map[string]bool{"picture.webp": true}}
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{
|
||||||
|
Input: createRestoreFixture(t, []restoreFixtureObject{{key: "picture.webp", body: []byte("replacement")}}),
|
||||||
|
Yes: true,
|
||||||
|
Overwrite: true,
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Empty(t, store.headCalls, "显式 overwrite 不需要默认防覆盖 Head")
|
||||||
|
require.Len(t, store.puts, 1)
|
||||||
|
require.Equal(t, []byte("replacement"), store.puts[0].body)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("dry run without yes", func(t *testing.T) {
|
||||||
|
store := &fakeRestoreStore{}
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{
|
||||||
|
Input: createRestoreFixture(t, []restoreFixtureObject{{key: "picture.webp", body: []byte("dry-run")}}),
|
||||||
|
DryRun: true,
|
||||||
|
TargetPrefix: "drill/",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, []string{"drill/picture.webp"}, store.headCalls)
|
||||||
|
require.Empty(t, store.puts)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("actual write requires yes", func(t *testing.T) {
|
||||||
|
store := &fakeRestoreStore{}
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{
|
||||||
|
Input: createRestoreFixture(t, []restoreFixtureObject{{key: "picture.webp", body: []byte("write")}}),
|
||||||
|
})
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Contains(t, err.Error(), "--yes")
|
||||||
|
require.Empty(t, store.headCalls)
|
||||||
|
require.Empty(t, store.puts)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestoreRejectsTamperedFilesBeforeAnyRemoteCall(t *testing.T) {
|
||||||
|
wrongSize := int64(999)
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
second restoreFixtureObject
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "SHA256 tamper with unchanged size",
|
||||||
|
second: restoreFixtureObject{
|
||||||
|
key: "second.webp", body: []byte("second"), storedBody: []byte("SECOND"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "size tamper",
|
||||||
|
second: restoreFixtureObject{
|
||||||
|
key: "second.webp", body: []byte("second"), manifestSize: &wrongSize,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "manifest hash tamper",
|
||||||
|
second: restoreFixtureObject{
|
||||||
|
key: "second.webp", body: []byte("second"), manifestSHA: strings.Repeat("0", 64),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
store := &fakeRestoreStore{}
|
||||||
|
input := createRestoreFixture(t, []restoreFixtureObject{
|
||||||
|
{key: "first.webp", body: []byte("first")},
|
||||||
|
tt.second,
|
||||||
|
})
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{Input: input, Yes: true})
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Empty(t, store.headCalls, "全部本地文件应在任何远端操作前完成校验")
|
||||||
|
require.Empty(t, store.puts, "首个有效文件也不能在后续文件校验失败前上传")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReadImageManifestRejectsInvalidStructure(t *testing.T) {
|
||||||
|
validHash := strings.Repeat("a", 64)
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
json string
|
||||||
|
}{
|
||||||
|
{name: "null", json: `null`},
|
||||||
|
{name: "trailing JSON", json: `[] {}`},
|
||||||
|
{name: "trailing garbage", json: `[] garbage`},
|
||||||
|
{name: "duplicate key", json: fmt.Sprintf(`[{"key":"same","size":0,"sha256":%q},{"key":"same","size":0,"sha256":%q}]`, validHash, validHash)},
|
||||||
|
{name: "short hash", json: `[{"key":"image","size":1,"sha256":"abc"}]`},
|
||||||
|
{name: "non hex hash", json: fmt.Sprintf(`[{"key":"image","size":1,"sha256":%q}]`, strings.Repeat("z", 64))},
|
||||||
|
{name: "negative size", json: fmt.Sprintf(`[{"key":"image","size":-1,"sha256":%q}]`, validHash)},
|
||||||
|
{name: "path traversal", json: fmt.Sprintf(`[{"key":"../image","size":1,"sha256":%q}]`, validHash)},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "images-manifest.json")
|
||||||
|
// 非数组用例直接保留原 JSON;其余用例同样应被解析层拒绝。
|
||||||
|
require.NoError(t, os.WriteFile(path, []byte(tt.json), 0o600))
|
||||||
|
_, err := readImageManifest(path)
|
||||||
|
require.Error(t, err)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestoreRejectsSymlinkBeforeRemoteCalls(t *testing.T) {
|
||||||
|
input := createRestoreFixture(t, []restoreFixtureObject{{key: "picture.webp", body: []byte("content")}})
|
||||||
|
source := filepath.Join(input, "images", "picture.webp")
|
||||||
|
outside := filepath.Join(t.TempDir(), "outside.webp")
|
||||||
|
require.NoError(t, os.WriteFile(outside, []byte("content"), 0o600))
|
||||||
|
require.NoError(t, os.Remove(source))
|
||||||
|
if err := os.Symlink(outside, source); err != nil {
|
||||||
|
t.Skipf("当前平台无法创建 symlink:%v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
store := &fakeRestoreStore{}
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{Input: input, DryRun: true})
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Empty(t, store.headCalls)
|
||||||
|
require.Empty(t, store.puts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestoreReportsPartialFailureWithoutClaimingTransaction(t *testing.T) {
|
||||||
|
store := &fakeRestoreStore{putFailures: map[string]error{"second.webp": errors.New("injected put failure")}}
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{
|
||||||
|
Input: createRestoreFixture(t, []restoreFixtureObject{
|
||||||
|
{key: "first.webp", body: []byte("first")},
|
||||||
|
{key: "second.webp", body: []byte("second")},
|
||||||
|
}),
|
||||||
|
Yes: true,
|
||||||
|
})
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Contains(t, err.Error(), "1 successful uploads")
|
||||||
|
require.Contains(t, err.Error(), "not transactional")
|
||||||
|
require.Len(t, store.puts, 2)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestoreRejectsUnsafeTargetPrefixBeforeRemoteCalls(t *testing.T) {
|
||||||
|
store := &fakeRestoreStore{}
|
||||||
|
err := restoreImagesWithStore(context.Background(), store, "bucket", restoreImagesOptions{
|
||||||
|
Input: createRestoreFixture(t, []restoreFixtureObject{{key: "picture.webp", body: []byte("content")}}),
|
||||||
|
Yes: true,
|
||||||
|
TargetPrefix: "../outside/",
|
||||||
|
})
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Empty(t, store.headCalls)
|
||||||
|
require.Empty(t, store.puts)
|
||||||
|
}
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"flag"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"github.com/urfave/cli/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRunBackupRejectsMutuallyExclusiveModes(t *testing.T) {
|
||||||
|
set := flag.NewFlagSet("backup", flag.ContinueOnError)
|
||||||
|
set.Bool("db-only", true, "")
|
||||||
|
set.Bool("images-only", true, "")
|
||||||
|
|
||||||
|
err := runBackup(cli.NewContext(nil, set, nil))
|
||||||
|
require.EqualError(t, err, "--db-only and --images-only cannot be used together")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteFileAtomically(t *testing.T) {
|
||||||
|
t.Run("success", func(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
target := filepath.Join(dir, "backup.sql")
|
||||||
|
require.NoError(t, writeFileAtomically(target, func(w io.Writer) error {
|
||||||
|
_, err := io.WriteString(w, "complete backup")
|
||||||
|
return err
|
||||||
|
}))
|
||||||
|
|
||||||
|
content, err := os.ReadFile(target)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "complete backup", string(content))
|
||||||
|
requireNoPartFiles(t, dir)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("failure preserves previous backup", func(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
target := filepath.Join(dir, "backup.sql")
|
||||||
|
require.NoError(t, os.WriteFile(target, []byte("known good"), 0o600))
|
||||||
|
writeErr := errors.New("dump failed")
|
||||||
|
|
||||||
|
err := writeFileAtomically(target, func(w io.Writer) error {
|
||||||
|
_, _ = io.WriteString(w, "partial")
|
||||||
|
return writeErr
|
||||||
|
})
|
||||||
|
require.ErrorIs(t, err, writeErr)
|
||||||
|
|
||||||
|
content, err := os.ReadFile(target)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "known good", string(content))
|
||||||
|
requireNoPartFiles(t, dir)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("failure leaves no new backup", func(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
target := filepath.Join(dir, "backup.sql")
|
||||||
|
writeErr := errors.New("dump failed")
|
||||||
|
|
||||||
|
err := writeFileAtomically(target, func(w io.Writer) error {
|
||||||
|
_, _ = io.WriteString(w, "partial")
|
||||||
|
return writeErr
|
||||||
|
})
|
||||||
|
require.ErrorIs(t, err, writeErr)
|
||||||
|
require.NoFileExists(t, target)
|
||||||
|
requireNoPartFiles(t, dir)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireNoPartFiles(t *testing.T, dir string) {
|
||||||
|
t.Helper()
|
||||||
|
matches, err := filepath.Glob(filepath.Join(dir, "*.part"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Empty(t, matches)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestImageManifestEntryIntegrityFields(t *testing.T) {
|
||||||
|
entry := imageManifestEntry{
|
||||||
|
Key: "tamabox/image.webp",
|
||||||
|
Size: 3,
|
||||||
|
SHA256: "actual-content-sha256",
|
||||||
|
ETag: "provider-etag",
|
||||||
|
}
|
||||||
|
encoded, err := json.Marshal(entry)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.JSONEq(t, `{
|
||||||
|
"key":"tamabox/image.webp",
|
||||||
|
"size":3,
|
||||||
|
"sha256":"actual-content-sha256",
|
||||||
|
"etag":"provider-etag"
|
||||||
|
}`, string(encoded))
|
||||||
|
require.NotContains(t, string(encoded), "md5")
|
||||||
|
}
|
||||||
+3
-1
@@ -95,7 +95,9 @@ func runBan(ctx *cli.Context) error {
|
|||||||
return errors.Wrap(err, "ban user and cleanup")
|
return errors.Wrap(err, "ban user and cleanup")
|
||||||
}
|
}
|
||||||
|
|
||||||
logrus.WithField("user_id", user.ID).Info("用户已封禁,关联数据已清理")
|
// 数据库事务已经提交,先明确输出这一事实,再进行非事务性的 R2 清理。
|
||||||
|
logrus.WithField("user_id", user.ID).Info("数据库封禁与关联数据清理已提交")
|
||||||
|
drainR2OutboxAfterBan(ctx.Context)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,365 @@
|
|||||||
|
// Copyright 2026 tamakyi. All rights reserved.
|
||||||
|
// Use of this source code is governed by a MIT-style
|
||||||
|
// license that can be found in the LICENSE file.
|
||||||
|
|
||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
"github.com/sirupsen/logrus"
|
||||||
|
"github.com/urfave/cli/v2"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/db"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
r2OutboxDefaultLimit = 100
|
||||||
|
r2OutboxWorkerInterval = time.Minute
|
||||||
|
r2OutboxInitialBackoff = time.Minute
|
||||||
|
r2OutboxMaxBackoff = time.Hour
|
||||||
|
r2OutboxMaxLastErrorSize = 1024
|
||||||
|
)
|
||||||
|
|
||||||
|
var errR2OutboxNotConfigured = errors.New("R2 configuration is incomplete")
|
||||||
|
|
||||||
|
// R2Outbox 有界处理一批持久化的 R2 对象删除任务。
|
||||||
|
var R2Outbox = &cli.Command{
|
||||||
|
Name: "r2-outbox",
|
||||||
|
Usage: "Drain a bounded batch of pending R2 object deletion tasks",
|
||||||
|
Action: runR2Outbox,
|
||||||
|
Flags: []cli.Flag{
|
||||||
|
&cli.IntFlag{
|
||||||
|
Name: "limit",
|
||||||
|
Value: r2OutboxDefaultLimit,
|
||||||
|
Usage: "Maximum number of due tasks to process",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
type r2OutboxTaskStore interface {
|
||||||
|
ListDue(ctx context.Context, limit int) ([]*db.R2DeleteTask, error)
|
||||||
|
Delete(id uint) error
|
||||||
|
MarkFailed(id uint, nextRetry time.Time, lastError string) error
|
||||||
|
HasReferences(ctx context.Context, bucket, key string) (bool, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type r2ObjectDeleter interface {
|
||||||
|
DeleteObject(ctx context.Context, bucket, key string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
type r2S3ObjectDeleter struct {
|
||||||
|
client *s3.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *r2S3ObjectDeleter) DeleteObject(ctx context.Context, bucket, key string) error {
|
||||||
|
return storage.DeleteObject(ctx, d.client, bucket, key)
|
||||||
|
}
|
||||||
|
|
||||||
|
type r2OutboxResult struct {
|
||||||
|
Listed int
|
||||||
|
Removed int
|
||||||
|
ObjectsDeleted int
|
||||||
|
ReferenceOnly int
|
||||||
|
Failed int
|
||||||
|
Skipped int
|
||||||
|
Remaining int
|
||||||
|
}
|
||||||
|
|
||||||
|
// processR2Outbox 串行处理一批到期任务。单个任务失败不会中断同批后续任务;
|
||||||
|
// context 取消则立即停止,且不会把关闭过程中的取消误记为一次任务失败。
|
||||||
|
func processR2Outbox(ctx context.Context, store r2OutboxTaskStore, deleter r2ObjectDeleter, limit int) (result r2OutboxResult, err error) {
|
||||||
|
defer func() {
|
||||||
|
result.Remaining = result.Listed - result.Removed
|
||||||
|
}()
|
||||||
|
|
||||||
|
tasks, err := store.ListDue(ctx, limit)
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return result, ctx.Err()
|
||||||
|
}
|
||||||
|
return result, fmt.Errorf("list R2 outbox tasks: %s", sanitizeR2OutboxError(err))
|
||||||
|
}
|
||||||
|
result.Listed = len(tasks)
|
||||||
|
|
||||||
|
for _, task := range tasks {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return result, err
|
||||||
|
}
|
||||||
|
if task == nil {
|
||||||
|
result.Failed++
|
||||||
|
logrus.Warn("R2 outbox 返回了空任务,已跳过")
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
logger := logrus.WithField("task_id", task.ID)
|
||||||
|
if task.Bucket != conf.Upload.ImageBucket {
|
||||||
|
result.Skipped++
|
||||||
|
logger.Warn("R2 outbox 任务 bucket 与当前配置不匹配,任务已保留")
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
referenced, refErr := store.HasReferences(ctx, task.Bucket, task.Key)
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return result, ctx.Err()
|
||||||
|
}
|
||||||
|
if refErr != nil {
|
||||||
|
result.Failed++
|
||||||
|
markR2OutboxTaskFailed(store, task, refErr)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if referenced {
|
||||||
|
// 还有引用时只清理过期 outbox 记录,绝不删除对象。最后一个引用被
|
||||||
|
// 删除时,对应数据库事务会重新入队一条任务。
|
||||||
|
if err := store.Delete(task.ID); err != nil {
|
||||||
|
result.Failed++
|
||||||
|
logger.WithField("error", sanitizeR2OutboxError(err)).Warn("R2 outbox 引用任务清理失败,任务已保留")
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
result.Removed++
|
||||||
|
result.ReferenceOnly++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
deleteErr := deleter.DeleteObject(ctx, task.Bucket, task.Key)
|
||||||
|
if deleteErr != nil && !isS3ObjectNotFound(deleteErr) {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return result, ctx.Err()
|
||||||
|
}
|
||||||
|
result.Failed++
|
||||||
|
markR2OutboxTaskFailed(store, task, deleteErr)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// storage.DeleteObject 已将 404 归一为成功;这里再次识别 404,保证其他
|
||||||
|
// deleter 实现也遵循相同的幂等语义。只有任务记录也删除后才算完成。
|
||||||
|
if err := store.Delete(task.ID); err != nil {
|
||||||
|
result.Failed++
|
||||||
|
logger.WithField("error", sanitizeR2OutboxError(err)).Warn("R2 对象已删除,但 outbox 任务清理失败;后续将幂等重试")
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
result.Removed++
|
||||||
|
result.ObjectsDeleted++
|
||||||
|
}
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func markR2OutboxTaskFailed(store r2OutboxTaskStore, task *db.R2DeleteTask, cause error) {
|
||||||
|
nextRetry := time.Now().Add(r2OutboxBackoff(task.Attempts))
|
||||||
|
lastError := sanitizeR2OutboxError(cause)
|
||||||
|
if err := store.MarkFailed(task.ID, nextRetry, lastError); err != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"task_id": task.ID,
|
||||||
|
"processing_error": lastError,
|
||||||
|
"persist_error": sanitizeR2OutboxError(err),
|
||||||
|
}).Warn("R2 outbox 任务失败且重试状态写入失败,任务已保留")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"task_id": task.ID,
|
||||||
|
"error": lastError,
|
||||||
|
"next_retry": nextRetry,
|
||||||
|
}).Warn("R2 outbox 任务处理失败,已安排重试")
|
||||||
|
}
|
||||||
|
|
||||||
|
func r2OutboxBackoff(attempts int) time.Duration {
|
||||||
|
if attempts < 0 {
|
||||||
|
attempts = 0
|
||||||
|
}
|
||||||
|
// 1、2、4、8、16、32 分钟后,下一次本应到 64 分钟,直接封顶一小时。
|
||||||
|
if attempts >= 6 {
|
||||||
|
return r2OutboxMaxBackoff
|
||||||
|
}
|
||||||
|
return r2OutboxInitialBackoff << attempts
|
||||||
|
}
|
||||||
|
|
||||||
|
// sanitizeR2OutboxError 在持久化或记录错误前移除已知配置与凭据,并限制为
|
||||||
|
// 1024 字节。截断保持 UTF-8 边界,避免向 text 列写入半个多字节字符。
|
||||||
|
func sanitizeR2OutboxError(err error) string {
|
||||||
|
if err == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
message := err.Error()
|
||||||
|
sensitiveValues := []string{
|
||||||
|
conf.Database.DSN,
|
||||||
|
conf.Database.Password,
|
||||||
|
conf.Upload.ImageEndpoint,
|
||||||
|
conf.Upload.ImageAccessID,
|
||||||
|
conf.Upload.ImageAccessSecret,
|
||||||
|
conf.Upload.ImageBucket,
|
||||||
|
conf.Upload.ImageBucketCDNHost,
|
||||||
|
}
|
||||||
|
for _, value := range sensitiveValues {
|
||||||
|
if value != "" {
|
||||||
|
message = strings.ReplaceAll(message, value, "[redacted]")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(message) <= r2OutboxMaxLastErrorSize {
|
||||||
|
return message
|
||||||
|
}
|
||||||
|
end := r2OutboxMaxLastErrorSize
|
||||||
|
for end > 0 && !utf8.ValidString(message[:end]) {
|
||||||
|
end--
|
||||||
|
}
|
||||||
|
return message[:end]
|
||||||
|
}
|
||||||
|
|
||||||
|
func r2OutboxConfigured() bool {
|
||||||
|
return strings.TrimSpace(conf.Upload.ImageEndpoint) != "" &&
|
||||||
|
strings.TrimSpace(conf.Upload.ImageAccessID) != "" &&
|
||||||
|
strings.TrimSpace(conf.Upload.ImageAccessSecret) != "" &&
|
||||||
|
strings.TrimSpace(conf.Upload.ImageBucket) != ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func newR2OutboxObjectDeleter(ctx context.Context) (r2ObjectDeleter, error) {
|
||||||
|
if !r2OutboxConfigured() {
|
||||||
|
return nil, errR2OutboxNotConfigured
|
||||||
|
}
|
||||||
|
client, err := storage.NewS3Client(
|
||||||
|
ctx,
|
||||||
|
conf.Upload.ImageEndpoint,
|
||||||
|
conf.Upload.ImageBucket,
|
||||||
|
conf.Upload.ImageAccessID,
|
||||||
|
conf.Upload.ImageAccessSecret,
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("initialize R2 client: %s", sanitizeR2OutboxError(err))
|
||||||
|
}
|
||||||
|
return &r2S3ObjectDeleter{client: client}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// startR2OutboxWorker 只能在 db.Init 成功后调用。goroutine 由调用方提供的
|
||||||
|
// WaitGroup 跟踪,Web 退出时先取消 ctx,再等待正在执行的一轮收尾。
|
||||||
|
func startR2OutboxWorker(ctx context.Context, wg *sync.WaitGroup) {
|
||||||
|
if !r2OutboxConfigured() {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
deleter, err := newR2OutboxObjectDeleter(ctx)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithField("error", sanitizeR2OutboxError(err)).Warn("R2 outbox worker 启动失败,Web 服务继续运行")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
runR2OutboxWorker(ctx, db.R2DeleteTasks, deleter)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
func runR2OutboxWorker(ctx context.Context, store r2OutboxTaskStore, deleter r2ObjectDeleter) {
|
||||||
|
ticker := time.NewTicker(r2OutboxWorkerInterval)
|
||||||
|
defer ticker.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
// 启动先立即跑一轮;处理期间不启动并发轮次。若一轮超过一分钟,
|
||||||
|
// ticker 只会合并唤醒,不会形成多个 worker 同时消费。
|
||||||
|
_, err := processR2Outbox(ctx, store, deleter, r2OutboxDefaultLimit)
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
logrus.WithField("error", sanitizeR2OutboxError(err)).Warn("R2 outbox 本轮处理失败,将在下一分钟重试")
|
||||||
|
}
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func drainR2Outbox(ctx context.Context, limit int) (r2OutboxResult, error) {
|
||||||
|
deleter, err := newR2OutboxObjectDeleter(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return r2OutboxResult{}, err
|
||||||
|
}
|
||||||
|
return processR2Outbox(ctx, db.R2DeleteTasks, deleter, limit)
|
||||||
|
}
|
||||||
|
|
||||||
|
func runR2Outbox(ctx *cli.Context) error {
|
||||||
|
limit := ctx.Int("limit")
|
||||||
|
if limit <= 0 {
|
||||||
|
return errors.New("--limit must be greater than zero")
|
||||||
|
}
|
||||||
|
if err := conf.Init(); err != nil {
|
||||||
|
return fmt.Errorf("load configuration: %w", err)
|
||||||
|
}
|
||||||
|
if !r2OutboxConfigured() {
|
||||||
|
return errR2OutboxNotConfigured
|
||||||
|
}
|
||||||
|
|
||||||
|
dbType := conf.Database.Type
|
||||||
|
var dsn string
|
||||||
|
switch dbType {
|
||||||
|
case "mysql", "":
|
||||||
|
dsn = conf.MySQLDsn()
|
||||||
|
case "postgres":
|
||||||
|
dsn = conf.PostgresDsn()
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("unknown database type: %q", dbType)
|
||||||
|
}
|
||||||
|
conf.Database.DSN = dsn
|
||||||
|
gdb, err := db.Init(dbType, dsn)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("connect to database: %w", err)
|
||||||
|
}
|
||||||
|
if sqlDB, dbErr := gdb.DB(); dbErr == nil {
|
||||||
|
defer func() {
|
||||||
|
if err := sqlDB.Close(); err != nil {
|
||||||
|
logrus.WithField("error", sanitizeR2OutboxError(err)).Warn("关闭数据库连接池失败")
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := drainR2Outbox(ctx.Context, limit)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("drain R2 outbox: %s", sanitizeR2OutboxError(err))
|
||||||
|
}
|
||||||
|
logR2OutboxResult("R2 outbox 本轮处理完成", result)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func logR2OutboxResult(message string, result r2OutboxResult) {
|
||||||
|
entry := logrus.WithFields(logrus.Fields{
|
||||||
|
"listed": result.Listed,
|
||||||
|
"removed": result.Removed,
|
||||||
|
"object_deletes": result.ObjectsDeleted,
|
||||||
|
"reference_only": result.ReferenceOnly,
|
||||||
|
"failed": result.Failed,
|
||||||
|
"skipped": result.Skipped,
|
||||||
|
"remaining": result.Remaining,
|
||||||
|
})
|
||||||
|
if result.Remaining > 0 {
|
||||||
|
entry.Warn(message + ",仍有任务待重试或保留")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
entry.Info(message)
|
||||||
|
}
|
||||||
|
|
||||||
|
// drainR2OutboxAfterBan 在封禁事务提交后执行。R2 失败只影响对象清理进度,
|
||||||
|
// 不向 runBan 返回错误,避免运维误以为需要重新执行不可逆的数据库封禁。
|
||||||
|
func drainR2OutboxAfterBan(ctx context.Context) {
|
||||||
|
result, err := drainR2Outbox(ctx, r2OutboxDefaultLimit)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithField("error", sanitizeR2OutboxError(err)).Warn(
|
||||||
|
"数据库封禁与清理已提交,但 R2 outbox 本轮未能处理;任务仍保留,无需重跑 ban,可稍后运行 r2-outbox",
|
||||||
|
)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if result.Remaining > 0 {
|
||||||
|
logR2OutboxResult("数据库封禁与清理已提交;R2 对象清理将在后续重试,无需重跑 ban", result)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
logR2OutboxResult("数据库封禁与清理已提交;R2 outbox 本轮处理完成", result)
|
||||||
|
}
|
||||||
@@ -0,0 +1,270 @@
|
|||||||
|
package cmd
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/db"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeR2OutboxFailure struct {
|
||||||
|
nextRetry time.Time
|
||||||
|
lastError string
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeR2OutboxStore struct {
|
||||||
|
tasks []*db.R2DeleteTask
|
||||||
|
references map[string]bool
|
||||||
|
referenceErrs map[string]error
|
||||||
|
referenceCalls []string
|
||||||
|
deleted map[uint]bool
|
||||||
|
deleteErr error
|
||||||
|
failures map[uint]fakeR2OutboxFailure
|
||||||
|
markFailedErr error
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFakeR2OutboxStore(tasks ...*db.R2DeleteTask) *fakeR2OutboxStore {
|
||||||
|
return &fakeR2OutboxStore{
|
||||||
|
tasks: tasks,
|
||||||
|
references: make(map[string]bool),
|
||||||
|
referenceErrs: make(map[string]error),
|
||||||
|
deleted: make(map[uint]bool),
|
||||||
|
failures: make(map[uint]fakeR2OutboxFailure),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *fakeR2OutboxStore) ListDue(ctx context.Context, limit int) ([]*db.R2DeleteTask, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
tasks := s.tasks
|
||||||
|
if limit >= 0 && len(tasks) > limit {
|
||||||
|
tasks = tasks[:limit]
|
||||||
|
}
|
||||||
|
return append([]*db.R2DeleteTask(nil), tasks...), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *fakeR2OutboxStore) Delete(id uint) error {
|
||||||
|
if s.deleteErr != nil {
|
||||||
|
return s.deleteErr
|
||||||
|
}
|
||||||
|
s.deleted[id] = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *fakeR2OutboxStore) MarkFailed(id uint, nextRetry time.Time, lastError string) error {
|
||||||
|
if s.markFailedErr != nil {
|
||||||
|
return s.markFailedErr
|
||||||
|
}
|
||||||
|
s.failures[id] = fakeR2OutboxFailure{nextRetry: nextRetry, lastError: lastError}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *fakeR2OutboxStore) HasReferences(ctx context.Context, _, key string) (bool, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
s.referenceCalls = append(s.referenceCalls, key)
|
||||||
|
return s.references[key], s.referenceErrs[key]
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeR2DeleteCall struct {
|
||||||
|
bucket string
|
||||||
|
key string
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeR2ObjectDeleter struct {
|
||||||
|
calls []fakeR2DeleteCall
|
||||||
|
errs map[string]error
|
||||||
|
onDelete func(context.Context, string, string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *fakeR2ObjectDeleter) DeleteObject(ctx context.Context, bucket, key string) error {
|
||||||
|
d.calls = append(d.calls, fakeR2DeleteCall{bucket: bucket, key: key})
|
||||||
|
if d.onDelete != nil {
|
||||||
|
return d.onDelete(ctx, bucket, key)
|
||||||
|
}
|
||||||
|
return d.errs[key]
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeR2HTTPStatusError int
|
||||||
|
|
||||||
|
func (e fakeR2HTTPStatusError) Error() string {
|
||||||
|
return fmt.Sprintf("R2 returned HTTP %d", int(e))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e fakeR2HTTPStatusError) HTTPStatusCode() int {
|
||||||
|
return int(e)
|
||||||
|
}
|
||||||
|
|
||||||
|
func setR2OutboxTestConfig(t *testing.T) {
|
||||||
|
t.Helper()
|
||||||
|
old := conf.Upload
|
||||||
|
conf.Upload.ImageEndpoint = "https://r2.example.test"
|
||||||
|
conf.Upload.ImageAccessID = "test-access-id"
|
||||||
|
conf.Upload.ImageAccessSecret = "test-secret"
|
||||||
|
conf.Upload.ImageBucket = "configured-bucket"
|
||||||
|
conf.Upload.ImageBucketCDNHost = "cdn.example.test"
|
||||||
|
t.Cleanup(func() {
|
||||||
|
conf.Upload = old
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessR2OutboxSuccessAndNotFound(t *testing.T) {
|
||||||
|
setR2OutboxTestConfig(t)
|
||||||
|
for _, tt := range []struct {
|
||||||
|
name string
|
||||||
|
deleteErr error
|
||||||
|
}{
|
||||||
|
{name: "success"},
|
||||||
|
{name: "not found is success", deleteErr: fakeR2HTTPStatusError(404)},
|
||||||
|
} {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
first := &db.R2DeleteTask{ID: 1, Bucket: conf.Upload.ImageBucket, Key: "first.webp"}
|
||||||
|
second := &db.R2DeleteTask{ID: 2, Bucket: conf.Upload.ImageBucket, Key: "second.webp"}
|
||||||
|
store := newFakeR2OutboxStore(first, second)
|
||||||
|
deleter := &fakeR2ObjectDeleter{errs: map[string]error{"second.webp": tt.deleteErr}}
|
||||||
|
|
||||||
|
result, err := processR2Outbox(context.Background(), store, deleter, 100)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, 2, result.Listed)
|
||||||
|
require.Equal(t, 2, result.Removed)
|
||||||
|
require.Equal(t, 2, result.ObjectsDeleted)
|
||||||
|
require.Zero(t, result.Failed)
|
||||||
|
require.Zero(t, result.Remaining)
|
||||||
|
require.Equal(t, []fakeR2DeleteCall{
|
||||||
|
{bucket: conf.Upload.ImageBucket, key: "first.webp"},
|
||||||
|
{bucket: conf.Upload.ImageBucket, key: "second.webp"},
|
||||||
|
}, deleter.calls, "任务必须按 ListDue 顺序串行处理")
|
||||||
|
require.True(t, store.deleted[first.ID])
|
||||||
|
require.True(t, store.deleted[second.ID])
|
||||||
|
require.Empty(t, store.failures)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessR2OutboxReferencedTaskDeletesNoObject(t *testing.T) {
|
||||||
|
setR2OutboxTestConfig(t)
|
||||||
|
task := &db.R2DeleteTask{ID: 11, Bucket: conf.Upload.ImageBucket, Key: "shared.webp"}
|
||||||
|
store := newFakeR2OutboxStore(task)
|
||||||
|
store.references[task.Key] = true
|
||||||
|
deleter := &fakeR2ObjectDeleter{}
|
||||||
|
|
||||||
|
result, err := processR2Outbox(context.Background(), store, deleter, 100)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Empty(t, deleter.calls, "仍有引用时绝不能调用 R2 删除")
|
||||||
|
require.True(t, store.deleted[task.ID], "过期任务本身应删除,最后引用移除时会重新入队")
|
||||||
|
require.Equal(t, 1, result.ReferenceOnly)
|
||||||
|
require.Equal(t, 1, result.Removed)
|
||||||
|
require.Zero(t, result.ObjectsDeleted)
|
||||||
|
require.Zero(t, result.Remaining)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessR2OutboxReferenceErrorSchedulesRetry(t *testing.T) {
|
||||||
|
setR2OutboxTestConfig(t)
|
||||||
|
task := &db.R2DeleteTask{ID: 21, Bucket: conf.Upload.ImageBucket, Key: "unknown.webp"}
|
||||||
|
store := newFakeR2OutboxStore(task)
|
||||||
|
store.referenceErrs[task.Key] = errors.New("reference query failed")
|
||||||
|
deleter := &fakeR2ObjectDeleter{}
|
||||||
|
before := time.Now()
|
||||||
|
|
||||||
|
result, err := processR2Outbox(context.Background(), store, deleter, 100)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Empty(t, deleter.calls)
|
||||||
|
require.False(t, store.deleted[task.ID])
|
||||||
|
require.Contains(t, store.failures, task.ID)
|
||||||
|
failure := store.failures[task.ID]
|
||||||
|
require.Equal(t, "reference query failed", failure.lastError)
|
||||||
|
require.WithinDuration(t, before.Add(time.Minute), failure.nextRetry, 2*time.Second)
|
||||||
|
require.Equal(t, 1, result.Failed)
|
||||||
|
require.Equal(t, 1, result.Remaining)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessR2OutboxDeleteFailureUsesBackoffAndBoundsLastError(t *testing.T) {
|
||||||
|
setR2OutboxTestConfig(t)
|
||||||
|
task := &db.R2DeleteTask{ID: 31, Bucket: conf.Upload.ImageBucket, Key: "retry.webp", Attempts: 2}
|
||||||
|
store := newFakeR2OutboxStore(task)
|
||||||
|
longErr := fmt.Errorf("delete failed endpoint=%s credential=%s %s",
|
||||||
|
conf.Upload.ImageEndpoint,
|
||||||
|
conf.Upload.ImageAccessSecret,
|
||||||
|
strings.Repeat("x", 1100),
|
||||||
|
)
|
||||||
|
deleter := &fakeR2ObjectDeleter{errs: map[string]error{task.Key: longErr}}
|
||||||
|
before := time.Now()
|
||||||
|
|
||||||
|
result, err := processR2Outbox(context.Background(), store, deleter, 100)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, deleter.calls, 1)
|
||||||
|
require.False(t, store.deleted[task.ID])
|
||||||
|
require.Contains(t, store.failures, task.ID)
|
||||||
|
failure := store.failures[task.ID]
|
||||||
|
require.WithinDuration(t, before.Add(4*time.Minute), failure.nextRetry, 2*time.Second)
|
||||||
|
require.LessOrEqual(t, len(failure.lastError), r2OutboxMaxLastErrorSize)
|
||||||
|
require.NotContains(t, failure.lastError, conf.Upload.ImageEndpoint)
|
||||||
|
require.NotContains(t, failure.lastError, conf.Upload.ImageAccessSecret)
|
||||||
|
require.Equal(t, 1, result.Failed)
|
||||||
|
require.Equal(t, 1, result.Remaining)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessR2OutboxBucketMismatchIsRetainedWithoutDeletion(t *testing.T) {
|
||||||
|
setR2OutboxTestConfig(t)
|
||||||
|
task := &db.R2DeleteTask{ID: 41, Bucket: "different-bucket", Key: "foreign.webp"}
|
||||||
|
store := newFakeR2OutboxStore(task)
|
||||||
|
deleter := &fakeR2ObjectDeleter{}
|
||||||
|
|
||||||
|
result, err := processR2Outbox(context.Background(), store, deleter, 100)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Empty(t, store.referenceCalls, "bucket 不匹配时不应查询当前 bucket 的引用")
|
||||||
|
require.Empty(t, deleter.calls)
|
||||||
|
require.Empty(t, store.failures)
|
||||||
|
require.False(t, store.deleted[task.ID])
|
||||||
|
require.Equal(t, 1, result.Skipped)
|
||||||
|
require.Zero(t, result.Failed)
|
||||||
|
require.Equal(t, 1, result.Remaining)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessR2OutboxContextCancellationStopsWithoutMarkFailed(t *testing.T) {
|
||||||
|
setR2OutboxTestConfig(t)
|
||||||
|
first := &db.R2DeleteTask{ID: 51, Bucket: conf.Upload.ImageBucket, Key: "first.webp"}
|
||||||
|
second := &db.R2DeleteTask{ID: 52, Bucket: conf.Upload.ImageBucket, Key: "second.webp"}
|
||||||
|
store := newFakeR2OutboxStore(first, second)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
deleter := &fakeR2ObjectDeleter{
|
||||||
|
onDelete: func(ctx context.Context, _, _ string) error {
|
||||||
|
cancel()
|
||||||
|
return ctx.Err()
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := processR2Outbox(ctx, store, deleter, 100)
|
||||||
|
require.ErrorIs(t, err, context.Canceled)
|
||||||
|
require.Len(t, deleter.calls, 1, "取消后不得继续处理第二个任务")
|
||||||
|
require.Empty(t, store.failures, "优雅关闭导致的取消不应增加任务失败次数")
|
||||||
|
require.Empty(t, store.deleted)
|
||||||
|
require.Zero(t, result.Failed)
|
||||||
|
require.Equal(t, 2, result.Remaining)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestR2OutboxBackoffCapsAtOneHour(t *testing.T) {
|
||||||
|
for _, tt := range []struct {
|
||||||
|
attempts int
|
||||||
|
expected time.Duration
|
||||||
|
}{
|
||||||
|
{attempts: -1, expected: time.Minute},
|
||||||
|
{attempts: 0, expected: time.Minute},
|
||||||
|
{attempts: 1, expected: 2 * time.Minute},
|
||||||
|
{attempts: 2, expected: 4 * time.Minute},
|
||||||
|
{attempts: 5, expected: 32 * time.Minute},
|
||||||
|
{attempts: 6, expected: time.Hour},
|
||||||
|
{attempts: 100, expected: time.Hour},
|
||||||
|
} {
|
||||||
|
require.Equal(t, tt.expected, r2OutboxBackoff(tt.attempts))
|
||||||
|
}
|
||||||
|
}
|
||||||
+10
-1
@@ -10,6 +10,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
|
"sync"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -209,7 +210,15 @@ func runWeb(ctx *cli.Context) error {
|
|||||||
// 「提问已 INSERT、图片还没传完」「回答已写入、配额还没扣」这类半成品。
|
// 「提问已 INSERT、图片还没传完」「回答已写入、配额还没扣」这类半成品。
|
||||||
// systemd Restart=always 与 docker restart 都救不了这个。
|
// systemd Restart=always 与 docker restart 都救不了这个。
|
||||||
runCtx, stop := signal.NotifyContext(ctx.Context, syscall.SIGTERM, os.Interrupt)
|
runCtx, stop := signal.NotifyContext(ctx.Context, syscall.SIGTERM, os.Interrupt)
|
||||||
defer stop()
|
var r2WorkerWG sync.WaitGroup
|
||||||
|
defer func() {
|
||||||
|
// 先取消 worker 的 context,再等待当前一轮退出,避免无人等待的后台删除。
|
||||||
|
stop()
|
||||||
|
r2WorkerWG.Wait()
|
||||||
|
}()
|
||||||
|
|
||||||
|
// db.Init 已成功;R2 未完整配置时此函数直接返回,不影响 Web 启动。
|
||||||
|
startR2OutboxWorker(runCtx, &r2WorkerWG)
|
||||||
|
|
||||||
serveErr := make(chan error, 1)
|
serveErr := make(chan error, 1)
|
||||||
go func() {
|
go func() {
|
||||||
|
|||||||
@@ -6,22 +6,30 @@ package db
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
"github.com/sirupsen/logrus"
|
"github.com/sirupsen/logrus"
|
||||||
"github.com/thanhpk/randstr"
|
"github.com/thanhpk/randstr"
|
||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
|
||||||
"github.com/tamakyi/TamaBox/internal/dbutil"
|
"github.com/tamakyi/TamaBox/internal/dbutil"
|
||||||
)
|
)
|
||||||
|
|
||||||
var Conversations ConversationStore
|
var Conversations ConversationStore
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrDailyMessageLimit = errors.New("已达到每日私聊上限")
|
||||||
|
ErrConversationLimit = errors.New("会话已达到消息上限或已关闭")
|
||||||
|
)
|
||||||
|
|
||||||
type ConversationStore interface {
|
type ConversationStore interface {
|
||||||
Create(ctx context.Context, opts CreateConversationOptions) (*Conversation, error)
|
Create(ctx context.Context, opts CreateConversationOptions) (*Conversation, error)
|
||||||
// CreateWithFirstMessage 事务化地建会话 + 首条消息 + 消息计数。
|
// CreateWithFirstMessage 事务化地建会话 + 首条消息 + 消息计数。
|
||||||
// 单独写首条消息(用于「发第一条私聊」这类一次性动作),避免失败时留下空会话。
|
// 单独写首条消息(用于「发第一条私聊」这类一次性动作),避免失败时留下空会话。
|
||||||
CreateWithFirstMessage(ctx context.Context, opts CreateConversationOptions, senderID uint, content string) (*Conversation, *Message, error)
|
CreateWithFirstMessage(ctx context.Context, opts CreateConversationOptions, senderID uint, content string) (*Conversation, *Message, error)
|
||||||
|
SendMessage(ctx context.Context, opts SendConversationMessageOptions) (*Message, error)
|
||||||
GetByID(ctx context.Context, id uint) (*Conversation, error)
|
GetByID(ctx context.Context, id uint) (*Conversation, error)
|
||||||
GetByToken(ctx context.Context, token string) (*Conversation, error)
|
GetByToken(ctx context.Context, token string) (*Conversation, error)
|
||||||
GetByIDAndOwner(ctx context.Context, id, ownerID uint) (*Conversation, error)
|
GetByIDAndOwner(ctx context.Context, id, ownerID uint) (*Conversation, error)
|
||||||
@@ -74,6 +82,39 @@ type CreateConversationOptions struct {
|
|||||||
MaxMessages int
|
MaxMessages int
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type SendConversationMessageOptions struct {
|
||||||
|
ConversationID uint
|
||||||
|
SenderID uint
|
||||||
|
Content string
|
||||||
|
Image *CreateMessageImageOptions
|
||||||
|
}
|
||||||
|
|
||||||
|
func reserveDailyMessageTx(tx *gorm.DB, ownerID, senderID uint) error {
|
||||||
|
var sender User
|
||||||
|
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).Select("id").First(&sender, senderID).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "lock message sender")
|
||||||
|
}
|
||||||
|
var owner User
|
||||||
|
if err := tx.Select("daily_message_limit").First(&owner, ownerID).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "get owner daily message limit")
|
||||||
|
}
|
||||||
|
if owner.DailyMessageLimit <= 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
now := time.Now()
|
||||||
|
start := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, now.Location())
|
||||||
|
var count int64
|
||||||
|
if err := tx.Table("messages").Joins("JOIN conversations ON conversations.id = messages.conversation_id").
|
||||||
|
Where("conversations.owner_id = ? AND messages.sender_id = ? AND messages.created_at >= ?", ownerID, senderID, start).
|
||||||
|
Count(&count).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "count today messages")
|
||||||
|
}
|
||||||
|
if count >= int64(owner.DailyMessageLimit) {
|
||||||
|
return ErrDailyMessageLimit
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (db *conversations) Create(ctx context.Context, opts CreateConversationOptions) (*Conversation, error) {
|
func (db *conversations) Create(ctx context.Context, opts CreateConversationOptions) (*Conversation, error) {
|
||||||
conversation := &Conversation{
|
conversation := &Conversation{
|
||||||
OwnerID: opts.OwnerID,
|
OwnerID: opts.OwnerID,
|
||||||
@@ -99,6 +140,11 @@ func (db *conversations) CreateWithFirstMessage(
|
|||||||
var msg *Message
|
var msg *Message
|
||||||
|
|
||||||
err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||||
|
if senderID == opts.AskerID {
|
||||||
|
if err := reserveDailyMessageTx(tx, opts.OwnerID, senderID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
conv = &Conversation{
|
conv = &Conversation{
|
||||||
OwnerID: opts.OwnerID,
|
OwnerID: opts.OwnerID,
|
||||||
AskerID: opts.AskerID,
|
AskerID: opts.AskerID,
|
||||||
@@ -139,6 +185,59 @@ func (db *conversations) CreateWithFirstMessage(
|
|||||||
return conv, msg, nil
|
return conv, msg, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (db *conversations) SendMessage(ctx context.Context, opts SendConversationMessageOptions) (*Message, error) {
|
||||||
|
msg := &Message{ConversationID: opts.ConversationID, SenderID: opts.SenderID, Content: opts.Content}
|
||||||
|
err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||||
|
// 先锁发送者,再锁会话,保证与 CreateWithFirstMessage 的锁顺序一致,
|
||||||
|
// 避免并发下两条路径互相等待造成配额检查窗口。
|
||||||
|
var sender User
|
||||||
|
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).Select("id").First(&sender, opts.SenderID).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "lock message sender")
|
||||||
|
}
|
||||||
|
|
||||||
|
var conv Conversation
|
||||||
|
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).First(&conv, opts.ConversationID).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "lock conversation")
|
||||||
|
}
|
||||||
|
|
||||||
|
// 发送者必须是会话双方之一
|
||||||
|
if conv.OwnerID != opts.SenderID && conv.AskerID != opts.SenderID {
|
||||||
|
return errors.New("sender is not part of the conversation")
|
||||||
|
}
|
||||||
|
|
||||||
|
// 只有 asker 发送才消耗 owner 的每日配额
|
||||||
|
if conv.AskerID == opts.SenderID {
|
||||||
|
if err := reserveDailyMessageTx(tx, conv.OwnerID, opts.SenderID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
res := tx.Model(&Conversation{}).Where("id = ? AND message_count < max_messages AND is_closed = ?", conv.ID, false).
|
||||||
|
UpdateColumn("message_count", gorm.Expr("message_count + 1"))
|
||||||
|
if res.Error != nil {
|
||||||
|
return errors.Wrap(res.Error, "increment message count")
|
||||||
|
}
|
||||||
|
if res.RowsAffected == 0 {
|
||||||
|
return ErrConversationLimit
|
||||||
|
}
|
||||||
|
if err := tx.Create(msg).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "create message")
|
||||||
|
}
|
||||||
|
if opts.Image != nil {
|
||||||
|
image := *opts.Image
|
||||||
|
image.MessageID = msg.ID
|
||||||
|
if err := tx.Create(&MessageImage{MessageID: image.MessageID, URL: image.URL, ObjectKey: image.ObjectKey, FileName: image.FileName, FileSize: image.FileSize}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "create message image")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return msg, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (db *conversations) GetByID(ctx context.Context, id uint) (*Conversation, error) {
|
func (db *conversations) GetByID(ctx context.Context, id uint) (*Conversation, error) {
|
||||||
var conv Conversation
|
var conv Conversation
|
||||||
if err := db.WithContext(ctx).First(&conv, id).Error; err != nil {
|
if err := db.WithContext(ctx).First(&conv, id).Error; err != nil {
|
||||||
|
|||||||
+4
-1
@@ -36,7 +36,7 @@ func Ping(ctx context.Context) error {
|
|||||||
|
|
||||||
var AllTables = []interface{}{
|
var AllTables = []interface{}{
|
||||||
&User{}, &Question{}, &FollowUp{}, &CensorLog{}, &UploadImage{}, &UploadImageQuestion{},
|
&User{}, &Question{}, &FollowUp{}, &CensorLog{}, &UploadImage{}, &UploadImageQuestion{},
|
||||||
&Conversation{}, &Message{}, &MessageImage{}, &UserBlock{},
|
&Conversation{}, &Message{}, &MessageImage{}, &UserBlock{}, &R2DeleteTask{},
|
||||||
// 匿名投票
|
// 匿名投票
|
||||||
&Vote{}, &VoteOption{}, &VoteFormField{}, &VoteSubmission{},
|
&Vote{}, &VoteOption{}, &VoteFormField{}, &VoteSubmission{},
|
||||||
// 功能开关(提问 / 私聊 / 投票 的开关、定时、定量收集)
|
// 功能开关(提问 / 私聊 / 投票 的开关、定时、定量收集)
|
||||||
@@ -68,6 +68,7 @@ func Init(typ, dsn string) (*gorm.DB, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
db, err := gorm.Open(dialector, &gorm.Config{
|
db, err := gorm.Open(dialector, &gorm.Config{
|
||||||
|
Logger: newSafeLogger(),
|
||||||
SkipDefaultTransaction: true,
|
SkipDefaultTransaction: true,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -131,6 +132,7 @@ func Init(typ, dsn string) (*gorm.DB, error) {
|
|||||||
Conversations = NewConversationStore(db)
|
Conversations = NewConversationStore(db)
|
||||||
Messages = NewMessageStore(db)
|
Messages = NewMessageStore(db)
|
||||||
MessageImages = NewMessageImageStore(db)
|
MessageImages = NewMessageImageStore(db)
|
||||||
|
R2DeleteTasks = NewR2DeleteTaskStore(db)
|
||||||
Votes = NewVotesStore(db)
|
Votes = NewVotesStore(db)
|
||||||
VoteOptions = NewVoteOptionsStore(db)
|
VoteOptions = NewVoteOptionsStore(db)
|
||||||
VoteFormFields = NewVoteFormFieldsStore(db)
|
VoteFormFields = NewVoteFormFieldsStore(db)
|
||||||
@@ -155,6 +157,7 @@ func Init(typ, dsn string) (*gorm.DB, error) {
|
|||||||
|
|
||||||
if err := db.Use(otelgorm.NewPlugin(
|
if err := db.Use(otelgorm.NewPlugin(
|
||||||
otelgorm.WithDBName(conf.Database.Name),
|
otelgorm.WithDBName(conf.Database.Name),
|
||||||
|
otelgorm.WithoutQueryVariables(),
|
||||||
)); err != nil {
|
)); err != nil {
|
||||||
return nil, errors.Wrap(err, "register otelgorm plugin")
|
return nil, errors.Wrap(err, "register otelgorm plugin")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -20,6 +20,8 @@ import (
|
|||||||
|
|
||||||
var FeatureControls FeatureControlsStore
|
var FeatureControls FeatureControlsStore
|
||||||
|
|
||||||
|
var ErrFeatureQuotaReached = errors.New("功能收集配额已满")
|
||||||
|
|
||||||
var _ FeatureControlsStore = (*featureControls)(nil)
|
var _ FeatureControlsStore = (*featureControls)(nil)
|
||||||
|
|
||||||
// 受控功能标识(feature_controls.feature 取值)。
|
// 受控功能标识(feature_controls.feature 取值)。
|
||||||
@@ -517,6 +519,21 @@ func (db *featureControls) IncrementQuota(ctx context.Context, userID uint, feat
|
|||||||
// 于是计数比真实行数少,箱主的「已收集 7/10」永远偏低,达到上限后仍能继续收,
|
// 于是计数比真实行数少,箱主的「已收集 7/10」永远偏低,达到上限后仍能继续收,
|
||||||
// 而重置计数也校准不回来(重置是把计数清零,与实际行数更脱节)。
|
// 而重置计数也校准不回来(重置是把计数清零,与实际行数更脱节)。
|
||||||
// 现在内容 store 把这一步放进自己的事务里,任一步失败整体回滚,计数天然对齐。
|
// 现在内容 store 把这一步放进自己的事务里,任一步失败整体回滚,计数天然对齐。
|
||||||
|
func reserveFeatureQuotaTx(tx *gorm.DB, userID uint, feature string) error {
|
||||||
|
if err := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(DefaultFeatureControl(userID, feature)).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "initialize feature quota")
|
||||||
|
}
|
||||||
|
var fc FeatureControl
|
||||||
|
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
|
||||||
|
Where("user_id = ? AND feature = ?", userID, feature).First(&fc).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "lock feature quota")
|
||||||
|
}
|
||||||
|
if fc.QuotaLimit > 0 && fc.QuotaCount >= fc.QuotaLimit {
|
||||||
|
return ErrFeatureQuotaReached
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func incrementQuotaTx(tx *gorm.DB, userID uint, feature string, delta int64) error {
|
func incrementQuotaTx(tx *gorm.DB, userID uint, feature string, delta int64) error {
|
||||||
if delta == 0 {
|
if delta == 0 {
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -0,0 +1,79 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/sirupsen/logrus"
|
||||||
|
"gorm.io/gorm/logger"
|
||||||
|
)
|
||||||
|
|
||||||
|
type safeLogger struct {
|
||||||
|
logger.Config
|
||||||
|
}
|
||||||
|
|
||||||
|
func newSafeLogger() logger.Interface {
|
||||||
|
return &safeLogger{Config: logger.Config{
|
||||||
|
SlowThreshold: 200 * time.Millisecond,
|
||||||
|
LogLevel: logger.Warn,
|
||||||
|
IgnoreRecordNotFoundError: false,
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *safeLogger) LogMode(level logger.LogLevel) logger.Interface {
|
||||||
|
newLogger := *l
|
||||||
|
newLogger.LogLevel = level
|
||||||
|
return &newLogger
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *safeLogger) Info(ctx context.Context, _ string, _ ...interface{}) {
|
||||||
|
if l.LogLevel >= logger.Info {
|
||||||
|
logrus.WithContext(ctx).WithField("component", "gorm").Info("数据库信息事件")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *safeLogger) Warn(ctx context.Context, _ string, _ ...interface{}) {
|
||||||
|
if l.LogLevel >= logger.Warn {
|
||||||
|
logrus.WithContext(ctx).WithField("component", "gorm").Warn("数据库警告事件")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *safeLogger) Error(ctx context.Context, _ string, _ ...interface{}) {
|
||||||
|
if l.LogLevel >= logger.Error {
|
||||||
|
logrus.WithContext(ctx).WithField("component", "gorm").Error("数据库错误事件")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *safeLogger) Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error) {
|
||||||
|
if l.LogLevel <= logger.Silent {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
elapsed := time.Since(begin)
|
||||||
|
switch {
|
||||||
|
case err != nil && l.LogLevel >= logger.Error && (!errors.Is(err, logger.ErrRecordNotFound) || !l.IgnoreRecordNotFoundError):
|
||||||
|
_, rows := fc()
|
||||||
|
l.traceEntry(ctx, elapsed, rows).WithFields(logrus.Fields{
|
||||||
|
"error_type": fmt.Sprintf("%T", err),
|
||||||
|
"record_not_found": errors.Is(err, logger.ErrRecordNotFound),
|
||||||
|
}).Error("数据库操作失败")
|
||||||
|
case elapsed > l.SlowThreshold && l.SlowThreshold != 0 && l.LogLevel >= logger.Warn:
|
||||||
|
_, rows := fc()
|
||||||
|
l.traceEntry(ctx, elapsed, rows).
|
||||||
|
WithField("slow_threshold", l.SlowThreshold.String()).
|
||||||
|
Warn("数据库慢查询")
|
||||||
|
case l.LogLevel >= logger.Info:
|
||||||
|
_, rows := fc()
|
||||||
|
l.traceEntry(ctx, elapsed, rows).Info("数据库操作完成")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *safeLogger) traceEntry(ctx context.Context, elapsed time.Duration, rows int64) *logrus.Entry {
|
||||||
|
return logrus.WithContext(ctx).WithFields(logrus.Fields{
|
||||||
|
"component": "gorm",
|
||||||
|
"elapsed_ms": float64(elapsed.Nanoseconds()) / 1e6,
|
||||||
|
"rows_affected": rows,
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -26,6 +26,7 @@ type MessageImage struct {
|
|||||||
gorm.Model
|
gorm.Model
|
||||||
MessageID uint `gorm:"index" json:"message_id"`
|
MessageID uint `gorm:"index" json:"message_id"`
|
||||||
URL string `gorm:"size:500" json:"url"`
|
URL string `gorm:"size:500" json:"url"`
|
||||||
|
ObjectKey string `gorm:"type:text" json:"-"`
|
||||||
FileName string `gorm:"size:200" json:"file_name"`
|
FileName string `gorm:"size:200" json:"file_name"`
|
||||||
FileSize int64 `json:"file_size"`
|
FileSize int64 `json:"file_size"`
|
||||||
}
|
}
|
||||||
@@ -33,6 +34,7 @@ type MessageImage struct {
|
|||||||
type CreateMessageImageOptions struct {
|
type CreateMessageImageOptions struct {
|
||||||
MessageID uint
|
MessageID uint
|
||||||
URL string
|
URL string
|
||||||
|
ObjectKey string
|
||||||
FileName string
|
FileName string
|
||||||
FileSize int64
|
FileSize int64
|
||||||
}
|
}
|
||||||
@@ -41,6 +43,7 @@ func (db *messageImages) Create(ctx context.Context, opts CreateMessageImageOpti
|
|||||||
img := &MessageImage{
|
img := &MessageImage{
|
||||||
MessageID: opts.MessageID,
|
MessageID: opts.MessageID,
|
||||||
URL: opts.URL,
|
URL: opts.URL,
|
||||||
|
ObjectKey: opts.ObjectKey,
|
||||||
FileName: opts.FileName,
|
FileName: opts.FileName,
|
||||||
FileSize: opts.FileSize,
|
FileSize: opts.FileSize,
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMessageImageCreatePersistsObjectKey(t *testing.T) {
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
store := NewMessageImageStore(conn)
|
||||||
|
|
||||||
|
created, err := store.Create(ctx, CreateMessageImageOptions{
|
||||||
|
MessageID: 42,
|
||||||
|
URL: "https://cdn.example.test/assets/2026/11/private.webp",
|
||||||
|
ObjectKey: "2026/11/private.webp",
|
||||||
|
FileName: "private.webp",
|
||||||
|
FileSize: 1234,
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "2026/11/private.webp", created.ObjectKey)
|
||||||
|
|
||||||
|
var persisted MessageImage
|
||||||
|
require.NoError(t, conn.WithContext(ctx).First(&persisted, created.ID).Error)
|
||||||
|
require.Equal(t, created.ObjectKey, persisted.ObjectKey)
|
||||||
|
byMessage, err := store.GetByMessageIDs(ctx, []uint{42})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, byMessage[42], 1)
|
||||||
|
require.Equal(t, created.ObjectKey, byMessage[42][0].ObjectKey)
|
||||||
|
|
||||||
|
encoded, err := json.Marshal(&persisted)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NotContains(t, string(encoded), "ObjectKey")
|
||||||
|
require.NotContains(t, string(encoded), "object_key")
|
||||||
|
|
||||||
|
}
|
||||||
@@ -67,6 +67,13 @@ var migrations = []Migration{
|
|||||||
Name: "normalize harassment setting",
|
Name: "normalize harassment setting",
|
||||||
Up: normalizeHarassmentSetting,
|
Up: normalizeHarassmentSetting,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
// 新增列 AutoMigrate 只补 0,而 0 与「无授权」的零值撞车 —— 必须回填,
|
||||||
|
// 否则升级后存量用户的页面密码会被静默绕过。
|
||||||
|
Version: 6,
|
||||||
|
Name: "backfill page password version",
|
||||||
|
Up: backfillPagePasswordVer,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
// runMigrations 按版本号升序执行尚未应用的迁移。
|
// runMigrations 按版本号升序执行尚未应用的迁移。
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestBackfillPagePasswordVer 覆盖 2026-11-05 那个 P0 的数据面:
|
||||||
|
// 新增列 AutoMigrate 只补 0,而 0 恰好等于「session 里没有授权记录」的零值,
|
||||||
|
// 不回填的话存量页面密码在升级瞬间全部失效。
|
||||||
|
func TestBackfillPagePasswordVer(t *testing.T) {
|
||||||
|
conn, clean := newTestDB(t)
|
||||||
|
defer clean("users")
|
||||||
|
|
||||||
|
mustCreate := func(domain string, pagePassword string, pageVer uint, askPassword string, askVer uint) *User {
|
||||||
|
u := &User{
|
||||||
|
Name: domain,
|
||||||
|
Email: domain + "@example.com",
|
||||||
|
Password: "x",
|
||||||
|
Domain: domain,
|
||||||
|
PagePassword: pagePassword,
|
||||||
|
PagePasswordVer: pageVer,
|
||||||
|
AskPassword: askPassword,
|
||||||
|
AskPasswordVer: askVer,
|
||||||
|
}
|
||||||
|
require.NoError(t, conn.Create(u).Error)
|
||||||
|
return u
|
||||||
|
}
|
||||||
|
|
||||||
|
// 存量:有密码但版本号仍为默认 0 —— 必须被回填。
|
||||||
|
legacy := mustCreate("legacy", "hashed-page", 0, "hashed-ask", 0)
|
||||||
|
// 已轮换过密码:版本号非 0,不能被重置。
|
||||||
|
rotated := mustCreate("rotated", "hashed-page", 3, "hashed-ask", 5)
|
||||||
|
// 没有密码:版本号保持 0,回填不应凭空造出授权。
|
||||||
|
noPassword := mustCreate("nopassword", "", 0, "", 0)
|
||||||
|
|
||||||
|
require.NoError(t, backfillPagePasswordVer(conn))
|
||||||
|
|
||||||
|
var gotLegacy, gotRotated, gotNoPassword User
|
||||||
|
require.NoError(t, conn.First(&gotLegacy, legacy.ID).Error)
|
||||||
|
require.NoError(t, conn.First(&gotRotated, rotated.ID).Error)
|
||||||
|
require.NoError(t, conn.First(&gotNoPassword, noPassword.ID).Error)
|
||||||
|
|
||||||
|
require.EqualValues(t, 1, gotLegacy.PagePasswordVer, "存量页面密码应回填为 1")
|
||||||
|
require.EqualValues(t, 1, gotLegacy.AskPasswordVer, "存量提问密码应回填为 1")
|
||||||
|
|
||||||
|
require.EqualValues(t, 3, gotRotated.PagePasswordVer, "已轮换的版本号不能被覆盖")
|
||||||
|
require.EqualValues(t, 5, gotRotated.AskPasswordVer, "已轮换的版本号不能被覆盖")
|
||||||
|
|
||||||
|
require.EqualValues(t, 0, gotNoPassword.PagePasswordVer, "没设密码的账号不应被写版本号")
|
||||||
|
require.EqualValues(t, 0, gotNoPassword.AskPasswordVer, "没设密码的账号不应被写版本号")
|
||||||
|
|
||||||
|
// 幂等:再跑一次不应把已回填的行推到 2。
|
||||||
|
require.NoError(t, backfillPagePasswordVer(conn))
|
||||||
|
require.NoError(t, conn.First(&gotLegacy, legacy.ID).Error)
|
||||||
|
require.EqualValues(t, 1, gotLegacy.PagePasswordVer, "迁移必须幂等")
|
||||||
|
}
|
||||||
@@ -128,6 +128,9 @@ func (db *questions) Create(ctx context.Context, opts CreateQuestionOptions) (*Q
|
|||||||
IsPrivate: opts.IsPrivate,
|
IsPrivate: opts.IsPrivate,
|
||||||
}
|
}
|
||||||
err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||||
|
if err := reserveFeatureQuotaTx(tx, opts.UserID, FeatureQuestion); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
if err := tx.Create(&question).Error; err != nil {
|
if err := tx.Create(&question).Error; err != nil {
|
||||||
return errors.Wrap(err, "create question")
|
return errors.Wrap(err, "create question")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,163 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func createQuotaTestUser(t *testing.T, ctx context.Context, conn *gorm.DB, name string, dailyLimit int) *User {
|
||||||
|
t.Helper()
|
||||||
|
user := &User{Name: name, Domain: name, Email: name + "@example.test", Password: "test-only", DailyMessageLimit: dailyLimit}
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(user).Error)
|
||||||
|
return user
|
||||||
|
}
|
||||||
|
|
||||||
|
func setFeatureQuota(t *testing.T, ctx context.Context, conn *gorm.DB, userID uint, feature string, limit int64) {
|
||||||
|
t.Helper()
|
||||||
|
require.NoError(t, NewFeatureControlsStore(conn).Save(ctx, userID, feature, UpdateFeatureControlOptions{Enabled: true, QuotaLimit: limit}))
|
||||||
|
}
|
||||||
|
|
||||||
|
func runConcurrently(n int, fn func(int) error) []error {
|
||||||
|
start := make(chan struct{})
|
||||||
|
errs := make([]error, n)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := range n {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
<-start
|
||||||
|
errs[i] = fn(i)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
close(start)
|
||||||
|
wg.Wait()
|
||||||
|
return errs
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireOnlyQuotaErrors(t *testing.T, errs []error, quotaErr error) int {
|
||||||
|
t.Helper()
|
||||||
|
successes := 0
|
||||||
|
for _, err := range errs {
|
||||||
|
if err == nil {
|
||||||
|
successes++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
require.ErrorIs(t, err, quotaErr)
|
||||||
|
}
|
||||||
|
return successes
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestQuestionQuotaConcurrentLastSlot(t *testing.T) {
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
owner := createQuotaTestUser(t, ctx, conn, "quota-question-owner", 0)
|
||||||
|
setFeatureQuota(t, ctx, conn, owner.ID, FeatureQuestion, 1)
|
||||||
|
questions := NewQuestionsStore(conn)
|
||||||
|
errs := runConcurrently(12, func(i int) error {
|
||||||
|
_, err := questions.Create(ctx, CreateQuestionOptions{UserID: owner.ID, Content: fmt.Sprintf("question-%d", i)})
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
require.Equal(t, 1, requireOnlyQuotaErrors(t, errs, ErrFeatureQuotaReached))
|
||||||
|
var count int64
|
||||||
|
require.NoError(t, conn.Model(&Question{}).Where("user_id = ?", owner.ID).Count(&count).Error)
|
||||||
|
require.Equal(t, int64(1), count)
|
||||||
|
fc, err := NewFeatureControlsStore(conn).Get(ctx, owner.ID, FeatureQuestion)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, int64(1), fc.QuotaCount)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVoteQuotasConcurrentLastSlot(t *testing.T) {
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
owner := createQuotaTestUser(t, ctx, conn, "quota-vote-owner", 0)
|
||||||
|
setFeatureQuota(t, ctx, conn, owner.ID, FeatureVote, 1)
|
||||||
|
votes := NewVotesStore(conn)
|
||||||
|
options := NewVoteOptionsStore(conn)
|
||||||
|
submissions := NewVoteSubmissionsStore(conn)
|
||||||
|
ids := make([]uint, 2)
|
||||||
|
for i := range 2 {
|
||||||
|
vote, err := votes.Create(ctx, CreateVoteOptions{UserID: owner.ID, Title: fmt.Sprintf("vote-%d", i), Type: VoteTypeSingle, QuotaLimit: 1})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, votes.UpdateStatus(ctx, vote.ID, VoteStatusPublished))
|
||||||
|
option, err := options.Create(ctx, CreateVoteOptionOptions{VoteID: vote.ID, Content: "option"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
ids[i] = option.ID
|
||||||
|
}
|
||||||
|
errs := runConcurrently(12, func(i int) error {
|
||||||
|
token := fmt.Sprintf("voter-%d", i)
|
||||||
|
_, err := submissions.Create(ctx, CreateVoteSubmissionOptions{VoteID: uint(i%2 + 1), OptionID: ids[i%2], VoterToken: &token, QuotaUserID: owner.ID})
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
successes := 0
|
||||||
|
for _, err := range errs {
|
||||||
|
if err == nil {
|
||||||
|
successes++
|
||||||
|
} else {
|
||||||
|
require.Error(t, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
require.Equal(t, 1, successes)
|
||||||
|
var count int64
|
||||||
|
require.NoError(t, conn.Model(&VoteSubmission{}).Count(&count).Error)
|
||||||
|
require.Equal(t, int64(1), count)
|
||||||
|
fc, err := NewFeatureControlsStore(conn).Get(ctx, owner.ID, FeatureVote)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, int64(1), fc.QuotaCount)
|
||||||
|
var ended int64
|
||||||
|
require.NoError(t, conn.Model(&Vote{}).Where("status = ?", VoteStatusEnded).Count(&ended).Error)
|
||||||
|
require.Equal(t, int64(1), ended)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDailyMessageLimitAcrossConversationPaths(t *testing.T) {
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
owner := createQuotaTestUser(t, ctx, conn, "quota-chat-owner", 1)
|
||||||
|
asker := createQuotaTestUser(t, ctx, conn, "quota-chat-asker", 0)
|
||||||
|
conversations := NewConversationStore(conn)
|
||||||
|
existing, err := conversations.Create(ctx, CreateConversationOptions{OwnerID: owner.ID, AskerID: asker.ID, MaxMessages: 20})
|
||||||
|
require.NoError(t, err)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
start := make(chan struct{})
|
||||||
|
errs := make([]error, 2)
|
||||||
|
wg.Add(2)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
<-start
|
||||||
|
_, _, errs[0] = conversations.CreateWithFirstMessage(ctx, CreateConversationOptions{OwnerID: owner.ID, AskerID: asker.ID, MaxMessages: 20}, asker.ID, "first")
|
||||||
|
}()
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
<-start
|
||||||
|
_, errs[1] = conversations.SendMessage(ctx, SendConversationMessageOptions{ConversationID: existing.ID, SenderID: asker.ID, Content: "existing"})
|
||||||
|
}()
|
||||||
|
close(start)
|
||||||
|
wg.Wait()
|
||||||
|
require.Equal(t, 1, requireOnlyQuotaErrors(t, errs, ErrDailyMessageLimit))
|
||||||
|
var messages, convCount int64
|
||||||
|
require.NoError(t, conn.Model(&Message{}).Count(&messages).Error)
|
||||||
|
require.NoError(t, conn.Model(&Conversation{}).Count(&convCount).Error)
|
||||||
|
require.Equal(t, int64(1), messages)
|
||||||
|
require.Equal(t, int64(1), convCount)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOwnerReplyDoesNotConsumeAskerDailyQuota(t *testing.T) {
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
owner := createQuotaTestUser(t, ctx, conn, "quota-reply-owner", 1)
|
||||||
|
asker := createQuotaTestUser(t, ctx, conn, "quota-reply-asker", 0)
|
||||||
|
conversations := NewConversationStore(conn)
|
||||||
|
conv, _, err := conversations.CreateWithFirstMessage(ctx, CreateConversationOptions{OwnerID: owner.ID, AskerID: asker.ID, MaxMessages: 20}, asker.ID, "first")
|
||||||
|
require.NoError(t, err)
|
||||||
|
_, err = conversations.SendMessage(ctx, SendConversationMessageOptions{ConversationID: conv.ID, SenderID: owner.ID, Content: "reply"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
_, err = conversations.SendMessage(ctx, SendConversationMessageOptions{ConversationID: conv.ID, SenderID: asker.ID, Content: "over"})
|
||||||
|
require.ErrorIs(t, err, ErrDailyMessageLimit)
|
||||||
|
var count int64
|
||||||
|
require.NoError(t, conn.Model(&Message{}).Count(&count).Error)
|
||||||
|
require.Equal(t, int64(2), count)
|
||||||
|
}
|
||||||
@@ -0,0 +1,277 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/pkg/errors"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/dbutil"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
var R2DeleteTasks R2DeleteTaskStore
|
||||||
|
|
||||||
|
var _ R2DeleteTaskStore = (*r2DeleteTasks)(nil)
|
||||||
|
|
||||||
|
// R2DeleteTask 是 R2 对象删除的持久化 outbox 任务。
|
||||||
|
// 数据库事务只负责登记任务,实际对象删除由 worker 在事务外执行。
|
||||||
|
type R2DeleteTask struct {
|
||||||
|
ID uint `gorm:"primaryKey"`
|
||||||
|
Bucket string `gorm:"size:255;not null;uniqueIndex:uk_r2_delete_tasks_bucket_key_hash,priority:1"`
|
||||||
|
Key string `gorm:"type:text;not null"`
|
||||||
|
KeyHash string `gorm:"size:64;not null;uniqueIndex:uk_r2_delete_tasks_bucket_key_hash,priority:2"`
|
||||||
|
Attempts int `gorm:"not null;default:0"`
|
||||||
|
NextRetryAt time.Time `gorm:"not null;index:idx_r2_delete_tasks_next_retry_at"`
|
||||||
|
LastError string `gorm:"type:text"`
|
||||||
|
CreatedAt time.Time
|
||||||
|
UpdatedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
type R2DeleteTaskStore interface {
|
||||||
|
Enqueue(ctx context.Context, bucket string, keys []string) error
|
||||||
|
EnqueueValues(ctx context.Context, bucket string, values []string) error
|
||||||
|
ListDue(ctx context.Context, limit int) ([]*R2DeleteTask, error)
|
||||||
|
Delete(id uint) error
|
||||||
|
MarkFailed(id uint, nextRetry time.Time, lastError string) error
|
||||||
|
// HasReferences 仅供 worker 对当前配置的 bucket 调用。
|
||||||
|
// 它只查询数据库,不执行网络请求;查询或旧 URL 解析失败时必须返回错误。
|
||||||
|
HasReferences(ctx context.Context, bucket, key string) (bool, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewR2DeleteTaskStore(db *gorm.DB) R2DeleteTaskStore {
|
||||||
|
return &r2DeleteTasks{DB: db}
|
||||||
|
}
|
||||||
|
|
||||||
|
type r2DeleteTasks struct {
|
||||||
|
*gorm.DB
|
||||||
|
}
|
||||||
|
|
||||||
|
// EnqueueR2DeleteTasks 在调用方事务中登记对象删除任务。
|
||||||
|
// 相同 (bucket,key) 重复入队会重置重试状态,使新事务对应的删除请求立即可见。
|
||||||
|
func EnqueueR2DeleteTasks(tx *gorm.DB, bucket string, keys []string) error {
|
||||||
|
if bucket == "" || len(keys) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
tasks := make([]*R2DeleteTask, 0, len(keys))
|
||||||
|
seen := make(map[string]struct{}, len(keys))
|
||||||
|
for _, key := range keys {
|
||||||
|
if key == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := seen[key]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
sum := sha256.Sum256([]byte(key))
|
||||||
|
tasks = append(tasks, &R2DeleteTask{
|
||||||
|
Bucket: bucket,
|
||||||
|
Key: key,
|
||||||
|
KeyHash: hex.EncodeToString(sum[:]),
|
||||||
|
NextRetryAt: now,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if len(tasks) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := tx.Clauses(clause.OnConflict{
|
||||||
|
Columns: []clause.Column{{Name: "bucket"}, {Name: "key_hash"}},
|
||||||
|
DoUpdates: clause.Assignments(map[string]interface{}{
|
||||||
|
"attempts": 0,
|
||||||
|
"next_retry_at": now,
|
||||||
|
"last_error": "",
|
||||||
|
"updated_at": now,
|
||||||
|
}),
|
||||||
|
}).Create(&tasks).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "enqueue R2 delete tasks")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (db *r2DeleteTasks) Enqueue(ctx context.Context, bucket string, keys []string) error {
|
||||||
|
return EnqueueR2DeleteTasks(db.WithContext(ctx), bucket, keys)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (db *r2DeleteTasks) EnqueueValues(ctx context.Context, bucket string, values []string) error {
|
||||||
|
return enqueueR2DeleteTasksForValues(db.WithContext(ctx), bucket, values)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (db *r2DeleteTasks) ListDue(ctx context.Context, limit int) ([]*R2DeleteTask, error) {
|
||||||
|
tasks := make([]*R2DeleteTask, 0)
|
||||||
|
if limit <= 0 {
|
||||||
|
return tasks, nil
|
||||||
|
}
|
||||||
|
if err := db.WithContext(ctx).
|
||||||
|
Where("next_retry_at <= ?", time.Now()).
|
||||||
|
Order("next_retry_at ASC, id ASC").
|
||||||
|
Limit(limit).
|
||||||
|
Find(&tasks).Error; err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list due R2 delete tasks")
|
||||||
|
}
|
||||||
|
return tasks, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (db *r2DeleteTasks) Delete(id uint) error {
|
||||||
|
if err := db.DB.Delete(&R2DeleteTask{}, id).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete R2 delete task")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (db *r2DeleteTasks) MarkFailed(id uint, nextRetry time.Time, lastError string) error {
|
||||||
|
if err := db.Model(&R2DeleteTask{}).Where("id = ?", id).Updates(map[string]interface{}{
|
||||||
|
"attempts": gorm.Expr("attempts + 1"),
|
||||||
|
"next_retry_at": nextRetry,
|
||||||
|
"last_error": lastError,
|
||||||
|
"updated_at": time.Now(),
|
||||||
|
}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "mark R2 delete task failed")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (db *r2DeleteTasks) HasReferences(ctx context.Context, bucket, key string) (bool, error) {
|
||||||
|
if bucket == "" || key == "" {
|
||||||
|
return false, nil
|
||||||
|
}
|
||||||
|
defaultKey, owned, err := r2ObjectKeyFromValue(key)
|
||||||
|
if err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
if owned && isDefaultProfileObject(defaultKey) {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
quotedKey := dbutil.QuoteIdentifier(db.Dialector.Name(), "key")
|
||||||
|
var count int64
|
||||||
|
if err := db.WithContext(ctx).Unscoped().Model(&UploadImage{}).
|
||||||
|
Where(quotedKey+" = ?", key).Count(&count).Error; err != nil {
|
||||||
|
return false, errors.Wrap(err, "check upload image references")
|
||||||
|
}
|
||||||
|
if count > 0 {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
count = 0
|
||||||
|
if err := db.WithContext(ctx).Unscoped().Model(&MessageImage{}).
|
||||||
|
Where("object_key = ? OR url = ?", key, key).Count(&count).Error; err != nil {
|
||||||
|
return false, errors.Wrap(err, "check message image references")
|
||||||
|
}
|
||||||
|
if count > 0 {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var uploadURLs []string
|
||||||
|
if err := db.WithContext(ctx).Unscoped().Model(&UploadImage{}).
|
||||||
|
Where(quotedKey+" LIKE ? OR "+quotedKey+" LIKE ?", "%://%", "//%").
|
||||||
|
Pluck(quotedKey, &uploadURLs).Error; err != nil {
|
||||||
|
return false, errors.Wrap(err, "list legacy upload image URLs")
|
||||||
|
}
|
||||||
|
referenced, err := hasLegacyCDNReference(uploadURLs, key)
|
||||||
|
if err != nil || referenced {
|
||||||
|
return referenced, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var messageURLs []string
|
||||||
|
if err := db.WithContext(ctx).Unscoped().Model(&MessageImage{}).
|
||||||
|
Where("url LIKE ? OR url LIKE ?", "%://%", "//%").
|
||||||
|
Pluck("url", &messageURLs).Error; err != nil {
|
||||||
|
return false, errors.Wrap(err, "list legacy message image URLs")
|
||||||
|
}
|
||||||
|
referenced, err = hasLegacyCDNReference(messageURLs, key)
|
||||||
|
if err != nil || referenced {
|
||||||
|
return referenced, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var profileUsers []*User
|
||||||
|
if err := db.WithContext(ctx).Unscoped().Model(&User{}).
|
||||||
|
Select("avatar", "background", "background_image").
|
||||||
|
Where("avatar = ? OR background = ? OR background_image = ? OR avatar LIKE ? OR background LIKE ? OR background_image LIKE ?", key, key, key, "%://%", "%://%", "%://%").
|
||||||
|
Find(&profileUsers).Error; err != nil {
|
||||||
|
return false, errors.Wrap(err, "list profile image references")
|
||||||
|
}
|
||||||
|
for _, user := range profileUsers {
|
||||||
|
referenced, err = hasLegacyCDNReference([]string{user.Avatar, user.Background, user.BackgroundImage}, key)
|
||||||
|
if err != nil || referenced {
|
||||||
|
return referenced, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isDefaultProfileObject(key string) bool {
|
||||||
|
for _, value := range []string{conf.Upload.DefaultAvatarURL, conf.Upload.DefaultBackground, conf.Upload.DefaultBackgroundImageURL} {
|
||||||
|
defaultKey, owned, err := r2ObjectKeyFromValue(value)
|
||||||
|
if err == nil && owned && defaultKey == key {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasLegacyCDNReference(values []string, targetKey string) (bool, error) {
|
||||||
|
for _, value := range values {
|
||||||
|
key, owned, err := r2ObjectKeyFromValue(value)
|
||||||
|
if err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
if owned && key == targetKey {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// r2ObjectKeyFromValue 接受当前对象 key 或旧 CDN URL。
|
||||||
|
// 外部 URL 不属于当前 CDN,安全返回 owned=false;畸形 URL 返回错误以避免误判无引用。
|
||||||
|
func r2ObjectKeyFromValue(value string) (key string, owned bool, err error) {
|
||||||
|
if value == "" {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
if !strings.Contains(value, "://") && !strings.HasPrefix(value, "//") {
|
||||||
|
return value, true, nil
|
||||||
|
}
|
||||||
|
key, owned, err = storage.ObjectKeyFromCDNURL(value, conf.Upload.ImageBucketCDNHost)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, errors.Wrap(err, "parse legacy CDN URL")
|
||||||
|
}
|
||||||
|
if !owned || key == "" {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
return key, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func enqueueR2DeleteTasksForValues(tx *gorm.DB, bucket string, values []string, excludedKeys ...string) error {
|
||||||
|
if bucket == "" || len(values) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
excluded := make(map[string]struct{}, len(excludedKeys))
|
||||||
|
for _, key := range excludedKeys {
|
||||||
|
excluded[key] = struct{}{}
|
||||||
|
}
|
||||||
|
keys := make([]string, 0, len(values))
|
||||||
|
for _, value := range values {
|
||||||
|
key, owned, err := r2ObjectKeyFromValue(value)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !owned || isDefaultProfileObject(key) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, skip := excluded[key]; skip {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
return EnqueueR2DeleteTasks(tx, bucket, keys)
|
||||||
|
}
|
||||||
@@ -0,0 +1,141 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"fmt"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
)
|
||||||
|
|
||||||
|
func setR2TestConfig(t *testing.T) {
|
||||||
|
t.Helper()
|
||||||
|
oldBucket := conf.Upload.ImageBucket
|
||||||
|
oldCDNHost := conf.Upload.ImageBucketCDNHost
|
||||||
|
conf.Upload.ImageBucket = "test-r2-bucket"
|
||||||
|
conf.Upload.ImageBucketCDNHost = "cdn.example.test/assets"
|
||||||
|
oldDefaults := []string{conf.Upload.DefaultAvatarURL, conf.Upload.DefaultBackground, conf.Upload.DefaultBackgroundImageURL}
|
||||||
|
conf.Upload.DefaultAvatarURL = "https://cdn.example.test/assets/default-avatar.webp"
|
||||||
|
conf.Upload.DefaultBackground = "https://cdn.example.test/assets/default-background.webp"
|
||||||
|
conf.Upload.DefaultBackgroundImageURL = "https://cdn.example.test/assets/default-qr.webp"
|
||||||
|
t.Cleanup(func() {
|
||||||
|
conf.Upload.DefaultAvatarURL, conf.Upload.DefaultBackground, conf.Upload.DefaultBackgroundImageURL = oldDefaults[0], oldDefaults[1], oldDefaults[2]
|
||||||
|
conf.Upload.ImageBucket = oldBucket
|
||||||
|
conf.Upload.ImageBucketCDNHost = oldCDNHost
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestR2DeleteTaskStore(t *testing.T) {
|
||||||
|
setR2TestConfig(t)
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
store := NewR2DeleteTaskStore(conn)
|
||||||
|
|
||||||
|
require.NoError(t, store.Enqueue(ctx, "", []string{"ignored.webp"}))
|
||||||
|
require.NoError(t, store.Enqueue(ctx, conf.Upload.ImageBucket, nil))
|
||||||
|
require.NoError(t, store.Enqueue(ctx, conf.Upload.ImageBucket, []string{
|
||||||
|
"2026/11/shared.webp", "2026/11/shared.webp", "2026/11/other.webp", "",
|
||||||
|
}))
|
||||||
|
|
||||||
|
tasks, err := store.ListDue(ctx, 10)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, tasks, 2)
|
||||||
|
byKey := make(map[string]*R2DeleteTask, len(tasks))
|
||||||
|
for _, task := range tasks {
|
||||||
|
byKey[task.Key] = task
|
||||||
|
require.Equal(t, conf.Upload.ImageBucket, task.Bucket)
|
||||||
|
require.Len(t, task.KeyHash, 64)
|
||||||
|
}
|
||||||
|
shared := byKey["2026/11/shared.webp"]
|
||||||
|
other := byKey["2026/11/other.webp"]
|
||||||
|
require.NotNil(t, shared)
|
||||||
|
require.NotNil(t, other)
|
||||||
|
require.Equal(t, fmt.Sprintf("%x", sha256.Sum256([]byte(shared.Key))), shared.KeyHash)
|
||||||
|
|
||||||
|
future := time.Now().Add(time.Hour).Truncate(time.Microsecond)
|
||||||
|
require.NoError(t, store.MarkFailed(shared.ID, future, "temporary failure"))
|
||||||
|
due, err := store.ListDue(ctx, 10)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, due, 1)
|
||||||
|
require.Equal(t, other.ID, due[0].ID)
|
||||||
|
|
||||||
|
require.NoError(t, store.Enqueue(ctx, conf.Upload.ImageBucket, []string{shared.Key, shared.Key}))
|
||||||
|
var requeued R2DeleteTask
|
||||||
|
require.NoError(t, conn.WithContext(ctx).First(&requeued, shared.ID).Error)
|
||||||
|
require.Equal(t, shared.ID, requeued.ID)
|
||||||
|
require.Zero(t, requeued.Attempts)
|
||||||
|
require.Empty(t, requeued.LastError)
|
||||||
|
require.WithinDuration(t, time.Now(), requeued.NextRetryAt, 5*time.Second)
|
||||||
|
|
||||||
|
// 同一个 key 可在不同 bucket 各有一条任务,但同一 bucket 内仍保持唯一。
|
||||||
|
require.NoError(t, store.Enqueue(ctx, "another-bucket", []string{shared.Key}))
|
||||||
|
var count int64
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Model(&R2DeleteTask{}).Count(&count).Error)
|
||||||
|
require.Equal(t, int64(3), count)
|
||||||
|
|
||||||
|
require.NoError(t, store.Delete(other.ID))
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Model(&R2DeleteTask{}).Count(&count).Error)
|
||||||
|
require.Equal(t, int64(2), count)
|
||||||
|
empty, err := store.ListDue(ctx, 0)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NotNil(t, empty)
|
||||||
|
require.Empty(t, empty)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestR2DeleteTaskHasReferences(t *testing.T) {
|
||||||
|
setR2TestConfig(t)
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
store := NewR2DeleteTaskStore(conn)
|
||||||
|
bucket := conf.Upload.ImageBucket
|
||||||
|
|
||||||
|
upload := &UploadImage{Key: "upload.webp"}
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(upload).Error)
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Delete(upload).Error)
|
||||||
|
referenced, err := store.HasReferences(ctx, bucket, upload.Key)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.True(t, referenced, "软删除的 upload_images 仍必须通过 Unscoped 阻止对象删除")
|
||||||
|
|
||||||
|
message := &MessageImage{MessageID: 1, ObjectKey: "message.webp", URL: "https://cdn.example.test/assets/message.webp"}
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(message).Error)
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Delete(message).Error)
|
||||||
|
referenced, err = store.HasReferences(ctx, bucket, message.ObjectKey)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.True(t, referenced, "软删除的 message_images 仍必须通过 Unscoped 阻止对象删除")
|
||||||
|
|
||||||
|
legacyMessage := &MessageImage{MessageID: 2, URL: "https://cdn.example.test/assets/legacy-message.webp"}
|
||||||
|
legacyUpload := &UploadImage{Key: "https://cdn.example.test/assets/legacy-upload.webp"}
|
||||||
|
external := &MessageImage{MessageID: 3, URL: "https://external.example.net/external.webp"}
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(legacyMessage).Error)
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(legacyUpload).Error)
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(external).Error)
|
||||||
|
|
||||||
|
for _, key := range []string{"legacy-message.webp", "legacy-upload.webp"} {
|
||||||
|
referenced, err = store.HasReferences(ctx, bucket, key)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.True(t, referenced, key)
|
||||||
|
}
|
||||||
|
referenced, err = store.HasReferences(ctx, bucket, "external.webp")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.False(t, referenced)
|
||||||
|
referenced, err = store.HasReferences(ctx, bucket, "missing.webp")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.False(t, referenced)
|
||||||
|
|
||||||
|
profileUser := &User{Avatar: "https://cdn.example.test/assets/profile-avatar.webp", Background: "profile-background.webp", BackgroundImage: "https://external.example.net/external-profile.webp"}
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(profileUser).Error)
|
||||||
|
for _, key := range []string{"profile-avatar.webp", "profile-background.webp", "default-avatar.webp", "default-background.webp", "default-qr.webp"} {
|
||||||
|
referenced, err = store.HasReferences(ctx, bucket, key)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.True(t, referenced, key)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 第二个引用查询失败时必须显式返回错误,不能误报为无引用。
|
||||||
|
require.NoError(t, conn.Migrator().DropTable(&MessageImage{}))
|
||||||
|
_, err = store.HasReferences(ctx, bucket, "missing-again.webp")
|
||||||
|
require.Error(t, err)
|
||||||
|
}
|
||||||
@@ -10,6 +10,8 @@ import (
|
|||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
"github.com/samber/lo"
|
"github.com/samber/lo"
|
||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
)
|
)
|
||||||
|
|
||||||
var UploadImages UploadImagesStore
|
var UploadImages UploadImagesStore
|
||||||
@@ -23,12 +25,8 @@ type UploadImagesStore interface {
|
|||||||
// 用于账号导出等场景,将 N 次单题查询合并为少量 IN 查询,避免 N+1。
|
// 用于账号导出等场景,将 N 次单题查询合并为少量 IN 查询,避免 N+1。
|
||||||
GetByQuestionIDs(ctx context.Context, questionIDs []uint) (map[uint][]*UploadImage, error)
|
GetByQuestionIDs(ctx context.Context, questionIDs []uint) (map[uint][]*UploadImage, error)
|
||||||
GetByTypeQuestionID(ctx context.Context, typ UploadImageQuestionType, questionID uint) ([]*UploadImage, error)
|
GetByTypeQuestionID(ctx context.Context, typ UploadImageQuestionType, questionID uint) ([]*UploadImage, error)
|
||||||
// DeleteByQuestionID 删除某条提问时级联清理其全部上传图片记录与关联行。
|
// DeleteByQuestionID 先删除当前题目的关联,再物理删除不再被任何题目引用的主体。
|
||||||
//
|
// 实际删除的主体的对象 key 与元数据删除在同一事务中登记 R2 outbox。
|
||||||
// 用 Unscoped 真删:图片元数据(name/md5/key)随提问一并真删,与 questions.DeleteByID
|
|
||||||
// 对主表的真删口径一致。软删会留下带 deleted_at 的图片行,且关联行的 question_id
|
|
||||||
// 指向已不存在的提问。注意这里只清数据库记录,对象存储里的实际文件不在本函数删除
|
|
||||||
// (图片可能被多条记录复用,避免误删共享资源)。
|
|
||||||
DeleteByQuestionID(ctx context.Context, questionID uint) error
|
DeleteByQuestionID(ctx context.Context, questionID uint) error
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -87,9 +85,24 @@ func (db *uploadImages) Create(ctx context.Context, opts CreateUploadImageOption
|
|||||||
}
|
}
|
||||||
|
|
||||||
if opts.IsDeletingPrevious {
|
if opts.IsDeletingPrevious {
|
||||||
if err := tx.WithContext(ctx).Where("type = ? AND question_id = ?", opts.Type, opts.QuestionID).Delete(&UploadImageQuestion{}).Error; err != nil {
|
var previousLinks []*UploadImageQuestion
|
||||||
|
if err := tx.WithContext(ctx).
|
||||||
|
Where("type = ? AND question_id = ?", opts.Type, opts.QuestionID).
|
||||||
|
Find(&previousLinks).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "get previous image question links")
|
||||||
|
}
|
||||||
|
previousImageIDs := lo.Map(previousLinks, func(item *UploadImageQuestion, _ int) uint {
|
||||||
|
return item.UploadImageID
|
||||||
|
})
|
||||||
|
if err := tx.WithContext(ctx).
|
||||||
|
Where("type = ? AND question_id = ?", opts.Type, opts.QuestionID).
|
||||||
|
Delete(&UploadImageQuestion{}).Error; err != nil {
|
||||||
return errors.Wrap(err, "delete previous image question link")
|
return errors.Wrap(err, "delete previous image question link")
|
||||||
}
|
}
|
||||||
|
// 新图已经创建但尚未关联;即使它与旧图使用相同 key,也绝不加入删除队列。
|
||||||
|
if err := deleteUnreferencedUploadImages(ctx, tx, previousImageIDs, opts.Key); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if err := tx.WithContext(ctx).Create(&UploadImageQuestion{
|
if err := tx.WithContext(ctx).Create(&UploadImageQuestion{
|
||||||
Type: opts.Type,
|
Type: opts.Type,
|
||||||
@@ -167,10 +180,44 @@ func (db *uploadImages) GetByTypeQuestionID(ctx context.Context, typ UploadImage
|
|||||||
return db.getBy(ctx, "type = ? AND question_id = ?", typ, questionID)
|
return db.getBy(ctx, "type = ? AND question_id = ?", typ, questionID)
|
||||||
}
|
}
|
||||||
|
|
||||||
// DeleteByQuestionID 删除某条提问时级联清理其全部上传图片记录(软删除,与提问保持一致)。
|
// unreferencedUploadImagePredicate 是所有 UploadImage 删除入口共用的共享保护谓词。
|
||||||
// 注意:question_id 列在关联表 upload_image_questions 上,不在 upload_images 表上,
|
// 调用方必须先解除当前作用域的关联,再使用该谓词收集并删除真正的孤儿主体。
|
||||||
// 需先查关联行拿到 imageID 列表,再软删图片主体并删除关联行(同一事务内完成)。
|
const unreferencedUploadImagePredicate = `NOT EXISTS (
|
||||||
// 注:仅清理数据库记录,对象存储中的实际文件不在此处删除(避免误删复用资源)。
|
SELECT 1 FROM upload_image_questions AS shared_links
|
||||||
|
WHERE shared_links.upload_image_id = upload_images.id
|
||||||
|
)`
|
||||||
|
|
||||||
|
func deleteUnreferencedUploadImages(ctx context.Context, tx *gorm.DB, imageIDs []uint, excludedKeys ...string) error {
|
||||||
|
if len(imageIDs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var images []*UploadImage
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where("id IN ?", imageIDs).
|
||||||
|
Where(unreferencedUploadImagePredicate).
|
||||||
|
Find(&images).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "get unreferenced upload images")
|
||||||
|
}
|
||||||
|
if len(images) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
ids := lo.Map(images, func(image *UploadImage, _ int) uint { return image.ID })
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where("id IN ?", ids).
|
||||||
|
Where(unreferencedUploadImagePredicate).
|
||||||
|
Delete(&UploadImage{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete unreferenced upload images")
|
||||||
|
}
|
||||||
|
|
||||||
|
values := lo.Map(images, func(image *UploadImage, _ int) string { return image.Key })
|
||||||
|
if err := enqueueR2DeleteTasksForValues(tx.WithContext(ctx), conf.Upload.ImageBucket, values, excludedKeys...); err != nil {
|
||||||
|
return errors.Wrap(err, "enqueue deleted upload images")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (db *uploadImages) DeleteByQuestionID(ctx context.Context, questionID uint) error {
|
func (db *uploadImages) DeleteByQuestionID(ctx context.Context, questionID uint) error {
|
||||||
return db.Transaction(func(tx *gorm.DB) error {
|
return db.Transaction(func(tx *gorm.DB) error {
|
||||||
var links []*UploadImageQuestion
|
var links []*UploadImageQuestion
|
||||||
@@ -184,13 +231,10 @@ func (db *uploadImages) DeleteByQuestionID(ctx context.Context, questionID uint)
|
|||||||
imageIDs := lo.Map(links, func(item *UploadImageQuestion, _ int) uint {
|
imageIDs := lo.Map(links, func(item *UploadImageQuestion, _ int) uint {
|
||||||
return item.UploadImageID
|
return item.UploadImageID
|
||||||
})
|
})
|
||||||
if err := tx.WithContext(ctx).Where("id IN ?", imageIDs).Unscoped().Delete(&UploadImage{}).Error; err != nil {
|
// 关联表无 DeletedAt,Delete 本就是真删;先解除当前题目关联。
|
||||||
return errors.Wrap(err, "delete upload images by question id")
|
|
||||||
}
|
|
||||||
// 关联表 UploadImageQuestion 无 DeletedAt,Delete 本就是真删,无需 Unscoped。
|
|
||||||
if err := tx.WithContext(ctx).Where("question_id = ?", questionID).Delete(&UploadImageQuestion{}).Error; err != nil {
|
if err := tx.WithContext(ctx).Where("question_id = ?", questionID).Delete(&UploadImageQuestion{}).Error; err != nil {
|
||||||
return errors.Wrap(err, "delete image question links")
|
return errors.Wrap(err, "delete image question links")
|
||||||
}
|
}
|
||||||
return nil
|
return deleteUnreferencedUploadImages(ctx, tx, imageIDs)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func uploadImageCount(t *testing.T, conn *gorm.DB, id uint) int64 {
|
||||||
|
t.Helper()
|
||||||
|
var count int64
|
||||||
|
require.NoError(t, conn.Unscoped().Model(&UploadImage{}).Where("id = ?", id).Count(&count).Error)
|
||||||
|
return count
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadImagesDeleteByQuestionIDOutbox(t *testing.T) {
|
||||||
|
setR2TestConfig(t)
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
images := NewUploadImagesStore(conn)
|
||||||
|
tasks := NewR2DeleteTaskStore(conn)
|
||||||
|
|
||||||
|
first, err := images.Create(ctx, CreateUploadImageOptions{QuestionID: 1, Key: "unique.webp"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
duplicate, err := images.Create(ctx, CreateUploadImageOptions{QuestionID: 1, Key: "unique.webp"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
shared, err := images.Create(ctx, CreateUploadImageOptions{QuestionID: 1, Key: "shared.webp"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(&UploadImageQuestion{
|
||||||
|
UploadImageID: shared.ID,
|
||||||
|
QuestionID: 2,
|
||||||
|
}).Error)
|
||||||
|
|
||||||
|
require.NoError(t, images.DeleteByQuestionID(ctx, 1))
|
||||||
|
require.Zero(t, uploadImageCount(t, conn, first.ID))
|
||||||
|
require.Zero(t, uploadImageCount(t, conn, duplicate.ID))
|
||||||
|
require.Equal(t, int64(1), uploadImageCount(t, conn, shared.ID))
|
||||||
|
|
||||||
|
due, err := tasks.ListDue(ctx, 10)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, due, 1, "重复 key 必须去重,共享主体不得入队")
|
||||||
|
require.Equal(t, "unique.webp", due[0].Key)
|
||||||
|
|
||||||
|
require.NoError(t, images.DeleteByQuestionID(ctx, 2))
|
||||||
|
require.Zero(t, uploadImageCount(t, conn, shared.ID))
|
||||||
|
due, err = tasks.ListDue(ctx, 10)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, due, 2)
|
||||||
|
require.ElementsMatch(t, []string{"unique.webp", "shared.webp"}, []string{due[0].Key, due[1].Key})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadImagesReplaceDeletesOnlyOrphanedPrevious(t *testing.T) {
|
||||||
|
setR2TestConfig(t)
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
images := NewUploadImagesStore(conn)
|
||||||
|
tasks := NewR2DeleteTaskStore(conn)
|
||||||
|
|
||||||
|
old, err := images.Create(ctx, CreateUploadImageOptions{
|
||||||
|
Type: UploadImageQuestionTypeAnswer,
|
||||||
|
QuestionID: 10,
|
||||||
|
Key: "old.webp",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
shared, err := images.Create(ctx, CreateUploadImageOptions{
|
||||||
|
Type: UploadImageQuestionTypeAnswer,
|
||||||
|
QuestionID: 10,
|
||||||
|
Key: "shared-old.webp",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(&UploadImageQuestion{
|
||||||
|
Type: UploadImageQuestionTypeAnswer,
|
||||||
|
UploadImageID: shared.ID,
|
||||||
|
QuestionID: 11,
|
||||||
|
}).Error)
|
||||||
|
|
||||||
|
newImage, err := images.Create(ctx, CreateUploadImageOptions{
|
||||||
|
Type: UploadImageQuestionTypeAnswer,
|
||||||
|
QuestionID: 10,
|
||||||
|
Key: "new.webp",
|
||||||
|
IsDeletingPrevious: true,
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Zero(t, uploadImageCount(t, conn, old.ID))
|
||||||
|
require.Equal(t, int64(1), uploadImageCount(t, conn, shared.ID))
|
||||||
|
require.Equal(t, int64(1), uploadImageCount(t, conn, newImage.ID))
|
||||||
|
|
||||||
|
due, err := tasks.ListDue(ctx, 10)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, due, 1)
|
||||||
|
require.Equal(t, "old.webp", due[0].Key)
|
||||||
|
require.NotEqual(t, newImage.Key, due[0].Key)
|
||||||
|
}
|
||||||
+138
-11
@@ -133,6 +133,13 @@ type User struct {
|
|||||||
PagePassword string `json:"-" gorm:"column:page_password;type:varchar(255);default:''"`
|
PagePassword string `json:"-" gorm:"column:page_password;type:varchar(255);default:''"`
|
||||||
AskPassword string `json:"-" gorm:"column:ask_password;type:varchar(255);default:''"`
|
AskPassword string `json:"-" gorm:"column:ask_password;type:varchar(255);default:''"`
|
||||||
|
|
||||||
|
// PagePasswordVer / AskPasswordVer 是页面访问密码 / 提问密码的版本号。
|
||||||
|
// 每次修改对应密码时 +1,session 中保存授权时的版本号;
|
||||||
|
// 校验时若 session 中的版本号 != 当前版本号,则视为授权已失效。
|
||||||
|
// 这让「修改页面密码」能立即撤销已泄露的旧授权(SEC-06)。
|
||||||
|
PagePasswordVer uint `gorm:"default:0" json:"-"`
|
||||||
|
AskPasswordVer uint `gorm:"default:0" json:"-"`
|
||||||
|
|
||||||
// 私聊相关
|
// 私聊相关
|
||||||
AllowChat bool `json:"allow_chat" gorm:"default:false"`
|
AllowChat bool `json:"allow_chat" gorm:"default:false"`
|
||||||
// 每个私聊会话的消息条数上限(用户自设)。default:20 仅是 DB 层兜底;
|
// 每个私聊会话的消息条数上限(用户自设)。default:20 仅是 DB 层兜底;
|
||||||
@@ -386,10 +393,25 @@ func (db *users) Update(ctx context.Context, id uint, opts UpdateUserOptions) er
|
|||||||
"Language": opts.Language,
|
"Language": opts.Language,
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := db.WithContext(ctx).Model(&User{}).Where("id = ?", id).Updates(updates).Error; err != nil {
|
return db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||||
|
var current User
|
||||||
|
if err := tx.WithContext(ctx).Select("id", "avatar", "background", "background_image").First(¤t, id).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "get current profile images")
|
||||||
|
}
|
||||||
|
if err := tx.WithContext(ctx).Model(&User{}).Where("id = ?", id).Updates(updates).Error; err != nil {
|
||||||
return errors.Wrap(err, "update user")
|
return errors.Wrap(err, "update user")
|
||||||
}
|
}
|
||||||
|
oldValues := make([]string, 0, 3)
|
||||||
|
for _, pair := range [][2]string{{current.Avatar, opts.Avatar}, {current.Background, opts.Background}, {current.BackgroundImage, opts.BackgroundImage}} {
|
||||||
|
if pair[0] != pair[1] {
|
||||||
|
oldValues = append(oldValues, pair[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := enqueueR2DeleteTasksForValues(tx.WithContext(ctx), conf.Upload.ImageBucket, oldValues, opts.Avatar, opts.Background, opts.BackgroundImage); err != nil {
|
||||||
|
return errors.Wrap(err, "enqueue replaced profile images")
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// UpdateHarassmentType 仅更新骚扰防护类型(none / register_only)。
|
// UpdateHarassmentType 仅更新骚扰防护类型(none / register_only)。
|
||||||
@@ -863,13 +885,39 @@ func (db *users) GetHotTags(ctx context.Context, limit int) ([]string, error) {
|
|||||||
return hot, nil
|
return hot, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// backfillPagePasswordVer 给「已设置密码但版本号仍为 0」的存量账号补一次版本号。
|
||||||
|
//
|
||||||
|
// 背景:PagePasswordVer / AskPasswordVer 是 2026-11-05 新增的列,AutoMigrate 只补默认值 0。
|
||||||
|
// 而 0 同时是「session 里没有授权记录」时类型断言的零值,两者相等会让页面密码判定恒为通过
|
||||||
|
// (见 route/question/page.go 的 pagePasswordAuthorized 注释)。
|
||||||
|
// 把存量行统一置 1:语义与「改过一次密码」一致,升级前发出的旧授权(布尔 true)
|
||||||
|
// 也会因类型不符而自然作废,用户重输一次密码即可。
|
||||||
|
//
|
||||||
|
// 幂等:只更新仍为 0 的行,重复执行不会把版本号越推越大。
|
||||||
|
func backfillPagePasswordVer(db *gorm.DB) error {
|
||||||
|
if err := db.Model(&User{}).
|
||||||
|
Where("page_password <> '' AND page_password_ver = 0").
|
||||||
|
Update("page_password_ver", 1).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "backfill page password version")
|
||||||
|
}
|
||||||
|
if err := db.Model(&User{}).
|
||||||
|
Where("ask_password <> '' AND ask_password_ver = 0").
|
||||||
|
Update("ask_password_ver", 1).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "backfill ask password version")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (db *users) UpdatePagePassword(ctx context.Context, id uint, password string) error {
|
func (db *users) UpdatePagePassword(ctx context.Context, id uint, password string) error {
|
||||||
user, err := db.GetByID(ctx, id)
|
user, err := db.GetByID(ctx, id)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
user.SetPagePassword(password)
|
user.SetPagePassword(password)
|
||||||
return db.WithContext(ctx).Model(user).Update("page_password", user.PagePassword).Error
|
return db.WithContext(ctx).Model(user).Updates(map[string]interface{}{
|
||||||
|
"page_password": user.PagePassword,
|
||||||
|
"page_password_ver": gorm.Expr("page_password_ver + 1"),
|
||||||
|
}).Error
|
||||||
}
|
}
|
||||||
|
|
||||||
func (db *users) UpdateAskPassword(ctx context.Context, id uint, password string) error {
|
func (db *users) UpdateAskPassword(ctx context.Context, id uint, password string) error {
|
||||||
@@ -878,7 +926,10 @@ func (db *users) UpdateAskPassword(ctx context.Context, id uint, password string
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
user.SetAskPassword(password)
|
user.SetAskPassword(password)
|
||||||
return db.WithContext(ctx).Model(user).Update("ask_password", user.AskPassword).Error
|
return db.WithContext(ctx).Model(user).Updates(map[string]interface{}{
|
||||||
|
"ask_password": user.AskPassword,
|
||||||
|
"ask_password_ver": gorm.Expr("ask_password_ver + 1"),
|
||||||
|
}).Error
|
||||||
}
|
}
|
||||||
|
|
||||||
// 新增私聊设置
|
// 新增私聊设置
|
||||||
@@ -1011,7 +1062,8 @@ func (db *users) BanUserAndCleanup(ctx context.Context, userID uint) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// purgeUserContent 物理删除某用户的全部内容数据:追问、消息与消息图片、私聊会话、
|
// purgeUserContent 物理删除某用户的全部内容数据:追问、消息与消息图片、私聊会话、
|
||||||
// 提问-图片关联、提问、拉黑关系、功能开关。**不含用户行本身**(由调用方决定是软删除还是物理删除)。
|
// 提问图片元数据及关联、提问、投票及其子表、访问统计、拉黑关系、功能开关。
|
||||||
|
// **不含用户行本身**(由调用方决定是软删除还是物理删除),也不删除审计/处置记录。
|
||||||
//
|
//
|
||||||
// 全部使用 Unscoped 物理删除而非软删除:这些表多为软删除模型,而部分表的唯一索引
|
// 全部使用 Unscoped 物理删除而非软删除:这些表多为软删除模型,而部分表的唯一索引
|
||||||
// 不含 deleted_at(如 conversations.token),软删除会让被删行永久占据唯一槽位。
|
// 不含 deleted_at(如 conversations.token),软删除会让被删行永久占据唯一槽位。
|
||||||
@@ -1020,6 +1072,19 @@ func (db *users) BanUserAndCleanup(ctx context.Context, userID uint) error {
|
|||||||
// 直接复用会把上一条语句的 Where 条件串到下一条上(曾把 messages 的条件串进 follow_ups,
|
// 直接复用会把上一条语句的 Where 条件串到下一条上(曾把 messages 的条件串进 follow_ups,
|
||||||
// 报「字段 conversation_id 不存在」)。
|
// 报「字段 conversation_id 不存在」)。
|
||||||
func purgeUserContent(ctx context.Context, tx *gorm.DB, userID uint) error {
|
func purgeUserContent(ctx context.Context, tx *gorm.DB, userID uint) error {
|
||||||
|
var profile User
|
||||||
|
if err := tx.WithContext(ctx).Select("id", "avatar", "background", "background_image").First(&profile, userID).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "get profile images of user")
|
||||||
|
}
|
||||||
|
if err := enqueueR2DeleteTasksForValues(tx.WithContext(ctx), conf.Upload.ImageBucket, []string{profile.Avatar, profile.Background, profile.BackgroundImage}); err != nil {
|
||||||
|
return errors.Wrap(err, "enqueue profile images of user")
|
||||||
|
}
|
||||||
|
if err := tx.WithContext(ctx).Model(&User{}).Where("id = ?", userID).Updates(map[string]interface{}{
|
||||||
|
"avatar": "", "background": "", "background_image": "",
|
||||||
|
}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "clear profile images of user")
|
||||||
|
}
|
||||||
|
|
||||||
// 追问:属于该用户提问下的追问
|
// 追问:属于该用户提问下的追问
|
||||||
if err := tx.WithContext(ctx).Unscoped().
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
Where("question_id IN (SELECT id FROM questions WHERE user_id = ?)", userID).
|
Where("question_id IN (SELECT id FROM questions WHERE user_id = ?)", userID).
|
||||||
@@ -1027,18 +1092,38 @@ func purgeUserContent(ctx context.Context, tx *gorm.DB, userID uint) error {
|
|||||||
return errors.Wrap(err, "delete follow ups of user")
|
return errors.Wrap(err, "delete follow ups of user")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 消息图片必须先于消息删除,否则消息子查询将无法再匹配待删图片。
|
||||||
|
// 新数据优先使用物化 ObjectKey,旧数据回退到 URL;外部 URL 不会进入 R2 outbox。
|
||||||
|
messageImageScope := "message_id IN (SELECT id FROM messages WHERE conversation_id IN (SELECT id FROM conversations WHERE owner_id = ? OR asker_id = ?))"
|
||||||
|
var messageImages []*MessageImage
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where(messageImageScope, userID, userID).
|
||||||
|
Find(&messageImages).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "get message images of user")
|
||||||
|
}
|
||||||
|
messageImageValues := make([]string, 0, len(messageImages))
|
||||||
|
for _, image := range messageImages {
|
||||||
|
value := image.ObjectKey
|
||||||
|
if value == "" {
|
||||||
|
value = image.URL
|
||||||
|
}
|
||||||
|
messageImageValues = append(messageImageValues, value)
|
||||||
|
}
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where(messageImageScope, userID, userID).
|
||||||
|
Delete(&MessageImage{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete message images of user")
|
||||||
|
}
|
||||||
|
if err := enqueueR2DeleteTasksForValues(tx.WithContext(ctx), conf.Upload.ImageBucket, messageImageValues); err != nil {
|
||||||
|
return errors.Wrap(err, "enqueue deleted message images")
|
||||||
|
}
|
||||||
|
|
||||||
// 消息:属于该用户作为 owner 或 asker 的会话
|
// 消息:属于该用户作为 owner 或 asker 的会话
|
||||||
if err := tx.WithContext(ctx).Unscoped().
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
Where("conversation_id IN (SELECT id FROM conversations WHERE owner_id = ? OR asker_id = ?)", userID, userID).
|
Where("conversation_id IN (SELECT id FROM conversations WHERE owner_id = ? OR asker_id = ?)", userID, userID).
|
||||||
Delete(&Message{}).Error; err != nil {
|
Delete(&Message{}).Error; err != nil {
|
||||||
return errors.Wrap(err, "delete messages of user")
|
return errors.Wrap(err, "delete messages of user")
|
||||||
}
|
}
|
||||||
// 消息图片
|
|
||||||
if err := tx.WithContext(ctx).Unscoped().
|
|
||||||
Where("message_id IN (SELECT id FROM messages WHERE conversation_id IN (SELECT id FROM conversations WHERE owner_id = ? OR asker_id = ?))", userID, userID).
|
|
||||||
Delete(&MessageImage{}).Error; err != nil {
|
|
||||||
return errors.Wrap(err, "delete message images of user")
|
|
||||||
}
|
|
||||||
|
|
||||||
// 私聊会话
|
// 私聊会话
|
||||||
if err := tx.WithContext(ctx).Unscoped().
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
@@ -1047,12 +1132,25 @@ func purgeUserContent(ctx context.Context, tx *gorm.DB, userID uint) error {
|
|||||||
return errors.Wrap(err, "delete conversations of user")
|
return errors.Wrap(err, "delete conversations of user")
|
||||||
}
|
}
|
||||||
|
|
||||||
// 提问-图片关联(仅删除关联记录;UploadImage 本体为通用资源,保留)
|
// 先确定本用户上传或仅与当前作用域有关的候选主体,再解除本用户题目的关联。
|
||||||
|
// 最终是否可删除由与 UploadImages.DeleteByQuestionID 完全相同的共享保护谓词决定。
|
||||||
|
var uploadImageIDs []uint
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().Model(&UploadImage{}).
|
||||||
|
Where(`uploader_user_id = ? OR upload_images.id IN (
|
||||||
|
SELECT upload_image_id FROM upload_image_questions
|
||||||
|
WHERE question_id IN (SELECT id FROM questions WHERE user_id = ?)
|
||||||
|
)`, userID, userID).
|
||||||
|
Pluck("upload_images.id", &uploadImageIDs).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "get upload images of user")
|
||||||
|
}
|
||||||
if err := tx.WithContext(ctx).Unscoped().
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
Where("question_id IN (SELECT id FROM questions WHERE user_id = ?)", userID).
|
Where("question_id IN (SELECT id FROM questions WHERE user_id = ?)", userID).
|
||||||
Delete(&UploadImageQuestion{}).Error; err != nil {
|
Delete(&UploadImageQuestion{}).Error; err != nil {
|
||||||
return errors.Wrap(err, "delete upload image links of user")
|
return errors.Wrap(err, "delete upload image links of user")
|
||||||
}
|
}
|
||||||
|
if err := deleteUnreferencedUploadImages(ctx, tx, uploadImageIDs); err != nil {
|
||||||
|
return errors.Wrap(err, "delete upload images of user")
|
||||||
|
}
|
||||||
|
|
||||||
// 提问
|
// 提问
|
||||||
if err := tx.WithContext(ctx).Unscoped().
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
@@ -1061,6 +1159,35 @@ func purgeUserContent(ctx context.Context, tx *gorm.DB, userID uint) error {
|
|||||||
return errors.Wrap(err, "delete questions of user")
|
return errors.Wrap(err, "delete questions of user")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 投票子表必须先于投票主表删除;只处理该用户拥有的投票,不按参与者身份扩大范围。
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where("vote_id IN (SELECT id FROM votes WHERE user_id = ?)", userID).
|
||||||
|
Delete(&VoteSubmission{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete vote submissions of user")
|
||||||
|
}
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where("vote_id IN (SELECT id FROM votes WHERE user_id = ?)", userID).
|
||||||
|
Delete(&VoteFormField{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete vote form fields of user")
|
||||||
|
}
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where("vote_id IN (SELECT id FROM votes WHERE user_id = ?)", userID).
|
||||||
|
Delete(&VoteOption{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete vote options of user")
|
||||||
|
}
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where("user_id = ?", userID).
|
||||||
|
Delete(&Vote{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete votes of user")
|
||||||
|
}
|
||||||
|
|
||||||
|
// 提问箱访问统计:user_id 明确指向箱主,不属于站方审计/处置记录。
|
||||||
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
|
Where("user_id = ?", userID).
|
||||||
|
Delete(&BoxView{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete box views of user")
|
||||||
|
}
|
||||||
|
|
||||||
// 拉黑关系(该用户拉黑别人,或被别人拉黑)
|
// 拉黑关系(该用户拉黑别人,或被别人拉黑)
|
||||||
if err := tx.WithContext(ctx).Unscoped().
|
if err := tx.WithContext(ctx).Unscoped().
|
||||||
Where("user_id = ? OR blocked_id = ?", userID, userID).
|
Where("user_id = ? OR blocked_id = ?", userID, userID).
|
||||||
|
|||||||
@@ -0,0 +1,142 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
)
|
||||||
|
|
||||||
|
func createOutboxTestUser(t *testing.T, ctx context.Context, store UsersStore, email, domain string) *User {
|
||||||
|
t.Helper()
|
||||||
|
require.NoError(t, store.Create(ctx, CreateUserOptions{
|
||||||
|
Name: domain,
|
||||||
|
Password: "test-only-password",
|
||||||
|
Email: email,
|
||||||
|
Domain: domain,
|
||||||
|
}))
|
||||||
|
user, err := store.GetByEmail(ctx, email)
|
||||||
|
require.NoError(t, err)
|
||||||
|
return user
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPurgeUserContentEnqueuesOwnedR2Objects(t *testing.T) {
|
||||||
|
setR2TestConfig(t)
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
users := NewUsersStore(conn)
|
||||||
|
owner := createOutboxTestUser(t, ctx, users, "[email protected]", "outbox-owner")
|
||||||
|
other := createOutboxTestUser(t, ctx, users, "[email protected]", "outbox-other")
|
||||||
|
require.NoError(t, conn.Model(owner).Updates(map[string]interface{}{
|
||||||
|
"avatar": "https://cdn.example.test/assets/profile-owned.webp", "background": "https://cdn.example.test/assets/default-background.webp", "background_image": "https://cdn.example.test/assets/profile-shared.webp",
|
||||||
|
}).Error)
|
||||||
|
require.NoError(t, conn.Model(other).Update("background_image", "https://cdn.example.test/assets/profile-shared.webp").Error)
|
||||||
|
questions := NewQuestionsStore(conn)
|
||||||
|
question, err := questions.Create(ctx, CreateQuestionOptions{UserID: owner.ID, Content: "owned"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
otherQuestion, err := questions.Create(ctx, CreateQuestionOptions{UserID: other.ID, Content: "other"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
images := NewUploadImagesStore(conn)
|
||||||
|
owned, err := images.Create(ctx, CreateUploadImageOptions{QuestionID: question.ID, UploaderUserID: owner.ID, Key: "owned.webp"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
_, err = images.Create(ctx, CreateUploadImageOptions{QuestionID: question.ID, UploaderUserID: owner.ID, Key: "owned.webp"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
legacy, err := images.Create(ctx, CreateUploadImageOptions{
|
||||||
|
QuestionID: question.ID, UploaderUserID: owner.ID, Key: "https://cdn.example.test/assets/legacy.webp",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
external, err := images.Create(ctx, CreateUploadImageOptions{
|
||||||
|
QuestionID: question.ID, UploaderUserID: owner.ID, Key: "https://external.example.net/external.webp",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
shared, err := images.Create(ctx, CreateUploadImageOptions{QuestionID: question.ID, UploaderUserID: owner.ID, Key: "shared.webp"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Create(&UploadImageQuestion{
|
||||||
|
UploadImageID: shared.ID, QuestionID: otherQuestion.ID,
|
||||||
|
}).Error)
|
||||||
|
|
||||||
|
conversation, message, err := NewConversationStore(conn).CreateWithFirstMessage(ctx, CreateConversationOptions{
|
||||||
|
OwnerID: owner.ID, AskerID: other.ID, MaxMessages: 20,
|
||||||
|
}, other.ID, "private")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NotNil(t, conversation)
|
||||||
|
messageImages := NewMessageImageStore(conn)
|
||||||
|
for _, opts := range []CreateMessageImageOptions{
|
||||||
|
{MessageID: message.ID, ObjectKey: "message-key.webp", URL: "https://cdn.example.test/assets/message-key.webp"},
|
||||||
|
{MessageID: message.ID, ObjectKey: "owned.webp", URL: "https://cdn.example.test/assets/owned.webp"},
|
||||||
|
{MessageID: message.ID, URL: "https://cdn.example.test/assets/message-legacy.webp"},
|
||||||
|
{MessageID: message.ID, URL: "https://external.example.net/message-external.webp"},
|
||||||
|
} {
|
||||||
|
_, err = messageImages.Create(ctx, opts)
|
||||||
|
require.NoError(t, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, users.Deactivate(ctx, owner.ID))
|
||||||
|
require.Zero(t, uploadImageCount(t, conn, owned.ID))
|
||||||
|
require.Zero(t, uploadImageCount(t, conn, legacy.ID))
|
||||||
|
require.Zero(t, uploadImageCount(t, conn, external.ID))
|
||||||
|
require.Equal(t, int64(1), uploadImageCount(t, conn, shared.ID))
|
||||||
|
var sharedLinks int64
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Model(&UploadImageQuestion{}).
|
||||||
|
Where("upload_image_id = ? AND question_id = ?", shared.ID, otherQuestion.ID).Count(&sharedLinks).Error)
|
||||||
|
require.Equal(t, int64(1), sharedLinks)
|
||||||
|
|
||||||
|
due, err := NewR2DeleteTaskStore(conn).ListDue(ctx, 20)
|
||||||
|
require.NoError(t, err)
|
||||||
|
keys := make([]string, 0, len(due))
|
||||||
|
for _, task := range due {
|
||||||
|
keys = append(keys, task.Key)
|
||||||
|
}
|
||||||
|
require.ElementsMatch(t, []string{
|
||||||
|
"owned.webp", "legacy.webp", "message-key.webp", "message-legacy.webp", "profile-owned.webp", "profile-shared.webp",
|
||||||
|
}, keys)
|
||||||
|
require.NotContains(t, keys, "shared.webp")
|
||||||
|
require.NotContains(t, keys, "external.webp")
|
||||||
|
require.NotContains(t, keys, "message-external.webp")
|
||||||
|
require.NotContains(t, keys, "default-background.webp")
|
||||||
|
var deactivated User
|
||||||
|
require.NoError(t, conn.Unscoped().First(&deactivated, owner.ID).Error)
|
||||||
|
require.Empty(t, deactivated.Avatar)
|
||||||
|
require.Empty(t, deactivated.Background)
|
||||||
|
require.Empty(t, deactivated.BackgroundImage)
|
||||||
|
other, err = users.GetByID(ctx, other.ID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "https://cdn.example.test/assets/profile-shared.webp", other.BackgroundImage)
|
||||||
|
referenced, err := NewR2DeleteTaskStore(conn).HasReferences(ctx, conf.Upload.ImageBucket, "profile-shared.webp")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.True(t, referenced)
|
||||||
|
referenced, err = NewR2DeleteTaskStore(conn).HasReferences(ctx, conf.Upload.ImageBucket, "default-background.webp")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.True(t, referenced)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPurgeUserContentOutboxFailureRollsBack(t *testing.T) {
|
||||||
|
setR2TestConfig(t)
|
||||||
|
conn, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
users := NewUsersStore(conn)
|
||||||
|
owner := createOutboxTestUser(t, ctx, users, "[email protected]", "outbox-rollback")
|
||||||
|
question, err := NewQuestionsStore(conn).Create(ctx, CreateQuestionOptions{UserID: owner.ID, Content: "rollback"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
image, err := NewUploadImagesStore(conn).Create(ctx, CreateUploadImageOptions{
|
||||||
|
QuestionID: question.ID, UploaderUserID: owner.ID, Key: "rollback.webp",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
require.NoError(t, conn.Migrator().DropTable(&R2DeleteTask{}))
|
||||||
|
require.Error(t, users.Deactivate(ctx, owner.ID))
|
||||||
|
|
||||||
|
require.Equal(t, int64(1), uploadImageCount(t, conn, image.ID))
|
||||||
|
var links int64
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Model(&UploadImageQuestion{}).
|
||||||
|
Where("upload_image_id = ? AND question_id = ?", image.ID, question.ID).Count(&links).Error)
|
||||||
|
require.Equal(t, int64(1), links)
|
||||||
|
var questions int64
|
||||||
|
require.NoError(t, conn.WithContext(ctx).Unscoped().Model(&Question{}).Where("id = ?", question.ID).Count(&questions).Error)
|
||||||
|
require.Equal(t, int64(1), questions)
|
||||||
|
var user User
|
||||||
|
require.NoError(t, conn.WithContext(ctx).First(&user, owner.ID).Error)
|
||||||
|
require.False(t, user.DeletedAt.Valid)
|
||||||
|
}
|
||||||
@@ -552,6 +552,105 @@ func TestDeactivatePurgesUser(t *testing.T) {
|
|||||||
|
|
||||||
// TestBanUserAndCleanupRemovesRow 封禁与停用的唯一差别:封禁连用户行一起物理删除,
|
// TestBanUserAndCleanupRemovesRow 封禁与停用的唯一差别:封禁连用户行一起物理删除,
|
||||||
// 该邮箱/域名被彻底释放。
|
// 该邮箱/域名被彻底释放。
|
||||||
|
func TestDeactivatePurgesAllOwnedContent(t *testing.T) {
|
||||||
|
db, _ := newTestDB(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
usersStore := NewUsersStore(db)
|
||||||
|
|
||||||
|
createUser := func(email, domain string) *User {
|
||||||
|
require.Nil(t, usersStore.Create(ctx, CreateUserOptions{
|
||||||
|
Name: domain,
|
||||||
|
Password: "test-only-password",
|
||||||
|
Email: email,
|
||||||
|
Domain: domain,
|
||||||
|
}))
|
||||||
|
user, err := usersStore.GetByEmail(ctx, email)
|
||||||
|
require.Nil(t, err)
|
||||||
|
return user
|
||||||
|
}
|
||||||
|
owner := createUser("[email protected]", "purge-all")
|
||||||
|
other := createUser("[email protected]", "purge-other")
|
||||||
|
|
||||||
|
question, err := NewQuestionsStore(db).Create(ctx, CreateQuestionOptions{UserID: owner.ID, Content: "owned"})
|
||||||
|
require.Nil(t, err)
|
||||||
|
otherQuestion, err := NewQuestionsStore(db).Create(ctx, CreateQuestionOptions{UserID: other.ID, Content: "other"})
|
||||||
|
require.Nil(t, err)
|
||||||
|
|
||||||
|
images := NewUploadImagesStore(db)
|
||||||
|
ownedImage, err := images.Create(ctx, CreateUploadImageOptions{
|
||||||
|
QuestionID: question.ID, UploaderUserID: owner.ID, Key: "owned.webp",
|
||||||
|
})
|
||||||
|
require.Nil(t, err)
|
||||||
|
sharedImage, err := images.Create(ctx, CreateUploadImageOptions{
|
||||||
|
QuestionID: question.ID, UploaderUserID: owner.ID, Key: "shared.webp",
|
||||||
|
})
|
||||||
|
require.Nil(t, err)
|
||||||
|
require.Nil(t, db.WithContext(ctx).Create(&UploadImageQuestion{
|
||||||
|
UploadImageID: sharedImage.ID, QuestionID: otherQuestion.ID,
|
||||||
|
}).Error)
|
||||||
|
orphanImage := &UploadImage{UploaderUserID: owner.ID, Key: "orphan.webp"}
|
||||||
|
require.Nil(t, db.WithContext(ctx).Create(orphanImage).Error)
|
||||||
|
otherImage, err := images.Create(ctx, CreateUploadImageOptions{
|
||||||
|
QuestionID: otherQuestion.ID, UploaderUserID: other.ID, Key: "other.webp",
|
||||||
|
})
|
||||||
|
require.Nil(t, err)
|
||||||
|
|
||||||
|
conversations := NewConversationStore(db)
|
||||||
|
conversation, message, err := conversations.CreateWithFirstMessage(ctx, CreateConversationOptions{
|
||||||
|
OwnerID: owner.ID, AskerID: other.ID, MaxMessages: 20,
|
||||||
|
}, other.ID, "private")
|
||||||
|
require.Nil(t, err)
|
||||||
|
_, err = NewMessageImageStore(db).Create(ctx, CreateMessageImageOptions{
|
||||||
|
MessageID: message.ID, URL: "message.webp", FileName: "message.webp",
|
||||||
|
})
|
||||||
|
require.Nil(t, err)
|
||||||
|
|
||||||
|
vote, err := NewVotesStore(db).Create(ctx, CreateVoteOptions{
|
||||||
|
UserID: owner.ID, Title: "owned vote", Type: VoteTypeForm,
|
||||||
|
})
|
||||||
|
require.Nil(t, err)
|
||||||
|
require.Nil(t, NewVotesStore(db).UpdateStatus(ctx, vote.ID, VoteStatusPublished))
|
||||||
|
option, err := NewVoteOptionsStore(db).Create(ctx, CreateVoteOptionOptions{
|
||||||
|
VoteID: vote.ID, Content: "option",
|
||||||
|
})
|
||||||
|
require.Nil(t, err)
|
||||||
|
_, err = NewVoteFormFieldsStore(db).Create(ctx, CreateVoteFormFieldOptions{
|
||||||
|
VoteID: vote.ID, FieldType: VoteFormFieldTypeText, Label: "field",
|
||||||
|
})
|
||||||
|
require.Nil(t, err)
|
||||||
|
_, err = NewVoteSubmissionsStore(db).Create(ctx, CreateVoteSubmissionOptions{
|
||||||
|
VoteID: vote.ID, OptionID: option.ID,
|
||||||
|
})
|
||||||
|
require.Nil(t, err)
|
||||||
|
require.Nil(t, NewBoxViewsStore(db).Record(ctx, RecordBoxViewOptions{
|
||||||
|
UserID: owner.ID, ViewedOn: time.Now(), Visitor: "visitor",
|
||||||
|
}))
|
||||||
|
|
||||||
|
require.Nil(t, usersStore.Deactivate(ctx, owner.ID))
|
||||||
|
|
||||||
|
assertCount := func(model interface{}, query string, args []interface{}, want int64) {
|
||||||
|
t.Helper()
|
||||||
|
var count int64
|
||||||
|
require.Nil(t, db.WithContext(ctx).Unscoped().Model(model).Where(query, args...).Count(&count).Error)
|
||||||
|
require.Equal(t, want, count)
|
||||||
|
}
|
||||||
|
assertCount(&MessageImage{}, "message_id = ?", []interface{}{message.ID}, 0)
|
||||||
|
assertCount(&Message{}, "conversation_id = ?", []interface{}{conversation.ID}, 0)
|
||||||
|
assertCount(&Conversation{}, "id = ?", []interface{}{conversation.ID}, 0)
|
||||||
|
assertCount(&VoteSubmission{}, "vote_id = ?", []interface{}{vote.ID}, 0)
|
||||||
|
assertCount(&VoteFormField{}, "vote_id = ?", []interface{}{vote.ID}, 0)
|
||||||
|
assertCount(&VoteOption{}, "vote_id = ?", []interface{}{vote.ID}, 0)
|
||||||
|
assertCount(&Vote{}, "id = ?", []interface{}{vote.ID}, 0)
|
||||||
|
assertCount(&UploadImage{}, "id = ?", []interface{}{ownedImage.ID}, 0)
|
||||||
|
assertCount(&UploadImage{}, "id = ?", []interface{}{orphanImage.ID}, 0)
|
||||||
|
assertCount(&UploadImage{}, "id = ?", []interface{}{sharedImage.ID}, 1)
|
||||||
|
assertCount(&UploadImage{}, "id = ?", []interface{}{otherImage.ID}, 1)
|
||||||
|
assertCount(&UploadImageQuestion{}, "question_id = ?", []interface{}{question.ID}, 0)
|
||||||
|
assertCount(&UploadImageQuestion{}, "upload_image_id = ? AND question_id = ?", []interface{}{sharedImage.ID, otherQuestion.ID}, 1)
|
||||||
|
assertCount(&BoxView{}, "user_id = ?", []interface{}{owner.ID}, 0)
|
||||||
|
assertCount(&Question{}, "id = ?", []interface{}{otherQuestion.ID}, 1)
|
||||||
|
}
|
||||||
|
|
||||||
func TestBanUserAndCleanupRemovesRow(t *testing.T) {
|
func TestBanUserAndCleanupRemovesRow(t *testing.T) {
|
||||||
db, cleanup := newTestDB(t)
|
db, cleanup := newTestDB(t)
|
||||||
defer func() { _ = cleanup("users", "questions", "feature_controls") }()
|
defer func() { _ = cleanup("users", "questions", "feature_controls") }()
|
||||||
|
|||||||
@@ -34,11 +34,21 @@ type voteOptions struct {
|
|||||||
*gorm.DB
|
*gorm.DB
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// MaxVoteOptionLen 是选项内容在数据库中的最大字符数。
|
||||||
|
//
|
||||||
|
// 必须与 conf 的默认 vote_option_len 保持一致:配置曾经允许 300 而这里只有 200,
|
||||||
|
// 于是 201–300 字的选项「校验通过、写库失败」(MySQL 严格模式下报错,
|
||||||
|
// PostgreSQL 直接报 value too long)。改列宽而不是改配置,是因为配置本来
|
||||||
|
// 就是给箱主按需调小的,不该反过来限制它的上界。
|
||||||
|
//
|
||||||
|
// ⚠️ 改这个常量时必须同步改下面 Content 字段的 size 标签(结构体标签不能引用常量)。
|
||||||
|
const MaxVoteOptionLen = 300
|
||||||
|
|
||||||
// VoteOption 选项表:单选/多选/评分/排序共用。
|
// VoteOption 选项表:单选/多选/评分/排序共用。
|
||||||
type VoteOption struct {
|
type VoteOption struct {
|
||||||
dbutil.Model
|
dbutil.Model
|
||||||
VoteID uint `gorm:"index:idx_option_vote" json:"vote_id"`
|
VoteID uint `gorm:"index:idx_option_vote" json:"vote_id"`
|
||||||
Content string `gorm:"size:200" json:"content"`
|
Content string `gorm:"size:300" json:"content"`
|
||||||
SortOrder int `gorm:"default:0" json:"sort_order"`
|
SortOrder int `gorm:"default:0" json:"sort_order"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -114,3 +114,92 @@ func testVoteOptionsCountByVoteID(t *testing.T, ctx context.Context, db *voteOpt
|
|||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(0), got)
|
require.Equal(t, int64(0), got)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestVoteOptionMaxLengthRoundTrip 保证「配置允许的长度」真的能存下。
|
||||||
|
//
|
||||||
|
// 曾经 conf 允许 300 字而列宽只有 200:201–300 字的选项通过校验后在写库时才失败
|
||||||
|
// (PostgreSQL 报 value too long,MySQL 严格模式同样报错),用户看到的是「内部错误」。
|
||||||
|
func TestVoteOptionMaxLengthRoundTrip(t *testing.T) {
|
||||||
|
conn, cleanup := newTestDB(t)
|
||||||
|
t.Cleanup(func() {
|
||||||
|
if err := cleanup("vote_options"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
store := NewVoteOptionsStore(conn)
|
||||||
|
|
||||||
|
// 结构体标签不能引用常量,列宽与 MaxVoteOptionLen 是两处独立字面量,
|
||||||
|
// 这里显式比一次,防止只改了一边。
|
||||||
|
columns, err := conn.Migrator().ColumnTypes(&VoteOption{})
|
||||||
|
require.Nil(t, err)
|
||||||
|
for _, ct := range columns {
|
||||||
|
if ct.Name() != "content" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if length, ok := ct.Length(); ok {
|
||||||
|
require.EqualValues(t, MaxVoteOptionLen, length, "vote_options.content 列宽必须与 MaxVoteOptionLen 一致")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 用多字节字符填满上限,顺便验证按字符而非字节计数。
|
||||||
|
content := ""
|
||||||
|
for len([]rune(content)) < MaxVoteOptionLen {
|
||||||
|
content += "選"
|
||||||
|
}
|
||||||
|
|
||||||
|
option, err := store.Create(ctx, CreateVoteOptionOptions{VoteID: 1, Content: content})
|
||||||
|
require.Nil(t, err)
|
||||||
|
|
||||||
|
got, err := store.ListByVoteID(ctx, 1)
|
||||||
|
require.Nil(t, err)
|
||||||
|
require.Len(t, got, 1)
|
||||||
|
require.Equal(t, option.ID, got[0].ID)
|
||||||
|
require.Equal(t, MaxVoteOptionLen, len([]rune(got[0].Content)), "存进去的字符数应与上限一致,未被截断")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestVoteOptionColumnWidens 验证老库升级路径:已经存在 varchar(200) 的列,
|
||||||
|
// 跑一次 AutoMigrate 后必须变成 MaxVoteOptionLen。
|
||||||
|
//
|
||||||
|
// 只改 model 标签是不够的 —— 如果 AutoMigrate 不扩列,老部署上线后
|
||||||
|
// 201–300 字的选项依旧写不进去,而新库(测试库)永远发现不了这个问题。
|
||||||
|
func TestVoteOptionColumnWidens(t *testing.T) {
|
||||||
|
conn, cleanup := newTestDB(t)
|
||||||
|
t.Cleanup(func() {
|
||||||
|
if err := cleanup("vote_options"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
// 退回老库的列宽(两种方言语法不同)。
|
||||||
|
var alter string
|
||||||
|
switch conn.Dialector.Name() {
|
||||||
|
case "postgres":
|
||||||
|
alter = "ALTER TABLE vote_options ALTER COLUMN content TYPE varchar(200)"
|
||||||
|
case "mysql":
|
||||||
|
alter = "ALTER TABLE vote_options MODIFY COLUMN content varchar(200)"
|
||||||
|
default:
|
||||||
|
t.Skipf("未适配的方言:%s", conn.Dialector.Name())
|
||||||
|
}
|
||||||
|
require.Nil(t, conn.Exec(alter).Error)
|
||||||
|
|
||||||
|
contentLength := func() int64 {
|
||||||
|
columns, err := conn.Migrator().ColumnTypes(&VoteOption{})
|
||||||
|
require.Nil(t, err)
|
||||||
|
for _, ct := range columns {
|
||||||
|
if ct.Name() != "content" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
length, ok := ct.Length()
|
||||||
|
require.True(t, ok, "content 列应有明确长度")
|
||||||
|
return length
|
||||||
|
}
|
||||||
|
t.Fatal("未找到 content 列")
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
require.EqualValues(t, 200, contentLength(), "前置条件:列宽应已被改回 200")
|
||||||
|
|
||||||
|
require.Nil(t, conn.AutoMigrate(&VoteOption{}))
|
||||||
|
require.EqualValues(t, MaxVoteOptionLen, contentLength(), "AutoMigrate 必须把列宽扩到 MaxVoteOptionLen")
|
||||||
|
}
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import (
|
|||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
"gorm.io/datatypes"
|
"gorm.io/datatypes"
|
||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
|
||||||
"github.com/tamakyi/TamaBox/internal/dbutil"
|
"github.com/tamakyi/TamaBox/internal/dbutil"
|
||||||
)
|
)
|
||||||
@@ -25,6 +26,8 @@ type VoteSubmissionsStore interface {
|
|||||||
Create(ctx context.Context, opts CreateVoteSubmissionOptions) (*VoteSubmission, error)
|
Create(ctx context.Context, opts CreateVoteSubmissionOptions) (*VoteSubmission, error)
|
||||||
GetByID(ctx context.Context, id uint) (*VoteSubmission, error)
|
GetByID(ctx context.Context, id uint) (*VoteSubmission, error)
|
||||||
ListByVoteID(ctx context.Context, voteID uint, cursor *dbutil.Cursor) ([]*VoteSubmission, error)
|
ListByVoteID(ctx context.Context, voteID uint, cursor *dbutil.Cursor) ([]*VoteSubmission, error)
|
||||||
|
// ListByVoteIDs 批量查询多个投票的全部提交记录(IN 查询 + 内存分组),避免逐投票 N+1。
|
||||||
|
ListByVoteIDs(ctx context.Context, voteIDs []uint) (map[uint][]*VoteSubmission, error)
|
||||||
CountByVoteID(ctx context.Context, voteID uint) (int64, error)
|
CountByVoteID(ctx context.Context, voteID uint) (int64, error)
|
||||||
CountByVoteIDs(ctx context.Context, voteIDs []uint) (map[uint]int64, error)
|
CountByVoteIDs(ctx context.Context, voteIDs []uint) (map[uint]int64, error)
|
||||||
CountByOptionID(ctx context.Context, voteID, optionID uint) (int64, error)
|
CountByOptionID(ctx context.Context, voteID, optionID uint) (int64, error)
|
||||||
@@ -77,6 +80,7 @@ var (
|
|||||||
// ErrVoteAlreadyVoted 同一投票被同一身份重复提交(数据库唯一索引兜底返回)。
|
// ErrVoteAlreadyVoted 同一投票被同一身份重复提交(数据库唯一索引兜底返回)。
|
||||||
ErrVoteAlreadyVoted = errors.New("你已参与过该投票")
|
ErrVoteAlreadyVoted = errors.New("你已参与过该投票")
|
||||||
ErrVoteSubmissionNotExist = errors.New("投票记录不存在")
|
ErrVoteSubmissionNotExist = errors.New("投票记录不存在")
|
||||||
|
ErrVoteNotAccepting = errors.New("投票未开放或已达到收集上限")
|
||||||
)
|
)
|
||||||
|
|
||||||
// Create 写入一条投票/提交记录。若命中复合唯一索引(同一 user 或同一 token 重复提交),
|
// Create 写入一条投票/提交记录。若命中复合唯一索引(同一 user 或同一 token 重复提交),
|
||||||
@@ -92,6 +96,26 @@ func (db *voteSubmissions) Create(ctx context.Context, opts CreateVoteSubmission
|
|||||||
CustomText: opts.CustomText,
|
CustomText: opts.CustomText,
|
||||||
}
|
}
|
||||||
err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||||
|
if opts.QuotaUserID != 0 {
|
||||||
|
if err := reserveFeatureQuotaTx(tx, opts.QuotaUserID, FeatureVote); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var vote Vote
|
||||||
|
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).First(&vote, opts.VoteID).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "lock vote")
|
||||||
|
}
|
||||||
|
now := time.Now()
|
||||||
|
if vote.Status != VoteStatusPublished || (vote.StartAt != nil && now.Before(*vote.StartAt)) || (vote.EndAt != nil && !now.Before(*vote.EndAt)) {
|
||||||
|
return ErrVoteNotAccepting
|
||||||
|
}
|
||||||
|
var count int64
|
||||||
|
if err := tx.Model(&VoteSubmission{}).Where("vote_id = ?", opts.VoteID).Count(&count).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "count vote submissions")
|
||||||
|
}
|
||||||
|
if vote.QuotaLimit > 0 && count >= int64(vote.QuotaLimit) {
|
||||||
|
return ErrVoteNotAccepting
|
||||||
|
}
|
||||||
if err := tx.Create(submission).Error; err != nil {
|
if err := tx.Create(submission).Error; err != nil {
|
||||||
if isDuplicateKeyError(err) {
|
if isDuplicateKeyError(err) {
|
||||||
return ErrVoteAlreadyVoted
|
return ErrVoteAlreadyVoted
|
||||||
@@ -107,6 +131,11 @@ func (db *voteSubmissions) Create(ctx context.Context, opts CreateVoteSubmission
|
|||||||
return errors.Wrap(err, "increment vote quota")
|
return errors.Wrap(err, "increment vote quota")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if vote.QuotaLimit > 0 && count+1 >= int64(vote.QuotaLimit) {
|
||||||
|
if err := tx.Model(&Vote{}).Where("id = ?", vote.ID).Update("status", VoteStatusEnded).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "end vote on quota")
|
||||||
|
}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -148,6 +177,26 @@ func (db *voteSubmissions) ListByVoteID(ctx context.Context, voteID uint, cursor
|
|||||||
return submissions, nil
|
return submissions, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ListByVoteIDs 批量查询多个投票的全部提交记录(IN 查询 + 内存分组),避免逐投票 N+1。
|
||||||
|
// 返回 map[voteID][]*VoteSubmission;未出现在结果中的 voteID 视为无提交。
|
||||||
|
func (db *voteSubmissions) ListByVoteIDs(ctx context.Context, voteIDs []uint) (map[uint][]*VoteSubmission, error) {
|
||||||
|
out := make(map[uint][]*VoteSubmission, len(voteIDs))
|
||||||
|
if len(voteIDs) == 0 {
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
var submissions []*VoteSubmission
|
||||||
|
if err := db.WithContext(ctx).
|
||||||
|
Where("vote_id IN ?", voteIDs).
|
||||||
|
Order("created_at ASC").
|
||||||
|
Find(&submissions).Error; err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list vote submissions by vote ids")
|
||||||
|
}
|
||||||
|
for _, s := range submissions {
|
||||||
|
out[s.VoteID] = append(out[s.VoteID], s)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
// CountByVoteID 统计某投票的总参与人数。
|
// CountByVoteID 统计某投票的总参与人数。
|
||||||
func (db *voteSubmissions) CountByVoteID(ctx context.Context, voteID uint) (int64, error) {
|
func (db *voteSubmissions) CountByVoteID(ctx context.Context, voteID uint) (int64, error) {
|
||||||
var count int64
|
var count int64
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
|
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"gorm.io/datatypes"
|
"gorm.io/datatypes"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
|
||||||
"github.com/tamakyi/TamaBox/internal/dbutil"
|
"github.com/tamakyi/TamaBox/internal/dbutil"
|
||||||
)
|
)
|
||||||
@@ -53,10 +54,41 @@ func TestVoteSubmissions(t *testing.T) {
|
|||||||
func userIDPtr(id uint) *uint { return &id }
|
func userIDPtr(id uint) *uint { return &id }
|
||||||
func tokenPtr(s string) *string { return &s }
|
func tokenPtr(s string) *string { return &s }
|
||||||
|
|
||||||
|
// seedPublishedVotes 创建 n 个已发布的投票,每个投票带两个选项。
|
||||||
|
// 返回 voteID 列表;调用方用 votes[i] 作为 VoteID、options[voteID][j] 作为 OptionID。
|
||||||
|
// VoteSubmissions.Create 现在要求投票必须 published 才接受提交,旧的测试夹具
|
||||||
|
// 直接用 VoteID=1 的虚拟行会被锁行失败拒绝,因此需要真实投票。
|
||||||
|
func seedPublishedVotes(t *testing.T, ctx context.Context, conn *gorm.DB, userID uint, n int) ([]uint, map[uint][]uint) {
|
||||||
|
t.Helper()
|
||||||
|
votesStore := NewVotesStore(conn)
|
||||||
|
optionsStore := NewVoteOptionsStore(conn)
|
||||||
|
voteIDs := make([]uint, 0, n)
|
||||||
|
optionIDs := make(map[uint][]uint, n)
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
vote, err := votesStore.Create(ctx, CreateVoteOptions{
|
||||||
|
UserID: userID, Title: "seed vote", Type: VoteTypeSingle,
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, votesStore.UpdateStatus(ctx, vote.ID, VoteStatusPublished))
|
||||||
|
voteIDs = append(voteIDs, vote.ID)
|
||||||
|
optionIDs[vote.ID] = make([]uint, 0, 2)
|
||||||
|
for j := 0; j < 2; j++ {
|
||||||
|
opt, err := optionsStore.Create(ctx, CreateVoteOptionOptions{
|
||||||
|
VoteID: vote.ID, Content: "option",
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
optionIDs[vote.ID] = append(optionIDs[vote.ID], opt.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return voteIDs, optionIDs
|
||||||
|
}
|
||||||
|
|
||||||
func testVoteSubmissionsCreate(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsCreate(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 1)
|
||||||
|
voteID := voteIDs[0]
|
||||||
got, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
got, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 2,
|
OptionID: optionIDs[voteID][1],
|
||||||
UserID: userIDPtr(10),
|
UserID: userIDPtr(10),
|
||||||
VoterIP: "114.5.1.4",
|
VoterIP: "114.5.1.4",
|
||||||
})
|
})
|
||||||
@@ -67,9 +99,11 @@ func testVoteSubmissionsCreate(t *testing.T, ctx context.Context, db *voteSubmis
|
|||||||
|
|
||||||
// 核心防刷:同一登录用户对同一投票重复提交必须被数据库唯一索引拦截。
|
// 核心防刷:同一登录用户对同一投票重复提交必须被数据库唯一索引拦截。
|
||||||
func testVoteSubmissionsUniqueUser(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsUniqueUser(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 1)
|
||||||
|
voteID := voteIDs[0]
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
UserID: userIDPtr(10),
|
UserID: userIDPtr(10),
|
||||||
VoterIP: "114.5.1.4",
|
VoterIP: "114.5.1.4",
|
||||||
})
|
})
|
||||||
@@ -77,8 +111,8 @@ func testVoteSubmissionsUniqueUser(t *testing.T, ctx context.Context, db *voteSu
|
|||||||
|
|
||||||
t.Run("duplicate rejected", func(t *testing.T) {
|
t.Run("duplicate rejected", func(t *testing.T) {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 2,
|
OptionID: optionIDs[voteID][1],
|
||||||
UserID: userIDPtr(10),
|
UserID: userIDPtr(10),
|
||||||
VoterIP: "114.5.1.4",
|
VoterIP: "114.5.1.4",
|
||||||
})
|
})
|
||||||
@@ -87,8 +121,8 @@ func testVoteSubmissionsUniqueUser(t *testing.T, ctx context.Context, db *voteSu
|
|||||||
|
|
||||||
t.Run("different user allowed", func(t *testing.T) {
|
t.Run("different user allowed", func(t *testing.T) {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
UserID: userIDPtr(11),
|
UserID: userIDPtr(11),
|
||||||
VoterIP: "114.5.1.5",
|
VoterIP: "114.5.1.5",
|
||||||
})
|
})
|
||||||
@@ -98,9 +132,11 @@ func testVoteSubmissionsUniqueUser(t *testing.T, ctx context.Context, db *voteSu
|
|||||||
|
|
||||||
// 核心防刷:同一匿名指纹对同一投票重复提交必须被拦截。
|
// 核心防刷:同一匿名指纹对同一投票重复提交必须被拦截。
|
||||||
func testVoteSubmissionsUniqueToken(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsUniqueToken(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 1)
|
||||||
|
voteID := voteIDs[0]
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
VoterToken: tokenPtr("fp-token-001"),
|
VoterToken: tokenPtr("fp-token-001"),
|
||||||
VoterIP: "114.5.1.4",
|
VoterIP: "114.5.1.4",
|
||||||
})
|
})
|
||||||
@@ -108,8 +144,8 @@ func testVoteSubmissionsUniqueToken(t *testing.T, ctx context.Context, db *voteS
|
|||||||
|
|
||||||
t.Run("duplicate rejected", func(t *testing.T) {
|
t.Run("duplicate rejected", func(t *testing.T) {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 2,
|
OptionID: optionIDs[voteID][1],
|
||||||
VoterToken: tokenPtr("fp-token-001"),
|
VoterToken: tokenPtr("fp-token-001"),
|
||||||
VoterIP: "114.5.1.4",
|
VoterIP: "114.5.1.4",
|
||||||
})
|
})
|
||||||
@@ -118,8 +154,8 @@ func testVoteSubmissionsUniqueToken(t *testing.T, ctx context.Context, db *voteS
|
|||||||
|
|
||||||
t.Run("different token allowed", func(t *testing.T) {
|
t.Run("different token allowed", func(t *testing.T) {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
VoterToken: tokenPtr("fp-token-002"),
|
VoterToken: tokenPtr("fp-token-002"),
|
||||||
VoterIP: "114.5.1.4",
|
VoterIP: "114.5.1.4",
|
||||||
})
|
})
|
||||||
@@ -129,17 +165,19 @@ func testVoteSubmissionsUniqueToken(t *testing.T, ctx context.Context, db *voteS
|
|||||||
|
|
||||||
// 登录用户与匿名指纹可在同一投票下并存(user_id 为 NULL 与 voter_token 为 NULL 不冲突)。
|
// 登录用户与匿名指纹可在同一投票下并存(user_id 为 NULL 与 voter_token 为 NULL 不冲突)。
|
||||||
func testVoteSubmissionsUserAndAnonCoexist(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsUserAndAnonCoexist(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 1)
|
||||||
|
voteID := voteIDs[0]
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
UserID: userIDPtr(10),
|
UserID: userIDPtr(10),
|
||||||
VoterIP: "114.5.1.4",
|
VoterIP: "114.5.1.4",
|
||||||
})
|
})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
|
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err = db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 2,
|
OptionID: optionIDs[voteID][1],
|
||||||
VoterToken: tokenPtr("fp-anon-001"),
|
VoterToken: tokenPtr("fp-anon-001"),
|
||||||
VoterIP: "114.5.1.6",
|
VoterIP: "114.5.1.6",
|
||||||
})
|
})
|
||||||
@@ -147,25 +185,26 @@ func testVoteSubmissionsUserAndAnonCoexist(t *testing.T, ctx context.Context, db
|
|||||||
|
|
||||||
// 同一 user 换匿名 token 再来:user_id 相同仍应被拦截
|
// 同一 user 换匿名 token 再来:user_id 相同仍应被拦截
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err = db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
UserID: userIDPtr(10),
|
UserID: userIDPtr(10),
|
||||||
VoterToken: tokenPtr("fp-anon-002"),
|
VoterToken: tokenPtr("fp-anon-002"),
|
||||||
VoterIP: "114.5.1.7",
|
VoterIP: "114.5.1.7",
|
||||||
})
|
})
|
||||||
require.Equal(t, ErrVoteAlreadyVoted, err)
|
require.Equal(t, ErrVoteAlreadyVoted, err)
|
||||||
|
|
||||||
count, err := db.CountByVoteID(ctx, 1)
|
count, err := db.CountByVoteID(ctx, voteID)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(2), count)
|
require.Equal(t, int64(2), count)
|
||||||
}
|
}
|
||||||
|
|
||||||
// 同一用户 / 同一 token 对不同投票不受约束。
|
// 同一用户 / 同一 token 对不同投票不受约束。
|
||||||
func testVoteSubmissionsCrossVote(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsCrossVote(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
for voteID := uint(1); voteID <= 2; voteID++ {
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 2)
|
||||||
|
for _, voteID := range voteIDs {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: voteID,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
UserID: userIDPtr(10),
|
UserID: userIDPtr(10),
|
||||||
VoterIP: "114.5.1.4",
|
VoterIP: "114.5.1.4",
|
||||||
})
|
})
|
||||||
@@ -173,50 +212,50 @@ func testVoteSubmissionsCrossVote(t *testing.T, ctx context.Context, db *voteSub
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 同一 token 投两个不同投票也允许
|
// 同一 token 投两个不同投票也允许
|
||||||
|
for _, voteID := range voteIDs {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
VoterToken: tokenPtr("fp-x"),
|
|
||||||
VoterIP: "114.5.1.8",
|
|
||||||
})
|
|
||||||
require.Nil(t, err)
|
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{
|
|
||||||
VoteID: 2,
|
|
||||||
OptionID: 1,
|
|
||||||
VoterToken: tokenPtr("fp-x"),
|
VoterToken: tokenPtr("fp-x"),
|
||||||
VoterIP: "114.5.1.8",
|
VoterIP: "114.5.1.8",
|
||||||
})
|
})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
|
}
|
||||||
|
|
||||||
count, err := db.CountByVoteID(ctx, 1)
|
count, err := db.CountByVoteID(ctx, voteIDs[0])
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(2), count)
|
require.Equal(t, int64(2), count)
|
||||||
}
|
}
|
||||||
|
|
||||||
func testVoteSubmissionsCount(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsCount(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
// 投票 1:选项 1 两票,选项 2 一票
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 2)
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 1, UserID: userIDPtr(1), VoterIP: "a"})
|
vote1, vote2 := voteIDs[0], voteIDs[1]
|
||||||
|
opt1a, opt1b := optionIDs[vote1][0], optionIDs[vote1][1]
|
||||||
|
opt2a := optionIDs[vote2][0]
|
||||||
|
|
||||||
|
// 投票 1:选项 A 两票,选项 B 一票
|
||||||
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: vote1, OptionID: opt1a, UserID: userIDPtr(1), VoterIP: "a"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 1, UserID: userIDPtr(2), VoterIP: "b"})
|
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: vote1, OptionID: opt1a, UserID: userIDPtr(2), VoterIP: "b"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 2, UserID: userIDPtr(3), VoterIP: "c"})
|
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: vote1, OptionID: opt1b, UserID: userIDPtr(3), VoterIP: "c"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
// 投票 2:一票
|
// 投票 2:一票
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 2, OptionID: 1, UserID: userIDPtr(4), VoterIP: "d"})
|
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: vote2, OptionID: opt2a, UserID: userIDPtr(4), VoterIP: "d"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
|
|
||||||
t.Run("total", func(t *testing.T) {
|
t.Run("total", func(t *testing.T) {
|
||||||
got, err := db.CountByVoteID(ctx, 1)
|
got, err := db.CountByVoteID(ctx, vote1)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(3), got)
|
require.Equal(t, int64(3), got)
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("by option", func(t *testing.T) {
|
t.Run("by option", func(t *testing.T) {
|
||||||
got, err := db.CountByOptionID(ctx, 1, 1)
|
got, err := db.CountByOptionID(ctx, vote1, opt1a)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(2), got)
|
require.Equal(t, int64(2), got)
|
||||||
|
|
||||||
got, err = db.CountByOptionID(ctx, 1, 2)
|
got, err = db.CountByOptionID(ctx, vote1, opt1b)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(1), got)
|
require.Equal(t, int64(1), got)
|
||||||
})
|
})
|
||||||
@@ -229,54 +268,62 @@ func testVoteSubmissionsCount(t *testing.T, ctx context.Context, db *voteSubmiss
|
|||||||
}
|
}
|
||||||
|
|
||||||
func testVoteSubmissionsHasUserVoted(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsHasUserVoted(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 1, UserID: userIDPtr(10), VoterIP: "a"})
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 1)
|
||||||
|
voteID := voteIDs[0]
|
||||||
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: voteID, OptionID: optionIDs[voteID][0], UserID: userIDPtr(10), VoterIP: "a"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
|
|
||||||
got, err := db.HasUserVoted(ctx, 1, 10)
|
got, err := db.HasUserVoted(ctx, voteID, 10)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.True(t, got)
|
require.True(t, got)
|
||||||
|
|
||||||
got, err = db.HasUserVoted(ctx, 1, 11)
|
got, err = db.HasUserVoted(ctx, voteID, 11)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.False(t, got)
|
require.False(t, got)
|
||||||
}
|
}
|
||||||
|
|
||||||
func testVoteSubmissionsHasTokenVoted(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsHasTokenVoted(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 1, VoterToken: tokenPtr("fp-1"), VoterIP: "a"})
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 1)
|
||||||
|
voteID := voteIDs[0]
|
||||||
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: voteID, OptionID: optionIDs[voteID][0], VoterToken: tokenPtr("fp-1"), VoterIP: "a"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
|
|
||||||
got, err := db.HasTokenVoted(ctx, 1, "fp-1")
|
got, err := db.HasTokenVoted(ctx, voteID, "fp-1")
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.True(t, got)
|
require.True(t, got)
|
||||||
|
|
||||||
got, err = db.HasTokenVoted(ctx, 1, "fp-2")
|
got, err = db.HasTokenVoted(ctx, voteID, "fp-2")
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.False(t, got)
|
require.False(t, got)
|
||||||
}
|
}
|
||||||
|
|
||||||
func testVoteSubmissionsCountByIP(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsCountByIP(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 1)
|
||||||
|
voteID := voteIDs[0]
|
||||||
since := time.Now().Add(-1 * time.Hour)
|
since := time.Now().Add(-1 * time.Hour)
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 1, UserID: userIDPtr(1), VoterIP: "1.2.3.4"})
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: voteID, OptionID: optionIDs[voteID][0], UserID: userIDPtr(1), VoterIP: "1.2.3.4"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 1, UserID: userIDPtr(2), VoterIP: "1.2.3.4"})
|
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: voteID, OptionID: optionIDs[voteID][0], UserID: userIDPtr(2), VoterIP: "1.2.3.4"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 1, UserID: userIDPtr(3), VoterIP: "5.6.7.8"})
|
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: voteID, OptionID: optionIDs[voteID][0], UserID: userIDPtr(3), VoterIP: "5.6.7.8"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
|
|
||||||
got, err := db.CountByIP(ctx, 1, "1.2.3.4", since)
|
got, err := db.CountByIP(ctx, voteID, "1.2.3.4", since)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(2), got)
|
require.Equal(t, int64(2), got)
|
||||||
|
|
||||||
got, err = db.CountByIP(ctx, 1, "5.6.7.8", since)
|
got, err = db.CountByIP(ctx, voteID, "5.6.7.8", since)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(1), got)
|
require.Equal(t, int64(1), got)
|
||||||
}
|
}
|
||||||
|
|
||||||
func testVoteSubmissionsListByVoteID(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsListByVoteID(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 1)
|
||||||
|
voteID := voteIDs[0]
|
||||||
for i := 0; i < 3; i++ {
|
for i := 0; i < 3; i++ {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{
|
||||||
VoteID: 1,
|
VoteID: voteID,
|
||||||
OptionID: 1,
|
OptionID: optionIDs[voteID][0],
|
||||||
UserID: userIDPtr(uint(100 + i)),
|
UserID: userIDPtr(uint(100 + i)),
|
||||||
VoterIP: "x",
|
VoterIP: "x",
|
||||||
FormData: datatypes.JSON(`{"nickname":"user` + string(rune('0'+i)) + `"}`),
|
FormData: datatypes.JSON(`{"nickname":"user` + string(rune('0'+i)) + `"}`),
|
||||||
@@ -284,25 +331,27 @@ func testVoteSubmissionsListByVoteID(t *testing.T, ctx context.Context, db *vote
|
|||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
got, err := db.ListByVoteID(ctx, 1, &dbutil.Cursor{Value: 2, PageSize: 10})
|
got, err := db.ListByVoteID(ctx, voteID, &dbutil.Cursor{Value: 2, PageSize: 10})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Len(t, got, 1)
|
require.Len(t, got, 1)
|
||||||
require.Equal(t, uint(1), got[0].ID, "游标取 id < 2 的记录")
|
require.Equal(t, uint(1), got[0].ID, "游标取 id < 2 的记录")
|
||||||
}
|
}
|
||||||
|
|
||||||
func testVoteSubmissionsDeleteByVoteID(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
func testVoteSubmissionsDeleteByVoteID(t *testing.T, ctx context.Context, db *voteSubmissions) {
|
||||||
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 1, OptionID: 1, UserID: userIDPtr(1), VoterIP: "a"})
|
voteIDs, optionIDs := seedPublishedVotes(t, ctx, db.DB, 1, 2)
|
||||||
|
vote1, vote2 := voteIDs[0], voteIDs[1]
|
||||||
|
_, err := db.Create(ctx, CreateVoteSubmissionOptions{VoteID: vote1, OptionID: optionIDs[vote1][0], UserID: userIDPtr(1), VoterIP: "a"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: 2, OptionID: 1, UserID: userIDPtr(2), VoterIP: "b"})
|
_, err = db.Create(ctx, CreateVoteSubmissionOptions{VoteID: vote2, OptionID: optionIDs[vote2][0], UserID: userIDPtr(2), VoterIP: "b"})
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
|
|
||||||
require.Nil(t, db.DeleteByVoteID(ctx, 1))
|
require.Nil(t, db.DeleteByVoteID(ctx, vote1))
|
||||||
|
|
||||||
got, err := db.CountByVoteID(ctx, 1)
|
got, err := db.CountByVoteID(ctx, vote1)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(0), got)
|
require.Equal(t, int64(0), got)
|
||||||
|
|
||||||
got, err = db.CountByVoteID(ctx, 2)
|
got, err = db.CountByVoteID(ctx, vote2)
|
||||||
require.Nil(t, err)
|
require.Nil(t, err)
|
||||||
require.Equal(t, int64(1), got)
|
require.Equal(t, int64(1), got)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -61,6 +61,9 @@ const (
|
|||||||
|
|
||||||
type VotesStore interface {
|
type VotesStore interface {
|
||||||
Create(ctx context.Context, opts CreateVoteOptions) (*Vote, error)
|
Create(ctx context.Context, opts CreateVoteOptions) (*Vote, error)
|
||||||
|
// CreateWithOptions 在单个事务里创建投票 + 选项 + 表单字段。
|
||||||
|
// 任一步失败整体回滚,避免留下「有投票无选项」或「有选项无字段」的半成品。
|
||||||
|
CreateWithOptions(ctx context.Context, opts CreateVoteOptions, optionContents []string, formFields []CreateVoteFormFieldOptions) (*Vote, error)
|
||||||
GetByID(ctx context.Context, id uint) (*Vote, error)
|
GetByID(ctx context.Context, id uint) (*Vote, error)
|
||||||
GetByToken(ctx context.Context, token string) (*Vote, error)
|
GetByToken(ctx context.Context, token string) (*Vote, error)
|
||||||
ListByUserID(ctx context.Context, userID uint, opts ListVotesOptions) ([]*Vote, error)
|
ListByUserID(ctx context.Context, userID uint, opts ListVotesOptions) ([]*Vote, error)
|
||||||
@@ -68,6 +71,9 @@ type VotesStore interface {
|
|||||||
ListPublishedPublic(ctx context.Context, limit int) ([]*Vote, error)
|
ListPublishedPublic(ctx context.Context, limit int) ([]*Vote, error)
|
||||||
GetHot(ctx context.Context, limit int) ([]*Vote, error)
|
GetHot(ctx context.Context, limit int) ([]*Vote, error)
|
||||||
Update(ctx context.Context, id uint, opts UpdateVoteOptions) error
|
Update(ctx context.Context, id uint, opts UpdateVoteOptions) error
|
||||||
|
// UpdateWithOptions 在单个事务里更新投票元信息 + 替换选项 + 替换表单字段。
|
||||||
|
// 仅在草稿状态调用(发布后选项/字段锁定,避免破坏历史数据归因)。
|
||||||
|
UpdateWithOptions(ctx context.Context, id uint, opts UpdateVoteOptions, optionContents []string, formFields []CreateVoteFormFieldOptions) error
|
||||||
UpdateStatus(ctx context.Context, id uint, status string) error
|
UpdateStatus(ctx context.Context, id uint, status string) error
|
||||||
DeleteByID(ctx context.Context, id uint) error
|
DeleteByID(ctx context.Context, id uint) error
|
||||||
// DeleteWithCascade 在一个事务里真删投票及其选项、表单字段、提交记录。
|
// DeleteWithCascade 在一个事务里真删投票及其选项、表单字段、提交记录。
|
||||||
@@ -160,6 +166,69 @@ func (db *votes) Create(ctx context.Context, opts CreateVoteOptions) (*Vote, err
|
|||||||
return vote, nil
|
return vote, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CreateWithOptions 在单个事务里创建投票 + 选项 + 表单字段。
|
||||||
|
// 任一步失败整体回滚,避免留下「有投票无选项」或「有选项无字段」的半成品。
|
||||||
|
func (db *votes) CreateWithOptions(ctx context.Context, opts CreateVoteOptions, optionContents []string, formFields []CreateVoteFormFieldOptions) (*Vote, error) {
|
||||||
|
var vote *Vote
|
||||||
|
err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||||
|
v := &Vote{
|
||||||
|
UserID: opts.UserID,
|
||||||
|
Title: opts.Title,
|
||||||
|
Description: opts.Description,
|
||||||
|
Type: opts.Type,
|
||||||
|
Visibility: opts.Visibility,
|
||||||
|
VoterScope: opts.VoterScope,
|
||||||
|
ShowResult: opts.ShowResult,
|
||||||
|
AllowCustom: opts.AllowCustom,
|
||||||
|
MaxChoices: opts.MaxChoices,
|
||||||
|
StartAt: opts.StartAt,
|
||||||
|
EndAt: opts.EndAt,
|
||||||
|
MaxVotesPerFingerprint: opts.MaxVotesPerFingerprint,
|
||||||
|
RequireCaptcha: opts.RequireCaptcha,
|
||||||
|
CensorMetadata: opts.CensorMetadata,
|
||||||
|
QuotaLimit: opts.QuotaLimit,
|
||||||
|
}
|
||||||
|
if err := tx.Create(v).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "create vote")
|
||||||
|
}
|
||||||
|
vote = v
|
||||||
|
|
||||||
|
if len(optionContents) > 0 {
|
||||||
|
options := make([]*VoteOption, 0, len(optionContents))
|
||||||
|
for i, content := range optionContents {
|
||||||
|
options = append(options, &VoteOption{VoteID: v.ID, Content: content, SortOrder: i})
|
||||||
|
}
|
||||||
|
if err := tx.Create(&options).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "create vote options")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(formFields) > 0 {
|
||||||
|
fields := make([]*VoteFormField, 0, len(formFields))
|
||||||
|
for _, f := range formFields {
|
||||||
|
fields = append(fields, &VoteFormField{
|
||||||
|
VoteID: v.ID,
|
||||||
|
FieldType: f.FieldType,
|
||||||
|
Label: f.Label,
|
||||||
|
Required: f.Required,
|
||||||
|
MaxLength: f.MaxLength,
|
||||||
|
Options: f.Options,
|
||||||
|
SortOrder: f.SortOrder,
|
||||||
|
AllowCustom: f.AllowCustom,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if err := tx.Create(&fields).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "create vote form fields")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return vote, nil
|
||||||
|
}
|
||||||
|
|
||||||
var ErrVoteNotExist = errors.New("投票不存在")
|
var ErrVoteNotExist = errors.New("投票不存在")
|
||||||
|
|
||||||
func (db *votes) GetByID(ctx context.Context, id uint) (*Vote, error) {
|
func (db *votes) GetByID(ctx context.Context, id uint) (*Vote, error) {
|
||||||
@@ -335,6 +404,79 @@ func (db *votes) Update(ctx context.Context, id uint, opts UpdateVoteOptions) er
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UpdateWithOptions 在单个事务里更新投票元信息 + 替换选项 + 替换表单字段。
|
||||||
|
// 仅在草稿状态调用(发布后选项/字段锁定,避免破坏历史数据归因)。
|
||||||
|
// 任一步失败整体回滚,避免「元信息已更新、字段只更新一部分」的不一致草稿。
|
||||||
|
func (db *votes) UpdateWithOptions(ctx context.Context, id uint, opts UpdateVoteOptions, optionContents []string, formFields []CreateVoteFormFieldOptions) error {
|
||||||
|
return db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||||
|
updates := map[string]interface{}{
|
||||||
|
"title": opts.Title,
|
||||||
|
"description": opts.Description,
|
||||||
|
"visibility": opts.Visibility,
|
||||||
|
"voter_scope": opts.VoterScope,
|
||||||
|
"show_result": opts.ShowResult,
|
||||||
|
"allow_custom": opts.AllowCustom,
|
||||||
|
"start_at": opts.StartAt,
|
||||||
|
"end_at": opts.EndAt,
|
||||||
|
"max_votes_per_fingerprint": opts.MaxVotesPerFingerprint,
|
||||||
|
"require_captcha": opts.RequireCaptcha,
|
||||||
|
"censor_metadata": opts.CensorMetadata,
|
||||||
|
"quota_limit": opts.QuotaLimit,
|
||||||
|
}
|
||||||
|
if opts.Type != nil {
|
||||||
|
updates["type"] = *opts.Type
|
||||||
|
}
|
||||||
|
if opts.MaxChoices != nil {
|
||||||
|
updates["max_choices"] = *opts.MaxChoices
|
||||||
|
}
|
||||||
|
res := tx.Model(&Vote{}).Where("id = ?", id).Updates(updates)
|
||||||
|
if res.Error != nil {
|
||||||
|
return errors.Wrap(res.Error, "update vote")
|
||||||
|
}
|
||||||
|
if res.RowsAffected == 0 {
|
||||||
|
return ErrVoteNotExist
|
||||||
|
}
|
||||||
|
|
||||||
|
// 替换选项(先删后建)
|
||||||
|
if err := tx.Where("vote_id = ?", id).Delete(&VoteOption{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete old vote options")
|
||||||
|
}
|
||||||
|
if len(optionContents) > 0 {
|
||||||
|
options := make([]*VoteOption, 0, len(optionContents))
|
||||||
|
for i, content := range optionContents {
|
||||||
|
options = append(options, &VoteOption{VoteID: id, Content: content, SortOrder: i})
|
||||||
|
}
|
||||||
|
if err := tx.Create(&options).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "create vote options")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 替换表单字段(先删后建)
|
||||||
|
if err := tx.Where("vote_id = ?", id).Delete(&VoteFormField{}).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "delete old vote form fields")
|
||||||
|
}
|
||||||
|
if len(formFields) > 0 {
|
||||||
|
fields := make([]*VoteFormField, 0, len(formFields))
|
||||||
|
for _, f := range formFields {
|
||||||
|
fields = append(fields, &VoteFormField{
|
||||||
|
VoteID: id,
|
||||||
|
FieldType: f.FieldType,
|
||||||
|
Label: f.Label,
|
||||||
|
Required: f.Required,
|
||||||
|
MaxLength: f.MaxLength,
|
||||||
|
Options: f.Options,
|
||||||
|
SortOrder: f.SortOrder,
|
||||||
|
AllowCustom: f.AllowCustom,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if err := tx.Create(&fields).Error; err != nil {
|
||||||
|
return errors.Wrap(err, "create vote form fields")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
// UpdateStatus 更新投票状态(draft → published → ended → archived)。
|
// UpdateStatus 更新投票状态(draft → published → ended → archived)。
|
||||||
func (db *votes) UpdateStatus(ctx context.Context, id uint, status string) error {
|
func (db *votes) UpdateStatus(ctx context.Context, id uint, status string) error {
|
||||||
res := db.WithContext(ctx).Model(&Vote{}).Where("id = ?", id).Update("status", status)
|
res := db.WithContext(ctx).Model(&Vote{}).Where("id = ?", id).Update("status", status)
|
||||||
|
|||||||
@@ -235,6 +235,9 @@ func SendMessage(ctx context.Context, cache cache.Cache) {
|
|||||||
todayCount, err := db.Users.CountTodayMessagesFromUser(ctx.Request().Context(), pageUser.ID, ctx.User.ID)
|
todayCount, err := db.Users.CountTodayMessagesFromUser(ctx.Request().Context(), pageUser.ID, ctx.User.ID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logrus.WithError(err).Error("Failed to count today messages")
|
logrus.WithError(err).Error("Failed to count today messages")
|
||||||
|
ctx.SetInternalErrorFlash()
|
||||||
|
ctx.Redirect(fmt.Sprintf("/_/%s/chat/%s", domain, token))
|
||||||
|
return
|
||||||
}
|
}
|
||||||
if todayCount >= int64(pageUser.DailyMessageLimit) {
|
if todayCount >= int64(pageUser.DailyMessageLimit) {
|
||||||
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "chat.errDailyLimit", pageUser.DailyMessageLimit))
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "chat.errDailyLimit", pageUser.DailyMessageLimit))
|
||||||
@@ -277,7 +280,8 @@ func SendMessage(ctx context.Context, cache cache.Cache) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 上传图片(如有)
|
// 上传图片(如有)
|
||||||
var imageURL string
|
var imageURL, imageKey string
|
||||||
|
imageAttached := false
|
||||||
if imageFile != nil {
|
if imageFile != nil {
|
||||||
uploadedURL, err := storage.UploadPictureToS3(imageFile, imageHeader)
|
uploadedURL, err := storage.UploadPictureToS3(imageFile, imageHeader)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -291,6 +295,31 @@ func SendMessage(ctx context.Context, cache cache.Cache) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
imageURL = uploadedURL
|
imageURL = uploadedURL
|
||||||
|
defer func() {
|
||||||
|
if imageAttached || imageKey != "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := db.R2DeleteTasks.EnqueueValues(stdctx.Background(), conf.Upload.ImageBucket, []string{imageURL}); err != nil {
|
||||||
|
logrus.WithError(err).Error("Failed to enqueue unresolved uploaded message image compensation")
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
parsedKey, owned, err := storage.ObjectKeyFromCDNURL(imageURL, conf.Upload.ImageBucketCDNHost)
|
||||||
|
if err != nil || !owned || parsedKey == "" {
|
||||||
|
logrus.WithError(err).WithField("owned", owned).Error("Failed to resolve uploaded message image key")
|
||||||
|
ctx.SetInternalErrorFlash()
|
||||||
|
ctx.Redirect(fmt.Sprintf("/_/%s/chat/%s", domain, token))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
imageKey = parsedKey
|
||||||
|
// 只有消息图片元数据成功落库才标记 attached;此前任意失败出口都补偿新对象。
|
||||||
|
defer func() {
|
||||||
|
if imageAttached {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := db.R2DeleteTasks.Enqueue(stdctx.Background(), conf.Upload.ImageBucket, []string{imageKey}); err != nil {
|
||||||
|
logrus.WithError(err).WithField("key", imageKey).Error("Failed to enqueue uploaded message image compensation")
|
||||||
|
}
|
||||||
|
}()
|
||||||
}
|
}
|
||||||
|
|
||||||
// 内容安全检测(仅文本)
|
// 内容安全检测(仅文本)
|
||||||
@@ -304,49 +333,27 @@ func SendMessage(ctx context.Context, cache cache.Cache) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 原子地递增消息计数:真正的上限检查点(先于消息落库,防止超额消息写入)。
|
var image *db.CreateMessageImageOptions
|
||||||
// 并发场景(对话双方同时发送)下,仅当 message_count < max 时才 +1 成功。
|
if imageURL != "" {
|
||||||
ok, err := db.Conversations.IncrementMessageCountIfBelow(ctx.Request().Context(), conv.ID, conv.MaxMessages)
|
image = &db.CreateMessageImageOptions{URL: imageURL, ObjectKey: imageKey, FileName: imageHeader.Filename, FileSize: imageHeader.Size}
|
||||||
if err != nil {
|
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to increment message count")
|
|
||||||
ctx.SetInternalErrorFlash()
|
|
||||||
ctx.Redirect(fmt.Sprintf("/_/%s/chat/%s", domain, token))
|
|
||||||
return
|
|
||||||
}
|
}
|
||||||
if !ok {
|
_, err = db.Conversations.SendMessage(ctx.Request().Context(), db.SendConversationMessageOptions{
|
||||||
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "chat.errLimitReached"))
|
ConversationID: conv.ID, SenderID: ctx.User.ID, Content: content, Image: image,
|
||||||
ctx.Redirect(fmt.Sprintf("/_/%s/chat/%s", domain, token))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// 创建消息
|
|
||||||
msg, err := db.Messages.Create(ctx.Request().Context(), db.CreateMessageOptions{
|
|
||||||
ConversationID: conv.ID,
|
|
||||||
SenderID: ctx.User.ID,
|
|
||||||
Content: content,
|
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
// 计数已递增但消息落库失败:回滚计数,避免占用配额
|
switch {
|
||||||
if rerr := db.Conversations.DecrementMessageCount(ctx.Request().Context(), conv.ID); rerr != nil {
|
case errors.Is(err, db.ErrDailyMessageLimit):
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(rerr).Error("Failed to rollback message count")
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "chat.errDailyLimit", pageUser.DailyMessageLimit))
|
||||||
}
|
case errors.Is(err, db.ErrConversationLimit):
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create message")
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "chat.errLimitReached"))
|
||||||
|
default:
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to send message")
|
||||||
ctx.SetInternalErrorFlash()
|
ctx.SetInternalErrorFlash()
|
||||||
|
}
|
||||||
ctx.Redirect(fmt.Sprintf("/_/%s/chat/%s", domain, token))
|
ctx.Redirect(fmt.Sprintf("/_/%s/chat/%s", domain, token))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
imageAttached = true
|
||||||
// 保存图片记录(如有)
|
|
||||||
if imageURL != "" {
|
|
||||||
if _, err := db.MessageImages.Create(ctx.Request().Context(), db.CreateMessageImageOptions{
|
|
||||||
MessageID: msg.ID,
|
|
||||||
URL: imageURL,
|
|
||||||
FileName: imageHeader.Filename,
|
|
||||||
FileSize: imageHeader.Size,
|
|
||||||
}); err != nil {
|
|
||||||
logrus.WithError(err).Error("Failed to save message image")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// 达到上限时关闭对话(幂等,多实例同时执行也安全)
|
// 达到上限时关闭对话(幂等,多实例同时执行也安全)
|
||||||
if err := db.Conversations.MaybeClose(ctx.Request().Context(), conv.ID, conv.MaxMessages); err != nil {
|
if err := db.Conversations.MaybeClose(ctx.Request().Context(), conv.ID, conv.MaxMessages); err != nil {
|
||||||
|
|||||||
+24
-32
@@ -48,6 +48,29 @@ import (
|
|||||||
"github.com/tamakyi/TamaBox/templates"
|
"github.com/tamakyi/TamaBox/templates"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func sessionOptions() session.Options {
|
||||||
|
options := session.Options{
|
||||||
|
Initer: session.MemoryIniter(),
|
||||||
|
Cookie: session.CookieOptions{
|
||||||
|
HTTPOnly: true,
|
||||||
|
SameSite: http.SameSiteLaxMode,
|
||||||
|
Secure: strings.HasPrefix(conf.App.ExternalURL, "https"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if conf.Database.DSN != "" && (conf.Database.Type == "" || conf.Database.Type == "mysql") {
|
||||||
|
options.Initer = mysql.Initer()
|
||||||
|
options.Config = mysql.Config{DSN: conf.Database.DSN, Lifetime: 7 * 24 * time.Hour}
|
||||||
|
}
|
||||||
|
if conf.Redis.Addr != "" {
|
||||||
|
options.Initer = sessionRedis.Initer()
|
||||||
|
options.Config = sessionRedis.Config{
|
||||||
|
Options: &sessionRedis.Options{Addr: conf.Redis.Addr, Password: conf.Redis.Password, DB: 1},
|
||||||
|
Lifetime: 7 * 24 * time.Hour,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return options
|
||||||
|
}
|
||||||
|
|
||||||
// indexNowKeyPattern 限制 IndexNow 密钥文件名的合法形态:32 位小写十六进制 + ".txt"。
|
// indexNowKeyPattern 限制 IndexNow 密钥文件名的合法形态:32 位小写十六进制 + ".txt"。
|
||||||
// 用于防止把路由参数直接拼进 embed FS 路径时发生路径穿越(如 "../robots.txt")。
|
// 用于防止把路由参数直接拼进 embed FS 路径时发生路径穿越(如 "../robots.txt")。
|
||||||
var indexNowKeyPattern = regexp.MustCompile(`^[0-9a-f]{32}\.txt$`)
|
var indexNowKeyPattern = regexp.MustCompile(`^[0-9a-f]{32}\.txt$`)
|
||||||
@@ -243,38 +266,7 @@ func New() *flamego.Flame {
|
|||||||
gob.Register(time.Time{})
|
gob.Register(time.Time{})
|
||||||
gob.Register(context.Flash{})
|
gob.Register(context.Flash{})
|
||||||
gob.Register(captcha.StoredAnswer{})
|
gob.Register(captcha.StoredAnswer{})
|
||||||
var sessionStorage interface{}
|
sessioner := session.Sessioner(sessionOptions())
|
||||||
initer := session.MemoryIniter()
|
|
||||||
if conf.Database.DSN != "" {
|
|
||||||
initer = mysql.Initer()
|
|
||||||
sessionStorage = mysql.Config{
|
|
||||||
DSN: conf.Database.DSN,
|
|
||||||
Lifetime: 7 * 24 * time.Hour,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if conf.Redis.Addr != "" {
|
|
||||||
initer = sessionRedis.Initer()
|
|
||||||
sessionStorage = sessionRedis.Config{
|
|
||||||
Options: &sessionRedis.Options{
|
|
||||||
Addr: conf.Redis.Addr,
|
|
||||||
Password: conf.Redis.Password,
|
|
||||||
DB: 1,
|
|
||||||
},
|
|
||||||
Lifetime: 7 * 24 * time.Hour,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
sessioner := session.Sessioner(session.Options{
|
|
||||||
Initer: initer,
|
|
||||||
Config: sessionStorage,
|
|
||||||
// 显式声明 cookie 安全属性:HTTPOnly 防 XSS 窃取会话;SameSite=Lax 阻断跨站表单
|
|
||||||
// POST;Secure 仅在 https 对外服务时下发,避免 http 本地开发下 cookie 不生效
|
|
||||||
// (缺失 Secure 时,HSTS 尚未落地的首次访问可被 MITM 诱导明文请求窃取会话)。
|
|
||||||
Cookie: session.CookieOptions{
|
|
||||||
HTTPOnly: true,
|
|
||||||
SameSite: http.SameSiteLaxMode,
|
|
||||||
Secure: strings.HasPrefix(conf.App.ExternalURL, "https"),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
|
|
||||||
// 安全响应头必须注册在静态资源之前:flamego.Static 命中后会终结请求链,
|
// 安全响应头必须注册在静态资源之前:flamego.Static 命中后会终结请求链,
|
||||||
// 放在它后面等于静态资源拿不到 nosniff / HSTS。
|
// 放在它后面等于静态资源拿不到 nosniff / HSTS。
|
||||||
|
|||||||
@@ -0,0 +1,52 @@
|
|||||||
|
package route
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/flamego/session/mysql"
|
||||||
|
sessionRedis "github.com/flamego/session/redis"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSessionOptionsSelectStorageByDatabaseAndRedis(t *testing.T) {
|
||||||
|
oldDatabase, oldRedis, oldExternalURL := conf.Database, conf.Redis, conf.App.ExternalURL
|
||||||
|
t.Cleanup(func() {
|
||||||
|
conf.Database, conf.Redis, conf.App.ExternalURL = oldDatabase, oldRedis, oldExternalURL
|
||||||
|
})
|
||||||
|
conf.Database.DSN = "test-database-dsn"
|
||||||
|
conf.App.ExternalURL = "https://example.test"
|
||||||
|
|
||||||
|
for _, tt := range []struct {
|
||||||
|
name string
|
||||||
|
databaseType string
|
||||||
|
redisAddr string
|
||||||
|
configType interface{}
|
||||||
|
}{
|
||||||
|
{name: "postgres without redis uses memory", databaseType: "postgres"},
|
||||||
|
{name: "postgres with redis uses redis", databaseType: "postgres", redisAddr: "127.0.0.1:6379", configType: sessionRedis.Config{}},
|
||||||
|
{name: "mysql without redis uses mysql", databaseType: "mysql", configType: mysql.Config{}},
|
||||||
|
{name: "mysql with redis uses redis", databaseType: "mysql", redisAddr: "127.0.0.1:6379", configType: sessionRedis.Config{}},
|
||||||
|
{name: "legacy empty database type uses mysql", configType: mysql.Config{}},
|
||||||
|
} {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
conf.Database.Type = tt.databaseType
|
||||||
|
conf.Redis.Addr = tt.redisAddr
|
||||||
|
conf.Redis.Password = "test-only-password"
|
||||||
|
opts := sessionOptions()
|
||||||
|
if tt.configType == nil {
|
||||||
|
require.Nil(t, opts.Config)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
require.IsType(t, tt.configType, opts.Config)
|
||||||
|
switch cfg := opts.Config.(type) {
|
||||||
|
case mysql.Config:
|
||||||
|
require.Equal(t, conf.Database.DSN, cfg.DSN)
|
||||||
|
case sessionRedis.Config:
|
||||||
|
require.Equal(t, tt.redisAddr, cfg.Options.Addr)
|
||||||
|
require.Equal(t, 1, cfg.Options.DB)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
|
"unicode"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ObjectKeyFromCDNURL 将属于 cdnBase 的 HTTPS URL 转成原始 S3 对象 key。
|
||||||
|
// cdnBase 可省略 scheme,也可带 base path。外站或不属于 base path 的 URL
|
||||||
|
// 返回 owned=false 且不报错;格式非法或试图伪装路径的 URL 返回错误。
|
||||||
|
func ObjectKeyFromCDNURL(rawURL, cdnBase string) (key string, owned bool, err error) {
|
||||||
|
base, err := parseCDNBase(cdnBase)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, err
|
||||||
|
}
|
||||||
|
parsed, err := parseCDNURL(rawURL, false)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, err
|
||||||
|
}
|
||||||
|
if !strings.EqualFold(parsed.Host, base.Host) {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
basePath := base.Path
|
||||||
|
if basePath != "" {
|
||||||
|
if parsed.Path == basePath {
|
||||||
|
return "", false, fmt.Errorf("CDN URL does not contain an object key")
|
||||||
|
}
|
||||||
|
prefix := basePath + "/"
|
||||||
|
if !strings.HasPrefix(parsed.Path, prefix) {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
key = strings.TrimPrefix(parsed.Path, prefix)
|
||||||
|
} else {
|
||||||
|
key = strings.TrimPrefix(parsed.Path, "/")
|
||||||
|
}
|
||||||
|
if key == "" {
|
||||||
|
return "", false, fmt.Errorf("CDN URL does not contain an object key")
|
||||||
|
}
|
||||||
|
|
||||||
|
// 只去掉 cdnBase 的路径和前导分隔符;不添加或移除 bucket,也不 Clean,
|
||||||
|
// 因而返回的解码路径仍是实际 S3 key。
|
||||||
|
return key, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseCDNBase(cdnBase string) (*url.URL, error) {
|
||||||
|
if cdnBase == "" {
|
||||||
|
return nil, fmt.Errorf("CDN base URL is empty")
|
||||||
|
}
|
||||||
|
rawBase := cdnBase
|
||||||
|
if strings.HasPrefix(rawBase, "//") {
|
||||||
|
rawBase = "https:" + rawBase
|
||||||
|
} else if !strings.Contains(rawBase, "://") {
|
||||||
|
rawBase = "https://" + rawBase
|
||||||
|
}
|
||||||
|
return parseCDNURL(rawBase, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseCDNURL(rawURL string, isBase bool) (*url.URL, error) {
|
||||||
|
if rawURL == "" {
|
||||||
|
return nil, fmt.Errorf("URL is empty")
|
||||||
|
}
|
||||||
|
if strings.Contains(rawURL, `\`) || containsControl(rawURL) {
|
||||||
|
return nil, fmt.Errorf("URL contains a backslash or control character")
|
||||||
|
}
|
||||||
|
parsed, err := url.Parse(rawURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("parse URL: %w", err)
|
||||||
|
}
|
||||||
|
if !parsed.IsAbs() || !strings.EqualFold(parsed.Scheme, "https") || parsed.Host == "" ||
|
||||||
|
parsed.Hostname() == "" || parsed.Opaque != "" {
|
||||||
|
return nil, fmt.Errorf("URL must be an absolute HTTPS URL")
|
||||||
|
}
|
||||||
|
if parsed.User != nil {
|
||||||
|
return nil, fmt.Errorf("URL must not contain userinfo")
|
||||||
|
}
|
||||||
|
if parsed.RawQuery != "" || parsed.ForceQuery {
|
||||||
|
return nil, fmt.Errorf("URL must not contain a query")
|
||||||
|
}
|
||||||
|
if parsed.Fragment != "" || strings.Contains(rawURL, "#") {
|
||||||
|
return nil, fmt.Errorf("URL must not contain a fragment")
|
||||||
|
}
|
||||||
|
|
||||||
|
// 必须在 url.Parse 的 Path 解码之外检查 EscapedPath:编码斜杠在 Path 中
|
||||||
|
// 与普通分隔符无法区分,若放行会导致 CDN 与 S3 对 key 的解释不一致。
|
||||||
|
escapedPath := strings.ToLower(parsed.EscapedPath())
|
||||||
|
if strings.Contains(escapedPath, "%2f") || strings.Contains(escapedPath, "%5c") {
|
||||||
|
return nil, fmt.Errorf("URL path contains an encoded slash or backslash")
|
||||||
|
}
|
||||||
|
if strings.Contains(parsed.Path, `\`) || containsControl(parsed.Path) {
|
||||||
|
return nil, fmt.Errorf("URL path contains a backslash or control character")
|
||||||
|
}
|
||||||
|
|
||||||
|
if isBase && strings.HasSuffix(parsed.Path, "/") {
|
||||||
|
// 配置中的 CDN base 允许一个末尾斜杠;其余空路径段仍会被拒绝。
|
||||||
|
parsed.Path = strings.TrimSuffix(parsed.Path, "/")
|
||||||
|
parsed.RawPath = ""
|
||||||
|
}
|
||||||
|
if parsed.Path == "" {
|
||||||
|
if isBase {
|
||||||
|
return parsed, nil
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("URL path is empty")
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(parsed.Path, "/") {
|
||||||
|
return nil, fmt.Errorf("URL path must be absolute")
|
||||||
|
}
|
||||||
|
for _, segment := range strings.Split(strings.TrimPrefix(parsed.Path, "/"), "/") {
|
||||||
|
if segment == "" {
|
||||||
|
return nil, fmt.Errorf("URL path contains an empty segment")
|
||||||
|
}
|
||||||
|
if segment == "." || segment == ".." {
|
||||||
|
return nil, fmt.Errorf("URL path contains a dot segment")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return parsed, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsControl(value string) bool {
|
||||||
|
for _, r := range value {
|
||||||
|
if unicode.IsControl(r) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestObjectKeyFromCDNURL(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
rawURL string
|
||||||
|
cdnBase string
|
||||||
|
wantKey string
|
||||||
|
owned bool
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "scheme less base", rawURL: "https://cdn.example.com/2026/11/pic.webp",
|
||||||
|
cdnBase: "cdn.example.com", wantKey: "2026/11/pic.webp", owned: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "base path", rawURL: "https://cdn.example.com/assets/2026/pic.webp",
|
||||||
|
cdnBase: "https://cdn.example.com/assets", wantKey: "2026/pic.webp", owned: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "base trailing slash and escaped space", rawURL: "https://cdn.example.com/tamabox/a%20b.webp",
|
||||||
|
cdnBase: "cdn.example.com/tamabox/", wantKey: "a b.webp", owned: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "host case insensitive", rawURL: "https://CDN.EXAMPLE.COM/a.webp",
|
||||||
|
cdnBase: "cdn.example.com", wantKey: "a.webp", owned: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "preserve bucket prefix", rawURL: "https://cdn.example.com/tamabox/2026/a.webp",
|
||||||
|
cdnBase: "cdn.example.com", wantKey: "tamabox/2026/a.webp", owned: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "external host not owned", rawURL: "https://outside.example.com/a.webp",
|
||||||
|
cdnBase: "cdn.example.com", owned: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "host suffix attack not owned", rawURL: "https://cdn.example.com.evil.test/a.webp",
|
||||||
|
cdnBase: "cdn.example.com", owned: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "base path complete segment", rawURL: "https://cdn.example.com/assets2/a.webp",
|
||||||
|
cdnBase: "cdn.example.com/assets", owned: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "base path case sensitive", rawURL: "https://cdn.example.com/Assets/a.webp",
|
||||||
|
cdnBase: "cdn.example.com/assets", owned: false,
|
||||||
|
},
|
||||||
|
{name: "HTTP rejected", rawURL: "http://cdn.example.com/a", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "relative rejected", rawURL: "/a", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "scheme relative rejected", rawURL: "//cdn.example.com/a", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "userinfo rejected", rawURL: "https://[email protected]/a", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "query rejected", rawURL: "https://cdn.example.com/a?x=1", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "empty query rejected", rawURL: "https://cdn.example.com/a?", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "fragment rejected", rawURL: "https://cdn.example.com/a#fragment", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "empty fragment rejected", rawURL: "https://cdn.example.com/a#", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "backslash rejected", rawURL: `https://cdn.example.com/a\b`, cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "encoded backslash rejected", rawURL: "https://cdn.example.com/a%5Cb", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "encoded slash rejected", rawURL: "https://cdn.example.com/a%2Fb", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "control rejected", rawURL: "https://cdn.example.com/a\x7fb", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "encoded control rejected", rawURL: "https://cdn.example.com/a%0Ab", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "dot rejected", rawURL: "https://cdn.example.com/a/./b", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "dot dot rejected", rawURL: "https://cdn.example.com/a/../b", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "encoded dot dot rejected", rawURL: "https://cdn.example.com/a/%2E%2E/b", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "empty path rejected", rawURL: "https://cdn.example.com", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "root path rejected", rawURL: "https://cdn.example.com/", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "empty middle segment rejected", rawURL: "https://cdn.example.com/a//b", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "empty last segment rejected", rawURL: "https://cdn.example.com/a/", cdnBase: "cdn.example.com", wantErr: true},
|
||||||
|
{name: "missing key below base rejected", rawURL: "https://cdn.example.com/assets", cdnBase: "cdn.example.com/assets", wantErr: true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
key, owned, err := ObjectKeyFromCDNURL(tt.rawURL, tt.cdnBase)
|
||||||
|
if tt.wantErr {
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.False(t, owned)
|
||||||
|
assert.Empty(t, key)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, tt.owned, owned)
|
||||||
|
assert.Equal(t, tt.wantKey, key)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestObjectKeyFromCDNURLRejectsInvalidBase(t *testing.T) {
|
||||||
|
for _, base := range []string{
|
||||||
|
"",
|
||||||
|
"http://cdn.example.com",
|
||||||
|
"cdn.example.com?x=1",
|
||||||
|
"cdn.example.com#fragment",
|
||||||
|
"cdn.example.com/assets/../other",
|
||||||
|
"cdn.example.com/assets//other",
|
||||||
|
"cdn.example.com/assets%2Fother",
|
||||||
|
"[email protected]",
|
||||||
|
} {
|
||||||
|
t.Run(base, func(t *testing.T) {
|
||||||
|
key, owned, err := ObjectKeyFromCDNURL("https://cdn.example.com/assets/a", base)
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.False(t, owned)
|
||||||
|
assert.Empty(t, key)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,9 +5,6 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
|
|
||||||
"github.com/aliyun/aliyun-oss-go-sdk/oss"
|
"github.com/aliyun/aliyun-oss-go-sdk/oss"
|
||||||
"github.com/aws/aws-sdk-go-v2/aws"
|
|
||||||
"github.com/aws/aws-sdk-go-v2/config"
|
|
||||||
"github.com/aws/aws-sdk-go-v2/credentials"
|
|
||||||
"github.com/aws/aws-sdk-go-v2/service/s3"
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
@@ -31,29 +28,13 @@ func GetS3Client() (*s3.Client, error) {
|
|||||||
s3ClientErr = errors.New("S3 endpoint not configured")
|
s3ClientErr = errors.New("S3 endpoint not configured")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
s3Client, s3ClientErr = NewS3Client(
|
||||||
r2Resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
|
context.Background(),
|
||||||
return aws.Endpoint{
|
conf.Upload.ImageEndpoint,
|
||||||
URL: conf.Upload.ImageEndpoint,
|
conf.Upload.ImageBucket,
|
||||||
HostnameImmutable: true,
|
|
||||||
Source: aws.EndpointSourceCustom,
|
|
||||||
}, nil
|
|
||||||
})
|
|
||||||
|
|
||||||
cfg, err := config.LoadDefaultConfig(context.Background(),
|
|
||||||
config.WithEndpointResolverWithOptions(r2Resolver),
|
|
||||||
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
|
|
||||||
conf.Upload.ImageAccessID,
|
conf.Upload.ImageAccessID,
|
||||||
conf.Upload.ImageAccessSecret,
|
conf.Upload.ImageAccessSecret,
|
||||||
"",
|
|
||||||
)),
|
|
||||||
config.WithRegion("auto"),
|
|
||||||
)
|
)
|
||||||
if err != nil {
|
|
||||||
s3ClientErr = errors.Wrap(err, "load S3 config")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
s3Client = s3.NewFromConfig(cfg)
|
|
||||||
})
|
})
|
||||||
return s3Client, s3ClientErr
|
return s3Client, s3ClientErr
|
||||||
}
|
}
|
||||||
|
|||||||
+18
-7
@@ -16,6 +16,7 @@ import (
|
|||||||
"mime/multipart"
|
"mime/multipart"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/aws/aws-sdk-go-v2/aws"
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
"github.com/aws/aws-sdk-go-v2/service/s3"
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
@@ -23,7 +24,6 @@ import (
|
|||||||
"github.com/disintegration/imaging"
|
"github.com/disintegration/imaging"
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
"github.com/sirupsen/logrus"
|
"github.com/sirupsen/logrus"
|
||||||
"github.com/wuhan005/gadget"
|
|
||||||
|
|
||||||
"github.com/tamakyi/TamaBox/internal/conf"
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
"github.com/tamakyi/TamaBox/internal/imagecheck"
|
"github.com/tamakyi/TamaBox/internal/imagecheck"
|
||||||
@@ -189,9 +189,9 @@ func uploadPictureToS3(file multipart.File, fileHeader *multipart.FileHeader, as
|
|||||||
return "", errors.Wrap(err, "get S3 client")
|
return "", errors.Wrap(err, "get S3 client")
|
||||||
}
|
}
|
||||||
|
|
||||||
// 7. 上传到 R2
|
// 7. 上传到 R2。现有上传 API 不接收调用方 context,故显式标记为 TODO;
|
||||||
if err := gadget.Retry(5, func() error {
|
// PutObject 内部会为每次尝试设置独立超时,并对可取消 context 做退避。
|
||||||
_, err := client.PutObject(context.Background(), &s3.PutObjectInput{
|
if err := PutObject(context.TODO(), client, &s3.PutObjectInput{
|
||||||
Bucket: aws.String(conf.Upload.ImageBucket),
|
Bucket: aws.String(conf.Upload.ImageBucket),
|
||||||
Key: aws.String(key),
|
Key: aws.String(key),
|
||||||
Body: bytes.NewReader(uploadData),
|
Body: bytes.NewReader(uploadData),
|
||||||
@@ -200,8 +200,6 @@ func uploadPictureToS3(file multipart.File, fileHeader *multipart.FileHeader, as
|
|||||||
// key 含纳秒时间戳且内容不可变 → 可安全启用 immutable 强缓存,
|
// key 含纳秒时间戳且内容不可变 → 可安全启用 immutable 强缓存,
|
||||||
// 避免每次页面加载都回源校验(曾导致头像/二维码 logo 每次重新下载)。
|
// 避免每次页面加载都回源校验(曾导致头像/二维码 logo 每次重新下载)。
|
||||||
CacheControl: aws.String("public, max-age=31536000, immutable"),
|
CacheControl: aws.String("public, max-age=31536000, immutable"),
|
||||||
})
|
|
||||||
return err
|
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
return "", errors.Wrap(err, "put object after retry")
|
return "", errors.Wrap(err, "put object after retry")
|
||||||
}
|
}
|
||||||
@@ -251,6 +249,12 @@ func fitsWithin(w, h, maxSide int) bool {
|
|||||||
// cwebp 长时间占用 CPU 与内存,故在此显式设限。
|
// cwebp 长时间占用 CPU 与内存,故在此显式设限。
|
||||||
const maxGIFFrames = 200
|
const maxGIFFrames = 200
|
||||||
|
|
||||||
|
// gifConvertTimeout 是单次 cwebp 转换的硬上限。
|
||||||
|
// 帧数限制挡不住「帧少但每帧极大」或畸形数据把 cwebp 拖住的情况;子进程一旦挂住,
|
||||||
|
// 调用它的上传 goroutine 会一直等,最终拖死整个上传路径。超时后 exec 会杀掉子进程,
|
||||||
|
// 调用方按错误回退到原 GIF,不影响上传本身成功。
|
||||||
|
const gifConvertTimeout = 30 * time.Second
|
||||||
|
|
||||||
// ConvertGifToAnimatedWebP 将 GIF 转换为动画 WebP(需要系统安装 cwebp)。
|
// ConvertGifToAnimatedWebP 将 GIF 转换为动画 WebP(需要系统安装 cwebp)。
|
||||||
// cwebp 不可用、GIF 损坏或帧数超限时返回错误,调用方应回退到原始 GIF。
|
// cwebp 不可用、GIF 损坏或帧数超限时返回错误,调用方应回退到原始 GIF。
|
||||||
func ConvertGifToAnimatedWebP(gifData []byte) ([]byte, error) {
|
func ConvertGifToAnimatedWebP(gifData []byte) ([]byte, error) {
|
||||||
@@ -284,9 +288,16 @@ func ConvertGifToAnimatedWebP(gifData []byte) ([]byte, error) {
|
|||||||
|
|
||||||
// -mixed:对透明帧用无损、其余有损,是动画的推荐组合。
|
// -mixed:对透明帧用无损、其余有损,是动画的推荐组合。
|
||||||
// -metadata none:剥离调色板外的元数据,进一步减小体积。
|
// -metadata none:剥离调色板外的元数据,进一步减小体积。
|
||||||
cmd := exec.Command("cwebp", "-q", "80", "-mixed", "-metadata", "none",
|
// 用 CommandContext 而非裸 Command:超时会杀进程,不会让上传 goroutine 无限等待。
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), gifConvertTimeout)
|
||||||
|
defer cancel()
|
||||||
|
cmd := exec.CommandContext(ctx, "cwebp", "-q", "80", "-mixed", "-metadata", "none",
|
||||||
"-mt", "-quiet", tmpGif.Name(), "-o", tmpWebp.Name())
|
"-mt", "-quiet", tmpGif.Name(), "-o", tmpWebp.Name())
|
||||||
if err := cmd.Run(); err != nil {
|
if err := cmd.Run(); err != nil {
|
||||||
|
if errors.Is(ctx.Err(), context.DeadlineExceeded) {
|
||||||
|
logrus.WithField("timeout", gifConvertTimeout).Warn("cwebp 转换超时,回退为原 GIF")
|
||||||
|
return nil, errors.Errorf("cwebp timed out after %s", gifConvertTimeout)
|
||||||
|
}
|
||||||
return nil, errors.Wrap(err, "run cwebp")
|
return nil, errors.Wrap(err, "run cwebp")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,112 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/credentials"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
s3DialTimeout = 10 * time.Second
|
||||||
|
s3TLSHandshakeTimeout = 10 * time.Second
|
||||||
|
s3ResponseHeaderTimeout = 30 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// NewS3Client 构造用于 Cloudflare R2 的 S3 客户端。
|
||||||
|
// endpoint 可以不带路径,也可以带且仅带 /<bucket>;后者会被剥离,
|
||||||
|
// 再由 path-style 请求恰好添加一次 bucket,避免出现 /bucket/bucket/key。
|
||||||
|
func NewS3Client(ctx context.Context, endpoint, bucket, accessID, accessSecret string) (*s3.Client, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
normalizedEndpoint, err := normalizeS3Endpoint(endpoint, bucket)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// 显式构造配置,不加载本机 AWS 配置或默认凭据链,避免离线环境意外访问
|
||||||
|
// 元数据服务,也保证这里只使用调用方提供的静态凭据。
|
||||||
|
cfg := aws.Config{
|
||||||
|
Region: "auto",
|
||||||
|
BaseEndpoint: aws.String(normalizedEndpoint),
|
||||||
|
Credentials: aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider(
|
||||||
|
accessID,
|
||||||
|
accessSecret,
|
||||||
|
"",
|
||||||
|
)),
|
||||||
|
HTTPClient: newS3HTTPClient(),
|
||||||
|
Retryer: func() aws.Retryer {
|
||||||
|
return aws.NopRetryer{}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return s3.NewFromConfig(cfg, func(options *s3.Options) {
|
||||||
|
options.UsePathStyle = true
|
||||||
|
}), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeS3Endpoint(endpoint, bucket string) (string, error) {
|
||||||
|
if bucket == "" || strings.ContainsAny(bucket, `/\`) {
|
||||||
|
return "", fmt.Errorf("invalid S3 bucket %q", bucket)
|
||||||
|
}
|
||||||
|
parsed, err := url.Parse(endpoint)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("parse S3 endpoint: %w", err)
|
||||||
|
}
|
||||||
|
if !parsed.IsAbs() || parsed.Host == "" || parsed.Hostname() == "" || parsed.Opaque != "" ||
|
||||||
|
(!strings.EqualFold(parsed.Scheme, "https") && !strings.EqualFold(parsed.Scheme, "http")) {
|
||||||
|
return "", fmt.Errorf("S3 endpoint must be an absolute HTTP(S) URL: %q", endpoint)
|
||||||
|
}
|
||||||
|
if parsed.User != nil {
|
||||||
|
return "", fmt.Errorf("S3 endpoint must not contain userinfo")
|
||||||
|
}
|
||||||
|
if parsed.RawQuery != "" || parsed.ForceQuery {
|
||||||
|
return "", fmt.Errorf("S3 endpoint must not contain a query")
|
||||||
|
}
|
||||||
|
if parsed.Fragment != "" || strings.Contains(endpoint, "#") {
|
||||||
|
return "", fmt.Errorf("S3 endpoint must not contain a fragment")
|
||||||
|
}
|
||||||
|
|
||||||
|
// EscapedPath 用于精确比较,拒绝用 %62ucket 等编码形式绕过路径约束。
|
||||||
|
// 空路径保持原样;唯一允许的非空路径是 /<bucket>,包括 "/" 在内的
|
||||||
|
// 其他路径均拒绝,避免静默改变 endpoint 的语义。
|
||||||
|
switch parsed.EscapedPath() {
|
||||||
|
case "":
|
||||||
|
return parsed.String(), nil
|
||||||
|
case "/" + bucket:
|
||||||
|
normalized := *parsed
|
||||||
|
normalized.Path = ""
|
||||||
|
normalized.RawPath = ""
|
||||||
|
return normalized.String(), nil
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("S3 endpoint path must be empty or exactly /%s", bucket)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newS3HTTPClient() *http.Client {
|
||||||
|
dialer := &net.Dialer{
|
||||||
|
Timeout: s3DialTimeout,
|
||||||
|
KeepAlive: 30 * time.Second,
|
||||||
|
}
|
||||||
|
transport := &http.Transport{
|
||||||
|
Proxy: http.ProxyFromEnvironment,
|
||||||
|
DialContext: dialer.DialContext,
|
||||||
|
ForceAttemptHTTP2: true,
|
||||||
|
MaxIdleConns: 100,
|
||||||
|
IdleConnTimeout: 90 * time.Second,
|
||||||
|
TLSHandshakeTimeout: s3TLSHandshakeTimeout,
|
||||||
|
ResponseHeaderTimeout: s3ResponseHeaderTimeout,
|
||||||
|
ExpectContinueTimeout: 1 * time.Second,
|
||||||
|
}
|
||||||
|
|
||||||
|
// 不设置 http.Client.Timeout:该超时覆盖读取/写入响应的整个生命周期,
|
||||||
|
// 会截断大对象传输。每次 PutObject/DeleteObject 尝试另有独立 60 秒上限。
|
||||||
|
return &http.Client{Transport: transport}
|
||||||
|
}
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNormalizeS3Endpoint(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
bucket string
|
||||||
|
want string
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{name: "empty path", endpoint: "https://account.example.com", bucket: "images", want: "https://account.example.com"},
|
||||||
|
{name: "strip exact bucket path", endpoint: "https://account.example.com/images", bucket: "images", want: "https://account.example.com"},
|
||||||
|
{name: "httptest HTTP supported", endpoint: "http://127.0.0.1:12345/images", bucket: "images", want: "http://127.0.0.1:12345"},
|
||||||
|
{name: "root path rejected", endpoint: "https://account.example.com/", bucket: "images", wantErr: true},
|
||||||
|
{name: "different path rejected", endpoint: "https://account.example.com/other", bucket: "images", wantErr: true},
|
||||||
|
{name: "bucket trailing slash rejected", endpoint: "https://account.example.com/images/", bucket: "images", wantErr: true},
|
||||||
|
{name: "encoded bucket rejected", endpoint: "https://account.example.com/%69mages", bucket: "images", wantErr: true},
|
||||||
|
{name: "relative rejected", endpoint: "account.example.com", bucket: "images", wantErr: true},
|
||||||
|
{name: "empty hostname rejected", endpoint: "https://:443", bucket: "images", wantErr: true},
|
||||||
|
{name: "unsupported scheme rejected", endpoint: "ftp://account.example.com", bucket: "images", wantErr: true},
|
||||||
|
{name: "query rejected", endpoint: "https://account.example.com?x=1", bucket: "images", wantErr: true},
|
||||||
|
{name: "empty query rejected", endpoint: "https://account.example.com?", bucket: "images", wantErr: true},
|
||||||
|
{name: "fragment rejected", endpoint: "https://account.example.com#fragment", bucket: "images", wantErr: true},
|
||||||
|
{name: "userinfo rejected", endpoint: "https://[email protected]", bucket: "images", wantErr: true},
|
||||||
|
{name: "empty bucket rejected", endpoint: "https://account.example.com", bucket: "", wantErr: true},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := normalizeS3Endpoint(tt.endpoint, tt.bucket)
|
||||||
|
if tt.wantErr {
|
||||||
|
require.Error(t, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, tt.want, got)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewS3ClientConfiguration(t *testing.T) {
|
||||||
|
client, err := NewS3Client(context.Background(), "https://account.example.com/images", "images", "test-id", "test-secret")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
options := client.Options()
|
||||||
|
assert.Equal(t, "auto", options.Region)
|
||||||
|
assert.True(t, options.UsePathStyle)
|
||||||
|
credentials, err := options.Credentials.Retrieve(context.Background())
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "test-id", credentials.AccessKeyID)
|
||||||
|
assert.Equal(t, "test-secret", credentials.SecretAccessKey)
|
||||||
|
|
||||||
|
httpClient, ok := options.HTTPClient.(*http.Client)
|
||||||
|
require.True(t, ok)
|
||||||
|
assert.Zero(t, httpClient.Timeout, "全局 Client.Timeout 会截断大对象传输")
|
||||||
|
transport, ok := httpClient.Transport.(*http.Transport)
|
||||||
|
require.True(t, ok)
|
||||||
|
assert.NotNil(t, transport.DialContext)
|
||||||
|
assert.Positive(t, transport.TLSHandshakeTimeout)
|
||||||
|
assert.Positive(t, transport.ResponseHeaderTimeout)
|
||||||
|
assert.Positive(t, transport.ExpectContinueTimeout)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestS3PutObjectRequestShape(t *testing.T) {
|
||||||
|
type capturedRequest struct {
|
||||||
|
method string
|
||||||
|
path string
|
||||||
|
body string
|
||||||
|
contentType string
|
||||||
|
cacheControl string
|
||||||
|
authorization string
|
||||||
|
}
|
||||||
|
captured := make(chan capturedRequest, 1)
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
body, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
captured <- capturedRequest{
|
||||||
|
method: r.Method,
|
||||||
|
path: r.URL.Path,
|
||||||
|
body: string(body),
|
||||||
|
contentType: r.Header.Get("Content-Type"),
|
||||||
|
cacheControl: r.Header.Get("Cache-Control"),
|
||||||
|
authorization: r.Header.Get("Authorization"),
|
||||||
|
}
|
||||||
|
w.Header().Set("ETag", `"test-etag"`)
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client, err := NewS3Client(context.Background(), server.URL+"/images", "images", "test-id", "test-secret")
|
||||||
|
require.NoError(t, err)
|
||||||
|
err = PutObject(context.Background(), client, &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String("images"),
|
||||||
|
Key: aws.String("2026/11/picture.webp"),
|
||||||
|
Body: bytes.NewReader([]byte("fake-image")),
|
||||||
|
ContentLength: aws.Int64(10),
|
||||||
|
ContentType: aws.String("image/webp"),
|
||||||
|
CacheControl: aws.String("public, max-age=31536000, immutable"),
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
request := <-captured
|
||||||
|
assert.Equal(t, http.MethodPut, request.method)
|
||||||
|
assert.Equal(t, "/images/2026/11/picture.webp", request.path, "bucket 必须且只能出现一次")
|
||||||
|
assert.Equal(t, "fake-image", request.body)
|
||||||
|
assert.Equal(t, "image/webp", request.contentType)
|
||||||
|
assert.Equal(t, "public, max-age=31536000, immutable", request.cacheControl)
|
||||||
|
assert.Contains(t, request.authorization, "Credential=test-id/")
|
||||||
|
assert.Contains(t, request.authorization, "/auto/s3/aws4_request")
|
||||||
|
}
|
||||||
@@ -0,0 +1,167 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
"github.com/aws/smithy-go"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
s3AttemptTimeout = 60 * time.Second
|
||||||
|
s3MaxAttempts = 3
|
||||||
|
s3InitialRetryBackoff = 100 * time.Millisecond
|
||||||
|
)
|
||||||
|
|
||||||
|
// PutObject 上传对象。每次尝试独立限时 60 秒,最多执行 3 次;
|
||||||
|
// SDK 内部重试被关闭,确保重试次数、Body 回绕和错误分类只由本层控制。
|
||||||
|
func PutObject(ctx context.Context, client *s3.Client, input *s3.PutObjectInput) error {
|
||||||
|
if client == nil {
|
||||||
|
return errors.New("nil S3 client")
|
||||||
|
}
|
||||||
|
if input == nil {
|
||||||
|
return errors.New("nil S3 PutObject input")
|
||||||
|
}
|
||||||
|
return doS3RequestWithRetry(ctx, input.Body, func(attemptCtx context.Context) error {
|
||||||
|
_, err := client.PutObject(attemptCtx, input, func(options *s3.Options) {
|
||||||
|
options.Retryer = aws.NopRetryer{}
|
||||||
|
})
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeleteObject 删除对象。S3 Delete 本身幂等;404 也表示目标已经不存在,
|
||||||
|
// 因而不重试并返回成功。其他错误采用与 PutObject 相同的超时和重试规则。
|
||||||
|
func DeleteObject(ctx context.Context, client *s3.Client, bucket, key string) error {
|
||||||
|
if client == nil {
|
||||||
|
return errors.New("nil S3 client")
|
||||||
|
}
|
||||||
|
err := doS3RequestWithRetry(ctx, nil, func(attemptCtx context.Context) error {
|
||||||
|
_, err := client.DeleteObject(attemptCtx, &s3.DeleteObjectInput{
|
||||||
|
Bucket: aws.String(bucket),
|
||||||
|
Key: aws.String(key),
|
||||||
|
}, func(options *s3.Options) {
|
||||||
|
options.Retryer = aws.NopRetryer{}
|
||||||
|
})
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
if statusCode, ok := s3ErrorHTTPStatus(err); ok && statusCode == 404 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func doS3RequestWithRetry(ctx context.Context, body io.Reader, operation func(context.Context) error) error {
|
||||||
|
var (
|
||||||
|
seeker io.Seeker
|
||||||
|
bodyStart int64
|
||||||
|
bodyStartErr error
|
||||||
|
)
|
||||||
|
if body != nil {
|
||||||
|
if candidate, ok := body.(io.Seeker); ok {
|
||||||
|
seeker = candidate
|
||||||
|
// 记录调用时的位置,而不是武断跳到 0,兼容从切片区间上传的 Reader。
|
||||||
|
bodyStart, bodyStartErr = seeker.Seek(0, io.SeekCurrent)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for attempt := 0; attempt < s3MaxAttempts; attempt++ {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if attempt > 0 && seeker != nil {
|
||||||
|
if bodyStartErr != nil {
|
||||||
|
return fmt.Errorf("determine S3 request body start: %w", bodyStartErr)
|
||||||
|
}
|
||||||
|
if _, err := seeker.Seek(bodyStart, io.SeekStart); err != nil {
|
||||||
|
return fmt.Errorf("rewind S3 request body: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
attemptCtx, cancel := context.WithTimeout(ctx, s3AttemptTimeout)
|
||||||
|
err := operation(attemptCtx)
|
||||||
|
cancel()
|
||||||
|
if err == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
if attempt == s3MaxAttempts-1 || !retryableS3Error(err) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
backoff := s3InitialRetryBackoff << attempt
|
||||||
|
if err := waitForS3Retry(ctx, backoff); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitForS3Retry(ctx context.Context, delay time.Duration) error {
|
||||||
|
timer := time.NewTimer(delay)
|
||||||
|
defer timer.Stop()
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
case <-timer.C:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type httpStatusCoder interface {
|
||||||
|
HTTPStatusCode() int
|
||||||
|
}
|
||||||
|
|
||||||
|
func s3ErrorHTTPStatus(err error) (int, bool) {
|
||||||
|
var statusErr httpStatusCoder
|
||||||
|
if errors.As(err, &statusErr) {
|
||||||
|
return statusErr.HTTPStatusCode(), true
|
||||||
|
}
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
|
||||||
|
func retryableS3Error(err error) bool {
|
||||||
|
if err == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if statusCode, ok := s3ErrorHTTPStatus(err); ok {
|
||||||
|
return statusCode == 408 || statusCode == 429 || statusCode >= 500 && statusCode <= 599
|
||||||
|
}
|
||||||
|
if errors.Is(err, context.Canceled) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// 父 context 已结束时会在调用方提前返回;这里的 DeadlineExceeded 是
|
||||||
|
// 单次尝试或传输层超时,可以在父 context 仍有效时重试。
|
||||||
|
if errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
var networkErr net.Error
|
||||||
|
if errors.As(err, &networkErr) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// 某些兼容 S3 的实现可能只返回错误码而没有可提取的 HTTP 状态。
|
||||||
|
var apiErr smithy.APIError
|
||||||
|
if errors.As(err, &apiErr) {
|
||||||
|
switch apiErr.ErrorCode() {
|
||||||
|
case "RequestTimeout", "RequestTimeoutException",
|
||||||
|
"Throttling", "ThrottlingException", "TooManyRequestsException", "SlowDown",
|
||||||
|
"InternalError", "InternalServerError", "ServiceUnavailable", "ServiceUnavailableException":
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
|
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPutObjectRetriesAndRewindsBody(t *testing.T) {
|
||||||
|
var attempts atomic.Int32
|
||||||
|
bodies := make(chan string, 3)
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
body, _ := io.ReadAll(r.Body)
|
||||||
|
bodies <- string(body)
|
||||||
|
if attempts.Add(1) < 3 {
|
||||||
|
http.Error(w, "temporary", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client, err := NewS3Client(context.Background(), server.URL, "images", "id", "secret")
|
||||||
|
require.NoError(t, err)
|
||||||
|
err = PutObject(context.Background(), client, &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String("images"),
|
||||||
|
Key: aws.String("retry.txt"),
|
||||||
|
Body: bytes.NewReader([]byte("complete-body")),
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, int32(3), attempts.Load())
|
||||||
|
close(bodies)
|
||||||
|
for body := range bodies {
|
||||||
|
assert.Equal(t, "complete-body", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPutObjectDoesNotRetryPermanentErrors(t *testing.T) {
|
||||||
|
for _, statusCode := range []int{http.StatusForbidden, http.StatusNotFound} {
|
||||||
|
t.Run(http.StatusText(statusCode), func(t *testing.T) {
|
||||||
|
var attempts atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
attempts.Add(1)
|
||||||
|
http.Error(w, "permanent", statusCode)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client, err := NewS3Client(context.Background(), server.URL, "images", "id", "secret")
|
||||||
|
require.NoError(t, err)
|
||||||
|
err = PutObject(context.Background(), client, &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String("images"),
|
||||||
|
Key: aws.String("permanent.txt"),
|
||||||
|
Body: bytes.NewReader([]byte("body")),
|
||||||
|
})
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Equal(t, int32(1), attempts.Load())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPutObjectRetriesTemporaryStatusCodes(t *testing.T) {
|
||||||
|
for _, statusCode := range []int{http.StatusRequestTimeout, http.StatusTooManyRequests, http.StatusBadGateway} {
|
||||||
|
t.Run(http.StatusText(statusCode), func(t *testing.T) {
|
||||||
|
var attempts atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if attempts.Add(1) == 1 {
|
||||||
|
http.Error(w, "temporary", statusCode)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client, err := NewS3Client(context.Background(), server.URL, "images", "id", "secret")
|
||||||
|
require.NoError(t, err)
|
||||||
|
err = PutObject(context.Background(), client, &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String("images"),
|
||||||
|
Key: aws.String("temporary.txt"),
|
||||||
|
Body: bytes.NewReader([]byte("body")),
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, int32(2), attempts.Load())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPutObjectStopsAfterThreeAttempts(t *testing.T) {
|
||||||
|
var attempts atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
attempts.Add(1)
|
||||||
|
http.Error(w, "still unavailable", http.StatusServiceUnavailable)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client, err := NewS3Client(context.Background(), server.URL, "images", "id", "secret")
|
||||||
|
require.NoError(t, err)
|
||||||
|
err = PutObject(context.Background(), client, &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String("images"),
|
||||||
|
Key: aws.String("unavailable.txt"),
|
||||||
|
Body: bytes.NewReader([]byte("body")),
|
||||||
|
})
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Equal(t, int32(3), attempts.Load())
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPutObjectHonorsCanceledContext(t *testing.T) {
|
||||||
|
var attempts atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
attempts.Add(1)
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client, err := NewS3Client(context.Background(), server.URL, "images", "id", "secret")
|
||||||
|
require.NoError(t, err)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
err = PutObject(ctx, client, &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String("images"),
|
||||||
|
Key: aws.String("canceled.txt"),
|
||||||
|
Body: bytes.NewReader([]byte("body")),
|
||||||
|
})
|
||||||
|
require.ErrorIs(t, err, context.Canceled)
|
||||||
|
assert.Zero(t, attempts.Load())
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRetryBackoffCanBeCanceled(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
done <- waitForS3Retry(ctx, time.Hour)
|
||||||
|
}()
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case err := <-done:
|
||||||
|
require.ErrorIs(t, err, context.Canceled)
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("退避未响应 context 取消")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNetworkErrorsAreRetryable(t *testing.T) {
|
||||||
|
assert.True(t, retryableS3Error(&net.DNSError{Err: "temporary resolver failure", IsTimeout: true}))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDeleteObjectNotFoundIsIdempotent(t *testing.T) {
|
||||||
|
var attempts atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
attempts.Add(1)
|
||||||
|
assert.Equal(t, http.MethodDelete, r.Method)
|
||||||
|
http.Error(w, "not found", http.StatusNotFound)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client, err := NewS3Client(context.Background(), server.URL, "images", "id", "secret")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, DeleteObject(context.Background(), client, "images", "already-gone.txt"))
|
||||||
|
assert.Equal(t, int32(1), attempts.Load())
|
||||||
|
}
|
||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
|
|
||||||
"github.com/flamego/flamego"
|
"github.com/flamego/flamego"
|
||||||
|
|
||||||
|
"go.opentelemetry.io/otel/attribute"
|
||||||
"go.opentelemetry.io/otel/propagation"
|
"go.opentelemetry.io/otel/propagation"
|
||||||
semconv "go.opentelemetry.io/otel/semconv/v1.12.0"
|
semconv "go.opentelemetry.io/otel/semconv/v1.12.0"
|
||||||
oteltrace "go.opentelemetry.io/otel/trace"
|
oteltrace "go.opentelemetry.io/otel/trace"
|
||||||
@@ -31,16 +32,39 @@ func Middleware(service string, opts ...Option) flamego.Handler {
|
|||||||
c.Request().Request = c.Request().Request.WithContext(savedCtx)
|
c.Request().Request = c.Request().Request.WithContext(savedCtx)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
|
request := c.Request().Request
|
||||||
ctx := cfg.Propagators.Extract(savedCtx, propagation.HeaderCarrier(c.Request().Header))
|
ctx := cfg.Propagators.Extract(savedCtx, propagation.HeaderCarrier(c.Request().Header))
|
||||||
|
method := request.Method
|
||||||
|
if method == "" {
|
||||||
|
method = http.MethodGet
|
||||||
|
}
|
||||||
|
httpAttrs := []attribute.KeyValue{
|
||||||
|
semconv.HTTPMethodKey.String(method),
|
||||||
|
semconv.HTTPTargetKey.String(request.URL.Path),
|
||||||
|
}
|
||||||
|
if service != "" {
|
||||||
|
httpAttrs = append(httpAttrs, semconv.HTTPServerNameKey.String(service))
|
||||||
|
}
|
||||||
|
if request.TLS != nil {
|
||||||
|
httpAttrs = append(httpAttrs, semconv.HTTPSchemeHTTPS)
|
||||||
|
} else {
|
||||||
|
httpAttrs = append(httpAttrs, semconv.HTTPSchemeHTTP)
|
||||||
|
}
|
||||||
|
switch request.ProtoMajor {
|
||||||
|
case 1:
|
||||||
|
httpAttrs = append(httpAttrs, semconv.HTTPFlavorKey.String(fmt.Sprintf("1.%d", request.ProtoMinor)))
|
||||||
|
case 2:
|
||||||
|
httpAttrs = append(httpAttrs, semconv.HTTPFlavorHTTP20)
|
||||||
|
}
|
||||||
opts := []oteltrace.SpanStartOption{
|
opts := []oteltrace.SpanStartOption{
|
||||||
oteltrace.WithAttributes(semconv.NetAttributesFromHTTPRequest("tcp", c.Request().Request)...),
|
oteltrace.WithAttributes(semconv.NetAttributesFromHTTPRequest("tcp", request)...),
|
||||||
oteltrace.WithAttributes(semconv.EndUserAttributesFromHTTPRequest(c.Request().Request)...),
|
oteltrace.WithAttributes(semconv.EndUserAttributesFromHTTPRequest(request)...),
|
||||||
oteltrace.WithAttributes(semconv.HTTPServerAttributesFromHTTPRequest(service, "", c.Request().Request)...),
|
oteltrace.WithAttributes(httpAttrs...),
|
||||||
oteltrace.WithSpanKind(oteltrace.SpanKindServer),
|
oteltrace.WithSpanKind(oteltrace.SpanKindServer),
|
||||||
}
|
}
|
||||||
// span 名只取 URL.Path,不含 query string:
|
// span 名只取 URL.Path,不含 query string:
|
||||||
// RequestURI 会把 ?t=<问题token>、找回密码 ?code= 等私密参数原样写入 trace。
|
// RequestURI 会把 ?t=<问题token>、找回密码 ?code= 等私密参数原样写入 trace。
|
||||||
spanName := c.Request().URL.Path
|
spanName := request.URL.Path
|
||||||
if spanName == "" {
|
if spanName == "" {
|
||||||
spanName = fmt.Sprintf("HTTP %s route not found", c.Request().Method)
|
spanName = fmt.Sprintf("HTTP %s route not found", c.Request().Method)
|
||||||
}
|
}
|
||||||
|
|||||||
+43
-9
@@ -5,6 +5,7 @@
|
|||||||
package auth
|
package auth
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -73,6 +74,20 @@ func ForgotPasswordAction(ctx context.Context, f form.ForgotPassword, cache cach
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SEC-05:签发新令牌前,先尝试删除该用户上一次的找回令牌。
|
||||||
|
// 用 userID → code 反查键实现:新令牌签发后旧令牌的反查键被覆盖,
|
||||||
|
// 旧 code 在 24h 内虽然物理上还在缓存里,但已经不在用户的"当前令牌"集合中,
|
||||||
|
// checkRecoverPasswordCode 会拒绝使用。
|
||||||
|
userCodeKey := "forgot-password-user-code:" + fmt.Sprint(user.ID)
|
||||||
|
if oldCodeItf, err := cache.Get(ctx.Request().Context(), userCodeKey); err == nil {
|
||||||
|
if oldCode, ok := oldCodeItf.(string); ok && oldCode != "" {
|
||||||
|
oldCodeKey := "forgot-password-recovery-code:" + oldCode
|
||||||
|
if err := cache.Delete(ctx.Request().Context(), oldCodeKey); err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Warn("Failed to delete old recovery code")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
code := randstr.String(64)
|
code := randstr.String(64)
|
||||||
recoveryCodeCacheKey := "forgot-password-recovery-code:" + code
|
recoveryCodeCacheKey := "forgot-password-recovery-code:" + code
|
||||||
if err := cache.Set(ctx.Request().Context(), recoveryCodeCacheKey, user.ID, 24*time.Hour); err != nil {
|
if err := cache.Set(ctx.Request().Context(), recoveryCodeCacheKey, user.ID, 24*time.Hour); err != nil {
|
||||||
@@ -82,6 +97,16 @@ func ForgotPasswordAction(ctx context.Context, f form.ForgotPassword, cache cach
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 记录该用户当前的找回令牌 code,供下次签发时撤销。
|
||||||
|
if err := cache.Set(ctx.Request().Context(), userCodeKey, code, 24*time.Hour); err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to set user recovery code cache")
|
||||||
|
// 尽力回滚刚写入的令牌,避免孤儿
|
||||||
|
_ = cache.Delete(ctx.Request().Context(), recoveryCodeCacheKey)
|
||||||
|
ctx.SetInternalErrorFlash()
|
||||||
|
ctx.Redirect("/forgot-password")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
if err := mail.SendPasswordRecoveryMail(user.Email, code); err != nil {
|
if err := mail.SendPasswordRecoveryMail(user.Email, code); err != nil {
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to send password recovery mail")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to send password recovery mail")
|
||||||
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "auth.errEmailFailed"))
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "auth.errEmailFailed"))
|
||||||
@@ -151,15 +176,10 @@ func RecoverPasswordAction(ctx context.Context, cache cache.Cache, f form.Recove
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
code := ctx.Query("code")
|
// SEC-05:先更新密码,成功后再删除令牌。
|
||||||
recoveryCodeCacheKey := "forgot-password-recovery-code:" + code
|
// 旧实现先删令牌再改密码:若密码更新失败,用户必须重新申请。
|
||||||
if err := cache.Delete(ctx.Request().Context(), recoveryCodeCacheKey); err != nil {
|
// 现在密码更新成功才删令牌;即使删令牌失败,密码已改,
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to delete password recovery code cache")
|
// 旧令牌也不再有用(重新登录需要新密码)。
|
||||||
ctx.SetInternalErrorFlash()
|
|
||||||
ctx.Redirect("/login")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := db.Users.UpdatePassword(ctx.Request().Context(), user.ID, f.NewPassword); err != nil {
|
if err := db.Users.UpdatePassword(ctx.Request().Context(), user.ID, f.NewPassword); err != nil {
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to update user password")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to update user password")
|
||||||
ctx.SetInternalErrorFlash()
|
ctx.SetInternalErrorFlash()
|
||||||
@@ -167,6 +187,20 @@ func RecoverPasswordAction(ctx context.Context, cache cache.Cache, f form.Recove
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
code := ctx.Query("code")
|
||||||
|
recoveryCodeCacheKey := "forgot-password-recovery-code:" + code
|
||||||
|
if err := cache.Delete(ctx.Request().Context(), recoveryCodeCacheKey); err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to delete password recovery code cache")
|
||||||
|
// 令牌删除失败不阻断流程:密码已改,旧令牌已经无法用于登录。
|
||||||
|
// 但记录日志便于排查缓存异常。
|
||||||
|
}
|
||||||
|
|
||||||
|
// 同时清理 user → code 反查键,避免下次申请时尝试删除已不存在的旧 code。
|
||||||
|
userCodeKey := "forgot-password-user-code:" + fmt.Sprint(user.ID)
|
||||||
|
if err := cache.Delete(ctx.Request().Context(), userCodeKey); err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Warn("Failed to delete user recovery code mapping")
|
||||||
|
}
|
||||||
|
|
||||||
ctx.SetSuccessFlash(i18n.T(ctx.Lang(), "auth.errPasswordChanged"))
|
ctx.SetSuccessFlash(i18n.T(ctx.Lang(), "auth.errPasswordChanged"))
|
||||||
ctx.Redirect("/login")
|
ctx.Redirect("/login")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -101,6 +101,29 @@ func AskStatusAPI(ctx context.Context) error {
|
|||||||
return ctx.ServerError()
|
return ctx.ServerError()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SEC-06:与 ListAPI(route/question/page.go)对齐,补上 L4 和页面密码检查。
|
||||||
|
// 否则持有正确问题 token 的请求者可以绕过页面访问边界,取得回答状态、
|
||||||
|
// 创建时间和前 60 字摘要。
|
||||||
|
// 两者都返回「用户不存在」以隐藏真实原因(不暴露处置状态与密码设置)。
|
||||||
|
emptyResult := func() error {
|
||||||
|
out := make([]askStatusResult, 0, len(req.Items))
|
||||||
|
for _, it := range req.Items {
|
||||||
|
out = append(out, askStatusResult{ID: it.ID})
|
||||||
|
}
|
||||||
|
return ctx.JSON(map[string]interface{}{"items": out})
|
||||||
|
}
|
||||||
|
// 处置查询失败时不拦(fail-open),与 Pager / ListAPI 的既定口径一致:
|
||||||
|
// 这一层不是内容外传通道,DB 异常时再 fail-closed 只会让所有提问箱一起白屏。
|
||||||
|
// 与它们唯一的差别是这里补了日志,不再静默吞掉错误。
|
||||||
|
if level, _, err := db.Restrictions.ActiveLevel(ctx.Request().Context(), owner.ID); err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to query user restriction for ask status")
|
||||||
|
} else if level == db.RestrictionL4 {
|
||||||
|
return emptyResult()
|
||||||
|
}
|
||||||
|
if !pagePasswordAuthorized(ctx, owner) {
|
||||||
|
return emptyResult()
|
||||||
|
}
|
||||||
|
|
||||||
// 收集合法 ID 并一次查回(IN 查询,避免逐条 N+1)。
|
// 收集合法 ID 并一次查回(IN 查询,避免逐条 N+1)。
|
||||||
ids := make([]uint, 0, len(req.Items))
|
ids := make([]uint, 0, len(req.Items))
|
||||||
for _, it := range req.Items {
|
for _, it := range req.Items {
|
||||||
|
|||||||
+49
-12
@@ -83,17 +83,15 @@ func Pager(ctx context.Context) {
|
|||||||
// 与背景图、颜色同属箱主个性化配置;/user 后台由 Contexter 按登录用户自己的配置注入。
|
// 与背景图、颜色同属箱主个性化配置;/user 后台由 Contexter 按登录用户自己的配置注入。
|
||||||
// 空串 = 不加载主题 css(站点默认就是基础外观时省掉一次请求)。
|
// 空串 = 不加载主题 css(站点默认就是基础外观时省掉一次请求)。
|
||||||
ctx.Data["ThemeSlug"] = theme.Resolve(pageUser.Theme)
|
ctx.Data["ThemeSlug"] = theme.Resolve(pageUser.Theme)
|
||||||
// 页面密码检查
|
// 页面密码检查:session 中保存授权时的 PagePasswordVer 版本号,
|
||||||
if pageUser.PagePassword != "" && !(ctx.IsLogged && ctx.User.ID == pageUser.ID) {
|
// 与当前版本号比对,修改密码后旧授权立即失效(SEC-06)。
|
||||||
passKey := fmt.Sprintf("page_pass_%d", pageUser.ID)
|
if !pagePasswordAuthorized(ctx, pageUser) {
|
||||||
if authorized, _ := ctx.Session.Get(passKey).(bool); !authorized {
|
|
||||||
// 记录原始请求地址(含 query,如带 token 的单题链接),验证通过后跳回此处而非默认首页
|
// 记录原始请求地址(含 query,如带 token 的单题链接),验证通过后跳回此处而非默认首页
|
||||||
ctx.Data["Next"] = ctx.Request().URL.RequestURI()
|
ctx.Data["Next"] = ctx.Request().URL.RequestURI()
|
||||||
ctx.Data["PageUser"] = pageUser
|
ctx.Data["PageUser"] = pageUser
|
||||||
ctx.Success("question/password")
|
ctx.Success("question/password")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
// 分页加载公开已回答的问题(首次加载 20 条,置顶优先)
|
// 分页加载公开已回答的问题(首次加载 20 条,置顶优先)
|
||||||
// 多取 1 条探测是否仍有下一页,供模板决定「加载更多」按钮的初始可见状态,
|
// 多取 1 条探测是否仍有下一页,供模板决定「加载更多」按钮的初始可见状态,
|
||||||
@@ -255,6 +253,32 @@ func PasswordPage(ctx context.Context) {
|
|||||||
ctx.Success("question/password")
|
ctx.Success("question/password")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// pagePasswordAuthorized 判断当前访客是否已通过该箱主的页面密码校验。
|
||||||
|
//
|
||||||
|
// ⚠️ 这里是页面密码的唯一判定入口,HTML 页面与列表 API 都必须走它,不要各写一份。
|
||||||
|
//
|
||||||
|
// 关键点:session 里没有对应键时必须判为**未授权**。
|
||||||
|
// 曾用 `sessionVer, _ := ctx.Session.Get(key).(uint); if sessionVer != user.PagePasswordVer`
|
||||||
|
// 的写法:断言失败同样得到零值 0,而 PagePasswordVer 对存量用户也是 0(新增列默认值),
|
||||||
|
// 于是 0 != 0 为假 —— 升级瞬间所有既有页面密码静默失效,匿名访客无需密码即可看箱主页。
|
||||||
|
// 因此必须先用 ok 断言「确实存在授权记录」,再比较版本号。
|
||||||
|
func pagePasswordAuthorized(ctx context.Context, owner *db.User) bool {
|
||||||
|
if owner.PagePassword == "" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
// 箱主本人不受页面密码限制。
|
||||||
|
if ctx.IsLogged && ctx.User.ID == owner.ID {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
ver, ok := ctx.Session.Get(pagePassKey(owner.ID)).(uint)
|
||||||
|
return ok && ver == owner.PagePasswordVer
|
||||||
|
}
|
||||||
|
|
||||||
|
// pagePassKey 是页面密码授权在 session 里的键名,读写两侧共用,避免格式漂移。
|
||||||
|
func pagePassKey(userID uint) string {
|
||||||
|
return fmt.Sprintf("page_pass_%d", userID)
|
||||||
|
}
|
||||||
|
|
||||||
func VerifyPagePassword(ctx context.Context) {
|
func VerifyPagePassword(ctx context.Context) {
|
||||||
domain := ctx.Param("domain")
|
domain := ctx.Param("domain")
|
||||||
pageUser, err := db.Users.GetByDomain(ctx.Request().Context(), domain)
|
pageUser, err := db.Users.GetByDomain(ctx.Request().Context(), domain)
|
||||||
@@ -277,8 +301,8 @@ func VerifyPagePassword(ctx context.Context) {
|
|||||||
password := ctx.Request().FormValue("password")
|
password := ctx.Request().FormValue("password")
|
||||||
if pageUser.CheckPagePassword(password) {
|
if pageUser.CheckPagePassword(password) {
|
||||||
pagePasswordGuard.Reset(guardKey)
|
pagePasswordGuard.Reset(guardKey)
|
||||||
passKey := fmt.Sprintf("page_pass_%d", pageUser.ID)
|
// 保存当前密码版本号到 session,修改密码后旧授权自动失效(SEC-06)
|
||||||
ctx.Session.Set(passKey, true)
|
ctx.Session.Set(pagePassKey(pageUser.ID), pageUser.PagePasswordVer)
|
||||||
// 验证成功后跳回原始页面(如带 token 的单题链接),避免一律回到首页。
|
// 验证成功后跳回原始页面(如带 token 的单题链接),避免一律回到首页。
|
||||||
// 必须走 shared.SafeRedirectTarget:这里原先自己判「以 / 开头、非 //、不含 ://」,
|
// 必须走 shared.SafeRedirectTarget:这里原先自己判「以 / 开头、非 //、不含 ://」,
|
||||||
// 但没拒反斜杠 —— `next=/\evil.example` 会被浏览器当成 https://evil.example/。
|
// 但没拒反斜杠 —— `next=/\evil.example` 会被浏览器当成 https://evil.example/。
|
||||||
@@ -328,12 +352,9 @@ func ListAPI(ctx context.Context) error {
|
|||||||
if level, _, err := db.Restrictions.ActiveLevel(ctx.Request().Context(), pageUser.ID); err == nil && level == db.RestrictionL4 {
|
if level, _, err := db.Restrictions.ActiveLevel(ctx.Request().Context(), pageUser.ID); err == nil && level == db.RestrictionL4 {
|
||||||
return ctx.JSONError(40400, i18n.T(ctx.Lang(), "auth.errUserNotFound"))
|
return ctx.JSONError(40400, i18n.T(ctx.Lang(), "auth.errUserNotFound"))
|
||||||
}
|
}
|
||||||
if pageUser.PagePassword != "" && !(ctx.IsLogged && ctx.User.ID == pageUser.ID) {
|
if !pagePasswordAuthorized(ctx, pageUser) {
|
||||||
passKey := fmt.Sprintf("page_pass_%d", pageUser.ID)
|
|
||||||
if authorized, _ := ctx.Session.Get(passKey).(bool); !authorized {
|
|
||||||
return ctx.JSONError(40100, i18n.T(ctx.Lang(), "password.desc"))
|
return ctx.JSONError(40100, i18n.T(ctx.Lang(), "password.desc"))
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
pageQuestions, err := db.Questions.GetByUserID(ctx.Request().Context(), pageUser.ID, db.GetQuestionsByUserIDOptions{
|
pageQuestions, err := db.Questions.GetByUserID(ctx.Request().Context(), pageUser.ID, db.GetQuestionsByUserIDOptions{
|
||||||
Cursor: &dbutil.Cursor{
|
Cursor: &dbutil.Cursor{
|
||||||
@@ -471,6 +492,9 @@ func New(ctx context.Context, f form.NewQuestion, pageUser *db.User, recaptcha r
|
|||||||
todayCount, err := db.Users.CountTodayMessagesFromUser(ctx.Request().Context(), pageUser.ID, ctx.User.ID)
|
todayCount, err := db.Users.CountTodayMessagesFromUser(ctx.Request().Context(), pageUser.ID, ctx.User.ID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logrus.WithError(err).Error("Failed to count today messages")
|
logrus.WithError(err).Error("Failed to count today messages")
|
||||||
|
ctx.SetInternalError(f)
|
||||||
|
ctx.Success("question/list")
|
||||||
|
return
|
||||||
}
|
}
|
||||||
if todayCount >= int64(pageUser.DailyMessageLimit) {
|
if todayCount >= int64(pageUser.DailyMessageLimit) {
|
||||||
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "question.errChatDailyLimit", pageUser.DailyMessageLimit))
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "question.errChatDailyLimit", pageUser.DailyMessageLimit))
|
||||||
@@ -558,6 +582,11 @@ func New(ctx context.Context, f form.NewQuestion, pageUser *db.User, recaptcha r
|
|||||||
f.Content,
|
f.Content,
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
if errors.Is(err, db.ErrDailyMessageLimit) {
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "question.errChatDailyLimit", pageUser.DailyMessageLimit))
|
||||||
|
ctx.Redirect("/_/" + pageUser.Domain)
|
||||||
|
return
|
||||||
|
}
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create conversation with first message")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create conversation with first message")
|
||||||
ctx.SetInternalError(f)
|
ctx.SetInternalError(f)
|
||||||
ctx.Success("question/list")
|
ctx.Success("question/list")
|
||||||
@@ -686,6 +715,11 @@ func New(ctx context.Context, f form.NewQuestion, pageUser *db.User, recaptcha r
|
|||||||
IsPrivate: isPrivate,
|
IsPrivate: isPrivate,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
if errors.Is(err, db.ErrFeatureQuotaReached) {
|
||||||
|
ctx.SetErrorFlash(i18n.FeatureClosedMessage(ctx.Lang(), db.FeatureQuestion, db.FeatureReasonQuotaReached))
|
||||||
|
ctx.Redirect("/_/" + pageUser.Domain)
|
||||||
|
return
|
||||||
|
}
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create new question")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create new question")
|
||||||
ctx.SetInternalError(f)
|
ctx.SetInternalError(f)
|
||||||
ctx.Success("question/list")
|
ctx.Success("question/list")
|
||||||
@@ -871,7 +905,7 @@ func uploadImage(ctx context.Context, opts uploadImageOptions) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Wrap(err, "get S3 client")
|
return errors.Wrap(err, "get S3 client")
|
||||||
}
|
}
|
||||||
if _, err := client.PutObject(ctx.Request().Context(), &s3.PutObjectInput{
|
if err := storage.PutObject(ctx.Request().Context(), client, &s3.PutObjectInput{
|
||||||
Bucket: aws.String(conf.Upload.ImageBucket),
|
Bucket: aws.String(conf.Upload.ImageBucket),
|
||||||
Key: aws.String(uploadKey),
|
Key: aws.String(uploadKey),
|
||||||
Body: bytes.NewReader(uploadData),
|
Body: bytes.NewReader(uploadData),
|
||||||
@@ -894,6 +928,9 @@ func uploadImage(ctx context.Context, opts uploadImageOptions) error {
|
|||||||
IsDeletingPrevious: opts.IsDeletingPrevious,
|
IsDeletingPrevious: opts.IsDeletingPrevious,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
if enqueueErr := db.R2DeleteTasks.Enqueue(ctx.Request().Context(), conf.Upload.ImageBucket, []string{uploadKey}); enqueueErr != nil {
|
||||||
|
return errors.Wrapf(enqueueErr, "compensate upload image after metadata failure: %v", err)
|
||||||
|
}
|
||||||
return errors.Wrap(err, "create upload image")
|
return errors.Wrap(err, "create upload image")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -0,0 +1,126 @@
|
|||||||
|
package question
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/context"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/db"
|
||||||
|
)
|
||||||
|
|
||||||
|
// stubSession 只实现 pagePasswordAuthorized 用得到的 Get/Set,其余方法给空实现。
|
||||||
|
type stubSession struct {
|
||||||
|
data map[interface{}]interface{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newStubSession() *stubSession {
|
||||||
|
return &stubSession{data: map[interface{}]interface{}{}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stubSession) ID() string { return "test-session" }
|
||||||
|
func (s *stubSession) RegenerateID(http.ResponseWriter, *http.Request) error {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
func (s *stubSession) Get(key interface{}) interface{} { return s.data[key] }
|
||||||
|
func (s *stubSession) Set(key, val interface{}) { s.data[key] = val }
|
||||||
|
func (s *stubSession) SetFlash(val interface{}) {}
|
||||||
|
func (s *stubSession) Delete(key interface{}) { delete(s.data, key) }
|
||||||
|
func (s *stubSession) Flush() { s.data = map[interface{}]interface{}{} }
|
||||||
|
func (s *stubSession) Encode() ([]byte, error) { return nil, nil }
|
||||||
|
func (s *stubSession) HasChanged() bool { return false }
|
||||||
|
|
||||||
|
// TestPagePasswordAuthorized 是 2026-11-05 P0 的回归用例。
|
||||||
|
//
|
||||||
|
// 旧实现 `ver, _ := Session.Get(key).(uint); if ver != user.PagePasswordVer`
|
||||||
|
// 把「session 里没有授权记录」(断言失败 → 0)与「版本号默认值 0」判成相等,
|
||||||
|
// 于是升级瞬间所有存量页面密码失效——匿名访客不带任何 cookie 就能看箱主页。
|
||||||
|
// 这里把「无 key + ver=0」钉成必须拒绝。
|
||||||
|
func TestPagePasswordAuthorized(t *testing.T) {
|
||||||
|
const userID = 7
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
pagePassword string
|
||||||
|
pageVer uint
|
||||||
|
sessionFill func(*stubSession) // 预置 session,nil 表示完全没有授权键
|
||||||
|
logged bool
|
||||||
|
viewerID uint
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "未设页面密码时直接放行",
|
||||||
|
pagePassword: "",
|
||||||
|
pageVer: 0,
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "存量账号 ver=0 且 session 无授权键必须拒绝",
|
||||||
|
pagePassword: "hashed",
|
||||||
|
pageVer: 0,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "升级前留下的布尔授权(类型不符)作废",
|
||||||
|
pagePassword: "hashed",
|
||||||
|
pageVer: 0,
|
||||||
|
sessionFill: func(s *stubSession) { s.Set(pagePassKey(userID), true) },
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "版本号一致的授权通过",
|
||||||
|
pagePassword: "hashed",
|
||||||
|
pageVer: 1,
|
||||||
|
sessionFill: func(s *stubSession) { s.Set(pagePassKey(userID), uint(1)) },
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "改过密码后旧版本授权失效",
|
||||||
|
pagePassword: "hashed",
|
||||||
|
pageVer: 2,
|
||||||
|
sessionFill: func(s *stubSession) { s.Set(pagePassKey(userID), uint(1)) },
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "session 里是别的类型(如字符串)一律拒绝",
|
||||||
|
pagePassword: "hashed",
|
||||||
|
pageVer: 1,
|
||||||
|
sessionFill: func(s *stubSession) { s.Set(pagePassKey(userID), "1") },
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "箱主本人不受页面密码限制",
|
||||||
|
pagePassword: "hashed",
|
||||||
|
pageVer: 9,
|
||||||
|
logged: true,
|
||||||
|
viewerID: userID,
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "其他登录用户仍需通过密码",
|
||||||
|
pagePassword: "hashed",
|
||||||
|
pageVer: 9,
|
||||||
|
logged: true,
|
||||||
|
viewerID: userID + 1,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
session := newStubSession()
|
||||||
|
if c.sessionFill != nil {
|
||||||
|
c.sessionFill(session)
|
||||||
|
}
|
||||||
|
ctx := context.Context{
|
||||||
|
Session: session,
|
||||||
|
IsLogged: c.logged,
|
||||||
|
User: &db.User{ID: c.viewerID},
|
||||||
|
}
|
||||||
|
owner := &db.User{ID: userID, PagePassword: c.pagePassword, PagePasswordVer: c.pageVer}
|
||||||
|
|
||||||
|
if got := pagePasswordAuthorized(ctx, owner); got != c.want {
|
||||||
|
t.Fatalf("pagePasswordAuthorized() = %v, want %v", got, c.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
+237
-5
@@ -5,6 +5,7 @@
|
|||||||
package user
|
package user
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
stdctx "context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/url"
|
"net/url"
|
||||||
@@ -15,6 +16,7 @@ import (
|
|||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
"github.com/sirupsen/logrus"
|
"github.com/sirupsen/logrus"
|
||||||
"github.com/xuri/excelize/v2"
|
"github.com/xuri/excelize/v2"
|
||||||
|
"gorm.io/datatypes"
|
||||||
|
|
||||||
"github.com/tamakyi/TamaBox/internal/conf"
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
"github.com/tamakyi/TamaBox/internal/context"
|
"github.com/tamakyi/TamaBox/internal/context"
|
||||||
@@ -121,6 +123,17 @@ func UpdateProfile(ctx context.Context, f form.UpdateProfile) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
var uploadedProfileImages []string
|
||||||
|
profileSaved := false
|
||||||
|
defer func() {
|
||||||
|
if profileSaved || len(uploadedProfileImages) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := db.R2DeleteTasks.EnqueueValues(stdctx.Background(), conf.Upload.ImageBucket, uploadedProfileImages); err != nil {
|
||||||
|
logrus.WithError(err).Error("Failed to enqueue uploaded profile image compensation")
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
// 处理头像上传(强制 1:1,见 storage.UploadAvatarToS3)
|
// 处理头像上传(强制 1:1,见 storage.UploadAvatarToS3)
|
||||||
avatarURL := currentUser.Avatar // 默认使用原值
|
avatarURL := currentUser.Avatar // 默认使用原值
|
||||||
avatarFile, avatarFileHeader, err := ctx.Request().FormFile("avatar")
|
avatarFile, avatarFileHeader, err := ctx.Request().FormFile("avatar")
|
||||||
@@ -146,6 +159,7 @@ func UpdateProfile(ctx context.Context, f form.UpdateProfile) {
|
|||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to upload avatar")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to upload avatar")
|
||||||
} else {
|
} else {
|
||||||
avatarURL = uploadedURL
|
avatarURL = uploadedURL
|
||||||
|
uploadedProfileImages = append(uploadedProfileImages, uploadedURL)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -177,6 +191,7 @@ func UpdateProfile(ctx context.Context, f form.UpdateProfile) {
|
|||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to upload background")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to upload background")
|
||||||
} else {
|
} else {
|
||||||
backgroundURL = uploadedURL
|
backgroundURL = uploadedURL
|
||||||
|
uploadedProfileImages = append(uploadedProfileImages, uploadedURL)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -200,6 +215,7 @@ func UpdateProfile(ctx context.Context, f form.UpdateProfile) {
|
|||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to upload QRbackgroundImage")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to upload QRbackgroundImage")
|
||||||
} else {
|
} else {
|
||||||
backgroundImageURL = uploadedURL
|
backgroundImageURL = uploadedURL
|
||||||
|
uploadedProfileImages = append(uploadedProfileImages, uploadedURL)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -268,6 +284,7 @@ func UpdateProfile(ctx context.Context, f form.UpdateProfile) {
|
|||||||
}
|
}
|
||||||
ctx.SetInternalErrorFlash()
|
ctx.SetInternalErrorFlash()
|
||||||
} else if passwordChanged {
|
} else if passwordChanged {
|
||||||
|
profileSaved = true
|
||||||
// 密码修改成功且其余资料也保存成功:强制登出重登。
|
// 密码修改成功且其余资料也保存成功:强制登出重登。
|
||||||
// 先 Flush 清会话再写入 flash,保证提示能带到登录页显示。
|
// 先 Flush 清会话再写入 flash,保证提示能带到登录页显示。
|
||||||
ctx.Session.Flush()
|
ctx.Session.Flush()
|
||||||
@@ -275,6 +292,7 @@ func UpdateProfile(ctx context.Context, f form.UpdateProfile) {
|
|||||||
ctx.Redirect("/login")
|
ctx.Redirect("/login")
|
||||||
return
|
return
|
||||||
} else {
|
} else {
|
||||||
|
profileSaved = true
|
||||||
ctx.SetSuccessFlash(i18n.T(ctx.Lang(), "admin.flash.profileUpdated"))
|
ctx.SetSuccessFlash(i18n.T(ctx.Lang(), "admin.flash.profileUpdated"))
|
||||||
}
|
}
|
||||||
ctx.Redirect("/user/profile")
|
ctx.Redirect("/user/profile")
|
||||||
@@ -482,8 +500,9 @@ func DeactivatedProfile(ctx context.Context) {
|
|||||||
ctx.Success("user/deactivated")
|
ctx.Success("user/deactivated")
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExportProfileJSON 以 JSON 格式全量导出账号数据(用户信息 + 提问/回答/追问 + 私聊消息),
|
// ExportProfileJSON 以 JSON 格式导出账号数据(用户信息 + 提问/回答/追问 + 投票 + 私聊消息),
|
||||||
// 符合可携带权要求,供用户备份/迁移。
|
// 符合可携带权要求,供用户备份/迁移。
|
||||||
|
// FUN-01 修复:补齐投票及提交、私聊图片;子查询失败时 fail-closed 而非静默生成缺数据的文件。
|
||||||
func ExportProfileJSON(ctx context.Context) {
|
func ExportProfileJSON(ctx context.Context) {
|
||||||
user, err := db.Users.GetByID(ctx.Request().Context(), ctx.User.ID)
|
user, err := db.Users.GetByID(ctx.Request().Context(), ctx.User.ID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -525,12 +544,16 @@ func ExportProfileJSON(ctx context.Context) {
|
|||||||
followUpMap, err := db.FollowUps.GetByQuestionIDs(ctx.Request().Context(), questionIDs)
|
followUpMap, err := db.FollowUps.GetByQuestionIDs(ctx.Request().Context(), questionIDs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to batch get follow ups for export")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to batch get follow ups for export")
|
||||||
followUpMap = make(map[uint][]*db.FollowUp)
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
}
|
}
|
||||||
imageMap, err := db.UploadImages.GetByQuestionIDs(ctx.Request().Context(), questionIDs)
|
imageMap, err := db.UploadImages.GetByQuestionIDs(ctx.Request().Context(), questionIDs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to batch get images for export")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to batch get images for export")
|
||||||
imageMap = make(map[uint][]*db.UploadImage)
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
exportQuestions := make([]exportedQuestion, 0, len(questions))
|
exportQuestions := make([]exportedQuestion, 0, len(questions))
|
||||||
@@ -562,6 +585,7 @@ func ExportProfileJSON(ctx context.Context) {
|
|||||||
Content string `json:"content"`
|
Content string `json:"content"`
|
||||||
SenderID uint `json:"sender_id"`
|
SenderID uint `json:"sender_id"`
|
||||||
CreatedAt time.Time `json:"created_at"`
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
Images []string `json:"images"`
|
||||||
}
|
}
|
||||||
type exportedConversation struct {
|
type exportedConversation struct {
|
||||||
OwnerID uint `json:"owner_id"`
|
OwnerID uint `json:"owner_id"`
|
||||||
@@ -575,9 +599,19 @@ func ExportProfileJSON(ctx context.Context) {
|
|||||||
allConvs := make([]*db.Conversation, 0)
|
allConvs := make([]*db.Conversation, 0)
|
||||||
if owned, err := db.Conversations.ListByOwner(ctx.Request().Context(), user.ID, 0, 0); err == nil {
|
if owned, err := db.Conversations.ListByOwner(ctx.Request().Context(), user.ID, 0, 0); err == nil {
|
||||||
allConvs = append(allConvs, owned...)
|
allConvs = append(allConvs, owned...)
|
||||||
|
} else {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to list owned conversations for export")
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
}
|
}
|
||||||
if asked, err := db.Conversations.ListByAsker(ctx.Request().Context(), user.ID, 0, 0); err == nil {
|
if asked, err := db.Conversations.ListByAsker(ctx.Request().Context(), user.ID, 0, 0); err == nil {
|
||||||
allConvs = append(allConvs, asked...)
|
allConvs = append(allConvs, asked...)
|
||||||
|
} else {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to list asked conversations for export")
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
}
|
}
|
||||||
// 去重并收集会话 ID,批量拉取全部消息(IN 查询 + 内存分组),避免逐会话查询造成 N+1。
|
// 去重并收集会话 ID,批量拉取全部消息(IN 查询 + 内存分组),避免逐会话查询造成 N+1。
|
||||||
seen := make(map[uint]bool)
|
seen := make(map[uint]bool)
|
||||||
@@ -594,8 +628,26 @@ func ExportProfileJSON(ctx context.Context) {
|
|||||||
msgMap, err := db.Messages.GetAllByConversationIDs(ctx.Request().Context(), convIDs)
|
msgMap, err := db.Messages.GetAllByConversationIDs(ctx.Request().Context(), convIDs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to batch get messages for export")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to batch get messages for export")
|
||||||
msgMap = make(map[uint][]*db.Message)
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 批量拉取全部消息的图片(IN 查询 + 内存分组),避免逐消息查询造成 N+1。
|
||||||
|
allMessageIDs := make([]uint, 0)
|
||||||
|
for _, msgs := range msgMap {
|
||||||
|
for _, m := range msgs {
|
||||||
|
allMessageIDs = append(allMessageIDs, m.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
msgImageMap, err := db.MessageImages.GetByMessageIDs(ctx.Request().Context(), allMessageIDs)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to batch get message images for export")
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
for _, conv := range dedupedConvs {
|
for _, conv := range dedupedConvs {
|
||||||
msgs := msgMap[conv.ID]
|
msgs := msgMap[conv.ID]
|
||||||
if msgs == nil {
|
if msgs == nil {
|
||||||
@@ -603,10 +655,15 @@ func ExportProfileJSON(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
exportMsgs := make([]exportedMessage, 0, len(msgs))
|
exportMsgs := make([]exportedMessage, 0, len(msgs))
|
||||||
for _, m := range msgs {
|
for _, m := range msgs {
|
||||||
|
imageKeys := make([]string, 0)
|
||||||
|
for _, img := range msgImageMap[m.ID] {
|
||||||
|
imageKeys = append(imageKeys, img.ObjectKey)
|
||||||
|
}
|
||||||
exportMsgs = append(exportMsgs, exportedMessage{
|
exportMsgs = append(exportMsgs, exportedMessage{
|
||||||
Content: m.Content,
|
Content: m.Content,
|
||||||
SenderID: m.SenderID,
|
SenderID: m.SenderID,
|
||||||
CreatedAt: m.CreatedAt,
|
CreatedAt: m.CreatedAt,
|
||||||
|
Images: imageKeys,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
exportConvs = append(exportConvs, exportedConversation{
|
exportConvs = append(exportConvs, exportedConversation{
|
||||||
@@ -618,12 +675,187 @@ func ExportProfileJSON(ctx context.Context) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 投票及提交(FUN-01:补齐投票导出)
|
||||||
|
type exportedVoteOption struct {
|
||||||
|
ID uint `json:"id"`
|
||||||
|
Content string `json:"content"`
|
||||||
|
SortOrder int `json:"sort_order"`
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
}
|
||||||
|
type exportedVoteFormField struct {
|
||||||
|
ID uint `json:"id"`
|
||||||
|
FieldType string `json:"field_type"`
|
||||||
|
Label string `json:"label"`
|
||||||
|
Required bool `json:"required"`
|
||||||
|
MaxLength int `json:"max_length"`
|
||||||
|
Options string `json:"options"`
|
||||||
|
SortOrder int `json:"sort_order"`
|
||||||
|
AllowCustom bool `json:"allow_custom"`
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
}
|
||||||
|
type exportedVoteSubmission struct {
|
||||||
|
ID uint `json:"id"`
|
||||||
|
OptionID uint `json:"option_id"`
|
||||||
|
// 参与者 IP 属于第三方个人数据,不随箱主自助导出外流。
|
||||||
|
// 库里照常保留(防刷、取证、站方 CLI 处置都要用),只是不进这份给箱主的文件。
|
||||||
|
FormData datatypes.JSON `json:"form_data"`
|
||||||
|
CustomText string `json:"custom_text"`
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
}
|
||||||
|
type exportedVote struct {
|
||||||
|
ID uint `json:"id"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Description string `json:"description"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
Status string `json:"status"`
|
||||||
|
Visibility string `json:"visibility"`
|
||||||
|
VoterScope string `json:"voter_scope"`
|
||||||
|
ShowResult string `json:"show_result"`
|
||||||
|
AllowCustom bool `json:"allow_custom"`
|
||||||
|
MaxChoices int `json:"max_choices"`
|
||||||
|
StartAt *time.Time `json:"start_at"`
|
||||||
|
EndAt *time.Time `json:"end_at"`
|
||||||
|
MaxVotesPerFingerprint int `json:"max_votes_per_fingerprint"`
|
||||||
|
RequireCaptcha bool `json:"require_captcha"`
|
||||||
|
QuotaLimit int `json:"quota_limit"`
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
UpdatedAt time.Time `json:"updated_at"`
|
||||||
|
Options []exportedVoteOption `json:"options"`
|
||||||
|
FormFields []exportedVoteFormField `json:"form_fields"`
|
||||||
|
Submissions []exportedVoteSubmission `json:"submissions"`
|
||||||
|
}
|
||||||
|
|
||||||
|
votes, err := db.Votes.ListByUserID(ctx.Request().Context(), user.ID, db.ListVotesOptions{})
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to list votes for export")
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
voteIDs := make([]uint, 0, len(votes))
|
||||||
|
for _, v := range votes {
|
||||||
|
voteIDs = append(voteIDs, v.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 批量拉取选项、字段、提交(IN 查询 + 内存分组),避免逐投票 N+1。
|
||||||
|
voteOptsMap := make(map[uint][]*db.VoteOption)
|
||||||
|
if len(voteIDs) > 0 {
|
||||||
|
for _, v := range votes {
|
||||||
|
opts, err := db.VoteOptions.ListByVoteID(ctx.Request().Context(), v.ID)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to get vote options for export")
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
voteOptsMap[v.ID] = opts
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
voteFieldsMap := make(map[uint][]*db.VoteFormField)
|
||||||
|
if len(voteIDs) > 0 {
|
||||||
|
for _, v := range votes {
|
||||||
|
fields, err := db.VoteFormFields.ListByVoteID(ctx.Request().Context(), v.ID)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to get vote form fields for export")
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
voteFieldsMap[v.ID] = fields
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
voteSubsMap, err := db.VoteSubmissions.ListByVoteIDs(ctx.Request().Context(), voteIDs)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to batch get vote submissions for export")
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "admin.export.errQuestion"))
|
||||||
|
ctx.Redirect("/user/profile")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
exportVotes := make([]exportedVote, 0, len(votes))
|
||||||
|
for _, v := range votes {
|
||||||
|
opts := voteOptsMap[v.ID]
|
||||||
|
if opts == nil {
|
||||||
|
opts = []*db.VoteOption{}
|
||||||
|
}
|
||||||
|
exportOpts := make([]exportedVoteOption, 0, len(opts))
|
||||||
|
for _, o := range opts {
|
||||||
|
exportOpts = append(exportOpts, exportedVoteOption{
|
||||||
|
ID: o.ID,
|
||||||
|
Content: o.Content,
|
||||||
|
SortOrder: o.SortOrder,
|
||||||
|
CreatedAt: o.CreatedAt,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fields := voteFieldsMap[v.ID]
|
||||||
|
if fields == nil {
|
||||||
|
fields = []*db.VoteFormField{}
|
||||||
|
}
|
||||||
|
exportFields := make([]exportedVoteFormField, 0, len(fields))
|
||||||
|
for _, f := range fields {
|
||||||
|
exportFields = append(exportFields, exportedVoteFormField{
|
||||||
|
ID: f.ID,
|
||||||
|
FieldType: f.FieldType,
|
||||||
|
Label: f.Label,
|
||||||
|
Required: f.Required,
|
||||||
|
MaxLength: f.MaxLength,
|
||||||
|
Options: f.Options,
|
||||||
|
SortOrder: f.SortOrder,
|
||||||
|
AllowCustom: f.AllowCustom,
|
||||||
|
CreatedAt: f.CreatedAt,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
subs := voteSubsMap[v.ID]
|
||||||
|
if subs == nil {
|
||||||
|
subs = []*db.VoteSubmission{}
|
||||||
|
}
|
||||||
|
exportSubs := make([]exportedVoteSubmission, 0, len(subs))
|
||||||
|
for _, s := range subs {
|
||||||
|
exportSubs = append(exportSubs, exportedVoteSubmission{
|
||||||
|
ID: s.ID,
|
||||||
|
OptionID: s.OptionID,
|
||||||
|
FormData: s.FormData,
|
||||||
|
CustomText: s.CustomText,
|
||||||
|
CreatedAt: s.CreatedAt,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
exportVotes = append(exportVotes, exportedVote{
|
||||||
|
ID: v.ID,
|
||||||
|
Title: v.Title,
|
||||||
|
Description: v.Description,
|
||||||
|
Type: v.Type,
|
||||||
|
Status: v.Status,
|
||||||
|
Visibility: v.Visibility,
|
||||||
|
VoterScope: v.VoterScope,
|
||||||
|
ShowResult: v.ShowResult,
|
||||||
|
AllowCustom: v.AllowCustom,
|
||||||
|
MaxChoices: v.MaxChoices,
|
||||||
|
StartAt: v.StartAt,
|
||||||
|
EndAt: v.EndAt,
|
||||||
|
MaxVotesPerFingerprint: v.MaxVotesPerFingerprint,
|
||||||
|
RequireCaptcha: v.RequireCaptcha,
|
||||||
|
QuotaLimit: v.QuotaLimit,
|
||||||
|
CreatedAt: v.CreatedAt,
|
||||||
|
UpdatedAt: v.UpdatedAt,
|
||||||
|
Options: exportOpts,
|
||||||
|
FormFields: exportFields,
|
||||||
|
Submissions: exportSubs,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
payload := map[string]interface{}{
|
payload := map[string]interface{}{
|
||||||
"exported_at": time.Now().Format(time.RFC3339),
|
"exported_at": time.Now().Format(time.RFC3339),
|
||||||
"version": 1,
|
"version": 2,
|
||||||
"user": user,
|
"user": user,
|
||||||
"questions": exportQuestions,
|
"questions": exportQuestions,
|
||||||
"conversations": exportConvs,
|
"conversations": exportConvs,
|
||||||
|
"votes": exportVotes,
|
||||||
}
|
}
|
||||||
|
|
||||||
fileName := fmt.Sprintf("提问箱账号数据-%s-%s.json", user.Domain, time.Now().Format("20060102150405"))
|
fileName := fmt.Sprintf("提问箱账号数据-%s-%s.json", user.Domain, time.Now().Format("20060102150405"))
|
||||||
|
|||||||
+67
-59
@@ -36,6 +36,19 @@ var voteTextRepliesCache = cache.NewTTLCache(64)
|
|||||||
|
|
||||||
const voteTextRepliesCacheTTL = 30 * time.Second
|
const voteTextRepliesCacheTTL = 30 * time.Second
|
||||||
|
|
||||||
|
// voteOptionMaxLen 是选项输入的实际上限。
|
||||||
|
//
|
||||||
|
// 配置项 vote_option_len 只保证 ≥1,可以调得比数据库列还大;而 vote_options.content
|
||||||
|
// 是 varchar(300),超出的内容会被数据库拒绝(PostgreSQL 报错、MySQL 严格模式报错)。
|
||||||
|
// 取「配置值」与「存储上限」的较小值,保证"校验通过"等于"能存下",
|
||||||
|
// 也让前端输入框的 maxlength 与实际可保存长度一致。
|
||||||
|
func voteOptionMaxLen() int {
|
||||||
|
if conf.Limits.VoteOptionLen < db.MaxVoteOptionLen {
|
||||||
|
return conf.Limits.VoteOptionLen
|
||||||
|
}
|
||||||
|
return db.MaxVoteOptionLen
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------- 表单解析 ----------------
|
// ---------------- 表单解析 ----------------
|
||||||
|
|
||||||
// voteFormFieldInput 表单字段的前端输入结构(vote-form.html 中的 JS 数组)。
|
// voteFormFieldInput 表单字段的前端输入结构(vote-form.html 中的 JS 数组)。
|
||||||
@@ -244,8 +257,8 @@ func parseOptions(ctx context.Context, form *voteForm, voteType string, maxChoic
|
|||||||
if o == "" {
|
if o == "" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if len([]rune(o)) > conf.Limits.VoteOptionLen {
|
if len([]rune(o)) > voteOptionMaxLen() {
|
||||||
return errors.New(i18n.T(ctx.Lang(), "vote.err.optionTooLong", conf.Limits.VoteOptionLen))
|
return errors.New(i18n.T(ctx.Lang(), "vote.err.optionTooLong", voteOptionMaxLen()))
|
||||||
}
|
}
|
||||||
// 选项内容审核(fail-closed):草稿跳过,发布时统一审核
|
// 选项内容审核(fail-closed):草稿跳过,发布时统一审核
|
||||||
if !draft {
|
if !draft {
|
||||||
@@ -348,8 +361,8 @@ func parseFormFields(ctx context.Context, form *voteForm, draft bool) error {
|
|||||||
return errors.New(i18n.T(ctx.Lang(), "vote.err.fieldReservedOption", i+1, label))
|
return errors.New(i18n.T(ctx.Lang(), "vote.err.fieldReservedOption", i+1, label))
|
||||||
}
|
}
|
||||||
// 长度与单选/多选选项同规则(前端 maxlength 同源,这里补后端兜底校验)
|
// 长度与单选/多选选项同规则(前端 maxlength 同源,这里补后端兜底校验)
|
||||||
if len([]rune(o)) > conf.Limits.VoteOptionLen {
|
if len([]rune(o)) > voteOptionMaxLen() {
|
||||||
return errors.New(i18n.T(ctx.Lang(), "vote.err.optionTooLong", conf.Limits.VoteOptionLen))
|
return errors.New(i18n.T(ctx.Lang(), "vote.err.optionTooLong", voteOptionMaxLen()))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(opts) < 1 {
|
if len(opts) < 1 {
|
||||||
@@ -516,7 +529,7 @@ func VoteList(ctx context.Context) {
|
|||||||
func VoteNew(ctx context.Context) {
|
func VoteNew(ctx context.Context) {
|
||||||
ctx.Data["Title"] = "新建投票"
|
ctx.Data["Title"] = "新建投票"
|
||||||
ctx.Data["IsEdit"] = false
|
ctx.Data["IsEdit"] = false
|
||||||
ctx.Data["MaxVoteOptionLen"] = conf.Limits.VoteOptionLen // 选项输入框 maxlength 同源(vote-form.html JS)
|
ctx.Data["MaxVoteOptionLen"] = voteOptionMaxLen() // 选项输入框 maxlength 同源(vote-form.html JS)
|
||||||
// 默认投票类型为表单收集(用户要求);同时给定模板零值兜底,避免切到多选时 max_choices 显示 0。
|
// 默认投票类型为表单收集(用户要求);同时给定模板零值兜底,避免切到多选时 max_choices 显示 0。
|
||||||
ctx.Data["Vote"] = &db.Vote{
|
ctx.Data["Vote"] = &db.Vote{
|
||||||
Type: db.VoteTypeForm,
|
Type: db.VoteTypeForm,
|
||||||
@@ -534,7 +547,7 @@ func VoteNew(ctx context.Context) {
|
|||||||
// 错误提示直接写 c.Data 而非 session flash:本项目的 flash 在请求开始时就已读进 c.Data,
|
// 错误提示直接写 c.Data 而非 session flash:本项目的 flash 在请求开始时就已读进 c.Data,
|
||||||
// 之后写入 session 的 flash 本次渲染读不到,还会残留到下次请求造成「陈旧错误」串页。
|
// 之后写入 session 的 flash 本次渲染读不到,还会残留到下次请求造成「陈旧错误」串页。
|
||||||
func renderVoteFormWithInput(ctx context.Context, isEdit bool, voteID uint, flashMsg string, internal bool) {
|
func renderVoteFormWithInput(ctx context.Context, isEdit bool, voteID uint, flashMsg string, internal bool) {
|
||||||
ctx.Data["MaxVoteOptionLen"] = conf.Limits.VoteOptionLen // 选项输入框 maxlength 同源(vote-form.html JS)
|
ctx.Data["MaxVoteOptionLen"] = voteOptionMaxLen() // 选项输入框 maxlength 同源(vote-form.html JS)
|
||||||
get := func(n string) string { return strings.TrimSpace(ctx.Request().FormValue(n)) }
|
get := func(n string) string { return strings.TrimSpace(ctx.Request().FormValue(n)) }
|
||||||
title := get("title")
|
title := get("title")
|
||||||
description := get("description")
|
description := get("description")
|
||||||
@@ -626,36 +639,22 @@ func VoteCreate(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
form.opts.CensorMetadata = form.censorMetadata
|
form.opts.CensorMetadata = form.censorMetadata
|
||||||
|
|
||||||
vote, err := db.Votes.Create(ctx.Request().Context(), form.opts)
|
// 事务化创建:投票 + 选项 + 表单字段。任一步失败整体回滚,
|
||||||
|
// 避免留下「有投票无选项」或「有选项无字段」的半成品。
|
||||||
|
vote, err := db.Votes.CreateWithOptions(ctx.Request().Context(), form.opts, form.options, form.formFields)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create vote")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create vote")
|
||||||
renderVoteFormWithInput(ctx, false, 0, "", true)
|
renderVoteFormWithInput(ctx, false, 0, "", true)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(form.options) > 0 {
|
|
||||||
if _, err := db.VoteOptions.BulkCreate(ctx.Request().Context(), vote.ID, form.options); err != nil {
|
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create vote options")
|
|
||||||
_ = db.Votes.DeleteByID(ctx.Request().Context(), vote.ID) // 回滚本次已建的空草稿,避免孤儿记录
|
|
||||||
renderVoteFormWithInput(ctx, false, 0, "", true)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if len(form.formFields) > 0 {
|
|
||||||
for i := range form.formFields {
|
|
||||||
form.formFields[i].VoteID = vote.ID
|
|
||||||
}
|
|
||||||
if _, err := db.VoteFormFields.BulkCreate(ctx.Request().Context(), vote.ID, form.formFields); err != nil {
|
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create vote form fields")
|
|
||||||
_ = db.Votes.DeleteByID(ctx.Request().Context(), vote.ID) // 回滚本次已建的空草稿,避免孤儿记录
|
|
||||||
renderVoteFormWithInput(ctx, false, 0, "", true)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
editURL := "/user/votes/" + fmt.Sprint(vote.ID) + "/edit"
|
editURL := "/user/votes/" + fmt.Sprint(vote.ID) + "/edit"
|
||||||
|
|
||||||
if action == "publish" {
|
if action == "publish" {
|
||||||
|
if !canPublishVote(ctx) {
|
||||||
|
ctx.Redirect(editURL)
|
||||||
|
return
|
||||||
|
}
|
||||||
if err := validateForPublish(ctx, vote); err != nil {
|
if err := validateForPublish(ctx, vote); err != nil {
|
||||||
ctx.SetErrorFlash(err.Error())
|
ctx.SetErrorFlash(err.Error())
|
||||||
ctx.Redirect(editURL)
|
ctx.Redirect(editURL)
|
||||||
@@ -688,7 +687,7 @@ func VoteEdit(ctx context.Context) {
|
|||||||
|
|
||||||
ctx.Data["Vote"] = vote
|
ctx.Data["Vote"] = vote
|
||||||
ctx.Data["IsEdit"] = true
|
ctx.Data["IsEdit"] = true
|
||||||
ctx.Data["MaxVoteOptionLen"] = conf.Limits.VoteOptionLen // 选项输入框 maxlength 同源(vote-form.html JS)
|
ctx.Data["MaxVoteOptionLen"] = voteOptionMaxLen() // 选项输入框 maxlength 同源(vote-form.html JS)
|
||||||
ctx.Data["Title"] = "编辑投票"
|
ctx.Data["Title"] = "编辑投票"
|
||||||
// 草稿保存成功仅通过本次 URL 参数驱动统一 alert,不使用 session flash。
|
// 草稿保存成功仅通过本次 URL 参数驱动统一 alert,不使用 session flash。
|
||||||
if ctx.Query("saved") == "1" {
|
if ctx.Query("saved") == "1" {
|
||||||
@@ -700,10 +699,22 @@ func VoteEdit(ctx context.Context) {
|
|||||||
|
|
||||||
// 选项 / 字段回填传结构化数据,模板经 toJSON(template.JS)注入 <script>,
|
// 选项 / 字段回填传结构化数据,模板经 toJSON(template.JS)注入 <script>,
|
||||||
// 避免 template.HTML 在 script 上下文被转义导致初始化失效。
|
// 避免 template.HTML 在 script 上下文被转义导致初始化失效。
|
||||||
options, _ := db.VoteOptions.ListByVoteID(ctx.Request().Context(), vote.ID)
|
options, err := db.VoteOptions.ListByVoteID(ctx.Request().Context(), vote.ID)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to list vote options")
|
||||||
|
ctx.SetInternalErrorFlash()
|
||||||
|
ctx.Redirect("/user/questions#votes")
|
||||||
|
return
|
||||||
|
}
|
||||||
ctx.Data["InitialOptions"] = optionsContent(options)
|
ctx.Data["InitialOptions"] = optionsContent(options)
|
||||||
|
|
||||||
fields, _ := db.VoteFormFields.ListByVoteID(ctx.Request().Context(), vote.ID)
|
fields, err := db.VoteFormFields.ListByVoteID(ctx.Request().Context(), vote.ID)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to list vote form fields")
|
||||||
|
ctx.SetInternalErrorFlash()
|
||||||
|
ctx.Redirect("/user/questions#votes")
|
||||||
|
return
|
||||||
|
}
|
||||||
ctx.Data["InitialFields"] = fieldsContent(fields)
|
ctx.Data["InitialFields"] = fieldsContent(fields)
|
||||||
|
|
||||||
// 时间回填:Go 模板直接输出 time.Time 不符合 datetime-local 格式,预先格式化
|
// 时间回填:Go 模板直接输出 time.Time 不符合 datetime-local 格式,预先格式化
|
||||||
@@ -825,7 +836,9 @@ func VoteUpdate(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := db.Votes.Update(ctx.Request().Context(), vote.ID, db.UpdateVoteOptions{
|
// 事务化更新:投票元信息 + 替换选项 + 替换表单字段。任一步失败整体回滚,
|
||||||
|
// 避免「元信息已更新、字段只更新一部分」的不一致草稿。
|
||||||
|
if err := db.Votes.UpdateWithOptions(ctx.Request().Context(), vote.ID, db.UpdateVoteOptions{
|
||||||
Title: form.opts.Title,
|
Title: form.opts.Title,
|
||||||
Description: form.opts.Description,
|
Description: form.opts.Description,
|
||||||
Type: &form.opts.Type,
|
Type: &form.opts.Type,
|
||||||
@@ -840,27 +853,12 @@ func VoteUpdate(ctx context.Context) {
|
|||||||
RequireCaptcha: form.opts.RequireCaptcha,
|
RequireCaptcha: form.opts.RequireCaptcha,
|
||||||
CensorMetadata: form.censorMetadata,
|
CensorMetadata: form.censorMetadata,
|
||||||
QuotaLimit: form.opts.QuotaLimit,
|
QuotaLimit: form.opts.QuotaLimit,
|
||||||
}); err != nil {
|
}, form.options, form.formFields); err != nil {
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to update vote")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to update vote with options")
|
||||||
renderVoteFormWithInput(ctx, true, vote.ID, "", true)
|
renderVoteFormWithInput(ctx, true, vote.ID, "", true)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// 事务重建选项与表单字段(草稿阶段无历史数据,删除重建安全)
|
|
||||||
if err := db.VoteOptions.ReplaceByVoteID(ctx.Request().Context(), vote.ID, form.options); err != nil {
|
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to replace vote options")
|
|
||||||
renderVoteFormWithInput(ctx, true, vote.ID, i18n.T(ctx.Lang(), "vote.err.optionUpdateFailed"), false)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
for i := range form.formFields {
|
|
||||||
form.formFields[i].VoteID = vote.ID
|
|
||||||
}
|
|
||||||
if err := db.VoteFormFields.ReplaceByVoteID(ctx.Request().Context(), vote.ID, form.formFields); err != nil {
|
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to replace vote form fields")
|
|
||||||
renderVoteFormWithInput(ctx, true, vote.ID, i18n.T(ctx.Lang(), "vote.err.fieldUpdateFailed"), false)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// 选项/字段已重建,结果聚合可能变化:主动失效该投票的结果缓存。
|
// 选项/字段已重建,结果聚合可能变化:主动失效该投票的结果缓存。
|
||||||
votepkg.InvalidateResultsCache(vote.ID)
|
votepkg.InvalidateResultsCache(vote.ID)
|
||||||
|
|
||||||
@@ -875,6 +873,10 @@ func VoteUpdate(ctx context.Context) {
|
|||||||
|
|
||||||
// action=publish:已保存为草稿,再执行发布校验;失败则保留草稿并停留在编辑页
|
// action=publish:已保存为草稿,再执行发布校验;失败则保留草稿并停留在编辑页
|
||||||
if action == "publish" {
|
if action == "publish" {
|
||||||
|
if !canPublishVote(ctx) {
|
||||||
|
ctx.Redirect(editURL)
|
||||||
|
return
|
||||||
|
}
|
||||||
// 用刚保存的内容刷新 vote 结构,供审核与结构校验读取最新标题/类型
|
// 用刚保存的内容刷新 vote 结构,供审核与结构校验读取最新标题/类型
|
||||||
vote.Title = form.opts.Title
|
vote.Title = form.opts.Title
|
||||||
vote.Description = form.opts.Description
|
vote.Description = form.opts.Description
|
||||||
@@ -900,6 +902,22 @@ func VoteUpdate(ctx context.Context) {
|
|||||||
ctx.Redirect(editURL + "?saved=1")
|
ctx.Redirect(editURL + "?saved=1")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// canPublishVote 统一投票发布的站方处置检查:L2 起不得发布新的公开内容。
|
||||||
|
// 查询失败时 fail-closed;错误提示在此统一设置,重定向目标由各发布入口保持原语义。
|
||||||
|
func canPublishVote(ctx context.Context) bool {
|
||||||
|
level, _, err := db.Restrictions.ActiveLevel(ctx.Request().Context(), ctx.User.ID)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to query user restriction")
|
||||||
|
ctx.SetInternalErrorFlash()
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if db.RestrictionAtLeast(level, db.RestrictionL2) {
|
||||||
|
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "vote.err.restrictedPublish"))
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
// VotePublish 发布投票(draft → published)。草稿阶段跳过的结构校验与 fail-closed
|
// VotePublish 发布投票(draft → published)。草稿阶段跳过的结构校验与 fail-closed
|
||||||
// 内容审核在此统一补齐,确保对外公开内容仍受管控。
|
// 内容审核在此统一补齐,确保对外公开内容仍受管控。
|
||||||
func VotePublish(ctx context.Context) {
|
func VotePublish(ctx context.Context) {
|
||||||
@@ -915,17 +933,7 @@ func VotePublish(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// 站方处置:L2 起不得再发布新的公开内容(与「设公开」同口径)。
|
if !canPublishVote(ctx) {
|
||||||
// 此前投票链路完全没有处置校验:被处置的箱主仍能发新投票并进探索页曝露。
|
|
||||||
level, _, err := db.Restrictions.ActiveLevel(ctx.Request().Context(), ctx.User.ID)
|
|
||||||
if err != nil {
|
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to query user restriction")
|
|
||||||
ctx.SetInternalErrorFlash()
|
|
||||||
ctx.Redirect("/user/questions#votes")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if db.RestrictionAtLeast(level, db.RestrictionL2) {
|
|
||||||
ctx.SetErrorFlash(i18n.T(ctx.Lang(), "vote.err.restrictedPublish"))
|
|
||||||
ctx.Redirect("/user/questions#votes")
|
ctx.Redirect("/user/questions#votes")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,36 @@
|
|||||||
|
package user
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/tamakyi/TamaBox/internal/conf"
|
||||||
|
"github.com/tamakyi/TamaBox/internal/db"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestVoteOptionMaxLen 钉住「配置值不能超过存储上限」这条约束。
|
||||||
|
//
|
||||||
|
// 背景:conf 的 vote_option_len 只校验 ≥1,可以填得比 vote_options.content
|
||||||
|
// 的列宽还大;那样校验会放行,写库才失败(PostgreSQL 直接报错)。
|
||||||
|
func TestVoteOptionMaxLen(t *testing.T) {
|
||||||
|
original := conf.Limits.VoteOptionLen
|
||||||
|
t.Cleanup(func() { conf.Limits.VoteOptionLen = original })
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
conf int
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{"配置小于存储上限时以配置为准", 100, 100},
|
||||||
|
{"配置等于存储上限", db.MaxVoteOptionLen, db.MaxVoteOptionLen},
|
||||||
|
{"配置超过存储上限时截到上限", db.MaxVoteOptionLen + 500, db.MaxVoteOptionLen},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
conf.Limits.VoteOptionLen = c.conf
|
||||||
|
if got := voteOptionMaxLen(); got != c.want {
|
||||||
|
t.Fatalf("voteOptionMaxLen() = %d, want %d", got, c.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
+29
-7
@@ -116,7 +116,10 @@ func computeResults(reqCtx stdctx.Context, v *db.Vote) (*Results, error) {
|
|||||||
|
|
||||||
switch v.Type {
|
switch v.Type {
|
||||||
case db.VoteTypeSingle:
|
case db.VoteTypeSingle:
|
||||||
options, _ := db.VoteOptions.ListByVoteID(reqCtx, v.ID)
|
options, err := db.VoteOptions.ListByVoteID(reqCtx, v.ID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list vote options")
|
||||||
|
}
|
||||||
// 一次分组查询替代逐选项 COUNT(避免 N+1)
|
// 一次分组查询替代逐选项 COUNT(避免 N+1)
|
||||||
counts, err := db.VoteSubmissions.CountGroupByOptionID(reqCtx, v.ID)
|
counts, err := db.VoteSubmissions.CountGroupByOptionID(reqCtx, v.ID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -132,9 +135,15 @@ func computeResults(reqCtx stdctx.Context, v *db.Vote) (*Results, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
case db.VoteTypeMultiple:
|
case db.VoteTypeMultiple:
|
||||||
options, _ := db.VoteOptions.ListByVoteID(reqCtx, v.ID)
|
options, err := db.VoteOptions.ListByVoteID(reqCtx, v.ID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list vote options")
|
||||||
|
}
|
||||||
|
submissions, err := db.VoteSubmissions.ListByVoteID(reqCtx, v.ID, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list vote submissions")
|
||||||
|
}
|
||||||
counts := make(map[uint]int64, len(options))
|
counts := make(map[uint]int64, len(options))
|
||||||
submissions, _ := db.VoteSubmissions.ListByVoteID(reqCtx, v.ID, nil)
|
|
||||||
var totalSelections int64
|
var totalSelections int64
|
||||||
for _, s := range submissions {
|
for _, s := range submissions {
|
||||||
var data struct {
|
var data struct {
|
||||||
@@ -163,9 +172,12 @@ func computeResults(reqCtx stdctx.Context, v *db.Vote) (*Results, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
case db.VoteTypeRating:
|
case db.VoteTypeRating:
|
||||||
|
submissions, err := db.VoteSubmissions.ListByVoteID(reqCtx, v.ID, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list vote submissions")
|
||||||
|
}
|
||||||
scoreCounts := make(map[int]int64)
|
scoreCounts := make(map[int]int64)
|
||||||
var sum int64
|
var sum int64
|
||||||
submissions, _ := db.VoteSubmissions.ListByVoteID(reqCtx, v.ID, nil)
|
|
||||||
for _, s := range submissions {
|
for _, s := range submissions {
|
||||||
var data struct {
|
var data struct {
|
||||||
Rating int `json:"rating"`
|
Rating int `json:"rating"`
|
||||||
@@ -184,10 +196,16 @@ func computeResults(reqCtx stdctx.Context, v *db.Vote) (*Results, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
case db.VoteTypeRank:
|
case db.VoteTypeRank:
|
||||||
options, _ := db.VoteOptions.ListByVoteID(reqCtx, v.ID)
|
options, err := db.VoteOptions.ListByVoteID(reqCtx, v.ID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list vote options")
|
||||||
|
}
|
||||||
|
submissions, err := db.VoteSubmissions.ListByVoteID(reqCtx, v.ID, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list vote submissions")
|
||||||
|
}
|
||||||
rankSums := make(map[uint]float64, len(options))
|
rankSums := make(map[uint]float64, len(options))
|
||||||
rankCounts := make(map[uint]int64, len(options))
|
rankCounts := make(map[uint]int64, len(options))
|
||||||
submissions, _ := db.VoteSubmissions.ListByVoteID(reqCtx, v.ID, nil)
|
|
||||||
for _, s := range submissions {
|
for _, s := range submissions {
|
||||||
var data struct {
|
var data struct {
|
||||||
Ranking map[string]float64 `json:"ranking"`
|
Ranking map[string]float64 `json:"ranking"`
|
||||||
@@ -215,7 +233,11 @@ func computeResults(reqCtx stdctx.Context, v *db.Vote) (*Results, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
case db.VoteTypeForm:
|
case db.VoteTypeForm:
|
||||||
res.FormFields, _ = db.VoteFormFields.ListByVoteID(reqCtx, v.ID)
|
fields, err := db.VoteFormFields.ListByVoteID(reqCtx, v.ID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "list vote form fields")
|
||||||
|
}
|
||||||
|
res.FormFields = fields
|
||||||
res.FormCount = total
|
res.FormCount = total
|
||||||
res.FormFieldStats = aggregateFormFieldStats(reqCtx, res.FormFields)
|
res.FormFieldStats = aggregateFormFieldStats(reqCtx, res.FormFields)
|
||||||
}
|
}
|
||||||
|
|||||||
+35
-22
@@ -724,6 +724,8 @@ func Submit(ctx context.Context, recaptcha recaptcha.RecaptchaV3) {
|
|||||||
if _, err := db.VoteSubmissions.Create(ctx.Request().Context(), *submission); err != nil {
|
if _, err := db.VoteSubmissions.Create(ctx.Request().Context(), *submission); err != nil {
|
||||||
if errors.Is(err, db.ErrVoteAlreadyVoted) {
|
if errors.Is(err, db.ErrVoteAlreadyVoted) {
|
||||||
ctx.SetErrorFlash(i18n.T(lang, "vote.page.alreadyVoted"))
|
ctx.SetErrorFlash(i18n.T(lang, "vote.page.alreadyVoted"))
|
||||||
|
} else if errors.Is(err, db.ErrVoteNotAccepting) || errors.Is(err, db.ErrFeatureQuotaReached) {
|
||||||
|
ctx.SetErrorFlash(i18n.T(lang, "vote.flash.ended"))
|
||||||
} else {
|
} else {
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create vote submission")
|
logrus.WithContext(ctx.Request().Context()).WithError(err).Error("Failed to create vote submission")
|
||||||
ctx.SetInternalErrorFlash()
|
ctx.SetInternalErrorFlash()
|
||||||
@@ -738,19 +740,6 @@ func Submit(ctx context.Context, recaptcha recaptcha.RecaptchaV3) {
|
|||||||
// 用 InvalidateResultsCache 会让每一票都变成下一个读请求的同步全量重算(O(N²))。
|
// 用 InvalidateResultsCache 会让每一票都变成下一个读请求的同步全量重算(O(N²))。
|
||||||
ExpireResultsCache(vote.ID)
|
ExpireResultsCache(vote.ID)
|
||||||
|
|
||||||
// 单份投票的份数上限:达到即自动结束该投票。
|
|
||||||
// (箱主级的累计计数已由 VoteSubmissions.Create 在同一事务里 +1,这里不再重复)
|
|
||||||
if vote.QuotaLimit > 0 {
|
|
||||||
count, err := db.VoteSubmissions.CountByVoteID(ctx.Request().Context(), vote.ID)
|
|
||||||
if err != nil {
|
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Warn("Failed to count vote submissions for quota")
|
|
||||||
} else if count >= int64(vote.QuotaLimit) {
|
|
||||||
if err := db.Votes.UpdateStatus(ctx.Request().Context(), vote.ID, db.VoteStatusEnded); err != nil {
|
|
||||||
logrus.WithContext(ctx.Request().Context()).WithError(err).Warn("Failed to auto end vote on quota reached")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// 表单类型投票:异步通知箱主查看新回复(开启邮件通知时)。owner 在上方已取过,直接复用。
|
// 表单类型投票:异步通知箱主查看新回复(开启邮件通知时)。owner 在上方已取过,直接复用。
|
||||||
if vote.Type == db.VoteTypeForm && owner.Notify == db.NotifyTypeEmail {
|
if vote.Type == db.VoteTypeForm && owner.Notify == db.NotifyTypeEmail {
|
||||||
email := owner.Email
|
email := owner.Email
|
||||||
@@ -810,6 +799,19 @@ func parseMultipleOptionIDs(ids []string, maxChoices int, validOptions map[uint]
|
|||||||
return chosen, nil
|
return chosen, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// appendUniqueValue 把 v 追加到 dst,但同一个值只保留第一次出现。
|
||||||
|
// 多选类字段(checkbox)依赖它去重:客户端可以重复提交同名字段,
|
||||||
|
// 不去重就会在结果统计里把同一选项计多次。
|
||||||
|
//
|
||||||
|
// seen 由调用方持有并跨多次调用复用,用来记录已追加过的值。
|
||||||
|
func appendUniqueValue(dst []string, seen map[string]bool, v string) []string {
|
||||||
|
if seen[v] {
|
||||||
|
return dst
|
||||||
|
}
|
||||||
|
seen[v] = true
|
||||||
|
return append(dst, v)
|
||||||
|
}
|
||||||
|
|
||||||
// parseSubmission 按投票类型解析并校验提交内容,构造 VoteSubmission。
|
// parseSubmission 按投票类型解析并校验提交内容,构造 VoteSubmission。
|
||||||
// owner 为投票发起人(用于其全局屏蔽词表对开放文本的本地过滤)。
|
// owner 为投票发起人(用于其全局屏蔽词表对开放文本的本地过滤)。
|
||||||
func parseSubmission(ctx context.Context, v *db.Vote, owner *db.User) (*db.CreateVoteSubmissionOptions, error) {
|
func parseSubmission(ctx context.Context, v *db.Vote, owner *db.User) (*db.CreateVoteSubmissionOptions, error) {
|
||||||
@@ -820,9 +822,14 @@ func parseSubmission(ctx context.Context, v *db.Vote, owner *db.User) (*db.Creat
|
|||||||
|
|
||||||
// 选项合法性集合
|
// 选项合法性集合
|
||||||
validOptions := map[uint]bool{}
|
validOptions := map[uint]bool{}
|
||||||
|
var voteOptions []*db.VoteOption
|
||||||
if v.Type == db.VoteTypeSingle || v.Type == db.VoteTypeMultiple || v.Type == db.VoteTypeRank {
|
if v.Type == db.VoteTypeSingle || v.Type == db.VoteTypeMultiple || v.Type == db.VoteTypeRank {
|
||||||
options, _ := db.VoteOptions.ListByVoteID(ctx.Request().Context(), v.ID)
|
var err error
|
||||||
for _, o := range options {
|
voteOptions, err = db.VoteOptions.ListByVoteID(ctx.Request().Context(), v.ID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "load vote options")
|
||||||
|
}
|
||||||
|
for _, o := range voteOptions {
|
||||||
validOptions[o.ID] = true
|
validOptions[o.ID] = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -853,13 +860,12 @@ func parseSubmission(ctx context.Context, v *db.Vote, owner *db.User) (*db.Creat
|
|||||||
formData["rating"] = score
|
formData["rating"] = score
|
||||||
|
|
||||||
case db.VoteTypeRank:
|
case db.VoteTypeRank:
|
||||||
options, _ := db.VoteOptions.ListByVoteID(ctx.Request().Context(), v.ID)
|
|
||||||
ranking := map[string]float64{}
|
ranking := map[string]float64{}
|
||||||
used := map[float64]bool{}
|
used := map[float64]bool{}
|
||||||
for _, o := range options {
|
for _, o := range voteOptions {
|
||||||
r, err := strconv.ParseFloat(get(fmt.Sprintf("rank_%d", o.ID)), 64)
|
r, err := strconv.ParseFloat(get(fmt.Sprintf("rank_%d", o.ID)), 64)
|
||||||
if err != nil || r < 1 || r > float64(len(options)) {
|
if err != nil || r < 1 || r > float64(len(voteOptions)) {
|
||||||
return nil, fmt.Errorf(i18n.T(lang, "vote.err.rankRequired"), len(options))
|
return nil, fmt.Errorf(i18n.T(lang, "vote.err.rankRequired"), len(voteOptions))
|
||||||
}
|
}
|
||||||
if used[r] {
|
if used[r] {
|
||||||
return nil, errors.New(i18n.T(lang, "vote.err.rankDuplicate"))
|
return nil, errors.New(i18n.T(lang, "vote.err.rankDuplicate"))
|
||||||
@@ -870,7 +876,10 @@ func parseSubmission(ctx context.Context, v *db.Vote, owner *db.User) (*db.Creat
|
|||||||
formData["ranking"] = ranking
|
formData["ranking"] = ranking
|
||||||
|
|
||||||
case db.VoteTypeForm:
|
case db.VoteTypeForm:
|
||||||
fields, _ := db.VoteFormFields.ListByVoteID(ctx.Request().Context(), v.ID)
|
fields, err := db.VoteFormFields.ListByVoteID(ctx.Request().Context(), v.ID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.Wrap(err, "load vote form fields")
|
||||||
|
}
|
||||||
for _, f := range fields {
|
for _, f := range fields {
|
||||||
// 字段选项集合
|
// 字段选项集合
|
||||||
validFieldOptions := map[string]bool{}
|
validFieldOptions := map[string]bool{}
|
||||||
@@ -928,6 +937,10 @@ func parseSubmission(ctx context.Context, v *db.Vote, owner *db.User) (*db.Creat
|
|||||||
// 多选 checkbox
|
// 多选 checkbox
|
||||||
if f.FieldType == db.VoteFormFieldTypeCheckbox {
|
if f.FieldType == db.VoteFormFieldTypeCheckbox {
|
||||||
var chosen []string
|
var chosen []string
|
||||||
|
// 同一字段名可以被客户端重复提交(field_x=A&field_x=A),
|
||||||
|
// 不去重就会在结果统计里把同一个选项计多次,等于让一个人刷高计数。
|
||||||
|
// 按最终落库的字符串去重:预置值按值去重,「其他」按用户填的文本去重。
|
||||||
|
chosenSeen := make(map[string]bool, len(values))
|
||||||
for _, val := range values {
|
for _, val := range values {
|
||||||
val = strings.TrimSpace(val)
|
val = strings.TrimSpace(val)
|
||||||
if val == "" {
|
if val == "" {
|
||||||
@@ -951,13 +964,13 @@ func parseSubmission(ctx context.Context, v *db.Vote, owner *db.User) (*db.Creat
|
|||||||
if err := censorCheck(ctx, owner, lang, custom); err != nil {
|
if err := censorCheck(ctx, owner, lang, custom); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
chosen = append(chosen, custom)
|
chosen = appendUniqueValue(chosen, chosenSeen, custom)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if !validFieldOptions[val] {
|
if !validFieldOptions[val] {
|
||||||
return nil, errors.New(i18n.T(lang, "vote.err.fieldInvalidOption", f.Label))
|
return nil, errors.New(i18n.T(lang, "vote.err.fieldInvalidOption", f.Label))
|
||||||
}
|
}
|
||||||
chosen = append(chosen, val)
|
chosen = appendUniqueValue(chosen, chosenSeen, val)
|
||||||
}
|
}
|
||||||
if len(chosen) == 0 && f.Required {
|
if len(chosen) == 0 && f.Required {
|
||||||
return nil, errors.New(i18n.T(lang, "vote.err.fieldRequired", f.Label))
|
return nil, errors.New(i18n.T(lang, "vote.err.fieldRequired", f.Label))
|
||||||
|
|||||||
@@ -0,0 +1,48 @@
|
|||||||
|
package vote
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestAppendUniqueValue 钉住多选字段的去重语义:
|
||||||
|
// 客户端重复提交同一选项时,落库与统计都只能算一次。
|
||||||
|
func TestAppendUniqueValue(t *testing.T) {
|
||||||
|
t.Run("重复值只保留第一次", func(t *testing.T) {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
var got []string
|
||||||
|
for _, v := range []string{"A", "A", "B", "A", "B", "C"} {
|
||||||
|
got = appendUniqueValue(got, seen, v)
|
||||||
|
}
|
||||||
|
want := []string{"A", "B", "C"}
|
||||||
|
if len(got) != len(want) {
|
||||||
|
t.Fatalf("got %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
for i := range want {
|
||||||
|
if got[i] != want[i] {
|
||||||
|
t.Fatalf("got %v, want %v(顺序应按首次出现)", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("空切片与单值", func(t *testing.T) {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
var got []string
|
||||||
|
got = appendUniqueValue(got, seen, "只有一项")
|
||||||
|
if len(got) != 1 || got[0] != "只有一项" {
|
||||||
|
t.Fatalf("got %v", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("seen 复用时跨批次仍然去重", func(t *testing.T) {
|
||||||
|
// checkbox 的「其他」文本与预置值走同一条追加路径,
|
||||||
|
// seen 必须在整个字段解析期间共享,而不是每轮新建。
|
||||||
|
seen := map[string]bool{}
|
||||||
|
var got []string
|
||||||
|
got = appendUniqueValue(got, seen, "预置")
|
||||||
|
got = appendUniqueValue(got, seen, "其他文本")
|
||||||
|
got = appendUniqueValue(got, seen, "预置")
|
||||||
|
if len(got) != 2 {
|
||||||
|
t.Fatalf("got %v, want 2 项", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -21,6 +21,13 @@ type mockVoteSubmissions struct {
|
|||||||
func (m *mockVoteSubmissions) ListByVoteID(ctx stdctx.Context, voteID uint, cursor *dbutil.Cursor) ([]*db.VoteSubmission, error) {
|
func (m *mockVoteSubmissions) ListByVoteID(ctx stdctx.Context, voteID uint, cursor *dbutil.Cursor) ([]*db.VoteSubmission, error) {
|
||||||
return m.submissions, nil
|
return m.submissions, nil
|
||||||
}
|
}
|
||||||
|
func (m *mockVoteSubmissions) ListByVoteIDs(ctx stdctx.Context, voteIDs []uint) (map[uint][]*db.VoteSubmission, error) {
|
||||||
|
out := make(map[uint][]*db.VoteSubmission)
|
||||||
|
for _, s := range m.submissions {
|
||||||
|
out[s.VoteID] = append(out[s.VoteID], s)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
func (m *mockVoteSubmissions) Create(ctx stdctx.Context, opts db.CreateVoteSubmissionOptions) (*db.VoteSubmission, error) {
|
func (m *mockVoteSubmissions) Create(ctx stdctx.Context, opts db.CreateVoteSubmissionOptions) (*db.VoteSubmission, error) {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -30,6 +30,8 @@ func main() {
|
|||||||
cmd.HidePublic,
|
cmd.HidePublic,
|
||||||
cmd.Honeypot,
|
cmd.Honeypot,
|
||||||
cmd.Backup,
|
cmd.Backup,
|
||||||
|
cmd.RestoreImages,
|
||||||
|
cmd.R2Outbox,
|
||||||
}
|
}
|
||||||
if err := app.Run(os.Args); err != nil {
|
if err := app.Run(os.Args); err != nil {
|
||||||
logrus.WithError(err).Fatal("Failed to start application")
|
logrus.WithError(err).Fatal("Failed to start application")
|
||||||
|
|||||||
@@ -157,7 +157,17 @@
|
|||||||
el: '#waline',
|
el: '#waline',
|
||||||
//后端服务器地址
|
//后端服务器地址
|
||||||
serverURL: 'https://waline.337845818.best/',
|
serverURL: 'https://waline.337845818.best/',
|
||||||
path: window.location.href,
|
// 评论线程键:必须剔除 ?t= 私密题 token。
|
||||||
|
// 原来直接传 location.href,私密题 /_/<domain>/<id>?t=<token> 的钥匙
|
||||||
|
// 会整条 URL 发到 Waline 服务端并落进它的访问日志;token 本身就能读该题,
|
||||||
|
// 等于凭空多出一份可被日志检索的凭证副本。
|
||||||
|
// 只删 t、保留其余参数:公开题的 page/sort 等分页参数不变,
|
||||||
|
// 已有评论线程的键不受影响。
|
||||||
|
path: (function () {
|
||||||
|
var u = new URL(window.location.href);
|
||||||
|
u.searchParams.delete('t');
|
||||||
|
return u.toString();
|
||||||
|
})(),
|
||||||
// 中文默认
|
// 中文默认
|
||||||
locale: {
|
locale: {
|
||||||
nick: '昵称',
|
nick: '昵称',
|
||||||
|
|||||||
Reference in New Issue
Block a user