生成器可检测与管理远端主题素材(列表 / 差异 / 覆盖 / 删除)
「传上去」之后还得知道「传得对不对」:同一 slug 传过三次、换过 slug 旧图还在、 重新导出后远端仍是旧版,这些在页面上都看不出来。 - serve.py:s3_request 支持 query;新增 ListObjectsV2(含分页)与 DeleteObject, 端点 /api/s3/list、/api/s3/delete。 - index.html:远端素材清单按对象键名跟本地比 —— 一致 / 旧图(同名但大小不同)/ 多余(当前主题已不引用)/ 缺失。「补齐差异」只传缺的与旧图,「删除多余」批量删, 单行可 ↻ 覆盖或 ✕ 删除;删除走 confirm 二次确认。改过 slug 会提示重新检测。 - 修:list 返回完整键而 upload/delete 收相对键,不剥前缀会删到不存在的键 (S3 回 204,看起来删成功其实没删)。加 relKey() 统一。 - check_sigv4.py 补带 query 的用例(ListObjectsV2 三参数、乱序 token)→ 8/8; selftest_upload.py 补列表/分页/幂等删除 → 20 项全过。
This commit is contained in:
@@ -225,6 +225,24 @@ WebP 体积 ≈ 像素数 × 质量。**真正的大头是像素**:CSS 里只
|
||||
需要自定义域或 CDN 域名,那种情况自己填「公开前缀」。测试里的「匿名可读」会给一条提醒,
|
||||
但不会拦上传 —— 很多人就是靠自定义域分发的。
|
||||
|
||||
### 检测与清理
|
||||
|
||||
传上去只是第一步,「传上去之后对不对」是另一回事:同一个 slug 传过三次、换过 slug
|
||||
旧图还留在桶里、重新导出后远端还是旧版 —— 这些在页面上都看不出来。所以还有一块
|
||||
「远端素材」,点「**检测**」列出 `<路径前缀>/<slug>/` 下所有对象,按对象键名跟本地比:
|
||||
|
||||
| 状态 | 判断依据 | 能做什么 |
|
||||
| --- | --- | --- |
|
||||
| 一致 | 远端大小 == 本地编码后大小 | — |
|
||||
| 旧图 | 同名但大小不同(多半重新导出过) | 行内 ↻ 用本地覆盖 |
|
||||
| 多余 | 远端有、当前主题已不再引用(换过 slug / 删过素材) | 行内 ✕ 或「删除多余」批量删 |
|
||||
| 缺失 | 本地有、远端没有 | 「补齐差异」一次传完 |
|
||||
|
||||
- 「**补齐差异**」只传缺失 + 旧图,一致的不重传(省流量也省时间);
|
||||
- 「**删除多余**」列清单后二次确认,走 `DeleteObject`(幂等,不存在的键也算成功);
|
||||
- 改过 slug 会提示「重新检测」—— 拿旧主题的清单去比新主题,会满屏「多余」;
|
||||
- 只动远端,本地素材与工程文件不受影响。
|
||||
|
||||
厂商怎么填(2026 年的常见值,各家控制台都有「S3 API」页面):
|
||||
|
||||
| 厂商 | Endpoint | Region | 路径风格 |
|
||||
|
||||
@@ -30,19 +30,25 @@ TS = "20130524T000000Z"
|
||||
EMPTY = serve.EMPTY_SHA256
|
||||
BODY_HASH = serve._sha256(b"TamaBox theme builder probe\n")
|
||||
|
||||
# (说明, method, endpoint, extra cfg, key)
|
||||
# (说明, method, endpoint, extra cfg, key, query)
|
||||
CASES = [
|
||||
("path-style 桶级 HEAD", "HEAD", "http://127.0.0.1:9000", dict(bucket="tb-assets"), ""),
|
||||
("path-style 桶级 HEAD", "HEAD", "http://127.0.0.1:9000", dict(bucket="tb-assets"), "", None),
|
||||
("path-style 带前缀对象", "PUT", "http://127.0.0.1:9000",
|
||||
dict(bucket="tb-assets", prefix="themes"), "shiroko/bg1.webp"),
|
||||
dict(bucket="tb-assets", prefix="themes"), "shiroko/bg1.webp", None),
|
||||
("virtual-host 对象", "GET", "https://s3.us-west-004.backblazeb2.com",
|
||||
dict(bucket="my-bucket", region="us-west-004", pathStyle=False), "themes/shiroko/bg1.webp"),
|
||||
dict(bucket="my-bucket", region="us-west-004", pathStyle=False), "themes/shiroko/bg1.webp", None),
|
||||
("R2 region=auto", "PUT", "https://acc.r2.cloudflarestorage.com",
|
||||
dict(bucket="cdn", region="auto", prefix="themes"), "x/a.webp"),
|
||||
dict(bucket="cdn", region="auto", prefix="themes"), "x/a.webp", None),
|
||||
("R2 虚拟主机", "GET", "https://acc.r2.cloudflarestorage.com",
|
||||
dict(bucket="cdn", region="auto", pathStyle=False), "themes/s/t.webp"),
|
||||
dict(bucket="cdn", region="auto", pathStyle=False), "themes/s/t.webp", None),
|
||||
("带端口 https", "DELETE", "https://minio.lan:9443",
|
||||
dict(bucket="static", region="us-east-1"), "themes/s/t.webp"),
|
||||
dict(bucket="static", region="us-east-1"), "themes/s/t.webp", None),
|
||||
# 带 query:ListObjectsV2 的 canonical query 要按键排序、逐项编码(/ 编成 %2F)
|
||||
("ListObjectsV2 三参数", "GET", "http://127.0.0.1:9000", dict(bucket="tb-assets"),
|
||||
"", {"list-type": "2", "max-keys": "1000", "prefix": "themes/shiroko/"}),
|
||||
("ListObjectsV2 分页(乱序传入)", "GET", "https://acc.r2.cloudflarestorage.com",
|
||||
dict(bucket="cdn", region="auto", pathStyle=False), "",
|
||||
{"prefix": "themes/", "continuation-token": "1/abc+d", "max-keys": "2", "list-type": "2"}),
|
||||
]
|
||||
|
||||
|
||||
@@ -68,20 +74,21 @@ def go_sign(payloads):
|
||||
|
||||
def main():
|
||||
payloads = []
|
||||
for _, method, endpoint, extra, key in CASES:
|
||||
for _, method, endpoint, extra, key, query in CASES:
|
||||
cfg, path = build(endpoint, extra, key)
|
||||
body = BODY_HASH if method == "PUT" else EMPTY
|
||||
payloads.append("\t".join([method, cfg.url(path), cfg.region, body, TS]))
|
||||
url = cfg.url(path) + (("?" + serve.canonical_query(query)) if query else "")
|
||||
payloads.append("\t".join([method, url, cfg.region, body, TS]))
|
||||
|
||||
go = go_sign(payloads)
|
||||
if len(go) != len(CASES):
|
||||
raise SystemExit("返回条数不对:go=%d cases=%d" % (len(go), len(CASES)))
|
||||
|
||||
bad = 0
|
||||
for (desc, method, endpoint, extra, key), g in zip(CASES, go):
|
||||
for (desc, method, endpoint, extra, key, query), g in zip(CASES, go):
|
||||
cfg, path = build(endpoint, extra, key)
|
||||
body = BODY_HASH if method == "PUT" else EMPTY
|
||||
py = serve.sign_v4(cfg, method, path, body, TS, cfg.host)
|
||||
py = serve.sign_v4(cfg, method, path, body, TS, cfg.host, serve.canonical_query(query))
|
||||
if g.rsplit("Signature=", 1)[-1] == py.rsplit("Signature=", 1)[-1]:
|
||||
print("OK %s" % desc)
|
||||
else:
|
||||
|
||||
@@ -117,6 +117,26 @@
|
||||
.s3acts button{flex:1}
|
||||
.s3acts button:disabled{opacity:.45;cursor:not-allowed}
|
||||
|
||||
/* 远端清单:key 名弹性、大小右对齐、状态徽标定宽,操作按钮只留两个小方块。
|
||||
240px 面板里四列已经是极限,再加列就会把文件名挤成省略号。 */
|
||||
.remote{border-top:1px solid var(--ui-line);margin-top:10px;padding-top:9px}
|
||||
.remote .rhead{display:flex;align-items:center;gap:6px;margin-bottom:6px}
|
||||
.remote .rhead span{font-size:11px;color:var(--ui-text2);flex:1}
|
||||
.remote .rlist{display:flex;flex-direction:column}
|
||||
.rrow{display:grid;grid-template-columns:minmax(0,1fr) auto auto auto auto;gap:5px;
|
||||
align-items:center;font-size:11px;padding:2px 0}
|
||||
.rrow .nm{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
||||
.rrow .sz{color:var(--ui-text2);font-variant-numeric:tabular-nums;font-size:10px}
|
||||
.rrow .bg{font-size:10px;padding:0 5px;border-radius:999px;border:1px solid var(--ui-line);
|
||||
color:var(--ui-text2);white-space:nowrap}
|
||||
.rrow .bg.same{color:var(--ui-ok);border-color:#2f5744}
|
||||
.rrow .bg.changed{color:var(--ui-warn);border-color:#5a4a2a}
|
||||
.rrow .bg.extra{color:var(--ui-bad);border-color:#5a3036}
|
||||
.rrow button{padding:0;width:20px;height:20px;line-height:1;font-size:11px;border-radius:4px}
|
||||
.remote .rsum{font-size:11px;color:var(--ui-text2);margin-top:6px;line-height:1.5}
|
||||
.remote .racts{display:flex;gap:6px;margin-top:7px}
|
||||
.remote .racts button{flex:1;font-size:11px;padding:4px 6px}
|
||||
|
||||
.assets{display:grid;grid-template-columns:repeat(4,1fr);gap:6px}
|
||||
.as{
|
||||
position:relative;aspect-ratio:1;border-radius:6px;overflow:hidden;background:#0b0f14;
|
||||
@@ -395,6 +415,9 @@ const S = {
|
||||
host: Object.assign({}, D.host),
|
||||
/* 运行时状态:测试/上传结果,不参与持久化 */
|
||||
s3: { kind: '', text: '', busy: false, canUpload: null },
|
||||
/* 远端清单(检测结果)。at=0 表示这次会话还没检测过;slug 变了要重新检测,
|
||||
否则会拿旧主题的 key 跟新主题的本地素材比,满屏「多余」。 */
|
||||
remote: { objects: [], at: 0, slug: '', prefix: '' },
|
||||
dark: Object.assign({}, D.dark),
|
||||
};
|
||||
|
||||
@@ -1868,6 +1891,9 @@ function renderHostEditor(){
|
||||
res.textContent = S.s3.text || '';
|
||||
el.appendChild(res);
|
||||
|
||||
// 远端清单放在状态框下面:先看「测试/上传结果」,再看「远端现在到底有什么」
|
||||
el.appendChild(renderRemoteBlock());
|
||||
|
||||
const tip = document.createElement('p');
|
||||
tip.className = 'hint';
|
||||
tip.textContent = can === false
|
||||
@@ -1906,29 +1932,33 @@ async function testS3(){
|
||||
}
|
||||
}
|
||||
|
||||
async function uploadAssets(){
|
||||
async function uploadAssets(onlyIds){
|
||||
const cfg = s3Config();
|
||||
const miss = s3Missing(cfg);
|
||||
if (miss.length){ setS3State('bad', '还差:' + miss.join('、')); return; }
|
||||
const slug = ($('slug').value || '').trim() || 'my-theme';
|
||||
const slug = currentSlug();
|
||||
// usedAssetIds 返回 Set:它有 size 没有 length/map,先展开成数组再用。
|
||||
const ids = [...usedAssetIds()];
|
||||
// onlyIds 是「补齐差异」传进来的子集 —— 那时只该传缺的/变了的。
|
||||
const ids = onlyIds ? onlyIds.slice() : [...usedAssetIds()];
|
||||
if (!ids.length){ setS3State('bad', '还没有素材:先在「图片池」里加图片并摆到图层/背景上'); return; }
|
||||
|
||||
const files = ids.map(id => {
|
||||
const a = assetById(id);
|
||||
if (!a) return null;
|
||||
return {
|
||||
key: `${slug}/${id}.webp`,
|
||||
name: `${id}.webp`,
|
||||
contentType: 'image/webp',
|
||||
data: (a.dataURL.split(',')[1] || ''),
|
||||
};
|
||||
}).filter(f => f.data);
|
||||
}).filter(f => f && f.data);
|
||||
if (!files.length){ setS3State('bad', '这些素材没有可上传的内容(dataURL 为空)'); return; }
|
||||
const mb = files.reduce((n, f) => n + f.data.length, 0) * 0.75 / 1048576;
|
||||
|
||||
S.s3.busy = true;
|
||||
setS3State('', `正在上传 ${files.length} 张(约 ${mb.toFixed(2)}MB)…` + '\n首次传较多时会慢一会儿,别关页面。');
|
||||
renderHostEditor();
|
||||
let done = false;
|
||||
try {
|
||||
const j = await sbApi('/api/s3/upload', { config: cfg, files });
|
||||
if (!j.ok){
|
||||
@@ -1940,6 +1970,7 @@ async function uploadAssets(){
|
||||
const pub = publicBaseOf(S.host);
|
||||
if (pub) S.assetBase = pub;
|
||||
saveDraft();
|
||||
done = true;
|
||||
setS3State('ok', `已上传 ${j.total} 张到 ${cfg.prefix || '桶根'}/` + slug + '/\n'
|
||||
+ '导出的 CSS 会写成 ' + (pub || '(未推导出公开前缀)') + '/' + slug + '/<id>.webp');
|
||||
log(`素材已上传 ${j.total} 张;素材前缀已设为 ${pub || '(空)'}`, 'ok');
|
||||
@@ -1949,6 +1980,247 @@ async function uploadAssets(){
|
||||
} finally {
|
||||
S.s3.busy = false; renderAll();
|
||||
}
|
||||
// 检测过远端就顺手重读:否则刚传完还显示「缺失 3」,等于让用户自己骗自己。
|
||||
if (done && S.remote.at) await refreshRemote();
|
||||
}
|
||||
|
||||
/* ---------------------- 远端素材(检测 / 覆盖 / 删除) ---------------------- */
|
||||
/* 上传只解决「传上去」,解决不了「传上去之后对不对」:同一个 slug 传过三次、
|
||||
换过 slug 旧图还留在桶里、重新导出后远端还是旧图 —— 这些在页面上一概看不出来。
|
||||
所以再给一份远端清单,按对象键名(= <slug>/<id>.webp)跟本地比:
|
||||
差在哪、怎么补、怎么删,都写进行里。 */
|
||||
function currentSlug(){
|
||||
return ($('slug').value || '').trim() || 'my-theme';
|
||||
}
|
||||
|
||||
/** 本地引用素材 vs 远端对象的差异。用大小当内容指纹:同一 id 的图重编过,大小就会变。 */
|
||||
function remoteDiff(){
|
||||
const local = new Map();
|
||||
[...usedAssetIds()].forEach(id => {
|
||||
const a = assetById(id);
|
||||
if (a && a.dataURL) local.set(id, dataURLToBytes(a.dataURL).length);
|
||||
});
|
||||
const rows = [];
|
||||
const seen = new Set();
|
||||
(S.remote.objects || []).forEach(o => {
|
||||
const id = o.key.split('/').pop().replace(/\.webp$/i, '');
|
||||
const bytes = local.get(id);
|
||||
if (bytes == null){
|
||||
rows.push({ kind: 'extra', id, o });
|
||||
} else {
|
||||
seen.add(id);
|
||||
rows.push({ kind: bytes === o.size ? 'same' : 'changed', id, o, bytes });
|
||||
}
|
||||
});
|
||||
const missing = [...local.keys()].filter(id => !seen.has(id));
|
||||
return { rows, missing };
|
||||
}
|
||||
|
||||
function fmtBytes(n){
|
||||
if (n >= 1048576) return (n / 1048576).toFixed(1) + 'MB';
|
||||
if (n >= 1024) return Math.round(n / 1024) + 'KB';
|
||||
return n + 'B';
|
||||
}
|
||||
|
||||
/** 列表返回的是完整对象键,删除接口要的是「相对路径前缀」的键(与上传同一个约定)。
|
||||
不剥这一层,服务端会再拼一次前缀 → 删的是一个不存在的键,S3 还回 204,看起来像成功。 */
|
||||
function relKey(key, prefix){
|
||||
const p = String(prefix || '').replace(/^\/+|\/+$/g, '');
|
||||
return p && key.startsWith(p + '/') ? key.slice(p.length + 1) : key;
|
||||
}
|
||||
|
||||
async function refreshRemote(){
|
||||
const cfg = s3Config();
|
||||
const miss = s3Missing(cfg);
|
||||
if (miss.length){ setS3State('bad', '还差:' + miss.join('、')); return; }
|
||||
const slug = currentSlug();
|
||||
S.s3.busy = true;
|
||||
setS3State('', '正在读取远端清单…');
|
||||
renderHostEditor();
|
||||
try {
|
||||
const j = await sbApi('/api/s3/list', { config: cfg, prefix: slug + '/' });
|
||||
if (!j.ok){
|
||||
setS3State('bad', '读取远端清单失败:' + (j.error || '未知错误') + (j.hint ? '\n' + j.hint : ''));
|
||||
log('读取远端清单失败:' + (j.error || ''), 'e');
|
||||
return;
|
||||
}
|
||||
S.remote = { objects: j.objects || [], at: Date.now(), slug, prefix: j.prefix };
|
||||
const d = remoteDiff();
|
||||
const same = d.rows.filter(r => r.kind === 'same').length;
|
||||
const changed = d.rows.filter(r => r.kind === 'changed').length;
|
||||
const extra = d.rows.filter(r => r.kind === 'extra').length;
|
||||
const clean = !changed && !extra && !d.missing.length;
|
||||
// 只说结论,数字交给下面的汇总行 —— 两处都列一遍只是噪声
|
||||
setS3State(clean ? 'ok' : 'warn',
|
||||
clean ? `远端与本地一致(${same} 个对象)。` : '有差异,见下方清单。');
|
||||
log(`远端检测:${d.rows.length} 个对象(一致 ${same} / 大小不同 ${changed} / 多余 ${extra} / 缺失 ${d.missing.length})`,
|
||||
clean ? 'ok' : 'w');
|
||||
} catch (e){
|
||||
setS3State('bad', '读取远端清单失败:' + e.message);
|
||||
} finally {
|
||||
S.s3.busy = false; renderHostEditor();
|
||||
}
|
||||
}
|
||||
|
||||
/** 补差异:只传「远端没有」和「大小不一致」的,一致的不重传。 */
|
||||
async function uploadDiff(){
|
||||
const d = remoteDiff();
|
||||
const ids = d.missing.concat(d.rows.filter(r => r.kind === 'changed').map(r => r.id));
|
||||
if (!ids.length){
|
||||
setS3State('ok', '远端与本地一致,没有要补的。');
|
||||
return;
|
||||
}
|
||||
await uploadAssets(ids);
|
||||
}
|
||||
|
||||
async function deleteRemoteKeys(keys, why){
|
||||
if (!keys.length) return false;
|
||||
const preview = keys.slice(0, 6).join('\n') + (keys.length > 6 ? `\n…… 另有 ${keys.length - 6} 个` : '');
|
||||
if (!confirm(`从对象存储删除 ${keys.length} 个对象?\n\n${preview}\n\n${why}\n\n只删远端,本地素材不受影响。`)) return false;
|
||||
|
||||
const cfg = s3Config();
|
||||
const rel = keys.map(k => relKey(k, cfg.prefix));
|
||||
S.s3.busy = true;
|
||||
setS3State('', `正在删除 ${keys.length} 个对象…`);
|
||||
renderHostEditor();
|
||||
try {
|
||||
const j = await sbApi('/api/s3/delete', { config: cfg, keys: rel });
|
||||
if (!j.ok){
|
||||
const bad = (j.files || []).filter(f => !f.ok).slice(0, 3)
|
||||
.map(f => `· ${f.key}:${f.error || f.code || '失败'}`);
|
||||
setS3State('bad', [`${j.failed} / ${j.total} 个删除失败:`].concat(bad).join('\n'));
|
||||
log(`远端删除失败 ${j.failed} 个`, 'e');
|
||||
} else {
|
||||
log(`已从远端删除 ${j.total} 个对象`, 'ok');
|
||||
}
|
||||
} catch (e){
|
||||
setS3State('bad', '请求本地服务失败:' + e.message);
|
||||
} finally {
|
||||
S.s3.busy = false;
|
||||
}
|
||||
await refreshRemote();
|
||||
return true;
|
||||
}
|
||||
|
||||
function deleteExtra(){
|
||||
const d = remoteDiff();
|
||||
return deleteRemoteKeys(d.rows.filter(r => r.kind === 'extra').map(r => r.o.key),
|
||||
'这些对象在远端存在、但当前主题已经不再引用。');
|
||||
}
|
||||
|
||||
function renderRemoteBlock(){
|
||||
const wrap = document.createElement('div');
|
||||
wrap.className = 'remote';
|
||||
|
||||
const head = document.createElement('div');
|
||||
head.className = 'rhead';
|
||||
const title = document.createElement('span');
|
||||
title.textContent = '远端素材';
|
||||
const btnCheck = document.createElement('button');
|
||||
btnCheck.textContent = '检测';
|
||||
btnCheck.title = '读取 <路径前缀>/<slug>/ 下的对象,跟本地比';
|
||||
btnCheck.disabled = !!S.s3.busy;
|
||||
btnCheck.onclick = refreshRemote;
|
||||
head.append(title, btnCheck);
|
||||
wrap.appendChild(head);
|
||||
|
||||
if (S.remote.at && S.remote.slug !== currentSlug()){
|
||||
const p = document.createElement('p');
|
||||
p.className = 'hint';
|
||||
p.style.margin = '0';
|
||||
p.textContent = `上次检测的是 ${S.remote.slug},slug 已改成 ${currentSlug()} —— 重新点「检测」。`;
|
||||
wrap.appendChild(p);
|
||||
return wrap;
|
||||
}
|
||||
|
||||
if (!S.remote.at){
|
||||
const p = document.createElement('p');
|
||||
p.className = 'hint';
|
||||
p.style.margin = '0';
|
||||
p.textContent = '点「检测」读取远端已有对象:哪些一致、哪些是旧图(大小不同)、'
|
||||
+ '哪些已经没用了(多余),都能在行里直接覆盖或删除。';
|
||||
wrap.appendChild(p);
|
||||
return wrap;
|
||||
}
|
||||
|
||||
const d = remoteDiff();
|
||||
const list = document.createElement('div');
|
||||
list.className = 'rlist';
|
||||
const show = d.rows.slice(0, 24);
|
||||
show.forEach(r => {
|
||||
const row = document.createElement('div');
|
||||
row.className = 'rrow';
|
||||
|
||||
const nm = document.createElement('span');
|
||||
nm.className = 'nm';
|
||||
nm.textContent = r.o.key.split('/').pop();
|
||||
nm.title = r.o.key;
|
||||
|
||||
const sz = document.createElement('span');
|
||||
sz.className = 'sz';
|
||||
sz.textContent = fmtBytes(r.o.size);
|
||||
sz.title = '远端 ' + fmtBytes(r.o.size)
|
||||
+ (r.bytes != null ? ` / 本地 ${fmtBytes(r.bytes)}` : '')
|
||||
+ (r.o.modified ? '\n最后修改 ' + r.o.modified : '');
|
||||
|
||||
const bg = document.createElement('span');
|
||||
bg.className = 'bg ' + r.kind;
|
||||
bg.textContent = r.kind === 'same' ? '一致' : r.kind === 'changed' ? '旧图' : '多余';
|
||||
bg.title = r.kind === 'same' ? '大小与本地一致'
|
||||
: r.kind === 'changed' ? '远端与本地大小不同:多半重新导出过,点 ↻ 覆盖'
|
||||
: '远端有这个对象,但当前主题不再引用';
|
||||
|
||||
const btnPush = document.createElement('button');
|
||||
btnPush.textContent = '↻';
|
||||
btnPush.title = '用本地这张覆盖远端';
|
||||
btnPush.disabled = r.kind === 'extra' || S.s3.busy;
|
||||
btnPush.onclick = () => uploadAssets([r.id]);
|
||||
|
||||
const btnDel = document.createElement('button');
|
||||
btnDel.textContent = '✕';
|
||||
btnDel.title = '从远端删除这个对象';
|
||||
btnDel.disabled = !!S.s3.busy;
|
||||
btnDel.onclick = () => deleteRemoteKeys([r.o.key], '这是你手动选中的对象。');
|
||||
|
||||
row.append(nm, sz, bg, btnPush, btnDel);
|
||||
list.appendChild(row);
|
||||
});
|
||||
wrap.appendChild(list);
|
||||
|
||||
if (d.rows.length > show.length){
|
||||
const more = document.createElement('p');
|
||||
more.className = 'rsum';
|
||||
more.textContent = `…… 另有 ${d.rows.length - show.length} 个未列出`;
|
||||
wrap.appendChild(more);
|
||||
}
|
||||
|
||||
const sum = document.createElement('div');
|
||||
sum.className = 'rsum';
|
||||
const same = d.rows.filter(r => r.kind === 'same').length;
|
||||
const changed = d.rows.filter(r => r.kind === 'changed').length;
|
||||
const extra = d.rows.filter(r => r.kind === 'extra').length;
|
||||
sum.textContent = `一致 ${same} · 旧图 ${changed} · 多余 ${extra} · 缺失 ${d.missing.length}`
|
||||
+ (d.missing.length ? `(${d.missing.slice(0, 4).join('、')}${d.missing.length > 4 ? '…' : ''})` : '');
|
||||
wrap.appendChild(sum);
|
||||
|
||||
const acts = document.createElement('div');
|
||||
acts.className = 'racts';
|
||||
const btnFill = document.createElement('button');
|
||||
btnFill.className = 'primary';
|
||||
btnFill.textContent = `补齐差异${d.missing.length + changed ? ` (${d.missing.length + changed})` : ''}`;
|
||||
btnFill.title = '上传远端缺失的、以及大小不一致的(一致的不重传)';
|
||||
btnFill.disabled = S.s3.busy || !(d.missing.length + changed);
|
||||
btnFill.onclick = uploadDiff;
|
||||
|
||||
const btnClean = document.createElement('button');
|
||||
btnClean.textContent = `删除多余${extra ? ` (${extra})` : ''}`;
|
||||
btnClean.title = '删除远端存在、但当前主题不再引用的对象';
|
||||
btnClean.disabled = S.s3.busy || !extra;
|
||||
btnClean.onclick = deleteExtra;
|
||||
|
||||
acts.append(btnFill, btnClean);
|
||||
wrap.appendChild(acts);
|
||||
return wrap;
|
||||
}
|
||||
|
||||
/** 改画质后重编全部素材,并把前后体积说清楚 —— 否则用户不知道有没有生效。 */
|
||||
|
||||
@@ -91,7 +91,15 @@ class MockS3(BaseHTTPRequestHandler):
|
||||
self.end_headers()
|
||||
|
||||
def do_GET(self):
|
||||
parsed = urllib.parse.urlsplit(self.path)
|
||||
qs = urllib.parse.parse_qs(parsed.query)
|
||||
bucket, key = self._split()
|
||||
if not key and qs.get("list-type") == ["2"]:
|
||||
if not self._check(bucket):
|
||||
return
|
||||
return self._list(qs.get("prefix", [""])[0],
|
||||
int(qs.get("max-keys", ["1000"])[0] or 1000),
|
||||
int(qs.get("continuation-token", ["0"])[0] or 0))
|
||||
MockS3.requests.append((self.command, self.path, dict(self.headers)))
|
||||
if key not in MockS3.store:
|
||||
self._xml(404, "NoSuchKey", "The specified key does not exist")
|
||||
@@ -103,6 +111,34 @@ class MockS3(BaseHTTPRequestHandler):
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def _list(self, prefix, max_keys, offset):
|
||||
# 只实现 ListObjectsV2 里服务端用到的字段;分页用 offset 当 token,够测真实分页逻辑。
|
||||
keys = sorted(k for k in MockS3.store if k.startswith(prefix))
|
||||
page = keys[offset:offset + max_keys]
|
||||
truncated = offset + len(page) < len(keys)
|
||||
items = []
|
||||
for k in page:
|
||||
data = MockS3.store[k][0]
|
||||
items.append(
|
||||
"<Contents><Key>%s</Key><Size>%d</Size>"
|
||||
"<LastModified>2026-10-21T00:00:00.000Z</LastModified><ETag>"mock"</ETag></Contents>"
|
||||
% (k, len(data)))
|
||||
token = ("<NextContinuationToken>%d</NextContinuationToken>" % (offset + len(page))) if truncated else ""
|
||||
body = (
|
||||
'<?xml version="1.0" encoding="UTF-8"?>'
|
||||
'<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">'
|
||||
"<Name>%s</Name><Prefix>%s</Prefix><KeyCount>%d</KeyCount>"
|
||||
"<MaxKeys>%d</MaxKeys><IsTruncated>%s</IsTruncated>%s%s"
|
||||
"</ListBucketResult>"
|
||||
% (BUCKET, prefix, len(page), max_keys, "true" if truncated else "false",
|
||||
token, "".join(items))
|
||||
).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/xml")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_DELETE(self):
|
||||
bucket, key = self._split()
|
||||
if not self._check(bucket):
|
||||
|
||||
@@ -100,7 +100,34 @@ def main():
|
||||
"data": base64.b64encode(b"x").decode()}])
|
||||
check("空前缀传桶根", up2["ok"] and "root.webp" in MockS3.store, str(list(MockS3.store)))
|
||||
|
||||
# 7) 虚拟主机风格的路径(不带桶名)
|
||||
# 7) 列远端:前缀过滤、大小、分页
|
||||
serve.upload_files(cfg, [{"key": "shiroko/asset3.webp", "contentType": "image/webp",
|
||||
"data": base64.b64encode(b"three").decode()}])
|
||||
ls = serve.list_objects(cfg, "shiroko/")
|
||||
keys = sorted(o["key"] for o in ls["objects"])
|
||||
check("按 <前缀>/<slug>/ 列出 3 个对象",
|
||||
keys == ["themes/shiroko/asset1.webp", "themes/shiroko/asset2.webp",
|
||||
"themes/shiroko/asset3.webp"], str(keys))
|
||||
size1 = [o["size"] for o in ls["objects"] if o["key"].endswith("asset1.webp")]
|
||||
check("对象大小与上传字节一致", size1 == [len(f1)], str(size1))
|
||||
paged = serve.list_objects(cfg, "shiroko/", max_keys=2)
|
||||
check("分页把 3 个都取回来了", len(paged["objects"]) == 3, str(len(paged["objects"])))
|
||||
everything = serve.list_objects(cfg, "")
|
||||
# 空前缀 = 列「路径前缀」之下(跨主题),不扫整个桶:桶里还可能有头像等别的东西。
|
||||
check("空前缀列出 <路径前缀>/ 下全部、且不越界",
|
||||
all(o["key"].startswith("themes/") for o in everything["objects"])
|
||||
and len(everything["objects"]) == 3,
|
||||
str([o["key"] for o in everything["objects"]]))
|
||||
|
||||
# 8) 删除:幂等(不存在的键也当成功),且不影响别的前缀
|
||||
del_res = serve.delete_objects(cfg, ["shiroko/asset3.webp", "shiroko/nonexistent.webp"])
|
||||
check("删除含不存在的键仍算成功", del_res["ok"] and del_res["total"] == 2, str(del_res))
|
||||
after = serve.list_objects(cfg, "shiroko/")
|
||||
check("删完只剩 2 个", len(after["objects"]) == 2,
|
||||
str([o["key"] for o in after["objects"]]))
|
||||
check("桶根对象没被误删", "root.webp" in MockS3.store)
|
||||
|
||||
# 9) 虚拟主机风格的路径(不带桶名)
|
||||
vh = serve.S3Config(dict(base, pathStyle=False))
|
||||
check("虚拟主机路径不含桶名", vh.object_path("themes/a.webp") == "/themes/a.webp", vh.object_path("x"))
|
||||
check("虚拟主机 host 带桶名", vh.host.startswith(BUCKET + "."), vh.host)
|
||||
|
||||
+119
-7
@@ -70,7 +70,26 @@ def _quote_key(key: str) -> str:
|
||||
return urllib.parse.quote(key, safe="/-_.~")
|
||||
|
||||
|
||||
def sign_v4(cfg, method, canonical_path, payload_hash, amz_date, host):
|
||||
def _quote_query(s):
|
||||
"""query 的编码规则比 path 更严:`/` 也要编码成 %2F,只有 A-Za-z0-9-_.~ 原样保留。"""
|
||||
return urllib.parse.quote(str(s), safe="-_.~")
|
||||
|
||||
|
||||
def canonical_query(params):
|
||||
"""按 AWS 规则拼 canonical query:先按键排序,同名多值再按值排序,逐项编码。
|
||||
|
||||
ListObjectsV2 的 `list-type` / `prefix` / `continuation-token` 全走这里。
|
||||
顺序错、或漏了编码都是 SignatureDoesNotMatch —— 而 S3 不会告诉你差在哪。
|
||||
"""
|
||||
pairs = []
|
||||
for k, v in (params or {}).items():
|
||||
for one in (v if isinstance(v, (list, tuple)) else [v]):
|
||||
pairs.append((_quote_query(k), _quote_query(one)))
|
||||
pairs.sort()
|
||||
return "&".join("%s=%s" % kv for kv in pairs)
|
||||
|
||||
|
||||
def sign_v4(cfg, method, canonical_path, payload_hash, amz_date, host, query=""):
|
||||
region = cfg.region
|
||||
date = amz_date[:8]
|
||||
signed_headers = "host;x-amz-content-sha256;x-amz-date"
|
||||
@@ -82,7 +101,7 @@ def sign_v4(cfg, method, canonical_path, payload_hash, amz_date, host):
|
||||
# CanonicalHeaders 自身每行以 \n 结尾,与 SignedHeaders 之间因此有一个空行,
|
||||
# 这是规范要求的形态,不是笔误。
|
||||
canonical_request = "\n".join([
|
||||
method, canonical_path, "", canonical_headers, signed_headers, payload_hash,
|
||||
method, canonical_path, query or "", canonical_headers, signed_headers, payload_hash,
|
||||
])
|
||||
scope = date + "/" + region + "/s3/aws4_request"
|
||||
string_to_sign = "\n".join([
|
||||
@@ -178,8 +197,9 @@ def _now():
|
||||
return datetime.datetime.now(datetime.timezone.utc)
|
||||
|
||||
|
||||
def s3_request(cfg, method, key="", body=b"", content_type=None, cache_control=None, timeout=25):
|
||||
"""发一个签名请求。key 为空表示桶级操作(HEAD bucket)。
|
||||
def s3_request(cfg, method, key="", body=b"", content_type=None, cache_control=None,
|
||||
query=None, timeout=25):
|
||||
"""发一个签名请求。key 为空表示桶级操作(HEAD bucket / ListObjectsV2)。
|
||||
|
||||
返回 (status, headers, body);4xx/5xx 不抛异常,交由调用方判断 —— 测试连接
|
||||
需要把「403 是权限问题」和「连不上是网络问题」区分开讲。
|
||||
@@ -188,11 +208,13 @@ def s3_request(cfg, method, key="", body=b"", content_type=None, cache_control=N
|
||||
path = cfg.object_path(key)
|
||||
else:
|
||||
path = ("/" + cfg.bucket) if cfg.path_style else "/"
|
||||
qs = canonical_query(query)
|
||||
payload_hash = _sha256(body) if body else EMPTY_SHA256
|
||||
amz_date = _now().strftime("%Y%m%dT%H%M%SZ")
|
||||
auth = sign_v4(cfg, method, path, payload_hash, amz_date, cfg.host)
|
||||
auth = sign_v4(cfg, method, path, payload_hash, amz_date, cfg.host, qs)
|
||||
|
||||
req = urllib.request.Request(cfg.url(path), data=body if body else None, method=method)
|
||||
url = cfg.url(path) + (("?" + qs) if qs else "")
|
||||
req = urllib.request.Request(url, data=body if body else None, method=method)
|
||||
req.add_header("x-amz-content-sha256", payload_hash)
|
||||
req.add_header("x-amz-date", amz_date)
|
||||
req.add_header("Authorization", auth)
|
||||
@@ -317,6 +339,92 @@ def upload_files(cfg, files):
|
||||
return {"ok": not failed, "files": results, "failed": len(failed), "total": len(results)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 远端清单 / 删除
|
||||
# ---------------------------------------------------------------------------
|
||||
def _local(tag):
|
||||
"""去掉 XML 命名空间。各家 S3 兼容实现都带 `http://s3.amazonaws.com/doc/2006-03-01/`,
|
||||
个别(老 MinIO / 某些网关)又不带 —— 只比 localname 两边都能认。"""
|
||||
return tag.rsplit("}", 1)[-1]
|
||||
|
||||
|
||||
def parse_list_xml(body):
|
||||
"""解析 ListObjectsV2 响应。返回 (对象数组, 下一页 token 或 None)。"""
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
root = ET.fromstring(body)
|
||||
objects, token, truncated = [], None, False
|
||||
for node in root:
|
||||
name = _local(node.tag)
|
||||
if name == "Contents":
|
||||
row = {}
|
||||
for f in node:
|
||||
row[_local(f.tag)] = f.text
|
||||
objects.append({
|
||||
"key": row.get("Key") or "",
|
||||
"size": int(row.get("Size") or 0),
|
||||
"modified": row.get("LastModified") or "",
|
||||
"etag": (row.get("ETag") or "").strip('"'),
|
||||
})
|
||||
elif name == "NextContinuationToken":
|
||||
token = node.text
|
||||
elif name == "IsTruncated":
|
||||
truncated = (node.text or "").strip().lower() == "true"
|
||||
return objects, (token if truncated else None)
|
||||
|
||||
|
||||
def list_objects(cfg, prefix="", max_pages=20, max_keys=1000):
|
||||
"""列 `<cfg.prefix>/<prefix>` 下所有对象。分页上限只是防呆,正常一个主题不到 100 个。"""
|
||||
key_prefix = full_key(cfg, prefix) if str(prefix or "").strip() else (cfg.prefix + "/" if cfg.prefix else "")
|
||||
|
||||
objects, token, pages = [], None, 0
|
||||
while pages < max_pages:
|
||||
pages += 1
|
||||
query = {"list-type": "2", "max-keys": str(max_keys)}
|
||||
if key_prefix:
|
||||
query["prefix"] = key_prefix
|
||||
if token:
|
||||
query["continuation-token"] = token
|
||||
status, _, body = s3_request(cfg, "GET", query=query)
|
||||
if status >= 400:
|
||||
code, msg = _parse_s3_error(body)
|
||||
raise S3Error("ListObjects", status, code or "HTTP%d" % status, msg)
|
||||
page, token = parse_list_xml(body)
|
||||
objects.extend(page)
|
||||
if not token:
|
||||
break
|
||||
return {"ok": True, "prefix": key_prefix, "objects": objects, "truncated": bool(token)}
|
||||
|
||||
|
||||
def delete_objects(cfg, keys):
|
||||
"""逐个 DeleteObject。
|
||||
|
||||
没用批量 POST ?delete:那要额外算 Content-MD5、拼 XML body,而一次最多几十个键,
|
||||
省下的往返抵不上多一条签名路径的风险。S3 的 DELETE 本身幂等(不存在的键也回 204)。
|
||||
"""
|
||||
if not keys:
|
||||
raise ValueError("没有要删除的对象")
|
||||
if len(keys) > 500:
|
||||
raise ValueError("一次最多删 500 个对象")
|
||||
|
||||
results = []
|
||||
for k in keys:
|
||||
try:
|
||||
key = full_key(cfg, k)
|
||||
status, _, body = s3_request(cfg, "DELETE", key)
|
||||
if status >= 400 and status != 404:
|
||||
code, msg = _parse_s3_error(body)
|
||||
results.append({"key": key, "ok": False, "status": status,
|
||||
"code": code or "HTTP%d" % status, "error": msg})
|
||||
else:
|
||||
results.append({"key": key, "ok": True})
|
||||
except Exception as e:
|
||||
results.append({"key": str(k), "ok": False, "error": str(e)})
|
||||
|
||||
failed = [r for r in results if not r["ok"]]
|
||||
return {"ok": not failed, "files": results, "failed": len(failed), "total": len(results)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HTTP 层
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -359,7 +467,7 @@ class Handler(http.server.SimpleHTTPRequestHandler):
|
||||
|
||||
def do_POST(self):
|
||||
route = self.path.split("?")[0]
|
||||
if route not in ("/api/s3/test", "/api/s3/upload"):
|
||||
if route not in ("/api/s3/test", "/api/s3/upload", "/api/s3/list", "/api/s3/delete"):
|
||||
return self._json({"ok": False, "error": "未知接口 %s" % route}, 404)
|
||||
try:
|
||||
payload = self._read_json()
|
||||
@@ -374,6 +482,10 @@ class Handler(http.server.SimpleHTTPRequestHandler):
|
||||
try:
|
||||
if route == "/api/s3/test":
|
||||
return self._json(test_connection(cfg))
|
||||
if route == "/api/s3/list":
|
||||
return self._json(list_objects(cfg, payload.get("prefix") or ""))
|
||||
if route == "/api/s3/delete":
|
||||
return self._json(delete_objects(cfg, payload.get("keys") or []))
|
||||
return self._json(upload_files(cfg, payload.get("files") or []))
|
||||
except S3Error as e:
|
||||
return self._json({
|
||||
|
||||
Reference in New Issue
Block a user