Skip to content

feat: route document uploads through ChittyStorage #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chitcommit merged 1 commit into from
Apr 8, 2026
Merged

feat: route document uploads through ChittyStorage #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ export type Env = {
AI: Ai;
HYPERDRIVE: Hyperdrive;
DOCUMENTS: R2Bucket;
SVC_STORAGE: Fetcher;
COMMAND_KV: KVNamespace;
ACTION_AGENT: DurableObjectNamespace;
DATABASE_URL?: string;
Expand Down
201 changes: 97 additions & 104 deletions src/routes/documents.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
import { Hono } from 'hono';
import type { Env } from '../index';
import { getDb } from '../lib/db';
import { evidenceClient } from '../lib/integrations';

export const documentRoutes = new Hono<{ Bindings: Env }>();
const UUID_V4ISH = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;

documentRoutes.get('/', async (c) => {
const sql = getDb(c.env);
Expand All @@ -19,18 +17,13 @@ const ALLOWED_TYPES = new Set([
]);
const MAX_FILE_SIZE = 25 * 1024 * 1024; // 25MB

// Upload document to R2 and create DB record
// Upload document via ChittyStorage (content-addressed, entity-linked)
documentRoutes.post('/upload', async (c) => {
const formData = await c.req.formData();
const file = formData.get('file') as unknown as File | null;
const linkedDisputeRaw = formData.get('linked_dispute_id');
const linkedDisputeId = typeof linkedDisputeRaw === 'string' && linkedDisputeRaw.trim().length > 0
? linkedDisputeRaw.trim()
: null;
const entitySlug = (formData.get('entity_slug') as string) ?? '';
const origin = (formData.get('origin') as string) ?? 'first-party';
if (!file || typeof file === 'string') return c.json({ error: 'No file provided' }, 400);
if (linkedDisputeId && !UUID_V4ISH.test(linkedDisputeId)) {
return c.json({ error: 'Invalid linked_dispute_id' }, 400);
}

if (!ALLOWED_TYPES.has(file.type)) {
return c.json({ error: 'Unsupported file type', allowed: [...ALLOWED_TYPES] }, 400);
Expand All @@ -39,137 +32,137 @@ documentRoutes.post('/upload', async (c) => {
return c.json({ error: `File too large (max ${MAX_FILE_SIZE / 1024 / 1024}MB)` }, 400);
}

// Sanitize filename: keep only alphanumeric, dash, underscore, dot
const safeName = file.name.replace(/[^a-zA-Z0-9._-]/g, '_');
const sql = getDb(c.env);
const r2Key = `documents/${Date.now()}_${safeName}`;

// Store in R2
await c.env.DOCUMENTS.put(r2Key, file.stream(), {
// Hash locally for chitty_id generation (temporary until ChittyIdentity integration)
const bytes = new Uint8Array(await file.arrayBuffer());
const hashBuf = await crypto.subtle.digest('SHA-256', bytes);
const contentHash = Array.from(new Uint8Array(hashBuf)).map(b => b.toString(16).padStart(2, '0')).join('');
const chittyId = `scan-${contentHash.slice(0, 12)}`;

// Submit to ChittyStorage via service binding
if (c.env.SVC_STORAGE) {
try {
const content_base64 = btoa(String.fromCharCode(...bytes));
const storageRes = await c.env.SVC_STORAGE.fetch('https://internal/mcp', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
jsonrpc: '2.0',
method: 'tools/call',
params: {
name: 'storage_ingest',
arguments: {
chitty_id: chittyId,
filename: safeName,
content_base64,
mime_type: file.type,
source_platform: 'chittycommand',
origin,
copyright: '©2026_IT-CAN-BE-LLC_ALL-RIGHTS-RESERVED',
entity_slugs: entitySlug ? [entitySlug] : [],
},
},
id: 1,
}),
});
// MCP response - extract result
const mcp = await storageRes.json() as any;
const result = mcp?.result?.content?.[0]?.text;
if (result) {
const parsed = JSON.parse(result);
// Track in local cc_documents for ChittyCommand UI
const [doc] = await sql`
INSERT INTO cc_documents (doc_type, source, filename, r2_key, processing_status, metadata)
VALUES ('upload', 'chittycommand', ${safeName}, ${parsed.r2_key ?? `sha256/${contentHash}`}, 'synced',
${JSON.stringify({ content_hash: contentHash, storage_chitty_id: chittyId, deduplicated: parsed.deduplicated })}::jsonb)
RETURNING *
`;
return c.json({ ...doc, content_hash: contentHash, storage: parsed }, 201);
}
} catch (err) {
console.error('[documents] ChittyStorage ingest failed, falling back to direct R2:', err);
}
}

// Fallback: direct R2 (legacy path — remove once SVC_STORAGE is confirmed stable)
const r2Key = `sha256/${contentHash}`;
await c.env.DOCUMENTS.put(r2Key, bytes, {
httpMetadata: { contentType: file.type },
customMetadata: { filename: safeName, source: 'chittycommand' },
});

// Create DB record
const [doc] = await sql`
INSERT INTO cc_documents (doc_type, source, filename, r2_key, linked_dispute_id, processing_status)
VALUES ('upload', 'manual', ${safeName}, ${r2Key}, ${linkedDisputeId}, 'pending')
INSERT INTO cc_documents (doc_type, source, filename, r2_key, processing_status)
VALUES ('upload', 'manual', ${safeName}, ${r2Key}, 'pending')
Comment on lines +96 to +97
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve linked_dispute_id when inserting uploads

The /upload path no longer persists linked_dispute_id, so documents uploaded from dispute context are silently detached from their dispute. The UI still sends this field (ui/src/lib/api.ts), and dispute detail fetches documents with WHERE linked_dispute_id = :id (src/routes/disputes.ts), so these uploads stop appearing in dispute timelines after this change.

Useful? React with 👍 / 👎.

RETURNING *
`;

// Fire-and-forget: push to ChittyEvidence pipeline
const evidence = evidenceClient(c.env);
if (evidence) {
evidence.submitDocument({
filename: safeName,
fileType: file.type,
fileSize: String(file.size),
description: `Uploaded via ChittyCommand`,
evidenceTier: 'BUSINESS_RECORDS',
}).then((ev) => {
if (ev?.id) {
sql`UPDATE cc_documents SET metadata = jsonb_build_object('ledger_evidence_id', ${ev.id}), processing_status = 'synced' WHERE id = ${doc.id}`
.catch((err) => console.error(`[documents] Failed to update metadata for doc ${doc.id}:`, err));
} else {
console.warn(`[documents] Evidence submission returned no ID for ${safeName}`);
}
}).catch((err) => console.error(`[documents] Evidence submission failed for ${safeName}:`, err));
}

return c.json(doc, 201);
});

// Batch upload multiple documents (with dedup)
// Batch upload via ChittyStorage
documentRoutes.post('/upload/batch', async (c) => {
const formData = await c.req.formData();
const files = formData.getAll('files') as unknown as File[];
const entitySlug = (formData.get('entity_slug') as string) ?? '';
if (!files.length) return c.json({ error: 'No files provided' }, 400);
if (files.length > 20) return c.json({ error: 'Maximum 20 files per batch' }, 400);

const sql = getDb(c.env);

// Fetch existing filenames for dedup check
const safeNames = files.map((f) => typeof f === 'string' ? '' : f.name.replace(/[^a-zA-Z0-9._-]/g, '_'));
const existing = await sql`SELECT filename FROM cc_documents WHERE filename = ANY(${safeNames})`;
const existingSet = new Set(existing.map((r: any) => r.filename));

const results: { filename: string; status: 'ok' | 'skipped' | 'error'; error?: string; doc?: any }[] = [];
const results: { filename: string; status: 'ok' | 'skipped' | 'error'; error?: string; content_hash?: string }[] = [];

for (const file of files) {
if (typeof file === 'string') {
results.push({ filename: '(invalid)', status: 'error', error: 'Not a file' });
continue;
}
if (!ALLOWED_TYPES.has(file.type)) {
results.push({ filename: file.name, status: 'error', error: `Unsupported type: ${file.type}` });
continue;
}
if (file.size > MAX_FILE_SIZE) {
results.push({ filename: file.name, status: 'error', error: 'File too large (max 25MB)' });
continue;
}
if (typeof file === 'string') { results.push({ filename: '(invalid)', status: 'error', error: 'Not a file' }); continue; }
if (!ALLOWED_TYPES.has(file.type)) { results.push({ filename: file.name, status: 'error', error: `Unsupported: ${file.type}` }); continue; }
if (file.size > MAX_FILE_SIZE) { results.push({ filename: file.name, status: 'error', error: 'Too large' }); continue; }

const safeName = file.name.replace(/[^a-zA-Z0-9._-]/g, '_');

// Dedup: skip if this filename was already uploaded
if (existingSet.has(safeName)) {
results.push({ filename: safeName, status: 'skipped', error: 'Already uploaded' });
continue;
}

const r2Key = `documents/${Date.now()}_${safeName}`;

try {
await c.env.DOCUMENTS.put(r2Key, file.stream(), {
httpMetadata: { contentType: file.type },
});
const [doc] = await sql`
INSERT INTO cc_documents (doc_type, source, filename, r2_key, processing_status)
VALUES ('upload', 'manual', ${safeName}, ${r2Key}, 'pending')
RETURNING *
`;
existingSet.add(safeName); // prevent dupes within same batch
results.push({ filename: safeName, status: 'ok', doc });
const bytes = new Uint8Array(await file.arrayBuffer());
const hashBuf = await crypto.subtle.digest('SHA-256', bytes);
const contentHash = Array.from(new Uint8Array(hashBuf)).map(b => b.toString(16).padStart(2, '0')).join('');

if (c.env.SVC_STORAGE) {
const content_base64 = btoa(String.fromCharCode(...bytes));
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid spreading file bytes into fromCharCode

String.fromCharCode(...bytes) passes one argument per byte, which exceeds JS argument limits for normal document sizes and throws RangeError before ingest. In this batch path that means otherwise valid files fail whenever SVC_STORAGE is enabled; use chunked/base64 encoding that does not spread the whole array into function arguments.

Useful? React with 👍 / 👎.

await c.env.SVC_STORAGE.fetch('https://internal/mcp', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
Comment on lines +127 to +130
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject batch success when storage ingest returns HTTP error

This batch branch awaits fetch but never checks response.ok (or MCP error payload), so a 4xx/5xx from ChittyStorage still falls through and is reported as status: 'ok'. That creates silent data-loss behavior where clients believe uploads succeeded even though ingest was rejected.

Useful? React with 👍 / 👎.

jsonrpc: '2.0', method: 'tools/call',
params: { name: 'storage_ingest', arguments: {
chitty_id: `scan-${contentHash.slice(0, 12)}`, filename: safeName,
content_base64, mime_type: file.type, source_platform: 'chittycommand',
origin: 'first-party', copyright: '©2026_IT-CAN-BE-LLC_ALL-RIGHTS-RESERVED',
entity_slugs: entitySlug ? [entitySlug] : [],
}}, id: 1,
}),
});
} else {
await c.env.DOCUMENTS.put(`sha256/${contentHash}`, bytes, {
httpMetadata: { contentType: file.type },
customMetadata: { filename: safeName, source: 'chittycommand' },
});
}
results.push({ filename: safeName, status: 'ok', content_hash: contentHash });
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Insert batch-ingested files into cc_documents

After a successful batch ingest, the handler only appends to results and never writes a cc_documents row. Since document list and gap endpoints read from cc_documents, batch-uploaded files become invisible to dashboard/gap workflows despite returning success.

Useful? React with 👍 / 👎.

} catch (err) {
results.push({ filename: safeName, status: 'error', error: String(err) });
}
}

const succeeded = results.filter((r) => r.status === 'ok').length;
const skipped = results.filter((r) => r.status === 'skipped').length;
return c.json({ total: files.length, succeeded, skipped, failed: files.length - succeeded - skipped, results }, 201);
return c.json({ total: files.length, succeeded: results.filter(r => r.status === 'ok').length, results }, 201);
});

// Identify missing documents / coverage gaps
documentRoutes.get('/gaps', async (c) => {
const sql = getDb(c.env);

// All obligation payees that should have statements
const payees = await sql`
SELECT DISTINCT payee, category, recurrence
FROM cc_obligations
WHERE status IN ('pending', 'overdue')
ORDER BY payee
`;

// Documents uploaded per payee (match by filename containing payee name)
const payees = await sql`SELECT DISTINCT payee, category, recurrence FROM cc_obligations WHERE status IN ('pending', 'overdue') ORDER BY payee`;
const docs = await sql`SELECT filename, created_at FROM cc_documents ORDER BY created_at DESC`;

const gaps: { payee: string; category: string; recurrence: string | null; has_document: boolean; last_upload: string | null }[] = [];

for (const p of payees) {
const payeeLower = (p.payee as string).toLowerCase().replace(/[^a-z0-9]/g, '');
const match = docs.find((d: any) =>
d.filename && (d.filename as string).toLowerCase().replace(/[^a-z0-9]/g, '').includes(payeeLower)
);
gaps.push({
payee: p.payee as string,
category: p.category as string,
recurrence: p.recurrence as string | null,
has_document: !!match,
last_upload: match ? (match.created_at as string) : null,
});
const match = docs.find((d: any) => d.filename && (d.filename as string).toLowerCase().replace(/[^a-z0-9]/g, '').includes(payeeLower));
gaps.push({ payee: p.payee as string, category: p.category as string, recurrence: p.recurrence as string | null, has_document: !!match, last_upload: match ? (match.created_at as string) : null });
}

const missing = gaps.filter((g) => !g.has_document);
return c.json({ total_payees: gaps.length, covered: gaps.length - missing.length, missing: missing.length, gaps });
return c.json({ total_payees: gaps.length, covered: gaps.length - gaps.filter(g => !g.has_document).length, missing: gaps.filter(g => !g.has_document).length, gaps });
});
91 changes: 91 additions & 0 deletions wrangler.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
name = "chittycommand"
account_id = "0bc21e3a5a9de1a4cc843be9c3e98121"
main = "src/index.ts"
compatibility_date = "2026-01-15"
compatibility_flags = ["nodejs_compat"]
# Custom domain (command.chitty.cc) managed via CF dashboard — no zone token needed

[placement]
mode = "smart"

[vars]
ENVIRONMENT = "production"
CHITTYAUTH_URL = "https://auth.chitty.cc"
CHITTYLEDGER_URL = "https://ledger.chitty.cc"
CHITTYFINANCE_URL = "https://finance.chitty.cc"
CHITTYCHARGE_URL = "https://charge.chitty.cc"
CHITTYCONNECT_URL = "https://connect.chitty.cc"
PLAID_ENV = "production"
CHITTYBOOKS_URL = "https://chittybooks.chitty.cc"
CHITTYASSETS_URL = "https://chittyassets.chitty.cc"
CHITTYSCRAPE_URL = "https://scrape.chitty.cc"
CHITTYROUTER_URL = "https://router.chitty.cc"
CHITTYAGENT_SCRAPE_URL = "https://chittyagent-scrape.ccorp.workers.dev"
CHITTYEVIDENCE_URL = "https://evidence.chitty.cc"
# Optional: chittyregister for beacon heartbeats
CHITTYREGISTER_URL = "https://register.chitty.cc"
# Optional: chittychat data API for MCP tools
# CHITTYCHAT_DATA_API = "https://chittychat-api.chitty.cc/api"
# Optional: ChittySchema and ChittyCert endpoints for MCP tools
# CHITTYSCHEMA_URL = "https://schema.chitty.cc"
# CHITTYCERT_URL = "https://cert.chitty.cc"
# PLAID_CLIENT_ID and PLAID_SECRET set via `wrangler secret put`

# Neon PostgreSQL via Hyperdrive
[[hyperdrive]]
binding = "HYPERDRIVE"
id = "6f6cba43540b430eb77045f79384ca00"

# R2 for document storage
[[r2_buckets]]
binding = "DOCUMENTS"
bucket_name = "chittycommand-documents"

# KV for sync state and caching
[[kv_namespaces]]
binding = "COMMAND_KV"
id = "64eef343b99b46ac909dbbcc1c4b2dee"

# AI Gateway binding
[ai]
binding = "AI"

# ActionAgent Durable Object (Agents SDK)
[durable_objects]
bindings = [
{ name = "ACTION_AGENT", class_name = "ActionAgent" }
]

[[migrations]]
tag = "v1"
new_sqlite_classes = ["ActionAgent"]

# Observability — match dashboard config
[observability]
enabled = true
head_sampling_rate = 1
[observability.logs]
enabled = true
head_sampling_rate = 1
invocation_logs = true
[observability.traces]
enabled = true
head_sampling_rate = 1

# ChittyTrack observability
[[tail_consumers]]
service = "chittytrack"

# ChittyStorage service binding for document ingest
[[services]]
binding = "SVC_STORAGE"
service = "chittystorage"

[triggers]
crons = [
"0 12 * * *", # Daily 6 AM CT: Plaid + ChittyFinance sync
"0 13 * * *", # Daily 7 AM CT: Court docket check
"0 14 * * 1", # Weekly Monday 8 AM CT: Utility scrapers
"0 15 1 * *" # Monthly 1st 9 AM CT: Mortgage, property tax
]

Loading