Services

How to Build a Document Management System: Chunked Uploads, Virus Scanning, Permissions and Retention

2026.09.13 · 36 views
How to Build a Document Management System: Chunked Uploads, Virus Scanning, Permissions and Retention

From a 2.3 GB drawing set that will not send, to a DMS with versioning, audit trails and automatic expiry — 4 alternatives, 6 traps, real NT$ pricing and a 90-day roadmap.

Share:

3 a.m., and the fourteenth failed upload

An engineering consultancy needed to hand a 2.3 GB set of as-built drawings to its client. Gmail caps attachments at 25 MB; files sent over LINE expire after seven days. So the package went out as nine zip archives on a free file host — the client missed one, and the drawing set was unusable. The firm later measured about 320 hours a year lost to re-sending, hunting and version-checking files. In payroll terms that is over NT$250,000, more than the entire first-year cost of owning a purpose-built document management system.

Where a custom DMS fits, and where it does not

✅ Build a custom DMS❌ Do not build
Single files routinely exceed 500 MB (drawings, video, scans)Files stay under 20 MB
External vendors need access without company accountsUnder 10 people, no external sharing
You must prove who downloaded which version, and whenNo version-tracking obligation
Files must attach to orders, tickets or contractsA plain file locker with nothing to connect to
Privacy rules or retention policies require automatic expiryNo retention policy

Row four is where most firms misjudge. If you only need somewhere to put files, Dropbox Business always wins on cost. But once the requirement becomes "this file must attach itself to that work order and lock once the client signs off," packaged tools demand layers of glue code and the total cost quietly overtakes custom work.

Alternatives compared

OptionStrengthsLimitsAnnual cost
Google Workspace / Dropbox BusinessZero development, familiar UINo link to business process; coarse external-share permissionsNT$3,600–6,000 per seat
SharePoint / OneDriveMature permission model, native AD integrationHigh consulting fees; customisation needs SPFxNT$150,000–400,000 rollout
Open source (Nextcloud, Paperless-ngx)No licence fee, large plugin ecosystemNeeds in-house DevOps; upgrades break things; uneven CJK OCRNT$60,000–120,000 ops
Custom Laravel DMSEmbeds into existing systems; permissions fully yoursHigher upfront cost; needs a clear specFrom NT$180,000

The build, stage by stage

StageDaysDeliverablesTools
1 Inventory and permission modelling5File-type inventory, role permission matrix, retention policyNotion, Miro
2 Storage and upload architecture3Bucket structure, chunking strategy, signed-URL flowS3 Multipart Upload, Cloudflare R2
3 Interface design5Upload, browse, share and audit screensFigma
4 Upload pipeline10Chunking, resume, dedupe, progress reportingtus, Uppy, Laravel Filesystem
5 Security and governance7Scan queue, signed-URL expiry, download audit logClamAV, Laravel Queue, Redis
6 Process binding8Attachment to tickets and contracts, sign-off locking, auto-archivalLaravel Scheduler
7 Load testing and launch55 GB upload report, concurrency test, ops handbookk6, UptimeRobot

That is 43 working days, or nine to ten calendar weeks. Stage 4 carries the project: files are sliced into 5–10 MB chunks and uploaded piece by piece, so a failure retries only that chunk. Without it, a 2.3 GB drawing set that drops at 97% starts over from zero. Browser-native Blob.slice handles the client-side splitting — no plugin required.

What it actually costs

  • ScriptWalker Document Management System (DMS) build: from NT$180,000 — upload, permissions, versioning, audit
  • Process-binding module: +NT$60,000–120,000. OCR and full-text search: +NT$80,000

The costs that get forgotten (modelled at 1 TB):

  • Object storage: Cloudflare R2 at about US$0.015 per GB-month (roughly NT$480 a month for 1 TB) with zero egress charges
  • On AWS S3 instead: about US$0.023 per GB-month, plus roughly US$0.09 per GB of data transfer out — download-heavy projects almost always blow the budget here
  • Scanning host: ClamAV is free to licence but still needs a 1 vCPU / 2 GB container, around NT$400 a month
  • Off-site backup adds 40–60% to storage cost; three-year audit-log retention adds around NT$300 a month
  • Annual maintenance: 15–20% of build cost, or NT$27,000–36,000 a year

First-year total cost of ownership lands between NT$210,000 and NT$240,000 — almost exactly what the consultancy above was already losing to file chaos.

Expectation versus reality

What clients expectWhat actually happens
"Upload is just a button"Chunking, resume, dedupe and retries consume 35% of project hours
"Drop in a virus-scanning package"Scanning must run asynchronously behind a pending state; a synchronous scan times out any 500 MB upload
"Two roles: manager and staff"A full inventory surfaces 7–9 roles, plus vendor accounts scoped per project
"Migrating old files will be quick"Fifteen years of network drives means duplicates and mangled filenames; cleanup takes two to three weeks
"Expired files just get deleted"Contracts rarely permit hard deletion: archive, notify the owner, destroy on approval

Six traps and how to avoid them

  • Routing files through the PHP application: exhausts memory and hits upload_max_filesize. → Upload straight to object storage with a signed URL; the backend receives only metadata.
  • Validating by file extension: renaming .php to .jpg defeats it. → Check MIME magic bytes, rename server-side, disable execution on the bucket.
  • Share links that never expire: one leaked link is permanent public access. → Default signed URLs to 15 minutes and cap download counts on sensitive files.
  • No deduplication: the same file uploaded by twelve people costs twelve times the storage. → Compare SHA-256 before upload and store a reference instead.
  • Versioning by filename: "final_really_final_v3.pdf" is where the trouble starts. → Keep a version chain in the system and generate filenames server-side.
  • Logging uploads but not downloads: when something leaks, the question is who downloaded it. → Log every signed-URL issuance with user, IP, timestamp and version.

Success metrics and the 90-day roadmap

  • Day 30 — stability. Is upload success above 99%? Has average time-to-find dropped from eight minutes to under one? This month is about the long tail of failures: weak networks, mobile devices, oversized single files.
  • Day 60 — adoption. Is weekly active usage above 70% per department? How much still goes by email attachment? The holdouts are almost always blocked by "my old files aren't in there yet," not "I can't use it."
  • Day 90 — governance. Track misconfigured permissions, overdue documents and the storage growth curve, then switch on automatic archival. Most projects find 60% of data is cold and can move to infrequent-access tiers for another 30–40% saving.

Decision checklist

  • ☐ Does our largest single file exceed 500 MB?
  • ☐ How often each month does someone miss the current version?
  • ☐ Do external vendors or clients need access?
  • ☐ Must we know who downloaded which version?
  • ☐ Must files attach to tickets or contracts?
  • ☐ Is there a retention period set by law or contract?
  • ☐ On expiry, archive or delete — and who approves?
  • ☐ How many terabytes of legacy files exist, and where?
  • ☐ Who owns permission administration after launch?
  • ☐ Do we need full-text search and OCR?
  • ☐ What is the annual storage and operations ceiling?

Seven or more ticks and a custom DMS usually pays back. Three or fewer, and Dropbox Business is the right answer.

Frequently asked questions

How large can a single file be?

With direct chunked upload, the ceiling comes from the storage layer. S3-compatible multipart upload supports objects up to 5 TB with chunks between 5 MB and 5 GB. Our standard load test covers 5 GB single files, which clears most video and drawing-set workloads.

Does virus scanning slow uploads down?

No, because scanning sits outside the upload path. The file lands, gets flagged as scanning, and a Laravel Queue worker processes it in the background — usually 10 to 60 seconds before it becomes downloadable. No share link can be generated until the scan clears.

Can we ship upload first and add permissions and audit later?

Not advisable. The permission model determines table structure and bucket path design, so bolting it on later is a rebuild. Complete upload, permissions and audit as one core release, then hold OCR, full-text search and e-signature for phase two.

Next step

ScriptWalker offers a free 30-minute file architecture review. Bring your current storage setup and your largest file size; we return a build-versus-buy recommendation and a phased budget range. The Document Management System (DMS) build starts at NT$180,000 and covers requirements interviews, permission modelling, the chunked upload pipeline and audit logging, delivered in nine to ten weeks.

Share: