File Upload and Media Library Systems: Large Files, Transcoding, CDN, and Hotlink Protection

One 800MB Video Took the Whole Site Down for 12 Minutes

An education client's admin panel worked fine until an instructor uploaded an 800MB course video. The file was buffered into PHP memory, returned 413 Request Entity Too Large, and after three retries the server ran out of memory. The site was down for 12 minutes. The invoice was worse: the video sat on the same disk as the application and was streamed through the app layer, so one month of bandwidth cost more than the feature itself. This is not bad luck. It is the predictable result of treating file upload as just another form field.

When This Is Worth Building, and When It Is Not

Worth a full media library when:

Users upload files larger than 100MB (video, design source files, scanned PDFs)
The same image must appear at several sizes (thumbnail, hero, OG share image)
Content sits behind a paywall and must not leak when a URL is copied
You expect over 10,000 assets and need tagging, search, versioning, duplicate detection
You have compliance needs: uploader audit trail, retention period, deletion records

Not worth the budget when:

It is a brochure site and you upload 30 images once, then never again
Total assets are under 2GB with no video, and the CMS media library covers it
Only one or two internal colleagues upload, and a shared drive link is acceptable
The product is still in validation and you do not know whether users will upload

Alternatives Matrix: Three Paths, Very Different Bills

Option	Pros	Cons	Cost band
A. Local server disk	Fastest to build, no cross-origin setup	No horizontal scaling, painful backups, traffic uses server bandwidth, a full disk means downtime	Build from NT$15,000; server upgrades rise linearly with volume
B. Object storage + CDN, in-house	Portable, lowest unit price, full control over transcoding and signing rules	You own direct upload, the transcoding queue, cache invalidation, and cleanup jobs	Build NT$60,000–150,000; Cloudflare R2 Standard at US$0.015/GB-month with zero egress fees, or S3 Standard at US$0.023/GB-month plus roughly US$0.09/GB egress
C. Managed media SaaS	Transcoding, thumbnails, player, hotlink protection out of the box	Higher unit price, deep lock-in, unpredictable overage billing	Build NT$40,000–80,000; Cloudinary's free plan includes 10GB storage and 20GB viewing bandwidth, paid tiers step up by usage

Rule of thumb: if monthly egress (GB) × unit price × 12 < the extra build cost of doing it yourself, pick SaaS. Otherwise pick B. Video-heavy projects almost always flip to B in year two.

The Build, Stage by Stage

Week 1 — Spec: list file types, maximum sizes, daily volume, retention, and the permission matrix. Deliverables: Media Spec Sheet and Permission Matrix, drafted in Notion.
Week 2 — Architecture: decide the direct-upload path, naming rules, and transcoding targets (video to H.264 720p/1080p HLS, images to WebP). Deliverables: architecture diagram and upload flow, drawn in Figma.
Weeks 3–5 — Core development: the backend issues presigned URLs, the browser uploads chunks straight to object storage, and nothing passes through the application server. A completion webhook writes the database row and enqueues the job. Tools: Laravel Queue or BullMQ, Uppy, FFmpeg.
Week 6 — Media library admin: tags, search, replace, versioning, bulk and soft delete. Deliverables: working admin panel and operations guide.
Week 7 — CDN and protection: attach Cloudflare caching, set a Referer allowlist and signature TTL (15 minutes for private assets), enable WAF rate limiting, wire up ClamAV.
Week 8 — Load test: simulate 50 concurrent 500MB uploads, measure failure and resume rates. Deliverables: acceptance report and a monitoring dashboard using UptimeRobot and Sentry.

Real Cost Breakdown, Including the Hidden Lines

Modelled on a project adding 300GB per month, holding 3TB, serving 5TB of egress monthly:

Build: direct upload with resume NT$25,000; transcoding queue NT$30,000; media library admin NT$35,000; permissions and signing NT$18,000
Storage: 3TB on R2 is roughly US$45/month; on S3 Standard roughly US$69/month
Egress, the biggest hidden line: R2 egress is US$0, while serving 5TB straight from S3 is roughly US$450/month — often the single most expensive item in the project
Transcoding compute: a self-hosted FFmpeg worker with 4 vCPU runs about NT$1,800/month
Routinely forgotten: object operation fees (R2 Class A at US$4.50 per million, Class B at US$0.36 per million), off-site backup copies, antivirus licensing, SSL and custom domains, building the orphan-file cleanup job, and 15–20% annual maintenance

ScriptWalker's File Upload and Media Library Build starts at NT$68,000 and covers direct upload, the transcoding queue, and a basic admin panel. Streaming delivery and DRM are quoted separately.

What Clients Expect vs What Actually Happens

Expectation: upload is just a button. Reality: large files need chunking, resume, interruption handling, and duplicate-submit protection. Upload alone is about 40% of the module's hours.
Expectation: the video plays as soon as it uploads. Reality: a 30-minute video takes 5–20 minutes to produce multi-resolution HLS, so you need a "processing" state and a notification.
Expectation: cloud storage is cheap. Reality: storage is cheap, egress is not. The same dataset on S3 versus R2 can differ annually by more than the entire build fee.
Expectation: the URL is long, nobody can guess it. Reality: one shared link leaks forever. Without a signature TTL you have no access control.
Expectation: once it launches we can forget it. Reality: without a cleanup job, 30–40% of stored objects become orphans within two years, and you pay for them monthly.

Traps and How to Avoid Them

Files pass through the application server. Fix: switch to presigned URLs so the browser uploads directly; the server only issues credentials and records metadata.
Validating type by file extension. Fix: check real MIME via magic bytes, rename to a random UUID, strip EXIF.
Transcoding inside the request thread. Fix: process asynchronously in a queue with retry limits, a dead-letter queue, and a manual re-run path.
Private content on public URLs. Fix: keep the bucket fully private, serve only short-lived signed URLs, add a Referer allowlist at the CDN.
No orphan-file cleanup. Fix: write uploads in a "pending" state, auto-delete anything unlinked after 24 hours, and run a monthly reconciliation report.
No virus scanning or upload quota. Fix: trigger ClamAV on upload and quarantine until it passes; cap daily count and volume per account and rate-limit at the CDN.

Success Metrics and the First 90 Days

Day 30: track upload success rate (target above 98%), average upload duration, and 413/timeout error counts. Tune chunk size and resume logic.
Day 60: track transcoding backlog (target median under 5 minutes), retry failure rate, and CDN cache hit ratio (target above 90%). Tune cache rules and trim transcoding profiles.
Day 90: track real cost per GB, orphan-file ratio (target under 5%), hotlink blocks, and storage growth. Introduce lifecycle policies to move cold data to infrequent-access tiers.

Decision Checklist

☐ Can you state the maximum allowed size for a single file?
☐ Do you know your monthly added volume and monthly egress?
☐ Does any uploaded content need permission control?
☐ Do you need video transcoding or multi-resolution streaming?
☐ Does one image need three or more rendered sizes?
☐ Are uploaders external or unauthenticated users?
☐ Do you have audit-trail or retention compliance requirements?
☐ Does your app run on more than one server instance?
☐ Can you accept 5–20 minutes of transcoding delay?
☐ Can you fund 15–20% annual maintenance?
☐ Do you have people to operate the queue and monitoring?
☐ Do you need portability across cloud providers?
☐ Have you modelled egress cost, not just storage cost?

Seven or more checks means build option B in-house. Three or fewer means your CMS media library is enough.

FAQ

What is the right technique for large file uploads?

Above 100MB, use multipart upload with direct-to-storage transfer from the browser. Uppy or the tus protocol handles chunking and resume on the front end; the backend only issues presigned URLs. If the connection drops, the transfer resumes from the failed chunk instead of restarting.

Should I choose Cloudflare R2 or AWS S3?

It depends on traffic shape. If egress far exceeds stored volume (video, download-heavy sites), R2 usually wins because it charges nothing for egress. If you already depend on many AWS services and need mature lifecycle and analytics tooling, S3 integrates better. Both speak the S3 API, so migration cost is mostly data transfer and URL redirects.

How do you actually stop hotlinking?

Three layers together: keep the bucket private, serve only short-lived signed URLs, and configure a Referer allowlist plus rate limiting at the CDN. An unguessable long URL is not a security control. For paid content, add single-use playback tokens or DRM.

Can an existing live system be retrofitted?

Yes, in two phases: route new uploads to object storage while keeping the legacy read path, then migrate old files in background batches and rewrite the database columns. Migrations under 3TB typically finish in two to three weeks with no downtime.

Next Step

Evaluating a media pipeline? We offer a free 30-minute architecture consultation. Bring your file types, monthly growth, and egress estimate, and we will calculate the two-year total cost of ownership for options A, B, and C on the spot, plus hand you a Media Spec Sheet template.