#18: A Mobile App Developer's Guide to Video Files

I mentioned recently that my friends and I are working on a new video-focused social app. With that comes some priorities we need to balance:

• Small files (i.e. low storage costs & fast upload/download)

• Acceptable video & audio quality.

• Compatible with all modern mobile phones.

It’s been super satisfying to look into this and get more familiar with video. I accumulated quite a few actionable insights and fun facts along the way that I’ll summarize for you here.

If you’re designing an app with video features, here are the 3 things you need to do.

1. Pick a performant and compatible file format.

A video file isn’t all video data. It’s also wrapped in metadata that allows programs on your computer — like your file browser or video player — to know how to sort, display and read the file.

So the video file format is actually the combination of two formats:

• A Codec — the algorithm used to encode the video.

• A Container Format — the format of the metadata that wraps the encoded video.

When we think about video file “formats” we think about the file extensions .mp4, .mov and .mkv. But these are actually just container formats.

What affects file size most is the video’s codec. How are we representing the video data in this file?

A codec will compress video data in one of two ways:

• Intraframe Compression — where each frame is encoded individually

• Interframe Compression — where only the differences between frames are encoded.

Codecs that use intraframe compression can be good choice for video editors because frames can be loaded in individually, whereas frames in a video using interframe compression depend on the frames that came before it.

Interframe compression is especially good for “talking head” videos, where the background is static. Nowadays almost all commonly-used codecs leverage interframe compression.

The best options for video codecs today are:

• H.264 — It compresses files well while maintaining high quality.

• H.265 (or HEVC)  — Compresses 50% more efficiently than H.264.

• V9 — Achieves even better compression than H.265 for the same quality.

So why not choose V9 every time? Compatibility.

V9 is used with the .webm container format and is supported by almost all browsers nowadays. But it isn’t natively supported on iOS, so it’s not the best choice for mobile apps.

HEVC is supported in all modern phones, but older phones may not support it. Meanwhile, H.264 can be considered universally supported.

The question my team and I need to ask ourselves is: What devices do we expect our target user base to have? Will any of them be using very old phones? Are we ok with that minority of users not being able to use our app well?

If you want to play it super safe and build a video app that works an any device, use the mp4/H.264 format.

In my team’s case, I think opting for HEVC is a very safe bet. No one in our target demographic is likely to be using a pre-iPhone 8 smartphone. And older phones may be able to perform software encoding and decoding of HEVC video.

2. Tune other parameters based on your use case.

Aside from the codec and container format, you still have some knobs to turn:

Resolution

The number of pixels in each frame of your video. Increasing resolution increases quality, but also increases the size of the file. In our app’s case, high video quality isn’t as important as low latency, so 720p is a fine choice.

Frame Rate

My favorite “fun fact” from my research had to do with frame rate.

I used to think the higher the frame rate, the better the experience for the viewer.

That’s not true.

For example, feature films are almost always displayed at 24 fps, since it gives them that cinematic feel. People have experimented with higher frame rate films and it has this “uncanny valley” effect of looking so real that it feels fake.

If you watched The Hobbit in cinemas, you know what I mean.

More frames mean bigger files. For our use case, 24 fps is just dandy.

Bit Rate

This is a tricky one.

Bitrate is the number of bits used to represent each second of video. For example, 1 mega-bit per second (1Mbps).

When encoding the video, your codec will try to stay below the bitrate you set. It will use few bits to represent simple frames and more bits to represent complex frames. If your bitrate is too low for your resolution and frame rate, you can end up with blurry-looking videos.

However, in many cases, using a low bitrate can achieve much smaller file sizes with imperceptible degradation in video quality.

It’s important to know what kind of videos you’re dealing with. For example, very low bitrates can be used for cartoons, since the color palette is very simple and each frame is mostly the same, lending itself well to interframe compression.

For our use case, we can start with a low bitrate of 1 Mbps and see what kind of quality that leaves us with.

3. Consider Other Optimizations

Adaptive Bitrate Streaming

Adaptive Bitrate Streaming is a technique that breaks a video into small segments encoded at multiple quality levels (e.g. 480p, 720p, 1080p, 4k), allowing a player to dynamically switch between these versions based on real-time network conditions. This ensures smooth playback with minimal buffering, providing the best possible quality for each viewer’s situation.

Implementing this yourself is probably a non-starter. There are great services like Cloudflare Stream and AWS Media Service that handle all this for you.

For the MVP of our app, this isn’t necessary.

Client-Side Compression & Decompression

By using another compression algorithm like zstd, we can shrink videos down prior to upload and/or decompress videos after download. This makes more efficient use of a user’s bandwidth.

One caveat here is that everyone’s device is different, and may be running many other computationally-expensive apps in parallel, leading to inconsistent performance across users. There’s also some latency overhead, especially when compressing/decompressing long videos. Simply uploading the file as is may be faster.

For our use case, no added compression is probably fine.

Server-Side Compression

After uploading a file, it can be compressed by the server. This has the benefit of making more efficient use of storage.

Unfortunately, the pros don’t outweigh the cons in most cases. Storage is usually quite cheap anyways, and this adds latency and requires the app developer to pay for the compute to compress/decompress. It’s also another microservice to manage.

For our use case, the juice isn’t worth the squeeze.

—

Hope this was interesting for you! I’m looking forward to sharing more progress on our video-focused social app in coming articles.

Now back to coding…

Quick reminder - If you appreciate my writing, please reply to this email or “add to address book”. These positive signals help my emails land in your inbox.

If you don't want these emails, you can unsubscribe below. If you were sent this email and want more, you can subscribe here.

See you next week — Rayhan