Cardinal Peak

Creating Single Frame Movies

My camera (an Olympus SP-570UZ) allows me to optionally record a four-second audio clip with each photo I take. I haven’t used this feature much, because I typically upload my photos to Flickr, and there’s been no good way to associate the audio with the video. Ideally, I would like an audio player to appear below the photo, but there aren’t really any public audio sharing websites with much longevity. And, in any case, Flickr won’t allow me to embed an audio player in my photo description.

Recently, it occurred to me that since Flickr allows short movies (up to 1:30 long), maybe I could create a single-frame movie with the still picture as the frame and the audio as the sound track. Then the Flickr movie player would serve as the control for the audio, and the audio and the video would stay associated with each other.

I decided to try to use ffmpeg to create the movie, since it seems to be able to do almost anything with video and audio. The command line for ffmpeg is a bit obscure, so this blog post documents about two hours of my time spent getting it to work.

My camera produces 3648×2736 JPEG images, and the audio files are 8 kHz sample rate, mono, 8 bit unsigned PCM samples in WAV file format. I decided my goal would be to create a motion JPEG (MJPEG) encoded AVI file with maximum quality.

I started by searching the web to see if anyone had done this before. By studying those examples and experimenting, I came up with the following ffmpeg command line:

ffmpeg.exe -loop_input -shortest -f image2 -r 0.25 -i P910033.jpg -i P910033.wav -vcodec mjpeg -qscale 1 -t 4 foo.avi

Most of my attempts caused ffmpeg to hang. But eventually, I got the error message below:

Duration: 00:00:04.00, start: 0.000000, bitrate: N/A

Stream #0.0: Video: mjpeg, yuvj422p, 3648x2736, 0.25 tbr, 0.25 tbn, 0.25 tbc

[wav @ 01a80050]Estimating duration from bitrate, this may be inaccurate

Input #1, wav, from 'P6060033.wav':

Duration: 00:00:04.02, bitrate: 64 kb/s

Stream #1.0: Audio: pcm_u8, 8000 Hz, 1 channels, u8, 64 kb/s

[mp2 @ 01ac6310]Sampling rate 8000 is not allowed in mp2

Output #0, avi, to 'foo.avi':

Stream #0.0: Video: mjpeg, yuvj422p, 3648x2736, q=2-31, 200 kb/s, 90k tbn, 0

.25 tbc

Stream #0.1: Audio: mp2, 8000 Hz, 1 channels, s16, 64 kb/s

Stream mapping:

Stream #0.0 -> #0.0

Stream #1.0 -> #0.1

Error while opening encoder for output stream #0.1 - maybe incorrect parameters such as bit_rate, rate, width or height

At last I understood the problem: ffmpeg needs the audio sampled at some rate other than 8 kHz. So I decided to use Audacity, another open source application, to upsample the sound. However, now Audacity was unhappy with this audio format.

So I used Project->Import Raw Data, and selected my WAV file. I set up the import with the following parameters:

I knew this would work, because the WAV file format consists of a header, followed by PCM data, in this case 8 kHz unsigned samples. So the result in the audio editor would be an audio file with the WAV header as a noisy sound at the start, followed by the data I wanted. The selected (darker) portion of the WAV file below is the header. I used Edit->Cut to remove it.

Finally, I tried to save the audio at a different sample rate. The audio file has a pulldown menu that lets you change the sample rate, but it doesn’t do what I wanted—what it does is play the audio file back at a different rate with aliasing.

Instead, after consulting the Audacity documentation, I discovered you use the menu in at the lower left corner of the main Audacity window to set the sample rate.

Change this to 48000, and choose File->Export as WAV to save at the new sample rate. I re-ran ffmpeg, and the resulting AVI file would play in QuickTime and VLC player (although VLC crashes afterwards), but it would not work in Windows Media Player (audio played, no video), divx, realplayer, or Flickr. So, I decided to try encoding to mp4 instead with the following command:

ffmpeg.exe -loop_input -shortest -f image2 -r 0.25 -i P910033.jpg -i P910033.wav bar.mp4

The resulting mp4 file plays in all the media players (although, again, VLC crashes after playing it), and Flickr can read it successfully as well. Here is what it looks like on Flickr:

Using size as a proxy for quality, however, the encoded video is much smaller than the input JPEG file. Can someone suggest additional flags to ffmpeg to improve the encoding quality?

Ben Mesander has more than 18 years of experience leading software development teams and implementing software. His strengths include Linux, C, C++, numerical methods, control systems and digital signal processing. His experience includes embedded software, scientific software and enterprise software development environments.

Uploading Kodak Zi8 Videos to Flickr

Recently my mom bought me a Kodak Zi8 pocket HD video camera for my birthday. Thanks, Mom! You know what an engineer likes! I love photography, and I upload my photos to the Flickr photo sharing site. But I think my mom wanted some more home movies of my daughter.

The first day I had it, I took some videos, and went to share them on Flickr. Unfortunately, the videos did not properly transcode to the Flickr Flash-based format, and Flickr displayed the following:

I checked out Flickr’s video uploading FAQ and discovered that the Kodak Zi6 and Zi8 are the only two unsupported cameras of all the video cameras in the universe. Great, so I have this fun new camera, and it won’t work with my favorite sharing site. The Flickr staff hasn’t come up with a solution in over a year. But I don’t give up easy.

I talked to my partner Howdy, and thought about the possible things that might cause problems with Flickr’s transcoding process. The Zi8 produces MOV files, and within the MOV file there is an audio stream and a video stream, each encoded with some codec. The problem could lie with the outer system level MOV container, or one or both of the codecs. I decided to look inside the files and see what codecs were in use. Howdy suggested using tools from mpeg4ip for this, but I decided to be lazy and use the VLC media player, as I already had it installed on my computer.

I opened up one of the movies recorded with the camera, and went to the Tools->Codec Information window in VLC. This showed me that the video codec was avc1 and the audio was mp4a.

Now, “avc1” is just h.264 by another name, and “mp4a” is AAC—standard MPEG-4 audio. Nothing looks too unusual. Well, MOV files are an Apple pseudo-standard that has evolved over time, so maybe the transcoder used by Flickr doesn’t like something about the MOV files produced by the Zi8.

Fortunately, I know I can convert from a MOV format container to a MPEG-4 container without transcoding either the video or the audio. Transcoding normally involves decoding and then encoding, so you want to avoid it for two reasons: One, it introduces an additional generation of lossy compression, so quality will suffer. And two, it’s computationally intensive, so therefore slow.

At the system layer, the MPEG-4 file format, formally specified in ISO/IEC 14496-14, is based on work originally done at Apple. As a result, the MOV and MP4 formats are very similar.

I decided to try ffmpeg, the open-source Swiss Army knife of video conversions. We have used ffmpeg on a number of different projects here at Cardinal Peak, and we’ve found it to be very reliable. It is available for Windows, Mac OS/X, and Linux. Because of its flexibility and its command line interface, it can be a little tricky to use, but I figured out after some experimentation that I could change the MOV file to an MP4 file with the following command:

ffmpeg -i input.MOV -f mp4 -vcodec copy -acodec copy output.mp4

In this command line, input.MOV is the file from the Zi8 or Zi6 (e.g., 115_0164.MOV), and output.mp4 is the name of the output file. The -f mp4 tells ffmpeg to convert the input to mp4 format, and the -acodec copy and -vcodec copy tell it to copy the audio and video data from the MOV container to the mp4 container without transcoding it.

The command executes quickly, because the audio and video formats are not changed. I tried uploading the resulting .mp4 files to Flickr, and they work fine! I’m happy because I can use my new video camera with my favorite sharing site, and hopefully help others out.

UPDATE 11/12/2009: It looks like Flickr might now be accepting the raw MOV files from the Zi8 directly. They have removed a previous caveat from their video uploading FAQ linked above. Please leave a comment if you have more information.

Ben Mesander has more than 18 years of experience leading software development teams and implementing software. His strengths include Linux, C, C++, numerical methods, control systems and digital signal processing. His experience includes embedded software, scientific software and enterprise software development environments.