Preserving quality of mini DV SD cassettes for long term storage (camcorder vs dedicated player)

[spellbinder2050]

I have some mini DV cassettes from a camcorder I had in 2004-2006.

The camcorder I used was the Samsung SCD103 (which broke years ago):

https/www.amazon.com/Samsung-Digital-Camcorder-Discontinued-Manufacturer/dp/B0001DRRUM

And the cassettes were Sony DVM60's.

I want to convert them into a digital file using a firewire cable. I'm not too familiar with the mini dv format, so before I start:

1. The tapes themselves were recorded in SP mode, what resolution would this be considered? Would that even matter once they're converted into a digital format or would I still aim for a specific resolution?

2. What would yield the better result, re-buying the same camcorder again, buying a higher quality camcorder, or buying a dedicated player, all of which would be connected via firewire?

Expanding on that, what difference would a higher quality camera or dedicated player make if any?

3. In order to yield a maximum result the first time, should I clean the tapes and/or the camcorder (or player) before I stream copy the tapes to a file? How would I go about doing that?

4. Is there any other information I should know about to get the best result possible?

 

[molletts]

1) The data on the tapes will be in standard-definition DV format which is 720x480 in NTSC countries and 720x576 in PAL/SECAM. It's likely to be interlaced, although some camcorders were capable of recording progressive video. The audio will normally be 16-bit 48kHz stereo PCM (although other sample rates are permissible). When you read them in, you will most likely end up with either a raw DV file or an AVI file containing DV data, at 25Mbps for the video stream. The transfer will be a lossless process, although the DV format is itself lossy - it's a bit like a more advanced version of Motion JPEG (in other words, the file should contain exactly what is on the tape with no further quality loss).

2) In theory, it should make no difference to quality what device is used to read the tapes (as long as they are still in good condition) - they are pure digital media so the data returned should be identical to what was originally recorded (just like burning a CD/DVD on one PC then reading it on another). The only problem that could conceivably occur would be if the original recording device had gone "out of alignment" in some way (head azimuth, etc.) or the recordings have "faded" or become damaged in some other way over time in which case different playback devices may cope better or worse with them - this would vary from being unable to read them at all to returning occasional corrupt frames.

3a) Cleaning the tapes is generally not something you should attempt unless you find serious problems with them and certainly isn't a "standard procedure". I have occasionally cleaned very old tapes (¼" audio reels from the 1960s) using a jury-rigged tape-cleaner prior to transferring them to CDs or files. When transferring Betamax or VHS tapes, I've often found that simply playing the tape through cleans it sufficiently so if I get an unacceptable level of static on the first try, a second go is normally better.

3b) If the replay device is second-hand, you probably should clean it but how you go about this depends on how confident you are fiddling with precision micro-engineering! The simplest and safest way is to use an off-the-shelf MiniDV cleaning cassette if you can get hold of one. Personally, I would strip the machine down and carefully clean all the parts using high-purity isopropanol on a soft lint-free cleaning implement. If you attempt any kind of manual cleaning, beware that the video heads in particular are extremely delicate.

4) I'd suggest defragmenting your hard disk before beginning capture; avoid doing anything that makes it or your PC too busy during capture or you may end up dropping frames (it is a time-critical process - the camcorder will send each frame once, whether or not the PC is ready to receive it). Expect to get about 12GB of data per hour of video.

 

[spellbinder2050]

I wasn't aware that these camcorders recorded digital information to the tapes. It says digital directly on the packaging online. I'm assuming that means these camcorders have small processors in them. Would the age of one of these camcorders make it possible for it to stutter?

1)The only problem that could conceivably occur would be if the original recording device had gone "out of alignment" in some way (head azimuth, etc.) or the recordings have "faded" or become damaged in some other way over time in which case different playback devices may cope better or worse with them - this would vary from being unable to read them at all to returning occasional corrupt frames.

When you say this, are you saying that a problem would be created if the original camcorder had "gone out of alignment" as it was recording, affecting the final information on the tape, or are you saying that this "out of alignment" issue would affect the playback of a previously recorded tape?

If the former, I assume I would have noticed some weird stuff happening when playing the tapes back years ago, which I didn't.
--------------------------------

2. It looks like these mini DV cams are not made anymore. Is there any specific camcorder I should get for my purpose that has "stood the test of time", so to speak, that would be appropriate for my purpose?

3. Would it be better to use a bundled proprietary software, some other software, or what I have available now: Windows Movie Maker.

 

[molletts]

Would the age of one of these camcorders make it possible for it to stutter?

Only in as much as the mechanism and playback heads may be worn which may affect its ability to read the tape reliably. On the electronics side, most or all of the datastream handling would be done by dedicated chips rather than a general-purpose CPU (although there will be at least one CPU coordinating things, turning motors on and off, managing the display, reading user input from the various buttons, etc.)

1) ... if the original recording device had gone "out of alignment" ... or the recordings have "faded" or become damaged in some other way over time

When you say this, are you saying that a problem would be created if the original camcorder had "gone out of alignment" as it was recording, affecting the final information on the tape, or are you saying that this "out of alignment" issue would affect the playback of a previously recorded tape?

I guess it could be either, really, although I was originally just thinking of the original recording. If the tape was originally recorded on a device which had gone out of alignment then a correctly-aligned machine might struggle to read it but likewise, an out-of-alignment playback machine could struggle to read tapes recorded on a correctly-aligned machine.

That said, I've never had playback issues with MiniDV or Digital8 (Sony's version, which recorded DV-format data on 8mm cassettes) - I guess the technology was sufficiently mature by that stage that, barring mishandling or abnormally heavy usage, the machines are very reliable.

If the former, I assume I would have noticed some weird stuff happening when playing the tapes back years ago, which I didn't.

Maybe, maybe not... A badly-aligned machine will generally play back its own tapes OK (unless they were recorded prior to it going out of alignment).

2. It looks like these mini DV cams are not made anymore. Is there any specific camcorder I should get for my purpose that has "stood the test of time", so to speak, that would be appropriate for my purpose?

I'm not sufficiently familiar with the format to be able to recommend a specific model, but I've always found Panasonic and Canon hardware to be very reliable. I don't know how good Sony MiniDV hardware was - they used their own 8mm-based format for a while (which I found to be very reliable) so I never used a Sony DV camcorder.

3. Would it be better to use a bundled proprietary software, some other software, or what I have available now: Windows Movie Maker.

Windows Movie Maker is pretty rudimentary; I'm not even sure if it can import DV directly. When I used Windows (I'm a Linux guy these days), I used to use VirtualDub (free) to import the DV (and do basic edits, cutting out the bits I didn't want, cropping black bars, etc. - I used a DV capture box to get analogue tapes onto the PC) then use its frameserver mode to feed the result directly into an MPEG encoder so I could use VDub's deinterlacer and other filters without having to save to another file, which would both lose quality by re-compressing the video and take up huge amounts of disk space (those were the days when a 40GB disk was considered to be very big). These days, you could just do the cuts in VDub then use Handbrake (also free) to deinterlace, encode to H.264, etc. If you've got a fast second hard drive, you could export the cut video to a new DV file then encode that rather than using the frameserver - as long as you only cut, you won't lose quality as it can simply copy the frames unchanged.

(----------
A bit of technical background - you can skip this if it's too in-depth!

One advantage of DV over things like MPEG-4 when editing is that, in a DV file, every frame is a so-called Intra frame (I-frame), in other words, an entire standalone picture, which means that you can easily cut the video anywhere you like without having to re-encode sections of it.

Other formats like the MPEG-2 codec used on DVDs store most frames as "what has moved or changed since the last I-frame" (so-called "predicted frames" or P-frames). More advanced versions like H.264 (officially known as MPEG-4 Advanced Video Coding) have "bidirectionally-predicted frames" or B-frames which look ahead to the next I-frame, or even to other P- or B-frames. Cutting the stream at anywhere other than one of the occasional I-frames requires part of it be re-encoded to create a new I-frame and sequence of P/B-frames at the cut. In case you had ever wondered, this is why you sometimes see, for example, a bit of wall moving in the shape of someone's face for a few seconds when a digital video stream gets damaged - the "differences" get applied to the wrong I-frame until a new one comes along.
----------)

For archival purposes, I'd aim to try and store the DV files (either the raw tape dumps or the "trimmed" ones) somewhere - they're big but it will save you having to go back to the tapes again if you want to re-encode them in a newer format (or cut them in a different way) in future. (I don't know how durable BD-R discs are; maybe a big "green"-type hard drive might be the most reliable and cost-effective option.)

Well, that was a bit longer than I originally intended...

Hope I haven't bored you to death!
Stephen

 

[spellbinder2050]

I actually appreciate the detailed response. It's giving me an overview of the information I need. Then I can fill in the gaps using the net. You're a sharp guy.

These days, you could just do the cuts in VDub then use Handbrake (also free) to deinterlace, encode to H.264, etc. If you've got a fast second hard drive, you could export the cut video to a new DV file then encode that rather than using the frameserver - as long as you only cut, you won't lose quality as it can simply copy the frames unchanged.

1. Can you explain what a frameserver is using a minidv camcorder, virtual dub, and the other software you're alluding to as an example?

(----------
A bit of technical background - you can skip this if it's too in-depth!

One advantage of DV over things like MPEG-4 when editing is that, in a DV file, every frame is a so-called Intra frame (I-frame), in other words, an entire standalone picture, which means that you can easily cut the video anywhere you like without having to re-encode sections of it.

Other formats like the MPEG-2 codec used on DVDs store most frames as "what has moved or changed since the last I-frame" (so-called "predicted frames" or P-frames). More advanced versions like H.264 (officially known as MPEG-4 Advanced Video Coding) have "bidirectionally-predicted frames" or B-frames which look ahead to the next I-frame, or even to other P- or B-frames. Cutting the stream at anywhere other than one of the occasional I-frames requires part of it be re-encoded to create a new I-frame and sequence of P/B-frames at the cut. In case you had ever wondered, this is why you sometimes see, for example, a bit of wall moving in the shape of someone's face for a few seconds when a digital video stream gets damaged - the "differences" get applied to the wrong I-frame until a new one comes along.
----------)

For archival purposes, I'd aim to try and store the DV files (either the raw tape dumps or the "trimmed" ones) somewhere - they're big but it will save you having to go back to the tapes again if you want to re-encode them in a newer format (or cut them in a different way) in future. (I don't know how durable BD-R discs are; maybe a big "green"-type hard drive might be the most reliable and cost-effective option.)

I'm planning to take the raw files and burn them directly to some blurays for storage. If they don't last for more than 10-15 years, I will still have the tapes as a backup.

2. I'm assuming the files VirtualDub would create would be the "dv" files you alluded to previously. Is dv considered a codec, file container, or both? Is this an uncompressed "lossless" format? Would deinterlacing a dv file in VirtualDub affect any edits or cuts I make later? Should I even bother to deinterlace?

----------

It's to my understanding now that compression using a codec such as H.264 prevents edits or re-compression without further loss of quality because the file would have to be decoded and re-encoded.

3. Does the term "lossless" compression imply that any compression that has occurred would be Spatial (intraframe) compression, so that further cuts would not affect the quality of the entire video? Is the term "lossless" even used to describe a raw, uncompressed file?
-----------

4. For the I/P/B frames, does this refer to the key (or master) and delta (equivalent to p frames you described) frames? I found the following on the web about key and delta frames:

"Temporal compression relies on the placement of key frames interspersed throughout the frames sequence. The key frames are used as masters against which the following frames (called delta frames) are compared."

https/docstore.mik.ua/orelly/web2/wdesign/ch25_02.htm

 

[molletts]

1. Can you explain what a frameserver is using a minidv camcorder, virtual dub, and the other software you're alluding to as an example?

The frameserver is a very handy feature of VirtualDub. In your workflow, you would first capture the DV stream from the camcorder into a file using VDub's capture mode then apply any cuts/filters you want to it. Instead of saving the post-processed video as a new file, the frameserver allows you to save a kind of "signpost" AVI file. This file doesn't actually contain any audio or video data but the codec descriptors it specifies (the FourCC for the video and the equivalent for the audio) cause the program that opens it (such as Handbrake) to load a "pseudo-codec" which, instead of reading A/V data from the file as a normal codec would, connects to the open copy of VirtualDub and reads the post-processed data directly out of it.

Back when I was doing video stuff with VDub, the frameserver not only meant that I didn't have to find (more) tens of gigabytes of disk space to export the cut, deinterlaced, cropped, denoised video to another file, it also eliminated the additional loss of quality from re-compressing the post-processed video into the "holding" file prior to the final encoding stage. (An added benefit was that both CPUs in my dual-processor system got used - one running the VDub processing pipeline and the other doing the MPEG encoding. There was very little multi-threaded software in those days.)

I sometimes did at least some of the cuts the "old-fashioned" way, at least with DV or MJPEG source material, and archived the cut file, especially if I was just trimming off "cruft" from before and after the "real" recording.

2. I'm assuming the files VirtualDub would create would be the "dv" files you alluded to previously. Is dv considered a codec, file container, or both? Is this an uncompressed "lossless" format? Would deinterlacing a dv file in VirtualDub affect any edits or cuts I make later? Should I even bother to deinterlace?

From memory, I think VirtualDub uses AVI as its container file almost exclusively.

DV is both a codec and a stream format (a type of container). The raw data stored on a DV or D8 tape and returned over the FireWire link is in the stream format, which encapsulates both video (in the DV codec) and audio data. When it is saved in an AVI file, the DV stream is multiplexed unchanged into the video stream of the AVI file (including the audio!) and, optionally, the audio data is copied into the file's audio stream to improve compatibility with older software at the expense of being a bit bigger. (A Type 1 DV AVI has just the combined stream; a Type 2 has both streams.) I'm not sure what type VDub uses offhand - there's probably a setting for it.

3. Does the term "lossless" compression imply that any compression that has occurred would be Spatial (intraframe) compression, so that further cuts would not affect the quality of the entire video? Is the term "lossless" even used to describe a raw, uncompressed file?

"Lossless" refers to a type of codec that decodes to exactly the same as "what went in" so you can compress, decompress and re-compress the data as many times as you like and it will never lose any quality. Loosely, you could describe raw, uncompressed video (or audio) as "lossless" (as long as the sample format doesn't get changed - but that's a whole other topic!) Some lossless video codecs, such as HuffYUV and Lossless MJPEG, do only use spatial intra compression but others, such as H.264 in lossless mode, also use temporal delta compression. On the audio side, the leading lossless codec is probably FLAC.

DV is lossy even though it uses all intra frames. You can cut the video wherever and however many times you like and, as long as the editing program is smart and simply copies the frame data, the quality will be maintained, but as soon as you start to actually re-encode frames (if you apply a filter of some kind, for example), quality will be lost.

Conversely, editing using lossless H.264 would require (partial) re-encoding for cuts that didn't coincide with an I-frame, but no quality would be lost. It would just make the edits take longer. (Normal H.264, of course, would lose quality.)

4. For the I/P/B frames, does this refer to the key (or master) and delta (equivalent to p frames you described) frames?

In the MPEG family of codecs, I-frames are the key frames. P- and B-frames are delta frames. They include both motion vectors describing how parts of the image have moved relative to previous (and sometimes future) frames and "fresh" areas of image (compressed in the same way as an I-frame) when the encoder is unable to represent the changes to them using motion vectors.

My brain hurts! [searches the smiley list for a "mushroom cloud coming out of head" smiley] 8=:-o
Stephen

 

[kenrivers]

What is the maximum you are willing to spend for a used mini-DV camcorder? I only ask so I can help in the search. Sony mini-DV camcorders are as reliable as the older 8mm Sony's. I still own one of each (mini-DV and 8mm). Let me know if I can help.

 

[spellbinder2050]

The frameserver is a very handy feature of VirtualDub. In your workflow, you would first capture the DV stream from the camcorder into a file using VDub's capture mode then apply any cuts/filters you want to it. Instead of saving the post-processed video as a new file, the frameserver allows you to save a kind of "signpost" AVI file. This file doesn't actually contain any audio or video data but the codec descriptors it specifies (the FourCC for the video and the equivalent for the audio) cause the program that opens it (such as Handbrake) to load a "pseudo-codec" which, instead of reading A/V data from the file as a normal codec would, connects to the open copy of VirtualDub and reads the post-processed data directly out of it.

1. Does VirtualDub's frameserver actually save a full sized file that simply hasn't been fully encoded yet or does it save a smaller file with codec information, leaving the video file itself still stored in system memory in VirtualDub?

The raw data stored on a DV or D8 tape and returned over the FireWire link is in the stream format, which encapsulates both video (in the DV codec) and audio data. When it is saved in an AVI file, the DV stream is multiplexed unchanged into the video stream of the AVI file (including the audio!) and, optionally, the audio data is copied into the file's audio stream to improve compatibility with older software at the expense of being a bit bigger. (A Type 1 DV AVI has just the combined stream; a Type 2 has both streams.) I'm not sure what type VDub uses offhand - there's probably a setting for it.

2. To be clear, you're saying that "multiplexing" would combine the audio and video information on the file, but it's also possible to separate them into their own "streams" on the file in order to increase compatibility, but at the cost of increased file size?

3. Which would be better for long term storage? How would each scenario affect another encoding later on?

"Lossless" refers to a type of codec that decodes to exactly the same as "what went in" so you can compress, decompress and re-compress the data as many times as you like and it will never lose any quality. Loosely, you could describe raw, uncompressed video (or audio) as "lossless" (as long as the sample format doesn't get changed - but that's a whole other topic!) Some lossless video codecs, such as HuffYUV and Lossless MJPEG, do only use spatial intra compression but others, such as H.264 in lossless mode, also use temporal delta compression.

4. For "lossless" H.264, does this mean the the file is just H.264 encoded at a specifically high bitrate, or is there a literal "H.264 lossless" codec?

DV is lossy even though it uses all intra frames. You can cut the video wherever and however many times you like and, as long as the editing program is smart and simply copies the frame data, the quality will be maintained, but as soon as you start to actually re-encode frames (if you apply a filter of some kind, for example), quality will be lost.

5. Will the camcorder stream always be accepted in the DV format regardless of the software used? If so, would this mean that any cropping and deinterlacing would reduce the quality to some degree, or does cropping and deinterlacing not count as re-encoding frames as you described here?

Conversely, editing using lossless H.264 would require (partial) re-encoding for cuts that didn't coincide with an I-frame, but no quality would be lost. It would just make the edits take longer. (Normal H.264, of course, would lose quality.)

To be specific about my needs,

6. would there be any benefit to encoding in H.264 lossless over DV for my purpose? Which format do you believe has the better long term prospect as far as playback is concerned?

7. If H.264 lossless is preferred, would the method you described using VirtualDub with Handbrake be the best option for achieving my goal, or is there a single software option that would be better suited? Would different software simply have more features and fuctionality, but still produce the same end result. For example, if bitrate/codec were equivalent, would the output file basically be the same?

I'm not concerned with file size or compatibility with anything going further back than Windows 7, but I wouldn't mind if the video files were viewable on a Mac/iPhone.

----------

I greatly appreciate your help Stephen. I promise I won't bother you too much longer.

 

[spellbinder2050]

What is the maximum you are willing to spend for a used mini-DV camcorder? I only ask so I can help in the search. Sony mini-DV camcorders are as reliable as the older 8mm Sony's. I still own one of each (mini-DV and 8mm). Let me know if I can help.

Since the quality won't change with different camcorders, I'm thinking of just buying the same one over again, the Samsung SCD103. If you believe there could be a more reliable option, however, I'll definitely check it out. Firewire is necessary and any price below 200 is fine. I plan to resell the camera anyway.

 

[kenrivers]

Never having used a tape based camcorder other than a Sony I can't with any degree of accuracy say one manufacturer's camcorder is better than another manufacturer's camcorder or more reliable. I'll let you know if I find anything.

 

[molletts]

1. Does VirtualDub's frameserver actually save a full sized file that simply hasn't been fully encoded yet or does it save a smaller file with codec information, leaving the video file itself still stored in system memory in VirtualDub?

It saves a small file that simply points to the instance of VirtualDub that is running the frameserver. The video is processed as a stream - it is read on demand from the original file and any cuts and filtering are applied "on the fly" so there will normally only be one or a few frames in system memory at any one time.

2. To be clear, you're saying that "multiplexing" would combine the audio and video information on the file, but it's also possible to separate them into their own "streams" on the file in order to increase compatibility, but at the cost of increased file size?

Multiplexing is the process of putting several different data streams into one file. A simple video file will typically contain one video stream and one audio stream but, depending on the container format used, it's possible to have multiple video streams (e.g. different camera angles), multiple audio streams (e.g. different languages and/or a choice of stereo or 5.1 surround) and other data such as subtitles.

The DV stream is technically already a multiplex containing a video stream (using the DV codec) and an audio stream. Because of the way DV is specified, it isn't possible (as far as I am aware) to split the video and audio streams and store them separately in the corresponding parts of an AVI file. So a Type 1 DV AVI has no audio stream, according to the AVI headers - the audio is embedded in its video stream. This confuses some software (mainly older programs using the Video for Windows APIs, I think) so the Type 2 has both the embedded audio inside the video stream and a second copy, saved as a normal AVI audio stream. (Theoretically, you could put different audio in the second stream but that would get confusing! Possibly a way of pranking someone you know uses software that favours the secondary stream, when you use software that gives priority to the embedded one...)

3. Which would be better for long term storage? How would each scenario affect another encoding later on?

My personal preference is for Type 1 (no separate audio stream). I view Type 2 as a nasty kludge - the second copy of the audio stream is redundant and just makes the file bigger. All modern video software should be able to handle a Type 1 DV AVI. You could simply use raw DV files but stuffing it into an AVI improves compatibility without inflating the file size much. (Not all software will know what to do with a .dv file as it doesn't contain any information about what codecs are in use whereas the same DV stream in a .avi file has a header that says, "Use the DVSD codec to decode this".)

4. For "lossless" H.264, does this mean the the file is just H.264 encoded at a specifically high bitrate, or is there a literal "H.264 lossless" codec?

There is a specific lossless mode in H.264. If you are using an x264-based encoder (such as Handbrake), this mode is enabled by selecting Constant Rate Factor encoding (which, oddly, gives a variable bit rate in exchange for constant quality) with a rate factor of 0. (In Handbrake, select Constant Quality and drag the RF slider to 0, all the way to the right. Best compression will be achieved by selecting the "veryslow" preset. It will still be bigger than the original DV file, though. I just ran a 587MB .dv file through Handbrake with lossless H.264 selected, with "interlaced" specified as an extra option, and got a 767MB .m4v file out! It took ages, too - eight cores at 100% for nearly 10 minutes for a video clip lasting only 02:50. It was originally a capture from analogue video tape, though, so it was nowhere near as "clean" as a digital source would have been.)

5. Will the camcorder stream always be accepted in the DV format regardless of the software used? If so, would this mean that any cropping and deinterlacing would reduce the quality to some degree, or does cropping and deinterlacing not count as re-encoding frames as you described here?

Cropping and deinterlacing will result in frames being re-encoded (I'm not even sure if you can save non-standard cropped frame sizes in DV) because they change the content of the frames. Only a pure cut will result in the frames being copied unchanged (as long as the software is smart enough).

6. would there be any benefit to encoding in H.264 lossless over DV for my purpose? Which format do you believe has the better long term prospect as far as playback is concerned?

I think DV would be the best long-term archival format in this case - it's quick and easy to edit and you won't save any space by encoding to Lossless H.264 (as my test showed). DV is well-supported by open-source codecs and I don't see it disappearing any time soon. For archival purposes, I always try to stay as close to the original as possible, just cutting the sections of video to length and leaving all the post-processing for later. If a new, improved deinterlacer or a fancy new codec becomes available, I can go back to the archive and redo the "final cut". I've done this recently with some videos I originally deinterlaced with a simple linear blend and encoded to MPEG-4 ASP ("DivX") about 15 years ago, using a motion-compensating deinterlacer and H.264 for a smaller file that looks much better.

(The only time I've ever cropped the archive copies was on a few occasions in the early 2000s when I tried using raw uncompressed video capture. I then had to compress the archive copies anyway using high-bitrate MJPEG because the originals were simply too huge - over 20MBytes/sec or a full 40GB hard drive in half an hour. The quality difference was negligible so I went back to hardware MJPEG capture, then switched to DV a few years later.)

7. If H.264 lossless is preferred, would the method you described using VirtualDub with Handbrake be the best option for achieving my goal, or is there a single software option that would be better suited? Would different software simply have more features and fuctionality, but still produce the same end result. For example, if bitrate/codec were equivalent, would the output file basically be the same?

Whatever workflow you decide on, VirtualDub and Handbrake will be a very bare-bones "editing suite". If you're after more features (non-linear editing, effects, transitions, etc.), you'd really need to look at "proper" video editing packages, such as Vegas or Premiere (both have inexpensive "lite" versions which may well be good enough). Cinelerra is industrial-strength and free but hellishly hard to use; there are other free packages in various states of completeness and stability. There is a free version of Lightworks but it's got a steep learning curve, largely because of its heavyweight "pro" credentials (the full version has been used for editing major movies).

As for the output, quality-wise it would be the same if you used a lossless codec (by definition) but the size may vary depending on the encoder used. The open-source x264 encoder is generally regarded as one of the best H.264 implementations - in "normal" lossy mode, it's both very fast and produces very good quality.

I'm not concerned with file size or compatibility with anything going further back than Windows 7, but I wouldn't mind if the video files were viewable on a Mac/iPhone.

I'd probably aim to have both the archived "master copies" for long-term storage and "daily use" copies in a "sensible" format (.m4v files with normal lossy H.264 video and AAC audio, for example) which have been deinterlaced and had any other tweaks applied. Apart from anything else, you'll need to scale them a bit to correct the aspect ratio - DV doesn't have square pixels. It's always either 720x480 or 720x576 regardless of whether it's 4:3 or 16:9 aspect. (It is possible to embed an aspect ratio attribute in an MPEG stream but my experience is that players ignore it as often as not except when the stream is on a DVD.)

The iPhone especially has a reputation for being a bit picky about what it will play back - and the file size will likely be a showstopper if you use the DV originals, assuming it can even play them! (Quite apart from the problem of viewing interlaced video on a progressive screen.)

I greatly appreciate your help Stephen. I promise I won't bother you too much longer.

I was just thinking I ought to say that I'll be basically unavailable after the end of this week as I'm about to start a very intensive training course. (I would call it a crash course but it's aviation-related so that might not be appropriate!)

(For someone who hated writing essays at school, I certainly write a lot these days... I guess it's because it's something I'm interested in - get me started on one of my interests and I could write (or talk) for England!)

 

[spellbinder2050]

It saves a small file that simply points to the instance of VirtualDub that is running the frameserver. The video is processed as a stream - it is read on demand from the original file and any cuts and filtering are applied "on the fly" so there will normally only be one or a few frames in system memory at any one time."

1. When you say "original file", you're saying a large file that VirtualDub creates, which is secondary to the pointer file? The editing of this original file occurs while VirtualDub is still open or is it closed first?

I just ran a 587MB .dv file through Handbrake with lossless H.264 selected, with "interlaced" specified as an extra option, and got a 767MB .m4v file out! It took ages, too - eight cores at 100% for nearly 10 minutes for a video clip lasting only 02:50. It was originally a capture from analogue video tape, though, so it was nowhere near as "clean" as a digital source would have been.)

2. So is there any practical purposes for H.264 lossless?

3. Also, just out of curiosity, are these older analog and digital DV tapes stored as interlaced or full frame video? Also, did applying interlaced in your video edit have a direct affect on file size? If so, why? Lastly, what do you mean by not "clean" when describing your analog tape?

Cropping and deinterlacing will result in frames being re-encoded (I'm not even sure if you can save non-standard cropped frame sizes in DV) because they change the content of the frames. Only a pure cut will result in the frames being copied unchanged (as long as the software is smart enough).

4. When you say "pure cut" I'm assuming you mean simply spliting the video? If so, would VirtualDub fall in line with being "smart enough" to not affec the quality of my split videos?

I think DV would be the best long-term archival format in this case - it's quick and easy to edit and you won't save any space by encoding to Lossless H.264 (as my test showed). DV is well-supported by open-source codecs and I don't see it disappearing any time soon. For archival purposes, I always try to stay as close to the original as possible, just cutting the sections of video to length and leaving all the post-processing for later.

5. So again, just VirtualDub would be good enough for the archival aspect of my goal?

If a new, improved deinterlacer or a fancy new codec becomes available, I can go back to the archive and redo the "final cut". I've done this recently with some videos I originally deinterlaced with a simple linear blend and encoded to MPEG-4 ASP ("DivX") about 15 years ago, using a motion-compensating deinterlacer and H.264 for a smaller file that looks much better.

6. Care to share what this "motion-compensating deinterlacer" was and what software you used to apply it. =] What is it best suited for?

Whatever workflow you decide on, VirtualDub and Handbrake will be a very bare-bones "editing suite". If you're after more features (non-linear editing, effects, transitions, etc.), you'd really need to look at "proper" video editing packages, such as Vegas or Premiere (both have inexpensive "lite" versions which may well be good enough). Cinelerra is industrial-strength and free but hellishly hard to use; there are other free packages in various states of completeness and stability. There is a free version of Lightworks but it's got a steep learning curve, largely because of its heavyweight "pro" credentials (the full version has been used for editing major movies).

As for the output, quality-wise it would be the same if you used a lossless codec (by definition) but the size may vary depending on the encoder used. The open-source x264 encoder is generally regarded as one of the best H.264 implementations - in "normal" lossy mode, it's both very fast and produces very good quality.

7. Just to understand these technologies a little better, these codecs are a type of software library/method, but are coded in these software suites differently? What part of the codec internally is standardized if different software suites produce slightly different results with equivalent settings?

The open-source x264 encoder is generally regarded as one of the best H.264 implementations - in "normal" lossy mode, it's both very fast and produces very good quality.

8. I'm assuming you're talking about this: https/www.videolan.org/developers/x264.html

What difference would there be between using this and Handbrake?

I'd probably aim to have both the archived "master copies" for long-term storage and "daily use" copies in a "sensible" format (.m4v files with normal lossy H.264 video and AAC audio, for example) which have been deinterlaced and had any other tweaks applied. Apart from anything else, you'll need to scale them a bit to correct the aspect ratio - DV doesn't have square pixels. It's always either 720x480 or 720x576 regardless of whether it's 4:3 or 16:9 aspect. (It is possible to embed an aspect ratio attribute in an MPEG stream but my experience is that players ignore it as often as not except when the stream is on a DVD.)

9. What do you mean by "scaling" them? What does this have to do with the aspect ratio? What is the fix for dealing with non-square pixels when editing a video in Handbrake and x264?

 

[molletts]

1. When you say "original file", you're saying a large file that VirtualDub creates, which is secondary to the pointer file? The editing of this original file occurs while VirtualDub is still open or is it closed first?

By "original file", I'm referring to the original, unedited DV capture from the camcorder (or, if you choose to "top and tail" it first, a version of it that has been cut but had no other edits applied). VirtualDub stays open to run the frameserver and reads frames in from this file as required, applying any filters, etc. on the fly before passing the frames on to whatever application has opened the frameserver "signpost" file.

I just ran a 587MB .dv file through Handbrake with lossless H.264 selected, with "interlaced" specified as an extra option, and got a 767MB .m4v file out! ...

2. So is there any practical purposes for H.264 lossless?

It could be used to compress raw video as it is captured in order to tame the bitrate without sacrificing any quality -
that would probably require a hardware codec, although I haven't tested the faster x264 presets which may be able to encode in real time, albeit with a lower compression ratio. It could also be useful for storing intermediate results of editing (or a "master copy" of the final cut of something) without losing any quality as you would with a conventional lossy codec.

It got bigger in this case because I was starting with a file that was already encoded using a lossy codec. If I'd started with 2 minutes 50 of raw video at 720x576@25fps using the same sample format as DV, it would have come to something like 3.4GB. A lossless reduction to 767MB would look quite good then!

3. Also, just out of curiosity, are these older analog and digital DV tapes stored as interlaced or full frame video? Also, did applying interlaced in your video edit have a direct affect on file size? If so, why? Lastly, what do you mean by not "clean" when describing your analog tape?

Consumer analogue and DV tapes always store the video in interlaced form - although the video frames themselves could be non-interlaced, they would still be stored as two separate fields, one after the other.

Setting the "interlaced" flag for x264 told the codec to process the incoming frames (which were interlaced) as two separate fields. I didn't test it but I would expect the resultant file to be much larger if the codec had tried to process them as progressive frames - it would have spent a lot of bits encoding the "combing" effect down the edges of moving objects that characterises interlaced video when viewed on a progressive display. If the codec had been in "normal" lossy mode, the outcome without the "interlaced" flag would either have been a larger file (if the codec was running in constant-quality mode) or reduced quality (in constant-bitrate mode).

By "clean", I mean free from video noise ("fuzz"). Digital sources tend to produce clean video, except when the camera is struggling in low-light conditions, when the noise in the electronics gets amplified alongside the weak signals from the sensor and becomes more visible. Analogue sources are more prone to noise as it can creep in at any stage of the signal path - and the tape itself introduces noise: think back to the "hiss" that could be heard on audio cassettes, especially the cheaper Type I (ferric) ones.

4. When you say "pure cut" I'm assuming you mean simply spliting the video? If so, would VirtualDub fall in line with being "smart enough" to not affec the quality of my split videos?

Yes, just splitting the video at particular frame(s). With an all-key-frames codec like DV, it's just a matter of copying the right chunk of data from the input file to the output file. (Actually, that's not quite true - normally, the editor will also update the timestamp on each frame so that the output video stream's timestamps don't suddenly jump from 00:03:45 to 00:04:56 or whatever but that doesn't affect the content of the frame so it doesn't require any re-encoding.)

VirtualDub is definitely smart enough. It'll stream the video from the input file to the output one, skipping the bits that have been cut out, as fast as the source hard drive can deliver data and the destination one can take it. (If you're doing it on a single drive, it'll be much slower as the drive will have to keep seeking back and forth between the input file and the output one. If you've got a second drive, either internally or on USB, I'd recommend having the source file on one drive and exporting the cut version to the other.)

5. So again, just VirtualDub would be good enough for the archival aspect of my goal?

It'll certainly take care of that. It's also very simple to use - a more fully-featured program would be overkill (and it might be hard to ensure that it doesn't do any kind of re-encoding behind your back).

6. Care to share what this "motion-compensating deinterlacer" was and what software you used to apply it. =] What is it best suited for?

The most basic non-motion-compensating deinterlacers synthesise progressive frames either by discarding one field and filling in the gaps that are left by looking at the lines above and below them (which results in a loss of resolution) or by blending the two fields together (which still reduces the resolution, but it looks less severe, and results in blurring around moving objects - less ugly than "combing" but still not nice).

A motion-compensating deinterlacer looks for things that have actually moved from one field to the next (remember that the fields are not just "the odd lines" and "the even lines" from one frame - in 30 frames/sec video, the second field is "taken" by the camera 1/60 of a second later than the first) and, usually, attempts to reverse that motion to reconstruct the missing lines from the first field. Areas that don't appear to be moving can be left alone.

I used the QTGMC deinterlacer for AVISynth; you'll need to get your hands really dirty if you want to use AVISynth, though! It's also quite CPU-hungry.

Handbrake offers the Yadif deinterlacer, which is fast and gives very good results - see this page for some comparison screenshots including examples of discarding one field and blending the fields. I'm not convinced that I really gained very much from using a super-fancy deinterlacer (apart from a warmer room while it was running) - I doubt I'd spot the difference between Yadif and QTGMC without pausing the playback and looking closely at the screen! I'd suggest using a custom Yadif preset in Handbrake "mode=0" which does temporal and spatial checks to see which parts of the frame need deinterlacing then uses various interpolation methods to fill them in. I think VDub may have Yadif built in these days; there's also a Smart Deinterlace filter which is supposed to be good too.

7. Just to understand these technologies a little better, these codecs are a type of software library/method, but are coded in these software suites differently? What part of the codec internally is standardized if different software suites produce slightly different results with equivalent settings?

The different H.264 codecs are different programmers' implementations of the H.264 standard. The standard defines how the video stream must be represented in order that any compliant decoder can play it back - how the I-frames, motion vectors, etc. are to be stored and what they mean. It is left for the codec writers to decide how they actually go about converting a video stream into this form - and, indeed, how they convert the H.264 stream back into displayable video in the decoder. There are various "profiles" and "levels" which define exactly which features can be used when encoding, in order to ensure that the result is playable on a player which supports that particular profile and level (be that an app on a phone, a player program on a PC or a hardware device such as a Blu-Ray player).

In theory, one could write a compliant H.264 encoder which simply encoded every frame as an I-frame. It would be very fast but would give a very large output file and/or very poor quality. (Some early software MPEG encoders did exactly this - it would have taken a very long time to do full motion-estimation on an early-1990s-era PC. I didn't do video stuff in those days but I remember leaving the PC on for a couple of days once to rip a CD to MP3 format so I could give a pretty-rubbish-quality copy to a school friend on about 40 floppy disks!)

Some encoders make heavy use of the general-purpose compute capabilities of graphics cards; others are optimised for specific tasks such as low-delay real-time encoding or super-low bitrates.

Many modern graphics cards have dedicated hardware to assist with encoding and decoding common video formats. If you want super-fast encoding, this is the way to go but the quality isn't always as good as a good software encoder. (Likewise for decoding although the GPU can be useful to help with any post-processing that is necessary. nVidia graphics cards have a truly magnificent motion-adaptive temporal-spatial deinterlacer - if only I could use it while encoding video!)

The open-source x264 encoder is generally regarded as one of the best H.264 implementations - in "normal" lossy mode, it's both very fast and produces very good quality.

8. I'm assuming you're talking about this: https/www.videolan.org/developers/x264.html

What difference would there be between using this and Handbrake?

Yes, that's the one.

Handbrake uses x264 internally so there's not really any reason to use the standalone encoder instead. (Many other programs use it too - it's a popular choice because of its blend of speed and quality.)

... you'll need to scale them a bit to correct the aspect ratio - DV doesn't have square pixels. It's always either 720x480 or 720x576 regardless of whether it's 4:3 or 16:9 aspect.

9. What do you mean by "scaling" them? What does this have to do with the aspect ratio? What is the fix for dealing with non-square pixels when editing a video in Handbrake and x264?

By scaling, I mean resizing the video. Computer displays normally have square pixels and most playback software assumes that video streams being played back also have square pixels. This results in videos with non-square pixels being displayed with the wrong aspect ratio. It's not necessary to scale (or indeed deinterlace) DV when converting it to DVD, though, because DVD uses the same resolution and can store interlaced video (but make sure you use the MPEG2 encoder in interlaced mode, for the same reason as I passed the "interlaced" flag to x264 in my lossless test encode).

For example, a 4:3 NTSC DV stream with a resolution of 720x480 will appear to be slightly too wide. To get the correct shape, it will be necessary to either stretch it vertically, to 720x540 (see note below), or shrink it horizontally to 640x480. (Other 4:3 sizes would work too but I'm only changing one dimension for the purposes of the example.)

Conversely, a 16:9 NTSC DV stream will look too narrow because it'll still be in 720x480 resolution. It can either be stretched horizontally to 853x480 (see note below) or shrunk vertically to 720x405 (likewise). (Again, other 16:9 sizes would work fine.)

Note: When encoding to most modern formats, it can be beneficial to compromise a little on the exact dimensions for scaling. Most codecs divide the image into a grid of larger "macroblocks" which they then process. The block sizes can vary but they're often 8x8 or 16x16. H.264 uses 16x16 for its macroblocks although it also uses other smaller sizes for other purposes. It can improve encoding efficiency to have the width and height equal to multiples of the block size so it may be necessary to have a very-slightly-wrong aspect ratio (or possibly crop a few pixels off, or both) to achieve this.

720 ÷ 16 = 45 so this is fine.
540 ÷ 16 = 33.75 so 544 (34 × 16) would be better and isn't too far out.
853 ÷ 16 = 53.3125 so you'd have to use 848 (53 × 16) instead.
405 ÷ 16 = 25.3125 so 400 (25 × 16) would be better.

Bear in mind that scaling should always be done after deinterlacing if you are changing the vertical size (I wouldn't even try it using an interlaced scaler - it'll probably just confuse the deinterlacer). When increasing the horizontal size, there could theoretically be some benefit to deinterlacing after scaling, depending on the deinterlacer being used. I normally try to avoid upscaling if possible anyway - it just makes the encoder work harder to encode information that isn't really there anyway.

Looking at the Handbrake user interface, it appears that they seem to favour simply setting the aspect ratio attributes in the output file. Maybe players are better-behaved now than when I last did a multi-platform survey of them. I'll have to try and find the time to do a fresh survey.

It might be worth experimenting to see if you get playback in the right aspect ratio on all the devices that matter to you. Quality-wise, no scaling is preferable if playback works fine.

I need to charge my brain now!

Stephen

 

[spellbinder2050]

I just ran a 587MB .dv file through Handbrake with lossless H.264 selected, with "interlaced" specified as an extra option, and got a 767MB .m4v file out! ...

2. So is there any practical purposes for H.264 lossless?

It could be used to compress raw video as it is captured in order to tame the bitrate without sacrificing any quality - that would probably require a hardware codec, although I haven't tested the faster x264 presets which may be able to encode in real time, albeit with a lower compression ratio. It could also be useful for storing intermediate results of editing (or a "master copy" of the final cut of something) without losing any quality as you would with a conventional lossy codec.

1. I don't understand. Isn't the idea of compression to decrease the file size? In the context you described, the file sized increased.

"It got bigger in this case because I was starting with a file that was already encoded using a lossy codec. If I'd started with 2 minutes 50 of raw video at 720x576@25fps using the same sample format as DV, it would have come to something like 3.4GB. A lossless reduction to 767MB would look quite good then!"

2. I don't understand what you're saying here. Can you rephrase this?

Setting the "interlaced" flag for x264 told the codec to process the incoming frames (which were interlaced) as two separate fields. I didn't test it but I would expect the resultant file to be much larger if the codec had tried to process them as progressive frames - it would have spent a lot of bits encoding the "combing" effect down the edges of moving objects that characterises interlaced video when viewed on a progressive display. If the codec had been in "normal" lossy mode, the outcome without the "interlaced" flag would either have been a larger file (if the codec was running in constant-quality mode) or reduced quality (in constant-bitrate mode).

3. You're saying that by checking interlaced, you prevent the video from being deinterlaced (assuming by default)? The deinterlacing process is what would require it to be a greater file size?

4. Also, why would not marking interlacing reduce quality? Wouldn't deinterlacing increase quality?

VirtualDub is definitely smart enough. It'll stream the video from the input file to the output one, skipping the bits that have been cut out, as fast as the source hard drive can deliver data and the destination one can take it. (If you're doing it on a single drive, it'll be much slower as the drive will have to keep seeking back and forth between the input file and the output one. If you've got a second drive, either internally or on USB, I'd recommend having the source file on one drive and exporting the cut version to the other.)

5. How large are the files that these DV's produce using Virtualdub? I have an HDD, SSD, and an external 64GB USB 3.0 flash drive. Is there any specific combination I should use for reading and writing? I'm mainly worried about one of my drives stuttering as I'm copying the video from the DV cassettes. Is that something I should be concerned with?

The most basic non-motion-compensating deinterlacers synthesise progressive frames either by discarding one field and filling in the gaps that are left by looking at the lines above and below them (which results in a loss of resolution) or by blending the two fields together (which still reduces the resolution, but it looks less severe, and results in blurring around moving objects - less ugly than "combing" but still not nice).

A motion-compensating deinterlacer looks for things that have actually moved from one field to the next (remember that the fields are not just "the odd lines" and "the even lines" from one frame - in 30 frames/sec video, the second field is "taken" by the camera 1/60 of a second later than the first) and, usually, attempts to reverse that motion to reconstruct the missing lines from the first field. Areas that don't appear to be moving can be left alone.

I used the QTGMC deinterlacer for AVISynth; you'll need to get your hands really dirty if you want to use AVISynth, though! It's also quite CPU-hungry.

Handbrake offers the Yadif deinterlacer, which is fast and gives very good results - see this page for some comparison screenshots including examples of discarding one field and blending the fields.

6. So Yadif and QTGMC are both motion compensating deinterlacers? Also, were you referring to the QTGMC or both when you said "super fancy" deinterlacer?

Looking at the pictures, discarding seems to look superior to Yadif. On the Yadif photo there's some kind of artifacts that look like white specks. Blended looks completely blurry.

I'll just test these in motion once I have the camcorder.

In theory, one could write a compliant H.264 encoder which simply encoded every frame as an I-frame. It would be very fast but would give a very large output file and/or very poor quality.

7. Why would the quality be poorer if all of the i-frames were included? Wouldn't i-frames represent the frames that are closest to the original content (assuming that it was uncompressed)?

Many modern graphics cards have dedicated hardware to assist with encoding and decoding common video formats. If you want super-fast encoding, this is the way to go but the quality isn't always as good as a good software encoder.

8. Just to be clear on a differentiation,

is there is a difference between the encoding and decoding hardware found inside of a graphics card and the actual GPU's computing power's ability to help accelerate software encoding in various software suites, which I assume would be triggered by some kind of "hardware acceleration" setting in its menu?

nVidia graphics cards have a truly magnificent motion-adaptive temporal-spatial deinterlacer - if only I could use it while encoding video!

9. When is this deinterlacer used? What triggers it? Why was it built into Nvidia gpu's?

Bear in mind that scaling should always be done after deinterlacing if you are changing the vertical size (I wouldn't even try it using an interlaced scaler - it'll probably just confuse the deinterlacer). When increasing the horizontal size, there could theoretically be some benefit to deinterlacing after scaling, depending on the deinterlacer being used.

10. Wouldn't deinterlacing & scaling be done at the same time in VirtualDub/Handbrake?

Either way, from your description, it appears I would only need to shrink the video horizontally, which doesn't fall in line with either of these scenario.

 

[molletts]

1. I don't understand. Isn't the idea of compression to decrease the file size? In the context you described, the file sized increased.

That's because we're comparing the lossless-H.264-compressed file with an already-compressed file (compressed using a lossy algorithm) rather than with a raw uncompressed video stream.

It got bigger in this case because I was starting with a file that was already encoded using a lossy codec. If I'd started with 2 minutes 50 of raw video at 720x576@25fps using the same sample format as DV, it would have come to something like 3.4GB. A lossless reduction to 767MB would look quite good then!

2. I don't understand what you're saying here. Can you rephrase this?

Let us, for the moment, pretend that the DV file I used was from a DV camcorder, rather than being captured from a VHS tape... (The principles are the same but it's easier to explain with reference to a camcorder.)

When the video stream emerged from the digital circuitry around the camera's image sensor, it was in raw, uncompressed form, a string of binary numbers representing the brightness and colour of every pixel in every frame. In the case of my test file, the resolution was 720x576 (it's a PAL video stream). Each frame would therefore have 720 × 576 = 414,720 pixels. The sample format used for DV encoding results in 16 bits (2 bytes) per pixel so that's 829,440 bytes per frame. At 25 frames per second, that comes to 20,736,000 bytes (about 20MB) per second. Filming for 2:50 (or 170 seconds) that comes to about 3.3GB. (I estimated it before in my head using ballpark figures rather than calculating it properly - it's gratifying to know that I wasn't too far out.)

In the DV camcorder, that 3.3GB of data was then fed into a hardware DV codec (a dedicated, purpose-built chip) which compressed it, lossily, down to the 587MB stream that was written to the tape and ultimately ended up on my hard drive.

If I had been using a professional video camera that allowed me to access the raw uncompressed stream, I would have had a very unwieldy 3.3GB file on my hands at the end of that short scene. I could have chosen to compress it to DV myself but why would I want to use a lossy compression format if I'd spent a small fortune on a fancy camera? That's where lossless H.264 would come in - I could get the 3.3GB file down to a much more manageable 767MB without losing any quality. (In this case, I'd probably want to use a hardware H.264 codec so that the stream could be compressed "on the fly" as I filmed.)

In converting the DV file to lossless H.264, I had to first decompress it from DV back to the 3.3GB raw video stream then feed this into the H.264 encoder. (This was done for me by Handbrake, which didn't need to save the uncompressed stream anywhere - it just routed the output of the DV decoder directly into the H.264 encoder.) You wouldn't normally want to convert DV to lossless H.264 because that conversion serves no purpose - you've already got a compressed stream and it's not going to get any smaller by being converted, nor can the quality lost by the DV encoding process be restored by re-encoding to a lossless format. I simply wanted to find out how the two formats compared on compression ratio. (Actually, I was surprised just how small the lossless H.264 file came out. It may have been helped by the fact the original DV encoding process had already thrown away some of the image data which it judged to be unimportant.)

Setting the "interlaced" flag for x264 told the codec to process the incoming frames (which were interlaced) as two separate fields. I didn't test it but I would expect the resultant file to be much larger if the codec had tried to process them as progressive frames...

3. You're saying that by checking interlaced, you prevent the video from being deinterlaced (assuming by default)? The deinterlacing process is what would require it to be a greater file size?

No, I simply tell the codec that I'm sending it interlaced video (it expects progressive video by default) - I chose not to deinterlace the video before encoding it so that what ended up in the H.264 stream was identical to what was in the DV stream.

When the codec knows that what is coming in is interlaced, it changes the way it encodes the frames to take this into account. Instead of looking at each frame as a single image (which would have that nasty "combing" effect down the edges of moving objects), it looks at it as two images, one for each field, and produces an H.264 stream that encodes the two fields separately.

The codec doesn't deinterlace - that's not its job. Its job is to produce an H.264 video stream representing whatever video it is sent. Deinterlacing is carried out in a separate filtering stage prior to encoding. In Handbrake, there is a filtering tab to select what deinterlacer and other filters to use - these are then applied to the video stream after it has been decompressed from the original format (DV in this case) and before it is fed to the encoder.

4. Also, why would not marking interlacing reduce quality? Wouldn't deinterlacing increase quality?

Not marking the stream as interlaced would cause the codec to encode the frames as single images. The combing effect would be seen as a highly-detailed edge, and the encoder would need to work very hard (and, in this case, use a lot of data) to encode it. If I was using H.264 in its "normal" lossy mode, it would either blur the combing to save data or spend so much data trying to reproduce it faithfully ("hey, this edge is really detailed and high-contrast, it must be very important") that it wouldn't have enough bits left to reproduce other areas of the image properly.

Deinterlacing will improve the perceived quality of the image by eliminating the combing effect. A theoretical, perfect deinterlacer would keep the "quality" (i.e. the amount of information in the image by some mathematical definition) the same but a real-world deinterlacer will always lose some information, either by discarding it, by blending bits of the image together or by guessing either what colour to fill in the gaps with (interpolation) or how much to move something (motion compensation). As long as the perceived quality improvement outweighs the actual quality loss, that doesn't matter.

If you've got a second drive, either internally or on USB, I'd recommend having the source file on one drive and exporting the cut version to the other.

5. How large are the files that these DV's produce using Virtualdub? I have an HDD, SSD, and an external 64GB USB 3.0 flash drive. Is there any specific combination I should use for reading and writing? I'm mainly worried about one of my drives stuttering as I'm copying the video from the DV cassettes. Is that something I should be concerned with?

DV files come to about 12GB per hour of video. The data is received from the camcorder at 25Mbps or about 3MB/s which isn't very demanding really. The hard drive would probably be the most suited to direct capture (unless you have sufficient space on the SSD) - I find that USB flash drives tend to "pause" every so often while writing data which could lead to dropped frames.

I used the QTGMC deinterlacer for AVISynth; you'll need to get your hands really dirty if you want to use AVISynth, though! It's also quite CPU-hungry.

Handbrake offers the Yadif deinterlacer, which is fast and gives very good results - see this page for some comparison screenshots including examples of discarding one field and blending the fields.

6. So Yadif and QTGMC are both motion compensating deinterlacers? Also, were you referring to the QTGMC or both when you said "super fancy" deinterlacer?

QTGMC is a true motion-compensating deinterlacer - it attempts to track things moving around in the picture and move the relevant bits of image back to where they would be if the frame was a progressive one.

Yadif is a "smart" deinterlacer - it tries to leave alone bits that don't need deinterlacing then uses various tricks including blending and interpolation to do a pretty good job of deinterlacing the rest. It's fast enough that I can use it on my media PC (an elderly 1.8GHz Athlon Neo) to deinterlace DVDs in real time.

I used QTGMC on that particular occasion. It took ages and, as I said, I'm not convinced that it was really worth the extra effort.

Looking at the pictures, discarding seems to look superior to Yadif. On the Yadif photo there's some kind of artifacts that look like white specks. Blended looks completely blurry.

It's the horizontal things that suffer most with discarding - it simply reduces the resolution from 720x480 (for an NTSC file which I assume is what you'll be working with) to 720x240. That neatly removes the combing effect but throws half the image away at the same time.

Different deinterlacing techniques work better or worse with different types of image - as you say, it's best to try a few with your specific video stream and see which one looks best to you.

In theory, one could write a compliant H.264 encoder which simply encoded every frame as an I-frame. It would be very fast but would give a very large output file and/or very poor quality.

7. Why would the quality be poorer if all of the i-frames were included? Wouldn't i-frames represent the frames that are closest to the original content (assuming that it was uncompressed)?

If the quality was retained, the file would be very large because I-frames are very large compared to P- and B-frames (they have to encode the whole image, after all).

If, however, you asked the encoder to go for a small file (comparable to the file size you would expect to get from a "proper" H.264 encoder) it would have to dial the quality setting for producing the I-frames right back in order to achieve that.

As a quick demonstration, try loading a photo into an image editor. Save it out as a JPEG file with a fairly high quality setting (if your program offers a 0-100 scale with 100 being best, then 75 is pretty good). JPEG is fairly similar in principle to how an I-frame is compressed. Look at the file size of the resultant JPEG file and open it to see what the image quality looks like. Now go back to the original image and save it out as a JPEG again but reduce the quality setting to something really low (about 10 on the scale I mentioned before). Look at the file size and the image quality of this one. This is the equivalent of what the "dumb" I-frame-only H.264 encoder would have to do in order to achieve a small file size using only I-frames.

Many modern graphics cards have dedicated hardware to assist with encoding and decoding common video formats. If you want super-fast encoding, this is the way to go but the quality isn't always as good as a good software encoder.

8. Just to be clear on a differentiation,

is there is a difference between the encoding and decoding hardware found inside of a graphics card and the actual GPU's computing power's ability to help accelerate software encoding in various software suites, which I assume would be triggered by some kind of "hardware acceleration" setting in its menu?

I would expect encoding done by a software codec that makes use of GPU computing, to be as good as non-GPU-assisted encoding (unless the manual for the specific codec says otherwise) by the same codec.

"Hardware acceleration" is a bit ambiguous in this context - it can refer to using general-purpose GPU computing (which is really just software running on a GPU) or it can mean making use of actual dedicated encoder hardware bundled with the GPU. You'd have to look at the documentation for the software to find out exactly what is meant.

nVidia graphics cards have a truly magnificent motion-adaptive temporal-spatial deinterlacer - if only I could use it while encoding video!

9. When is this deinterlacer used? What triggers it? Why was it built into Nvidia gpu's?

I don't know when it gets used on Windows by default but on my Linux system, it's triggered by me turning it on in the player software. The VLC player has a menu option and a keypress and supports hardware deinterlacing on both Windows and Linux.

All GPUs (even the basic Intel integrated ones) have some kind of video playback capability these days and deinterlacing is a natural companion to that functionality as there are so many interlaced video streams out there - DVDs of TV shows for example. I don't know whether AMD/ATI or nVidia was first but once one had it, the other had to follow suit.

Bear in mind that scaling should always be done after deinterlacing...

10. Wouldn't deinterlacing & scaling be done at the same time in VirtualDub/Handbrake?

It would be done in the same operation but in more advanced software, you can choose to apply filters in any order you like. It's a while since I last used VirtualDub but I would expect that it would apply the filters in whatever order they were selected by the user, whether or not that makes sense. Handbrake is designed to be easy to use so it has the filter order hard-coded to something sensible.

Good luck with the video work - you can always play around with the settings, filter order, etc. for a short clip to see what happens if you do it "the wrong way".

Stephen