1. When you say "original file", you're saying a large file that VirtualDub creates, which is secondary to the pointer file? The editing of this original file occurs while VirtualDub is still open or is it closed first?
By "original file", I'm referring to the original, unedited DV capture from the camcorder (or, if you choose to "top and tail" it first, a version of it that has been cut but had no other edits applied). VirtualDub stays open to run the frameserver and reads frames in from this file as required, applying any filters, etc. on the fly before passing the frames on to whatever application has opened the frameserver "signpost" file.
I just ran a 587MB .dv file through Handbrake with lossless H.264 selected, with "interlaced" specified as an extra option, and got a 767MB .m4v file out! ...
2. So is there any practical purposes for H.264 lossless?
It could be used to compress raw video as it is captured in order to tame the bitrate without sacrificing any quality -
that would probably require a hardware codec, although I haven't tested the faster x264 presets which may be able to encode in real time, albeit with a lower compression ratio. It could also be useful for storing intermediate results of editing (or a "master copy" of the final cut of something) without losing any quality as you would with a conventional lossy codec.
It got bigger in this case because I was starting with a file that was already encoded using a lossy codec. If I'd started with 2 minutes 50 of raw video at 720x576@25fps using the same sample format as DV, it would have come to something like 3.4GB. A lossless reduction to 767MB would look quite good then!
3. Also, just out of curiosity, are these older analog and digital DV tapes stored as interlaced or full frame video? Also, did applying interlaced in your video edit have a direct affect on file size? If so, why? Lastly, what do you mean by not "clean" when describing your analog tape?
Consumer analogue and DV tapes always store the video in interlaced form - although the video frames themselves could be non-interlaced, they would still be stored as two separate fields, one after the other.
Setting the "interlaced" flag for x264 told the codec to process the incoming frames (which were interlaced) as two separate fields. I didn't test it but I would expect the resultant file to be much larger if the codec had tried to process them as progressive frames - it would have spent a lot of bits encoding the "combing" effect down the edges of moving objects that characterises interlaced video when viewed on a progressive display. If the codec had been in "normal" lossy mode, the outcome without the "interlaced" flag would either have been a larger file (if the codec was running in constant-quality mode) or reduced quality (in constant-bitrate mode).
By "clean", I mean free from video noise ("fuzz"). Digital sources tend to produce clean video, except when the camera is struggling in low-light conditions, when the noise in the electronics gets amplified alongside the weak signals from the sensor and becomes more visible. Analogue sources are more prone to noise as it can creep in at any stage of the signal path - and the tape itself introduces noise: think back to the "hiss" that could be heard on audio cassettes, especially the cheaper Type I (ferric) ones.
4. When you say "pure cut" I'm assuming you mean simply spliting the video? If so, would VirtualDub fall in line with being "smart enough" to not affec the quality of my split videos?
Yes, just splitting the video at particular frame(s). With an all-key-frames codec like DV, it's just a matter of copying the right chunk of data from the input file to the output file. (Actually, that's not quite true - normally, the editor will also update the timestamp on each frame so that the output video stream's timestamps don't suddenly jump from 00:03:45 to 00:04:56 or whatever but that doesn't affect the content of the frame so it doesn't require any re-encoding.)
VirtualDub is definitely smart enough. It'll stream the video from the input file to the output one, skipping the bits that have been cut out, as fast as the source hard drive can deliver data and the destination one can take it. (If you're doing it on a single drive, it'll be much slower as the drive will have to keep seeking back and forth between the input file and the output one. If you've got a second drive, either internally or on USB, I'd recommend having the source file on one drive and exporting the cut version to the other.)
5. So again, just VirtualDub would be good enough for the archival aspect of my goal?
It'll certainly take care of that. It's also very simple to use - a more fully-featured program would be overkill (and it might be hard to ensure that it doesn't do any kind of re-encoding behind your back).
6. Care to share what this "motion-compensating deinterlacer" was and what software you used to apply it. =] What is it best suited for?
The most basic non-motion-compensating deinterlacers synthesise progressive frames either by discarding one field and filling in the gaps that are left by looking at the lines above and below them (which results in a loss of resolution) or by blending the two fields together (which still reduces the resolution, but it looks less severe, and results in blurring around moving objects - less ugly than "combing" but still not nice).
A motion-compensating deinterlacer looks for things that have actually moved from one field to the next (remember that the fields are not just "the odd lines" and "the even lines" from one frame - in 30 frames/sec video, the second field is "taken" by the camera 1/60 of a second later than the first) and, usually, attempts to reverse that motion to reconstruct the missing lines from the first field. Areas that don't appear to be moving can be left alone.
I used the QTGMC deinterlacer
for AVISynth; you'll need to get your hands really dirty if you want to use AVISynth, though! It's also quite CPU-hungry.
Handbrake offers the Yadif deinterlacer, which is fast and gives very good results - see this page
for some comparison screenshots including examples of discarding one field and blending the fields. I'm not convinced that I really gained very much from using a super-fancy deinterlacer (apart from a warmer room while it was running) - I doubt I'd spot the difference between Yadif and QTGMC without pausing the playback and looking closely at the screen! I'd suggest using a custom Yadif preset in Handbrake "mode=0" which does temporal and spatial checks to see which parts of the frame need deinterlacing then uses various interpolation methods to fill them in. I think VDub may have Yadif built in these days; there's also a Smart Deinterlace filter which is supposed to be good too.
7. Just to understand these technologies a little better, these codecs are a type of software library/method, but are coded in these software suites differently? What part of the codec internally is standardized if different software suites produce slightly different results with equivalent settings?
The different H.264 codecs are different programmers' implementations of the H.264 standard. The standard defines how the video stream must be represented in order that any compliant decoder can play it back - how the I-frames, motion vectors, etc. are to be stored and what they mean. It is left for the codec writers to decide how they actually go about converting a video stream into this form - and, indeed, how they convert the H.264 stream back into displayable video in the decoder. There are various "profiles" and "levels" which define exactly which features can be used when encoding, in order to ensure that the result is playable on a player which supports that particular profile and level (be that an app on a phone, a player program on a PC or a hardware device such as a Blu-Ray player).
In theory, one could write a compliant H.264 encoder which simply encoded every frame as an I-frame. It would be very fast but would give a very large output file and/or very poor quality. (Some early software MPEG encoders did exactly this - it would have taken a very long time to do full motion-estimation on an early-1990s-era PC. I didn't do video stuff in those days but I remember leaving the PC on for a couple of days once to rip a CD to MP3 format so I could give a pretty-rubbish-quality copy to a school friend on about 40 floppy disks!)
Some encoders make heavy use of the general-purpose compute capabilities of graphics cards; others are optimised for specific tasks such as low-delay real-time encoding or super-low bitrates.
Many modern graphics cards have dedicated hardware to assist with encoding and decoding common video formats. If you want super-fast encoding, this is the way to go but the quality isn't always as good as a good software encoder. (Likewise for decoding although the GPU can be useful to help with any post-processing that is necessary. nVidia graphics cards have a truly magnificent motion-adaptive temporal-spatial deinterlacer - if only I could use it while encoding video!)
The open-source x264 encoder is generally regarded as one of the best H.264 implementations - in "normal" lossy mode, it's both very fast and produces very good quality.
8. I'm assuming you're talking about this: https/www.videolan.org/developers/x264.html
What difference would there be between using this and Handbrake?
Yes, that's the one.
Handbrake uses x264 internally so there's not really any reason to use the standalone encoder instead. (Many other programs use it too - it's a popular choice because of its blend of speed and quality.)
... you'll need to scale them a bit to correct the aspect ratio - DV doesn't have square pixels. It's always either 720x480 or 720x576 regardless of whether it's 4:3 or 16:9 aspect.
9. What do you mean by "scaling" them? What does this have to do with the aspect ratio? What is the fix for dealing with non-square pixels when editing a video in Handbrake and x264?
By scaling, I mean resizing the video. Computer displays normally have square pixels and most playback software assumes that video streams being played back also have square pixels. This results in videos with non-square pixels being displayed with the wrong aspect ratio. It's not necessary to scale (or indeed deinterlace) DV when converting it to DVD, though, because DVD uses the same resolution and can store interlaced video (but make sure you use the MPEG2 encoder in interlaced mode, for the same reason as I passed the "interlaced" flag to x264 in my lossless test encode).
For example, a 4:3 NTSC DV stream with a resolution of 720x480 will appear to be slightly too wide. To get the correct shape, it will be necessary to either stretch it vertically, to 720x540 (see note below), or shrink it horizontally to 640x480. (Other 4:3 sizes would work too but I'm only changing one dimension for the purposes of the example.)
Conversely, a 16:9 NTSC DV stream will look too narrow because it'll still be in 720x480 resolution. It can either be stretched horizontally to 853x480 (see note below) or shrunk vertically to 720x405 (likewise). (Again, other 16:9 sizes would work fine.)
When encoding to most modern formats, it can be beneficial to compromise a little on the exact dimensions for scaling. Most codecs divide the image into a grid of larger "macroblocks" which they then process. The block sizes can vary but they're often 8x8 or 16x16. H.264 uses 16x16 for its macroblocks although it also uses other smaller sizes for other purposes. It can improve encoding efficiency to have the width and height equal to multiples of the block size so it may be necessary to have a very-slightly-wrong aspect ratio (or possibly crop a few pixels off, or both) to achieve this.
720 ÷ 16 = 45 so this is fine.
540 ÷ 16 = 33.75 so 544 (34 × 16) would be better and isn't too far out.
853 ÷ 16 = 53.3125 so you'd have to use 848 (53 × 16) instead.
405 ÷ 16 = 25.3125 so 400 (25 × 16) would be better.
Bear in mind that scaling should always
be done after
deinterlacing if you are changing the vertical size (I wouldn't even try it using an interlaced scaler - it'll probably just confuse the deinterlacer). When increasing
the horizontal size, there could theoretically be some benefit to deinterlacing after scaling, depending on the deinterlacer being used. I normally try to avoid upscaling if possible anyway - it just makes the encoder work harder to encode information that isn't really there anyway.
Looking at the Handbrake user interface, it appears that they seem to favour simply setting the aspect ratio attributes in the output file. Maybe players are better-behaved now than when I last did a multi-platform survey of them. I'll have to try and find the time to do a fresh survey.
It might be worth experimenting to see if you get playback in the right aspect ratio on all the devices that matter to you. Quality-wise, no scaling is preferable if playback works fine.
I need to charge my brain now!