Category: Video

Moving Pictures..

Watching HD content (Part I)

stany

Introduction

One of my fields of interest is video making. I’ve been eyeing the HD stuff with a bit of curiosity, and pretty much convinced myself to step up to HD production. When I learned that Microsoft released an add-on to Xbox 360, consisting of an HD-DVD drive in an external USB-accessible enclosure, after some penny pinching, I took the plunge and bought it.

HD-DVD (and Blu-Ray) drives are part of the security layer designed by Advanced Access Control System (AACS), a new generation in viewer rights restriction, that successed Content Srambling System (CSS), implemented on DVDs. AACS is non-exclusively licensed by AACS Licensing Authority, and restricts what a buyer (in essence a long term renter of a right to watch) of a HD video disk can do.

Specifically, AACS mandates that a viewer should not be allowed to watch HD output from software player if there is an unencrypted path between the player and the video output device. In essence, if your video card doesn’t support High Defenition Content Protection (HDCP), and if your monitor is not connected to your video card over DVI or HDCP, but over analog or component outputs, your viewing experience will be “degraded”.

“Degradation” in plain terms means that if you have HDCP path to your monitor/TV, you will view content in it’s 1080p (1920×1080 resolution, deinterlaced) glory. If you have DVI outputs, well, you get 1080i (1920×1080, interlaced, but it’s OK, because output is 60Hz, and most decent HD TVs and monitors will do proper deinterlacing, in essence outputing 1080p). If you have component or analog out, you might be lucky, and video studio will let you watch the same video in a “pleb” 960×540 resolution. It is still better then 640×480 (or 720×480 – remember that TV pixels are not square, but are at 4:3 ratio, so depending on what your scaler does, resolution might differ) resolution of a Standard Defenition (SD) video signal, but you paid for 1080p, right? This is up to the studio, however, and very few are liberal enough to allow you even that. So most often you get to view the trailers and extra filler on disk in 540×900, but are not authorized to watch the main feature at all. These rights are both bitmasks on the video streams, and additional rights data in the encrypted XML data files on the disks.

I can think of a couple HD-DVD disks that were released totally unencrypted with no AACS bits. Problem is that there are talks about enforcing ROM Mark by the players. ROM Mark is a special sector that pressed disks have, and burned do not. Idea is that if the disk has ROM mark, it was pressed, and thus must have AACS. If the disk doesn’t have ROM mark, it was burned, and thus it must have unencrypted, uncopyrighted content (again, extra bits in the video stream that player checks for) obtained from, say, consumer video recorder. “Nature’s Colors” and “Running Scared” (EU release) are reportedly unencrypted. Depending on how stupid the implementation of the check is, these titles might stop working.

Ultimately, AACS consortium had a problem. How do you keep “pirates” from “stealing” your HD video, when your “customers” and “pirates” are one and the same? I mean, in the end, someone has to watch the stuff, right? Otherwise, why would anyone buy the videos? Hmmm. A problem indeed.

I have to give HD-DVD manufactureres a credit, however. Remember region protection? The annoying thing, that made you rip and burn the DVDs you bought in Machu Pichu, that told you about the history of the ruins, just so you could watch it? The thing that forced folks from UK wait 6 month to a year before “Lord of the Rings” DVDs would come out in UK? Well, in HD-DVDs it’s gone (Although there are rumors of it coming back in a year or so, but in current iteration nothing supports it). So I bought a bunch of Studio Canal HD-DVD titles from Amazon.fr, that are not offered in North America. They play fine. In Blu-Ray world, things are more complicated. There are three regions that break down thusly: USA + Japan, Europe and those thieving Chinese and Russians. So Blu-Ray player sactually don’t let one watch US version of “SpiderMan III” in Europe (And as far as I know, neither HD-DVD nor Blu-Ray made it to Russia or China yet, so there are only Region A and Region B, as far as Blu-Ray is concerned).

To keep those thieving “consumers” from “stealing” HD content, and god forbid end up with unprotected version of it, AACS created a rather complicated scheme.

The Scheme

This is my understanding of AACS, which is very high level, and probably very wrong. No warranties for the next few paragraphs.

Warning! Acronym overload coming right up.

At the lowest level, every element on the disk is encrypted with a Title Key (TK). Element can be the movie itself, trailers, various data files on the video disk, etc. Title Keys are unique to a particular movie, at least to a particular pressing of a disk (So EU version of “Bourne Supremacy” would have different TKs then US version of “Bourne Supremacy”, and then Japanese version of “Bourne Supremacy”.) Title Keys are stored on the disk itself, encrypted with a Volume Unique Key (VUK).

Muselix64’s hack, that made all the news in early January, consisted of a way to decrypt the video elements by taking the user supplied VUK, and using it to extract the TKs, and do decryption.

Now, VUK is generated by the software player. In order to generate the VUK, player has to use the Volume Identificator (VID), stored on the disk, and the Media Key, that is again unique per title, and hash the two together.

Sounds simple, right? But in order to obtain the Volume ID, some magic is involved. Volume ID is 16 bytes, and first 8 bytes can actually be extracted by reading the Burst Cutting Area (BCA) from the disk. No special tricks, really. The remaining 8 bytes can be obtained by reading the copyright data section on disk. Here things get harder.

Media Key is generated by using the Media Key Block (MKB) that is also stored on the disk, with a Device Key unique to the decrypting device (in case of a stand-alone player) or decrypting software (in case of a Home Theater PC).

But before the drive will let you read the data, it will want you to authenticate yourself to it using a key signed by AACS LA.

So here is part section 4.3 of the AACS spec 0.91 on how the drive and the software talk to each-other:


By the drive authentication, the drive and the PC host verify each counterpart is an AACS compliant device that has valid certificate signed by the AACS LA and can sign and verify digital signatures specified in this document. In addition, the drive and the PC host verify each counterpart is not revoked by checking the Host Revocation List (HRL) and the Drive Revocation List (DRL), respectively. To do this, the drive shall store the most recent HRL it has encountered and the PC host shall store the most recent DRL it has encountered. When the drive authentication is successful, the drive and the PC host have a shared Bus Key (BK), and can proceed to the further steps. Figure 4-6 shows the protocol flow for drive authentication and key sharing used in AACS.

AACS Common Specification, version 0.91. Pages 44 onwards document the intricacies of a drive talking to the system, where neither trusts each-other, and both trust things signed by AACS LA key.

So each device or software player has a different Device Key, but they all generate the same Media Key when combined with the Media Key Block.

So we generate a Media Key Block, then we convince the drive that we are for real, get the Volume ID, generate the Media Key, combine the two together, generate the VUK, use VUK to obtain the TKs, and then we decode each element as required, and get to see the Jack Black and Naomi Watts in all high resolution glory.

I swear, only mathematician can be excited by something this complicated. And we haven’t even looked at subset difference, key revocation, and other complicated things, designed to keep movie studios in control.

Content revocation? *blink*.

Yes, if you bought a copy of “Serenity” on HD-DVD (or Blu-Ray), and 4 years from now it is deemed that girls wearing dresses in martial arts scenes (Summer Glau is hawt!) are against the law, AACS LA would be able update the list of keys on the drive and Device Keys, to prevent you from being able to decrypt, and thus watch, content on “Serenity” HD-DVD. Since drive updates it’s list of revocations each time you insert new media into it, they will need to release a bunch of HD-DVDs with updated lists, and wait till you get one, and put it in a drive. Since Device Keys are supposed to cycle every 18 months or so, in about 18 months you will not be able to play back the new disks, since your software player won’t have the Device Keys for them (at this point old disks will still play fine). You will download a “free” software update for your PowerDVD, and it will remove the keys to play back “Serenity”. Easy, right?

Now you understand why I called you a “renter” in the beginning of this post.

At this point it’s almost 4 am, and I got to head to work in less then 5 hours. In next session I will talk about Xbox 360 HD-DVD add-on, and maybe get to cover some neat things you can do by sending CDBs to various devices. Or maybe something completely ifferent.

Video: DVD Studio Pro – I love you!

stany

Spent a good chunk of today fighting with DVD creation. My previous attempt was using iMovie HD 6 and iDVD 6. Now, I have some issues with iDVD:

Suppose your projects are saved on an external drive, both iMovie and iDVD. You’d think that when you are rendeering the final DVD, it would still keep all the bits and pieces on the external drive, where the data is saved. Oh, no. iDVD is different. iDVD is smarter then the user, and will try to save the intermediate audio track (before the muxing) in /tmp If you have PCM audio track (and all of my previous projects did, because I didn’t know any better. Me Ogg. Ogg stoopid, remember?). Now, imagine that you are running out of disk space on your internal drive as it is, and then out of nowhere another 1.5 – 2 gigs of stuff show up in /tmp. Of course iDVD would die at this point.

iDVD 6 is also temperamental about DVD mastering, and was refusing to even think about creating a dual layer DVD (ie something longer then 4.7 gigs in size) if I didn’t have a dual layer burner plugged into the iBook. Fiar enough, I gave it the drive.

Then after running all night, it would die with a numerical error code (I’ve googled for it, noone saw it before). I tried three times, as originally I thought that I might have exceeded the “TV safe” area on the menu (another famous was of getting iDVD to die) with DVD title, or somesuch. But no, it just wouldn’t work.

So I got access to a machine with Final Cut Studio installed on it.

Oh, what a joy.

Software actuallly uses the location you tell it to use, without arbitrarily using what it should not. Software tells you what it thinks you should do, but lets you overrule it if you think you know better. Sane software.

I’ve plugged in my external hard drive, and imported into DVD Studio Pro DV streams which I used in iMovie and iDVD without much success before. It happily dealt with them.

Quickly I created a timeline. I’ve had everything pretty much pre-rendered, so it was as simple as setting a bunch of chapter breaks, creating a menu and linking the buttons to actions (ie Play chapter 1).

System I was using is an elderly G4 1.5 Ghz which was kind of skipping frames when dealing with large streams, and thus creating chapters was a bit of an excercise in patience. I’ve opened the iMovie project, and looked at places where I’ve placed chapter breaks before. In DVD studio I’ve created a bunch of chapter breaks arbitrarily, and then adjusted the times, so they would match more or less what I had in iMovie.

Worked as it was supposed to. Beautiful.

DVD Studio was telling me that my project would compress down to 5.1 gigs. At this point I thought that I should just do it, and then run it through DVD2OneX or somesuch, and shrink it down to 4.7 gigs, and told it to go ahead and just do it. It happily rendered to hard drive (it also asked me where I want to set layer breaks in dual layer disk, which was really nice too).

Eventually I realized that there is such a thing as Compressor, that can take a component of a multiplexed stream, and convert it to a different format.

After taking two 12 gig DV streams, and running them through Compressor, I’ve converted the audio tracks on both streams from PCM audio to Dolby 2.0 AC3.

Once I’ve imported the AC3 streams into DVD Studio Pro project, deleted the PCM audio from the timeline, and added in ac3 audio, projected project size dropped from 5.1 gigs to 4.1 gigs, and actual project size (once assets were rendered) dropped from 4.9 gigs to 3.6 gigs (I’ve used crappy video as DV source, from video tapes that were sitting in storage for god knows how long, so they compressed a fair bit).

So overall, I am really really happy with DVD Studio, although I’ve not used even 1/10’s of it’s capabilities. It can create HD DVDs. It can embed web links in mpeg files. It can edit existing menus. Now I need to save up my shekels to buy it (899 CAD for student license for Final Cut Studio).

Video: Video archives

stany 1 Comment

I digitize various video footage, some of which is obtained from degrading video tapes, and needs to be color corrected.

Oldest footage so far was from a 1982 black and white Beta video, that was converted to VHS in 1988, timecode added, yet improperly stored since.

I keep the scenes that I feel are important, and eventually create DVDs in iMovie/iDVD, however I am not 100% sure that bits that I discard are not important.

Thus I am interested in preserving full footage as well, at least for 5, although realistically for over 10 years.

I can keep it on video tapes, yet am not certain that it’s a good idea, considering how much some of this footage degraded already. I am also concerned about availability of VCRs capable of reading video tapes in 10 year time frame.

One approach is to keep it in uncompressed DV format on a hard drive, drop the hard drive into a bank vault. As I am dealing with ~300 hours of video all together, this is not realistic, esp if I want a backup. Besides, will I be able to read HFS+, ext3fs, NTFS or whatever? EIDE? SATA? Firewire? USB?

I’ve been considering compressing it in full NTSC (or PAL) resolution to DivX 6 format, and burning it to DVDs (or on one or two external hard drives). Yes, I technically lose quality, but then again, most of my source material is not stellar as it is.

At this point I am concerned as well… Will I be able to decode it in 10 years? Will DVD drives still be available, or will they go the way of (5.25″) floppy drives?

Re-copying every 3 years?

Suggestions?

Is there anything better then DivX that I should look into? I am concerned about the disk space to quality ratio, and this project has no budget (ie I currently finance it out of my own pocket)

Video: Video digitization workflow

stany

It so happened that I got involved in video digitization project. Here is a quick description of my setup and workflow.

Key points

Video capture of DV video streams using QuickTime is not effective, as QT will try to encapsulate DV stream, resulting in both high cpu usage/dropped frames. Capture can be performed either using iMovie or using digital VCR application in FireWire SDK 22. This normally results in less then 50% CPU usage on a 1.2 Ghz iBook, and generates a proper DV stream.

In order to speed up exporting and importing data when working with iMovie, one can select “show package contents” in finder, and look inside the iMovie project. Capture.iMovieProject/Media/ contains the video streams, which can be moved out and edited in stand-alone applications, or moved in to speed up import procedure a great deal (it helps if streams that are being imported are of the same format as the project, although in iMovie 6 anything that QT supported seemed to work, I just pay for it in terms of conversion time at the final export).

Setup

Currently my tools are a Sony Hi8 video camera, essentially a consumer model, two different VCRs (A Mitsubishi and a Hitachi), and a Canopus ADVC-110 analog to digital adapter.

Canopus box emulates a DV camera, and speaks a subset of DV protocol, so as far as host system is concerned, it is a somewhat dumb DV camera. Somewhat dumb because it silently ignores any of the DV commands that have to do with reading the tape markers or rewinding the tape. Some of the commands tend to confuse it, and it in turn changes from “analog to digital” to “digital to analog” conversion. But in 99 out of a 100 cases it works really well.

Not all VCRs are born equal. My Hitachi generates grainier image using the same footage, however it is more resilient to video tape damage, and doesn’t loose tracking as easily as Mitsubishi does.

Usual way of performing digitization consists of plugging a video camera or VCR into Camopus box, which in turn gets plugged over firewire into an external hard drive, which in turn gets plugged in over firewire into an iBook, that runs iMovie.

iBook runs either iMovie HD 5 or 6 (depending on which iBook). iMovie project is set to “DV” quality, and saved on the the external drive (or wherever there is plenty of storage space). My rule of thumb is 12.5 gigs of disk space per 1 hour of DV video.

Difference between iMovie HD 5 and iMovie HD 6 that I’ve noticed is in the resilience to interference. iMovie 6 seems better at dealing with damaged tapes with dropping video frames. What iMovie HD 5 would show as a full frame dropout, iMovie 6 will only show as a horizontal black line across the image. No differences if source material is of reasonable quality.

Most problems I’ve had were primarily caused by length and quality of firewire cables. Short, well shielded cables and only a couple of devices on firewire bus work best in my experience.

Workflow

In iMovie I enable the video camera capture mode (should be automatic the moment it detects a video camera plugged in), and hit play. iMovie sends a “play” command to Canopus box, which Canopus box happily ignores.

Then I manually cue up the video tape to about where I want to start capturing from, keeping in mind that capturing more is better then capturing less, I can then cue and splice things up in software.

Once I am happy, I click “import” in iMovie, and “press play on tape” (str) , and watch for a while that things are happy.

iMovie imports video in 1 hour chunks, so 12 gig files are about the largest size I deal with.

Once I am done digitizing, I quit iMovie, and look inside the .iMovieProject/Media/ folder, that by now contains Capture nn.dv files. I move .dv files out of the iMovie project, and open these in QuickTime. If video was filmed with a mono audio track, before further editing, I adjust the audio signal to be center-center instead of left-right in the dv stream (Apple-J in QT Pro, followed by clicking on audio, and adjusting the position of the audio channels)

I do rough cuts of the stream in QuickTime as well.

Main .dv file is opened, and scenes out of it that are of interest are copied and pasted into new QT windows, and trimmed as needed.

At this point my usual procedure is to create a folder, and export scenes that are worth preserving out of QT in DV fromat into this folder, using xx0-filename.dv naming scheme. This naming convention is similar to naming lines in BASIC, and allowes me to roughly adjust the scene sequence before importing these back into iMovie.

Once captured streams are edited, I open iMovie again, and create a new project, again in DV format. I save the empty project and again quit iMovie. In my experience iMovie takes forever to import things as it both converts the stream that it imports to the project format, and copies it into project’s Media sub-folder, so it is alot faster to just drop .dv files into Media subfolder of the iMovie project, and re-start iMovie. On a re-start iMovie will complain that there are new tracks in the project, and asks if I want to delete them, or just move them to trash. I select to move them to trash, and once iMovie starts, move them out of the trash into the timeline.

It takes me about an hour and a half to cut and export a moderately representative sample of what I capture, and I discard on average between 40 and 60% of source material.

Video: NTSC vs PAL

stany

Technically SECAM cameras internally store data in PAL format, so for all practical purposes, when dealing with VHS signals over RGB signalling, PAL/SECAM is 25 fps, 576 lines per frame, NTSC is 30 fps, 480 lines per frame. Both are interlaced, PAL is higher resolutioin, lower frame per second rate.

The following deals somewhat with over the air transmissions.

NTSC PAL SECAM
National Television System(s) Committee Phase-Alternating Line Séquentiel couleur à mémoire
Varieties NTSC I, NTSC II
NTSC III came about due to digital television routing during the 1980s; all aspects of NTSC III are rigidly mathematically defined. 		PAL B, D, G, K, I, PAL-M, PAL-N and PAL-NC. Fear. French SECAM (SECAM-L), used in France and its former colonies
SECAM-B/G, used in the Middle East, former East Germany and Greece
SECAM D/K, used in the Commonwealth of Independent States and Eastern Europe (this is simply SECAM used with the D and K monochrome TV transmission standards
Lines per frame 525 lines per frame, of which 480 visible lines. 625 lines per frame, of which 576 visible lines. 625 lines per frame, of which 576 visible lines.
Refresh rate 59.94 interlaced fields (60 Hz/1.001), (29.97 full fps).  50 Hz – 50 interlaced fields (25 full fps). Note: Motion pictures are usually shot at 24 fps, resulting in 4.2% speedup when telecined. 50 Hz – 50 interlaced fields (25 full fps).
Color encoding luminance-chrominance encoding system. Quadrature amplitude modulated subcarrier carrying the chrominance information is added to luminance video signal.  Red and blue are transfered together. phase-alternating line encoding system.  Quadrature amplitude modulated subcarrier carrying the chrominance information is added to luminance video signal.  Red and blue are transmitted together resulting in bleeding in high contrast scenes. fixed frequency subcarrier modulated by color signal.  Frequency modulation is used to concode chrominance on the subcarrier, Red and Blue luminance are transmitted one at a time, separated by delay line.  As color information is approx. the same between two consequitive lines, information from preceeding line is used to make up for bandwidth shortage.
Benefits Great vertical color resolution, esp in NTSC VHS or NTSC laserdisks. higher then NTSC line resolution higher then NTSC line resolution, good color separation thanks to luminance bein transferred separately.
Shortcomings low number of lines per frame. color bleeding color bleeding Vertical color resolution is 1/2 of that  NTSC or PAL.