QuickTime (part 2)

stany

Another braindump.

Somehow I ended up at PBS Nova Science Now page. It had lots of shiny TV goodness that I wanted to watch. Of course there also was a warning on the page saying This program is not available for downloading due to rights reasons.. Rights. Right.

First restriction was trivially bypassed. Netblock I am using is registered in Eugine, Oregon (which has a funny side effect that some web sites insist on hooking me up with “hot girls in Eugine”), which is actually correct, as I lease this /24 from it’s american owner.

So as far as PBS was concerned, I were tax paying merkin, and thus can be permitted to watch their programming (produced by taxpayer money). I can’t verify it right now, however I believe that they out right don’t permit folks connecting from outside US to view videos. *sigh* By the way, BBC does the same thing to some of their on-line content.

So QuickTime video was happily streaming off their web page. Due to pecularities of my network setup, that lead me to believe that they use HTTP protocol for content delivery. I viewed source, and grabbed http://www.pbs.org/wgbh/nova/sciencenow/video/nsn-wrap-new.mov (Feedback please. Does it play in your browser when you click this link?) , which when played in QT quickly sent me to http://www.pbs.org/wgbh/nova/sciencenow/video/rights_restrictions.gif. Right. So it plays from inside the browser, streaming, but not from HD. Joy, PBS.

Eventually I gave up and sniffed traffic ( tcpdump -i en1 -s 0 -w cookie ; strings cookie ). Noticed the following insteresting file: http://www.pbs.org/wgbh/nova/sciencenow/video/3204-new.xml (Sorry, not a hyperlink, as I want you to copy/paste it, thus this post not ending up in referer field – it might raise questions, as it’s not meant to be accessed by a browser).

It refers to a bunch of .mov files, that the program consists of, plus the “captions” for each part of the main movie.

Groovy. 

wget --user-agent="QTS (qtver=7.0.1;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/wgbh/nova/sciencenow/video/3204-new.xml

wget --user-agent="QTS (qtver=7.0.1;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/wgbh/nova/sciencenow/video/3204-00-ref.mov

This one is another container file. I had to save it and strings on it, to figure out the main file name. It is available in two qualities: 3204-00-300.mov and 3204-00-56.mov

so 

wget --user-agent="QTS (qtver=7.0.1;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/media/wgbh/nova/sciencenow/video/3204-00-300.mov

worked.

root@gilva:~/pbs[04:27 AM]# grep vidURL 3204-new.xml |sed 's/ref.mov/300.mov/g ; s/^.*http/wget --user-agent="QTS (qtver=6.5.2;cpu=PPC;os=Mac 10.4.1)" http/g ; s/< .*$//g ; s//video//g ; s/wgbh/media/wgbh/g' 
wget --user-agent="QTS (qtver=6.5.2;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/media/wgbh/nova/sciencenow/3204-00-300.mov
wget --user-agent="QTS (qtver=6.5.2;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/media/wgbh/nova/sciencenow/3204-01-300.mov
wget --user-agent="QTS (qtver=6.5.2;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/media/wgbh/nova/sciencenow/3204-02-300.mov
wget --user-agent="QTS (qtver=6.5.2;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/media/wgbh/nova/sciencenow/3204-03-300.mov
wget --user-agent="QTS (qtver=6.5.2;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/media/wgbh/nova/sciencenow/3204-04-300.mov
wget --user-agent="QTS (qtver=6.5.2;cpu=PPC;os=Mac 10.4.1)" http://www.pbs.org/media/wgbh/nova/sciencenow/3204-05-300.mov
root@gilva:~/pbs[04:27 AM]#

Now, these don't play in stand-alone QT and refer you back to the "Rights" image. However VLC will happily play them.

*sigh*

I am going to bed now.

QuickTime (Part I)

stany

Note: This is just a quick braindump, so probably is inconclusive, and makes no sense

Situation

A few days ago LIVE8 concerts were held in major cities around the world. Most interesting (to me, YMMV, of course) was the reunion of Pink Floyd after over 10 years of not being around, with Roger Waters being on stage with the rest of the classic lineup for the first time in 24 years. Wow.

AOL has the license for the internet distribution of the videos, and has a reasonably nice site from which the clips can be streamed using QuckTime.

Clips are really good quality, where quality of the video was not sacrifised in favor of bandwidth. Thank you, AOL, you rock.

If one clicks on the little tab by the song name, a window pops up in which clip plays. One can view source, search for “mov”, and eventually find http URL to the actual file. So I grabbed the 4 Pink Floyd songs.

Problem

When I proceeded to play them in QuickTime, they played great. But every silver lining has a cloud – I wanted to build a playlist, where the songs would be played in sequence.

iTunes kind of helped – I am not a big iTunes user, but I imported .mov files into it, made a playlist, arranged them in sequence, and it kind of worked. There were two snags, however – there were ~2 second gaps between songs, and it was audio only. Grumble. I wanted something that could just play them all.

I could have used VLC.app, I guess. I just verified that it plays these tracks, and it has the concept of playlist down pat. But instead I fired up QT Pro 6.5.2, selected whole video, and wanted to paste it together with the next song, etc, to merge 4 songs into one 20 minute long video.

Of course nothing happened. QT had the copy and paste controls grayed out.

So I attempted to export it. It popped up a window telling me Couldn’t export “‘Breathe’ (LIVE 8)” because this movie doesn’t allow saving. Aaaarrrgggh!

Aimless wandering in the dark, searching for solution

So fater about half an hour of googling I learned that many others run into this problem. Seems like this “feature” of QuickTime got noticed when certain movie trailers (ST: Nemesis is one, apparently) were exported to QT with “do not allow modification” bit set. This had the added benefit of forbidding QT Pro to save the file to HD, and irked some folks to no end.

Hacker’s Guide to QuickTime (Which actually has lots of rather useless pointers, such as “open web page with QT component in browser, and then find the cached file in browser’s cache to save file to HD”, which doesn’t work as most of the time now browser just loads a small file (example) that in turn loads the rest of the content, if it feels like it, or folks actually deploy QuickTime Streaming Server, and browsers generally don’timplement RTSP protocol) mentions that:

Video editing programs like Cleaner allow authors to save movies in such a way that further changes to the movie are disallowed. When the author saves the movie, he simply enables the “disallow saving” check box. Some filmmakers chose to do this to prevent others from altering their work. Others chose this option to discourage users from making local copies of movies viewed online.

So this had a glimmer of hope: If I were to obtain the right software, I could make a small (2 – 3 seconds) source file, import it into video editing package, tell it to save once without disallowing saving, and once with, hexdump both files, and diff them. My stipulation is that it’s just a byte or two in the header, that QuickTime happily follows. If I were to know which ones, I potentially could just hexedit the restriction out, and solve my problem.

At this point for some reason I got diverted, and instead of investigating “Cleaner”, went and grabbed Sorenson Squeeze 4.1. Site e-mailed me confirmation and the above URL to the download package.

Sorensen Squeeze is VISE X packaged blob of data that has 30 day free trial, and that will watermark generated files (until you license it). I didn’t care about watermarking, as as long as it generates both protected and unprotected file identically, it’s not a big deal. I know save restriction doesn’t encrypt the file, as VLC.app happily plays them back.

After playing with Sorensen for a while, I realized that a) It does a rather poor job converting other QT files to requested form at(frame dropping. Gave it an 80K/sec mpeg4 inside QT container file (La Tortura from one of my earlier articles), and told it to generate 750K/sec result. Result had 8 frame/sec output, and was choppy as heck (source was 16 frames/sec). Maybe it’s another restriction of the 30 day demo) and b) I couldn’t find the menu to disable save in Squeeze’s features nor in documentation.

At this point I gave up in disgust, and uninstalled Sorensen Squeeze 4.1.

Another complaint about VISE X. Why the F*&^ does it demand that all other applications must be closed during uninstall of software? It demanded none such thing during install. I am not about to close Safari with 35 windows, nor X11 with 8 xterms. Aaargh, what a piece of crap. MS Media Player for Mac is also packaged with it, and in that case it actually demands admin password just to install an application into /Applications. WHY?

So this is as far I made it.

Questions

  • Is there a way to extract files from VISE installers, specifically out of Install.data, without running the installer? I always fear that it will spew files all over my system, and I’ll never find them.
  • Any advice about “Cleaner”? Admitedly I am reluctant to put this here, as I’m yet to google it.
  • Anyone has any experience dealing with QT restrictions?

CF and IO undefined symbols

stany

When building something probably written for Unix (I were fudging s10sh), you might eventually end up with undefined symbols such as:

gcc -O2 -Wall -g -I./libusb -o s10sh main.o crc.o usb.o serial.o common.o 
bar.o  -lreadline -ltermcap libusb/.libs/libusb.a
ld: Undefined symbols:
_CFRunLoopAddSource
_CFRunLoopGetCurrent
_CFRunLoopRun
_CFRunLoopStop
_CFUUIDGetConstantUUIDWithBytes
_CFUUIDGetUUIDBytes
_IOCreatePlugInInterfaceForService
_IOIteratorNext
_IOMasterPort
_IONotificationPortCreate
_IONotificationPortGetRunLoopSource
_IOObjectRelease
_IOServiceAddMatchingNotification
_IOServiceMatching
_kCFRunLoopDefaultMode
make: *** [s10sh] Error 1

The IOService errors can be fixed by -lIOKit thusly:

stany@gilva:~/src/s10sh-0.2.2[07:52 PM]$ gcc -O2 -Wall -g -I./libusb -o s10sh 
main.o crc.o usb.o serial.o common.o bar.o  -lreadline -ltermcap libusb/.libs/libusb.a -lIOKit
ld: Undefined symbols:
_CFRunLoopAddSource
_CFRunLoopGetCurrent
_CFRunLoopRun
_CFRunLoopStop
_CFUUIDGetConstantUUIDWithBytes
_CFUUIDGetUUIDBytes
_kCFRunLoopDefaultMode
stany@gilva:~/src/s10sh-0.2.2[07:53 PM]$

but the “proper” way to fix it is:

stany@gilva:~/src/s10sh-0.2.2[07:53 PM]$ gcc -O2 -Wall -g -I./libusb -o s10sh 
main.o crc.o usb.o serial.o common.o bar.o  -lreadline -ltermcap libusb/.libs/libusb.a  -framework IOKit -framework CoreFoundation
stany@gilva:~/src/s10sh-0.2.2[07:53 PM]$ 

stany@gilva:~/src/s10sh-0.2.2[07:54 PM]$ ./s10sh -u
USB mode enabled
S10sh -- version 0.2.2
Copyright (C) 2000-2001 by Salvatore Sanfilippo 
S10sh is FREE SOFTWARE under the terms of the GNU public license

[Canon PowerShot A75] > ls
ls error
[Canon PowerShot A75] >

*sigh*

Promise UltraTrak100 TX8 and rebuilding RAID

stany 1 Comment

This is from memory, as it happened about a year ago, but I figured I’d document it, in case it helps someone.

I own a Promise UltraTrak100 TX8 SCSI to IDE RAID array. If it helps, here are local mirrors of product manual and specifications.

I had a case once, when I shut the array down, and one of the drives did not spin back up.

Situation:

disk0: Good
disk1: Good
disk2: Did not Spin up
disk3: Good
disk4: Good
disk5: Good
disk6: Good
disk7: Good

Array (configured as 8 disk RAID5, Maxtor 120 gig drives, 800.5 GB of formated disk space) of course started beeping, so I grumbled, yanked a cold spare off the shelf, and put in in place of “failed” drive and went to sleep.

Situation:

disk0: Good
disk1: Good
disk2: replaced with good one, put onto shelf, resyncing
disk3: Good
disk4: Good
disk5: Good
disk6: Good
disk7: Good

About 3 hours into raid resync (no, UltraTrak100s are not really speedy), array instead of short beeps raised a rucus, and it’s crying woke me up. Turned out that another drive failed while into resync. So the nightmare happened – there were two failed drives in a RAID5, and of course the array is not designed to handle this.

Situation:

disk0: Good
disk1: Good
disk2: resyncing
disk3: Good
disk4: Good
disk5: failed with bad sectors
disk6: Good
disk7: Good

I had not backups. As an aside, when you have 800 gigs of on-line storage, all used, how do you back it up? DLT7K (which I also have) would take maybe 3 days, and at this point, do I trust the tapes? After all, when you have 20 tapes, probability of tape read failure would be raised to n^20. Then there is dust in the drive, SCSI cables (differential SCSI in my case), power fluctuations, etc. The only way to back up 1 TB is to put a second 1TB array near it, and mirror them, and start using filesystem snapshots (like NetApp does, or Solaris 8 and newer). Any way, backups are a subject of a rant of it’s own.

So I grumled, and cursed, but went ahead and examined the original drive, one that didn’t spin up. SMART was complaining that the drive takes too long to spin up, but in the end I managed to convince it to spin up. So now I had a case where I had an array with two “bad” drives, yet one drive was actually “good”, only market as bad in the NVRAM of the UltraTrak.

After a while on long distance calls to Promise, I got to talk to a chinese guy who actually was one of the developers. He told me of a magic way to try as last resort.

So don’t do this at home, this is serious evil, etc.

He told me to turn array off, yank all the drives out of the array, and put one new drive into it.
Upon power on, array would complain about lack of the original drives. Then he told me to delete the existing configuration, and power the array off.

After that, he told me to put the drives including drive that was originally having problems spinning up back into the array in the original order:

Situation:

disk0: Good
disk1: Good
disk2: Drive that not Spin up originally, but got convinced to spin up again
disk3: Good
disk4: Good
disk5: failed with bad sectors
disk6: Good
disk7: Good

Then he told me to go and configure the array again from scratch, RAID5, whole disks, etc.
At the moment when I were to commit the configuration of the array I had to be careful. Essentially at that point all of the lights on the disks in the array would flash in sequence, as the configuration of the array would be written to disks. After that there would be a 1 second pause. During it, I had to turn the array off.

This is a once time shot. If one doesn’t turn the array off at this 1 second interval, the array would proceed with formatting the disks, and all of the data would be lost.

As I did it, array wrote configuration of the array to disks, matching the configuration that I had before, but did not re-initialize the array. So the data was still there.

When I powered the array on, it span up all the drives, and proceeded to claim that it’s fully functional.

So I manually failed drive 5, that had bad sectors on it by yanking it out of the array, and replacing it with a cold spare.
About 10 hours later array re-initialized. Then I failed disk2, that had issues spinning up, and replaced it. Array re-initialized.

You have no idea how stressed I were until the first rebuild was done.

Any way, maybe this will help someone. Obviously this is not exactly a technique for the faint of heart, and is not supported by Promise. But it saved my ass. If you have spare disks, try building a test array (of like 2 disks) and practice on it first. And, have good backups.

This should work on UltraTrak100 TX4 as well, but I have no idea about any other models. Probably not. Talk to Promise, they can be nice to you.

This message will self-destruct in five….

stany

Many years ago (well, 1998, to be exact) when the science of computer forensics was basically in it’s infancy, there already were rumors of various TLAs being able to read data that was overwritten on the hard drive. Some folks were telling of tunneling microscopes, and other high tech gizmos that could recover data up to 7 overwritions ago. I don’t know how true it is, however am inclined to believe that it is true. Now a days, products such as Encase can do wonders, and Linux based and occasionally open source tools are close behind commercial vendors. Forensics field turned into science, with an entire industry to support it – hardware write blockers, special court proceedings, expert witnesses, data recovery software, etc. One of these days…

This time I’ll just address a simple question: What media should one store the data on, if one expects that one would need to destroy the data on the media some time in the future, and adversary with great financial and technical resources would be interested in reconstructing the data?

For something like this, I’d recommend rewritable CDs and DVDs. Primary reason is ease of data disposal – if one has 30 seconds to get rid of incriminating evidence, all one really has to do is to drop a styrofoam cup half-full of water with a CD on it into microwave, and tell it to reheat.

While I don’t know about recent models of microwaves, older systems would generate a satisfying arcing and media destruction in about 10 seconds. Media will not melt to slug, however the data layer would be covered by a spider web of cracks, that would pretty much be rendered unusable. Here is an example. A google search for “microwave CD” should provide plenty more links to images.

Now, to a physicist in me this looks like a rather complete way of getting rid of unwanted data irrecoverably.

However, this is a reason why I suggested CD-RW and DVD-RW in the first place:

As with CD-Rs, the read laser does not have enough power to change the state of the material in the recording layer — it’s a lot weaker than the write laser. The erase laser falls somewhere in between: While it isn’t strong enough to melt the material, it does have the necessary intensity to heat the material to the crystallization point. By holding the material at this temperature, the erase laser restores the compound to its crystalline state, effectively erasing the encoded 0. This clears the disc so new data can be encoded.

So my advice to dissidents world-wide – first erase the DVD-RW and CD-RWs, and then microwave them. After that, toss them out and don’t toss and turn while in bed 😛

Edit1: Note! When I talk about erasing CD-R or DVD-R, I mean about full erase, that takes ~15 minutes, NOT quick erase. Quick erase generally just zeros out first megabyte of data on disk, including TOC, so it seems like it’s a clean disk, yet all of the previously recorded data is still there!

Edit2: I wonder if the crystalline properties of the layer change from being melted and re-cooled during the erasure process. In other words, is it still possible to detect where data was based on the different structure of the “re-flowed” layer after erasure? Any material scientists around? 😛

That’s why I recommend microwaving of the disk, just to make reconstruction of the data just that much harder.

Installing Roundup on Mac OS X

David Rostenne

I decided that I needed to install an issue-tracker for the support work I am doing, as jobs were trying to fall through the cracks.

This is the process I went through to install Roundup on Mac OS X 10.4.1 client, minus all the swearing, googling and reading the mailing list archives.

I grabbed the latest version, roundup-0.8.3.tar.gz, from Sourceforge and extracted it into a temporary directory. The README.txt led me to the Install Guide located in the doc folder.

From there I:

loki:~$ python run_tests.py

Which gave no errors..

Then I went back and followed the Basic Install steps:

loki:~$ sudo python setup.py install --install-scripts=/Users/davidr/bin

This installed everything as normal, except the administration scripts, which were installed in my bin/ directory.

I then created the storage space form my trackers:

loki:~$ mkdir ~/Documents/roundup/trackers

This is where the Support tracker will live, and any others in the future…

Then I installed the tracker:

loki:~$ roundup-admin install
Enter tracker home: /Users/davidr/Documents/roundup/trackers/support
Templates: classic-demo,minimal, classic
Select template [classic]: 
Back ends: anydbm
Select backend [anydbm]: 

 You should now edit the tracker configuration file:
   /Users/davidr/Documents/roundup/trackers/support/config.ini
 ... at a minimum, you must set following options:
   [mail]: domain, host
   [tracker]: web
loki:~$

This is where things got confusing, as the references in the Documentation are to “MAILHOST, TRACKER_WEB, MAIL_DOMAIN and ADMIN_EMAIL.” Which do not exist by these names in the config.ini file.

Since the tracker will only be used by me, I am putting it on localhost, and email will be disabled.

I opened /Users/davidr/Documents/roundup/trackers/support/config.ini and edited the following:

instant_registration = yes
# Avoids email confirmation for new users

web = http://localhost:8080/support/
# This is where I will access my tracker

In the section [mail] I also had to set these so the tracker would run, even though I will not use email;
domain = local
host = loki.local # my machine’s local name

Then in the [nosy] section I also changed:
messages_to_author = no
add_author = no
This is likely not needed as nosy gets disabled soon…

Now to initialise the tracker database:

loki:~$ roundup-admin initialise
Enter tracker home: /Users/davidr/Documents/roundup/trackers/support
Admin Password: (You need to create one!)
Confirm: (re-enter the same one from the previous line)

After that you need to get to the web interface, I decided to use the built in server and so all I did was to run:

loki:~$ roundup-server support=/Users/davidr/Documents/roundup/trackers/support/

and that notified me that it was running: “Roundup server started on :8080”, so I went to the page I defined in config.ini: http://localhost:8080/support/

Create a new user, and you are almost ready to use the system.

After that I needed to disable the email, and create a more automatic way to launch the server:

Disabled the use of ‘nosy’ ie. the email interface, see FAQ.

loki:~$ mv /Users/davidr/Documents/roundup/trackers/support/detectors/nosyreaction.py /Users/davidr/Documents/roundup/trackers/support/detectors/nosyreaction.py_disabled

I then created a double-clickable terminal file which will launch the Roundup server. Open a new terminal window, and then go File:Save as: and named it “Support Roundup Server” and set ‘Execute this command’ to

/Users/davidr/bin/roundup-server support=/Users/davidr/Documents/roundup/trackers/support/

I also checked the ‘Execute this command in a shell’. Now when I double click the file it launches the roundup server and I can watch the log as it goes by. Once I get tired of it and am convinced it works without me looking at it.. I will figure out a way to launch it when I login, and run it in the background and have errors go to a logfile.

Microsoft Certified What?

stany

From pktech:

Arfa Karim Randhwa of Multan, Faisalabad, Pakistan has been accredited as the youngest Microsoft Certified Professional (MCP) at age 9. The requirements for MCP are by no means trivial. No details on which exam she wrote but Desktop Support Technician seems like one of the easier routes to MCP with 4 courses requiring installation, management and troubleshooting knowledge of the windows environment (hardware, filesystem, networking etc) and applications (office, outlook, etc). She studied through Applied Technologies (APTECH) (whose website is a disgrace) in a single summer vacation. She comes from a rural, agricultural background with her father working for United Nations in Congo and her mother acting as the landlady in his absence. She also won a national singing competition. Not an astonishing story, but it does set an excellent precedent for Pakistani youth — a little encouragement goes a long way. Pakistan could use a lot more intellectual capital. What can be done to promote the youth and build intellectual capital?

Personally, between her and Andy, I’d bet on her. Not because I have little faith in Andy, no. I’d bet on her because she’s younger, and if she is already where Andy is getting to, then she has additional 25 – 30 years to get even further ahead. I remember myself when I were working for iStar. 17, prime condition, were going to school from 8:30 to 15:30, worked from 16:00 to 2:00am, sleeping 5 hours a day, if not less, learning computer security, Solaris system administration and Cisco routing on my own time and that was when? Less then 10 years ago. Now a days, an allnighter would probably kill me, and I look in awe at folks who are 19 – 20 and are able to grok Galois Theory just like that, while still partying and drinking beer.

I am not stupid and slow because I am lazy. I am stupid and slow because brain gets rusty from lack of excercise. There are many theories why this is the case – some call it “age”, some think that certain neural links in brain have harder time forming as we get older. Maybe. In any event, the message I have today, esteemed readers (all three of you) is: Apply yourself, and do the best you can do, and then some. Take that chance to advance, to learn something new, to do something. Because if the opportunity will knock next time, you might just not be able to keep up with it. And $DEITY forbid, don’t drop out of school, don’t get seduced by the “real world”. Yes, academentia is demented. But you can always take the parts that are interesting, and build on them. Who knows, maybe you will end up doing something that will be truly innovative in your field in the end. In worst case you will be able to look back, and not regret the fact that you didn’t try.

New Scientist has an 11 step guide to better brain. I wonder…. In my experience Nootropil works, but you need to take it 3 times a day for at least 3 weeks to start seeing effect. And because it’s not sold in Canada, supplies I bring over usually last for only so long *sigh*.

P.S. I guess we need a category “Rant” for occasional things like this.

Tiger: Differences between stock and Apple OpenSSH

stany 3 Comments

Adam asked me to look if my OpenSSH install also does SRV lookups when attempting to resolve hostnames.

Under Tiger (10.4.1, md5sum of the OpenSSH binary is b582a5b1da5999b6832dec6cb9477917 /usr/bin/ssh, OpenSSH_3.8.1p1, OpenSSL 0.9.7b 10 Apr 2003) it indeed behaves the same way as Adam describes.

Under Panther (10.3.9, md5sum of OpenSSH binary is 878ef654570e14c103a20b54afe3c417 /usr/bin/ssh OpenSSH_3.6.1p1+CAN-2004-0175, SSH protocols 1.5/2.0, OpenSSL 0x0090702f) I am not seeing any SRV lookups at all.

So I started investigating.

I’ve pulled own OpenSSH from Darwin 8.1 (corresponding to 10.4.1) from http://darwinsource.opendarwin.org/tarballs/other/OpenSSH-56.tar.gz, and the “correct” build from OpenSSH site

diff file was 940K in size, (vast majority of the differences were due to the fact that newer version of autoconf was run on Apple’s sources, and regenerated all the “configure” framework anew) and let me put it this way – OpenSSH as Apple ships it has a whole lot of differences compared to portable OpenSSH 3.8.1p1.

patches subdirectory of Apple tree has most of the patches, but something is telling me that possibly not all of them.

So how is SSH as shipped by Apple differs from SSH as shipped by the portable team of the OpenSSH project?

  • Support for BSM (Basic Security Module) framework, under Solaris, and under MacOS X.
    As an aside – what’s up with BSM and auditing under MacOS X? Solaris (OK, I looked at Solaris 8, maybe things changed in Sol 10) has /etc/security/* with things like audit_class, audit_event, tools to enable and disable auditing, etc. 104.1 has /var/audit that is empty (Obviously – Auditing is not enabled), and prints two lines during kernel boot-up: 

    Jun 24 04:13:15 localhost kernel[0]: Security auditing service present
Jun 24 04:13:15 localhost kernel[0]: BSM auditing present

    Anyone has any idea how to actually tweak what gets audited, etc? /usr/include/bsm/ exists, so technically one can attempt to build Solaris BSM tools, but what would Brian Costello^W^WApple do?

  • HEIMDAL support
  • CCAPI – Credentials Caching
  • Fix for Mindrot bug 874 – Swapped parameters of SSH_FXP_SYMLINK packet of SFTP protocol 
    +/*
+ * "Blind" conversation function for password authentication.  Assumes that
+ * echo-off prompts are for the password and stores messages for later
+ * display.
+ */
  • PAM support for password authentication.
  • Under MacOS X uses Security/AuthSession.h AuthSession – APIs for managing login, authorization, and security Sessions.
    from sshd.c:

    +#ifdef USE_SECURITY_SESSION_API
+        /*
+         * Create a new security session for use by the new user login if
+         * the current session is the root session or we are not launched
+         * by inetd (eg: debugging mode or server mode).  We do not
+         * necessarily need to create a session if we are launched from
+         * inetd because Panther xinetd will create a session for us.
+         *
+         * The only case where this logic will fail is if there is an
+         * inetd running in a non-root session which is not creating
+         * new sessions for us.  Then all the users will end up in the
+         * same session (bad).
+         *
+         * When the client exits, the session will be destroyed for us
+         * automatically.
+         *
+         * We must create the session before any credentials are stored
+         * (including AFS pags, which happens a few lines below).
+         */
  • functional GSSAPI tie-in – tied into PAM and BSM, and HEIMDAL – part of making OpenSSH in 10.4.x kerberized. By default is turned on (refer to sshd_config man page under Tiger, applies to Protocol 2 ONLY) Supports lack of hostkey, reverts to “null” method of keying (from sshd.c) 
    +#ifndef GSSAPI
+       /* The GSSAPI key exchange can run without a host key */
        if ((options.protocol & SSH_PROTO_2) && !sensitive_data.have_ssh2_key) {
                logit("Disabling protocol version 2. Could not load host key");
                options.protocol &= ~SSH_PROTO_2;
        }
+#endif
  • Support for using memberd for resolving group memberships and to see if Serivce ACLs permit user to use ssh
  • Capabilities support. in sshd_config: 
    +# SACL options
+#SACLSupport yes
  • Extended attributes and resource fork support when copying between two 10.4.x systems – option -E for scp, implementation in copyfile.h and scp.c. Seems like metadata gets collected into an additional file, that gets transfered same way as a file would be, at gets re-applied on the other end.

    • Note: openssh/compat.c contains a list of all the “known” implementations of SSH clients, and what bugs they have. Quite an interesting read.

That’s about all I’ve noticed.

Now, regarding SRV lookups…. I’ve not noticed anything magic in the source that causes that to happen. Maybe that’s part of GSSAPI stuff – I frankly weren’t looking too closely. Maybe it’s something that libSystem.B.dylib does on behalf of ssh. Further investigation is needed, as it didn’t jump out at me. Sorry, Adam.

Update: I guess I should have been clearer. I did compile stock OpenSSH 3.8.1p1, and saw what kind of DNS queries it attempted. If the system in question is in /etc/hosts, it does no DNS quieries. If the system is not in /etc/hosts, all it looks at is 

17:28:37.287350 IP 10.9.15.194.51980 > 10.9.15.1.domain:  52104+ A? www.epals.com. (31)
17:28:37.593401 IP 10.9.15.1.domain > 10.9.15.194.51980:  52104 1/2/2 A www.epals.com (128)
17:28:38.211709 IP 10.9.15.194.51981 > 10.9.15.1.domain:  55591+ PTR? 1.15.9.10.in-addr.arpa. (40)
17:28:38.212701 IP 10.9.15.1.domain > 10.9.15.194.51981:  55591 NXDomain 0/1/0 (117)
17:28:38.217308 IP 10.9.15.194.51982 > 10.9.15.1.domain:  6539+ PTR? 116.141.26.64.in-addr.arpa. (44)
17:28:38.333627 IP 10.9.15.1.domain > 10.9.15.194.51982:  6539 2/2/2 CNAME 116.96-127.141.26.64.in-addr.arpa., PTR www.epals.com. (190)

(Yes, I enjoy attempting to ssh to epals.com, as most assuredly it would not be an IP address in my /etc/hosts)

On the other hand, Apple’s implementation of OpenSSH does these regardless if the system in question is in /etc/hosts or not:

17:30:25.107046 IP 10.9.15.194.51989 > 10.9.15.1.domain:  50351+ SRV? _telnet._tcp.iskra.ottix.net. (46)
17:30:25.108158 IP 10.9.15.1.domain > 10.9.15.194.51989:  50351 NXDomain 0/1/0 (86)
17:30:25.108981 IP 10.9.15.194.51990 > 10.9.15.1.domain:  3246+ SRV? _telnet._tcp.iskra.ottix.net. (46)
17:30:25.109571 IP 10.9.15.194.51991 > 10.9.15.1.domain:  3821+ SRV? _telnet._tcp.iskra.ottix.net.bhwireless.com. (61)
17:30:25.110614 IP 10.9.15.1.domain > 10.9.15.194.51990:  3246 NXDomain 0/1/0 (86)
17:30:25.110937 IP 10.9.15.1.domain > 10.9.15.194.51991:  3821 NXDomain 0/1/0 (134)
17:30:25.111186 IP 10.9.15.194.51992 > 10.9.15.1.domain:  7928+ SRV? _telnet._tcp.iskra.ottix.net.bhwireless.com. (61)
17:30:25.112891 IP 10.9.15.1.domain > 10.9.15.194.51992:  7928 NXDomain 0/1/0 (134)
[...]
stany@gilva:~/src/ssh/openssh-3.8.1p1[05:30 PM]$ grep ottix /etc/hosts 
192.231.228.2   iskra.ottix.net www.ottix.net
stany@gilva:~/src/ssh/openssh-3.8.1p1[05:30 PM]$ uname -a
Darwin gilva.local 8.1.0 Darwin Kernel Version 8.1.0: Tue May 10 18:16:08 PDT 2005; root:xnu-792.1.5.obj~4/RELEASE_PPC Power Macintosh powerpc
stany@gilva:~/src/ssh/openssh-3.8.1p1[05:30 PM]$

Another interesting side effect: Usually Apple’s sshd (enabled in control panels -> sharing -> Remote Login) registers itself with Rendez-Vous/ZeroConf. I have a piece of software called Rawr-Endezvous (0.6.b3, with my modifications to Growl framework 0.7. Newer versions of it just keep on dieing for me whenever I change location or enable/disable service,so I keep on waiting when Jereme Knope will fix it), that throws up a pop-up on my screen whenever new service is discovered. If one is to disable Remote Login in Sharing control panel, and start Apple’s ssh by hand, it registers the service as ZeroConf. If one starts up a stock OpenSSHD compiled from source, it doesn’t. I wonder if part of the problem is Apple’s patch to enable zeroconf in OpenSSH. 

stany@gilva:~/src/ssh/openssh-3.8.1p1[05:38 PM]$ sudo ./sshd -f /etc/sshd_config -h /etc/ssh_host_key -h /etc/ssh_host_rsa_key -h /etc/ssh_host_dsa_key -d
debug1: sshd version OpenSSH_3.8.1p1
debug1: private host key: #0 type 0 RSA1
debug1: read PEM private key done: type RSA
debug1: private host key: #1 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #2 type 2 DSA
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
Generating 768 bit RSA key.
RSA key generation complete.

(Remove -d if you want sshd to run in daemon mode).

MacOSX: Upgrading firmware for Pioneer DVD drives

stany

I did a large and opinionated post earlier about benefits of using cheap DVD drives over things like Pioneer. There is a benefit of paying for Pioneer drive too – ability to flash the firmware under MacOS X.

The software one needs for it is DVRflash compiled for Mac OS X, and a firmware image containing the kernel code necessary to get the drive into kernel mode (that permits flashing) and the (patched) firmware. Kernel code from any version of the firmware would do, as it is only used during the flashing to get the drive into receptive state. Firmware, on the other hand, should probably be either newer then the one you have already, or at the very least same version but with different features.

For the firmware for your particular model of the drive, you should probably look at RPC1.org web site. Also worth looking at are Pioneerdvd and Gradius’s web pages.

Here is the actual flashing session. 

#include <stddisclaimer.h> /* Not responsible for anything! */

I’ve put the Pioneer DVR-105 drive into an external USB/FW enclosure connected over FW. Operating system is MacOS X 10.4.1 (Still PPC, not yet mactel :-), drive is connected over firewire.

First I determine that the software sees the drive (as root):

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:16 PM]# ./DVRFlash

DVRFlash v2.0 : Pioneer DVR firmware flasher
by Agent Smith et al.,  July 2004

Commandline:
  ./DVRFlash 

Device parameter was not given, detecting all DVR drives:

     Device : B:
     Vendor : PIONEER 
      Model : DVD-RW  DVR-105 
   Revision : 1.00

Now run DVRFlash again, from the command prompt, using
one of the device(s) listed above as first parameter

Press the Return key to exit

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:19 PM]#

At this point I know that drive B: is the drive I want (Drive A is presumably the built in Matsushita combo), so I run the software again, this time with the right arguments:

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:19 PM]# ./DVRFlash 
-f PIONEER  R5100004.133 R5100104.133 

DVRFlash v2.0 : Pioneer DVR firmware flasher
by Agent Smith et al.,  July 2004

                       DISCLAIMER

THIS PROGRAM IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.

THE ENTIRE RISK AS TO THE ABILITY OF THIS PROGRAM TO FLASH A
PIONEER OR COMPATIBLE DVR DRIVE IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY
SERVICING, REPAIR OR CORRECTION.

THIS PROGRAM IS NOT ENDORSED BY PIONEER CORPORATION OR ANY
COMPANY RESELLING PIONEER EQUIPMENT AS THEIR OWN BRAND

IF YOU UNDERSTAND THE RISKS ASSOCIATED WITH THIS PROGRAM AND
DISCHARGE BOTH THE AUTHOR AND PIONEER CORPORATION FROM ANY
DAMAGE OCCURING AS THE RESULT OF ITS USE, PLEASE INDICATE SO
BY ANSWERING THE FOLLOWING QUESTION:

Do you understand and agree to the statement above (y/n)?
y

Commandline:
  ./DVRFlash -f PIONEER R5100004.133 R5100104.133 


Drive Information:
  Description    - PIONEER DVD-RW  DVR-105 
  Firmware Rev.  - 1.00
  Firmware Date  - 02/10/10
  Manufacturer   - PIONEER  
Drive is in normal mode.

Are you sure you want to flash this drive (y/n)?
y

Switching drive to Kernel mode:
  Description    - PIONEER DVD-RW  DVR-105 
  Firmware Rev.  - 0000
  Firmware Date  - 00/00/00
  Manufacturer   - PIONEER  
Drive is now in Kernel mode

Now sending the Kernel part...
Now internal Kernel reflashing. Please wait... OK.

Now sending the Normal part:
0%          25%          50%          75%         100%
|============|============|============|============|
Please hold your breath for about 30 seconds...

Now internal reflashing. Please wait... OK.

Updated Information:
  Description    - PIONEER DVD-RW  DVR-105 
  Firmware Rev.  - 1.33
  Firmware Date  - 03/05/26
  Manufacturer   - PIONEER  
Flashing operation successful ;)

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:20 PM]#

So after holding my breath for about 30 seconds, it finished. I went ahead and verified that firmware got updated:

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:20 PM]# ./DVRFlash 

DVRFlash v2.0 : Pioneer DVR firmware flasher
by Agent Smith et al.,  July 2004

Commandline:
  ./DVRFlash 

Device parameter was not given, detecting all DVR drives:

     Device : B:
     Vendor : PIONEER 
      Model : DVD-RW  DVR-105 
   Revision : 1.33

Now run DVRFlash again, from the command prompt, using
one of the device(s) listed above as first parameter

Press the Return key to exit

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:21 PM]#