Author: stany

CF and IO undefined symbols

stany

When building something probably written for Unix (I were fudging s10sh), you might eventually end up with undefined symbols such as:

gcc -O2 -Wall -g -I./libusb -o s10sh main.o crc.o usb.o serial.o common.o 
bar.o  -lreadline -ltermcap libusb/.libs/libusb.a
ld: Undefined symbols:
_CFRunLoopAddSource
_CFRunLoopGetCurrent
_CFRunLoopRun
_CFRunLoopStop
_CFUUIDGetConstantUUIDWithBytes
_CFUUIDGetUUIDBytes
_IOCreatePlugInInterfaceForService
_IOIteratorNext
_IOMasterPort
_IONotificationPortCreate
_IONotificationPortGetRunLoopSource
_IOObjectRelease
_IOServiceAddMatchingNotification
_IOServiceMatching
_kCFRunLoopDefaultMode
make: *** [s10sh] Error 1

The IOService errors can be fixed by -lIOKit thusly:

stany@gilva:~/src/s10sh-0.2.2[07:52 PM]$ gcc -O2 -Wall -g -I./libusb -o s10sh 
main.o crc.o usb.o serial.o common.o bar.o  -lreadline -ltermcap libusb/.libs/libusb.a -lIOKit
ld: Undefined symbols:
_CFRunLoopAddSource
_CFRunLoopGetCurrent
_CFRunLoopRun
_CFRunLoopStop
_CFUUIDGetConstantUUIDWithBytes
_CFUUIDGetUUIDBytes
_kCFRunLoopDefaultMode
stany@gilva:~/src/s10sh-0.2.2[07:53 PM]$

but the “proper” way to fix it is:

stany@gilva:~/src/s10sh-0.2.2[07:53 PM]$ gcc -O2 -Wall -g -I./libusb -o s10sh 
main.o crc.o usb.o serial.o common.o bar.o  -lreadline -ltermcap libusb/.libs/libusb.a  -framework IOKit -framework CoreFoundation
stany@gilva:~/src/s10sh-0.2.2[07:53 PM]$ 

stany@gilva:~/src/s10sh-0.2.2[07:54 PM]$ ./s10sh -u
USB mode enabled
S10sh -- version 0.2.2
Copyright (C) 2000-2001 by Salvatore Sanfilippo 
S10sh is FREE SOFTWARE under the terms of the GNU public license

[Canon PowerShot A75] > ls
ls error
[Canon PowerShot A75] >

*sigh*

Promise UltraTrak100 TX8 and rebuilding RAID

stany 1 Comment

This is from memory, as it happened about a year ago, but I figured I’d document it, in case it helps someone.

I own a Promise UltraTrak100 TX8 SCSI to IDE RAID array. If it helps, here are local mirrors of product manual and specifications.

I had a case once, when I shut the array down, and one of the drives did not spin back up.

Situation:

disk0: Good
disk1: Good
disk2: Did not Spin up
disk3: Good
disk4: Good
disk5: Good
disk6: Good
disk7: Good

Array (configured as 8 disk RAID5, Maxtor 120 gig drives, 800.5 GB of formated disk space) of course started beeping, so I grumbled, yanked a cold spare off the shelf, and put in in place of “failed” drive and went to sleep.

Situation:

disk0: Good
disk1: Good
disk2: replaced with good one, put onto shelf, resyncing
disk3: Good
disk4: Good
disk5: Good
disk6: Good
disk7: Good

About 3 hours into raid resync (no, UltraTrak100s are not really speedy), array instead of short beeps raised a rucus, and it’s crying woke me up. Turned out that another drive failed while into resync. So the nightmare happened – there were two failed drives in a RAID5, and of course the array is not designed to handle this.

Situation:

disk0: Good
disk1: Good
disk2: resyncing
disk3: Good
disk4: Good
disk5: failed with bad sectors
disk6: Good
disk7: Good

I had not backups. As an aside, when you have 800 gigs of on-line storage, all used, how do you back it up? DLT7K (which I also have) would take maybe 3 days, and at this point, do I trust the tapes? After all, when you have 20 tapes, probability of tape read failure would be raised to n^20. Then there is dust in the drive, SCSI cables (differential SCSI in my case), power fluctuations, etc. The only way to back up 1 TB is to put a second 1TB array near it, and mirror them, and start using filesystem snapshots (like NetApp does, or Solaris 8 and newer). Any way, backups are a subject of a rant of it’s own.

So I grumled, and cursed, but went ahead and examined the original drive, one that didn’t spin up. SMART was complaining that the drive takes too long to spin up, but in the end I managed to convince it to spin up. So now I had a case where I had an array with two “bad” drives, yet one drive was actually “good”, only market as bad in the NVRAM of the UltraTrak.

After a while on long distance calls to Promise, I got to talk to a chinese guy who actually was one of the developers. He told me of a magic way to try as last resort.

So don’t do this at home, this is serious evil, etc.

He told me to turn array off, yank all the drives out of the array, and put one new drive into it.
Upon power on, array would complain about lack of the original drives. Then he told me to delete the existing configuration, and power the array off.

After that, he told me to put the drives including drive that was originally having problems spinning up back into the array in the original order:

Situation:

disk0: Good
disk1: Good
disk2: Drive that not Spin up originally, but got convinced to spin up again
disk3: Good
disk4: Good
disk5: failed with bad sectors
disk6: Good
disk7: Good

Then he told me to go and configure the array again from scratch, RAID5, whole disks, etc.
At the moment when I were to commit the configuration of the array I had to be careful. Essentially at that point all of the lights on the disks in the array would flash in sequence, as the configuration of the array would be written to disks. After that there would be a 1 second pause. During it, I had to turn the array off.

This is a once time shot. If one doesn’t turn the array off at this 1 second interval, the array would proceed with formatting the disks, and all of the data would be lost.

As I did it, array wrote configuration of the array to disks, matching the configuration that I had before, but did not re-initialize the array. So the data was still there.

When I powered the array on, it span up all the drives, and proceeded to claim that it’s fully functional.

So I manually failed drive 5, that had bad sectors on it by yanking it out of the array, and replacing it with a cold spare.
About 10 hours later array re-initialized. Then I failed disk2, that had issues spinning up, and replaced it. Array re-initialized.

You have no idea how stressed I were until the first rebuild was done.

Any way, maybe this will help someone. Obviously this is not exactly a technique for the faint of heart, and is not supported by Promise. But it saved my ass. If you have spare disks, try building a test array (of like 2 disks) and practice on it first. And, have good backups.

This should work on UltraTrak100 TX4 as well, but I have no idea about any other models. Probably not. Talk to Promise, they can be nice to you.

This message will self-destruct in five….

stany

Many years ago (well, 1998, to be exact) when the science of computer forensics was basically in it’s infancy, there already were rumors of various TLAs being able to read data that was overwritten on the hard drive. Some folks were telling of tunneling microscopes, and other high tech gizmos that could recover data up to 7 overwritions ago. I don’t know how true it is, however am inclined to believe that it is true. Now a days, products such as Encase can do wonders, and Linux based and occasionally open source tools are close behind commercial vendors. Forensics field turned into science, with an entire industry to support it – hardware write blockers, special court proceedings, expert witnesses, data recovery software, etc. One of these days…

This time I’ll just address a simple question: What media should one store the data on, if one expects that one would need to destroy the data on the media some time in the future, and adversary with great financial and technical resources would be interested in reconstructing the data?

For something like this, I’d recommend rewritable CDs and DVDs. Primary reason is ease of data disposal – if one has 30 seconds to get rid of incriminating evidence, all one really has to do is to drop a styrofoam cup half-full of water with a CD on it into microwave, and tell it to reheat.

While I don’t know about recent models of microwaves, older systems would generate a satisfying arcing and media destruction in about 10 seconds. Media will not melt to slug, however the data layer would be covered by a spider web of cracks, that would pretty much be rendered unusable. Here is an example. A google search for “microwave CD” should provide plenty more links to images.

Now, to a physicist in me this looks like a rather complete way of getting rid of unwanted data irrecoverably.

However, this is a reason why I suggested CD-RW and DVD-RW in the first place:

As with CD-Rs, the read laser does not have enough power to change the state of the material in the recording layer — it’s a lot weaker than the write laser. The erase laser falls somewhere in between: While it isn’t strong enough to melt the material, it does have the necessary intensity to heat the material to the crystallization point. By holding the material at this temperature, the erase laser restores the compound to its crystalline state, effectively erasing the encoded 0. This clears the disc so new data can be encoded.

So my advice to dissidents world-wide – first erase the DVD-RW and CD-RWs, and then microwave them. After that, toss them out and don’t toss and turn while in bed 😛

Edit1: Note! When I talk about erasing CD-R or DVD-R, I mean about full erase, that takes ~15 minutes, NOT quick erase. Quick erase generally just zeros out first megabyte of data on disk, including TOC, so it seems like it’s a clean disk, yet all of the previously recorded data is still there!

Edit2: I wonder if the crystalline properties of the layer change from being melted and re-cooled during the erasure process. In other words, is it still possible to detect where data was based on the different structure of the “re-flowed” layer after erasure? Any material scientists around? 😛

That’s why I recommend microwaving of the disk, just to make reconstruction of the data just that much harder.

Microsoft Certified What?

stany

From pktech:

Arfa Karim Randhwa of Multan, Faisalabad, Pakistan has been accredited as the youngest Microsoft Certified Professional (MCP) at age 9. The requirements for MCP are by no means trivial. No details on which exam she wrote but Desktop Support Technician seems like one of the easier routes to MCP with 4 courses requiring installation, management and troubleshooting knowledge of the windows environment (hardware, filesystem, networking etc) and applications (office, outlook, etc). She studied through Applied Technologies (APTECH) (whose website is a disgrace) in a single summer vacation. She comes from a rural, agricultural background with her father working for United Nations in Congo and her mother acting as the landlady in his absence. She also won a national singing competition. Not an astonishing story, but it does set an excellent precedent for Pakistani youth — a little encouragement goes a long way. Pakistan could use a lot more intellectual capital. What can be done to promote the youth and build intellectual capital?

Personally, between her and Andy, I’d bet on her. Not because I have little faith in Andy, no. I’d bet on her because she’s younger, and if she is already where Andy is getting to, then she has additional 25 – 30 years to get even further ahead. I remember myself when I were working for iStar. 17, prime condition, were going to school from 8:30 to 15:30, worked from 16:00 to 2:00am, sleeping 5 hours a day, if not less, learning computer security, Solaris system administration and Cisco routing on my own time and that was when? Less then 10 years ago. Now a days, an allnighter would probably kill me, and I look in awe at folks who are 19 – 20 and are able to grok Galois Theory just like that, while still partying and drinking beer.

I am not stupid and slow because I am lazy. I am stupid and slow because brain gets rusty from lack of excercise. There are many theories why this is the case – some call it “age”, some think that certain neural links in brain have harder time forming as we get older. Maybe. In any event, the message I have today, esteemed readers (all three of you) is: Apply yourself, and do the best you can do, and then some. Take that chance to advance, to learn something new, to do something. Because if the opportunity will knock next time, you might just not be able to keep up with it. And $DEITY forbid, don’t drop out of school, don’t get seduced by the “real world”. Yes, academentia is demented. But you can always take the parts that are interesting, and build on them. Who knows, maybe you will end up doing something that will be truly innovative in your field in the end. In worst case you will be able to look back, and not regret the fact that you didn’t try.

New Scientist has an 11 step guide to better brain. I wonder…. In my experience Nootropil works, but you need to take it 3 times a day for at least 3 weeks to start seeing effect. And because it’s not sold in Canada, supplies I bring over usually last for only so long *sigh*.

P.S. I guess we need a category “Rant” for occasional things like this.

Tiger: Differences between stock and Apple OpenSSH

stany 3 Comments

Adam asked me to look if my OpenSSH install also does SRV lookups when attempting to resolve hostnames.

Under Tiger (10.4.1, md5sum of the OpenSSH binary is b582a5b1da5999b6832dec6cb9477917 /usr/bin/ssh, OpenSSH_3.8.1p1, OpenSSL 0.9.7b 10 Apr 2003) it indeed behaves the same way as Adam describes.

Under Panther (10.3.9, md5sum of OpenSSH binary is 878ef654570e14c103a20b54afe3c417 /usr/bin/ssh OpenSSH_3.6.1p1+CAN-2004-0175, SSH protocols 1.5/2.0, OpenSSL 0x0090702f) I am not seeing any SRV lookups at all.

So I started investigating.

I’ve pulled own OpenSSH from Darwin 8.1 (corresponding to 10.4.1) from http://darwinsource.opendarwin.org/tarballs/other/OpenSSH-56.tar.gz, and the “correct” build from OpenSSH site

diff file was 940K in size, (vast majority of the differences were due to the fact that newer version of autoconf was run on Apple’s sources, and regenerated all the “configure” framework anew) and let me put it this way – OpenSSH as Apple ships it has a whole lot of differences compared to portable OpenSSH 3.8.1p1.

patches subdirectory of Apple tree has most of the patches, but something is telling me that possibly not all of them.

So how is SSH as shipped by Apple differs from SSH as shipped by the portable team of the OpenSSH project?

  • Support for BSM (Basic Security Module) framework, under Solaris, and under MacOS X.
    As an aside – what’s up with BSM and auditing under MacOS X? Solaris (OK, I looked at Solaris 8, maybe things changed in Sol 10) has /etc/security/* with things like audit_class, audit_event, tools to enable and disable auditing, etc. 104.1 has /var/audit that is empty (Obviously – Auditing is not enabled), and prints two lines during kernel boot-up: 

    Jun 24 04:13:15 localhost kernel[0]: Security auditing service present
Jun 24 04:13:15 localhost kernel[0]: BSM auditing present

    Anyone has any idea how to actually tweak what gets audited, etc? /usr/include/bsm/ exists, so technically one can attempt to build Solaris BSM tools, but what would Brian Costello^W^WApple do?

  • HEIMDAL support
  • CCAPI – Credentials Caching
  • Fix for Mindrot bug 874 – Swapped parameters of SSH_FXP_SYMLINK packet of SFTP protocol 
    +/*
+ * "Blind" conversation function for password authentication.  Assumes that
+ * echo-off prompts are for the password and stores messages for later
+ * display.
+ */
  • PAM support for password authentication.
  • Under MacOS X uses Security/AuthSession.h AuthSession – APIs for managing login, authorization, and security Sessions.
    from sshd.c:

    +#ifdef USE_SECURITY_SESSION_API
+        /*
+         * Create a new security session for use by the new user login if
+         * the current session is the root session or we are not launched
+         * by inetd (eg: debugging mode or server mode).  We do not
+         * necessarily need to create a session if we are launched from
+         * inetd because Panther xinetd will create a session for us.
+         *
+         * The only case where this logic will fail is if there is an
+         * inetd running in a non-root session which is not creating
+         * new sessions for us.  Then all the users will end up in the
+         * same session (bad).
+         *
+         * When the client exits, the session will be destroyed for us
+         * automatically.
+         *
+         * We must create the session before any credentials are stored
+         * (including AFS pags, which happens a few lines below).
+         */
  • functional GSSAPI tie-in – tied into PAM and BSM, and HEIMDAL – part of making OpenSSH in 10.4.x kerberized. By default is turned on (refer to sshd_config man page under Tiger, applies to Protocol 2 ONLY) Supports lack of hostkey, reverts to “null” method of keying (from sshd.c) 
    +#ifndef GSSAPI
+       /* The GSSAPI key exchange can run without a host key */
        if ((options.protocol & SSH_PROTO_2) && !sensitive_data.have_ssh2_key) {
                logit("Disabling protocol version 2. Could not load host key");
                options.protocol &= ~SSH_PROTO_2;
        }
+#endif
  • Support for using memberd for resolving group memberships and to see if Serivce ACLs permit user to use ssh
  • Capabilities support. in sshd_config: 
    +# SACL options
+#SACLSupport yes
  • Extended attributes and resource fork support when copying between two 10.4.x systems – option -E for scp, implementation in copyfile.h and scp.c. Seems like metadata gets collected into an additional file, that gets transfered same way as a file would be, at gets re-applied on the other end.

    • Note: openssh/compat.c contains a list of all the “known” implementations of SSH clients, and what bugs they have. Quite an interesting read.

That’s about all I’ve noticed.

Now, regarding SRV lookups…. I’ve not noticed anything magic in the source that causes that to happen. Maybe that’s part of GSSAPI stuff – I frankly weren’t looking too closely. Maybe it’s something that libSystem.B.dylib does on behalf of ssh. Further investigation is needed, as it didn’t jump out at me. Sorry, Adam.

Update: I guess I should have been clearer. I did compile stock OpenSSH 3.8.1p1, and saw what kind of DNS queries it attempted. If the system in question is in /etc/hosts, it does no DNS quieries. If the system is not in /etc/hosts, all it looks at is 

17:28:37.287350 IP 10.9.15.194.51980 > 10.9.15.1.domain:  52104+ A? www.epals.com. (31)
17:28:37.593401 IP 10.9.15.1.domain > 10.9.15.194.51980:  52104 1/2/2 A www.epals.com (128)
17:28:38.211709 IP 10.9.15.194.51981 > 10.9.15.1.domain:  55591+ PTR? 1.15.9.10.in-addr.arpa. (40)
17:28:38.212701 IP 10.9.15.1.domain > 10.9.15.194.51981:  55591 NXDomain 0/1/0 (117)
17:28:38.217308 IP 10.9.15.194.51982 > 10.9.15.1.domain:  6539+ PTR? 116.141.26.64.in-addr.arpa. (44)
17:28:38.333627 IP 10.9.15.1.domain > 10.9.15.194.51982:  6539 2/2/2 CNAME 116.96-127.141.26.64.in-addr.arpa., PTR www.epals.com. (190)

(Yes, I enjoy attempting to ssh to epals.com, as most assuredly it would not be an IP address in my /etc/hosts)

On the other hand, Apple’s implementation of OpenSSH does these regardless if the system in question is in /etc/hosts or not:

17:30:25.107046 IP 10.9.15.194.51989 > 10.9.15.1.domain:  50351+ SRV? _telnet._tcp.iskra.ottix.net. (46)
17:30:25.108158 IP 10.9.15.1.domain > 10.9.15.194.51989:  50351 NXDomain 0/1/0 (86)
17:30:25.108981 IP 10.9.15.194.51990 > 10.9.15.1.domain:  3246+ SRV? _telnet._tcp.iskra.ottix.net. (46)
17:30:25.109571 IP 10.9.15.194.51991 > 10.9.15.1.domain:  3821+ SRV? _telnet._tcp.iskra.ottix.net.bhwireless.com. (61)
17:30:25.110614 IP 10.9.15.1.domain > 10.9.15.194.51990:  3246 NXDomain 0/1/0 (86)
17:30:25.110937 IP 10.9.15.1.domain > 10.9.15.194.51991:  3821 NXDomain 0/1/0 (134)
17:30:25.111186 IP 10.9.15.194.51992 > 10.9.15.1.domain:  7928+ SRV? _telnet._tcp.iskra.ottix.net.bhwireless.com. (61)
17:30:25.112891 IP 10.9.15.1.domain > 10.9.15.194.51992:  7928 NXDomain 0/1/0 (134)
[...]
stany@gilva:~/src/ssh/openssh-3.8.1p1[05:30 PM]$ grep ottix /etc/hosts 
192.231.228.2   iskra.ottix.net www.ottix.net
stany@gilva:~/src/ssh/openssh-3.8.1p1[05:30 PM]$ uname -a
Darwin gilva.local 8.1.0 Darwin Kernel Version 8.1.0: Tue May 10 18:16:08 PDT 2005; root:xnu-792.1.5.obj~4/RELEASE_PPC Power Macintosh powerpc
stany@gilva:~/src/ssh/openssh-3.8.1p1[05:30 PM]$

Another interesting side effect: Usually Apple’s sshd (enabled in control panels -> sharing -> Remote Login) registers itself with Rendez-Vous/ZeroConf. I have a piece of software called Rawr-Endezvous (0.6.b3, with my modifications to Growl framework 0.7. Newer versions of it just keep on dieing for me whenever I change location or enable/disable service,so I keep on waiting when Jereme Knope will fix it), that throws up a pop-up on my screen whenever new service is discovered. If one is to disable Remote Login in Sharing control panel, and start Apple’s ssh by hand, it registers the service as ZeroConf. If one starts up a stock OpenSSHD compiled from source, it doesn’t. I wonder if part of the problem is Apple’s patch to enable zeroconf in OpenSSH. 

stany@gilva:~/src/ssh/openssh-3.8.1p1[05:38 PM]$ sudo ./sshd -f /etc/sshd_config -h /etc/ssh_host_key -h /etc/ssh_host_rsa_key -h /etc/ssh_host_dsa_key -d
debug1: sshd version OpenSSH_3.8.1p1
debug1: private host key: #0 type 0 RSA1
debug1: read PEM private key done: type RSA
debug1: private host key: #1 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #2 type 2 DSA
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
Generating 768 bit RSA key.
RSA key generation complete.

(Remove -d if you want sshd to run in daemon mode).

MacOSX: Upgrading firmware for Pioneer DVD drives

stany

I did a large and opinionated post earlier about benefits of using cheap DVD drives over things like Pioneer. There is a benefit of paying for Pioneer drive too – ability to flash the firmware under MacOS X.

The software one needs for it is DVRflash compiled for Mac OS X, and a firmware image containing the kernel code necessary to get the drive into kernel mode (that permits flashing) and the (patched) firmware. Kernel code from any version of the firmware would do, as it is only used during the flashing to get the drive into receptive state. Firmware, on the other hand, should probably be either newer then the one you have already, or at the very least same version but with different features.

For the firmware for your particular model of the drive, you should probably look at RPC1.org web site. Also worth looking at are Pioneerdvd and Gradius’s web pages.

Here is the actual flashing session. 

#include <stddisclaimer.h> /* Not responsible for anything! */

I’ve put the Pioneer DVR-105 drive into an external USB/FW enclosure connected over FW. Operating system is MacOS X 10.4.1 (Still PPC, not yet mactel :-), drive is connected over firewire.

First I determine that the software sees the drive (as root):

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:16 PM]# ./DVRFlash

DVRFlash v2.0 : Pioneer DVR firmware flasher
by Agent Smith et al.,  July 2004

Commandline:
  ./DVRFlash 

Device parameter was not given, detecting all DVR drives:

     Device : B:
     Vendor : PIONEER 
      Model : DVD-RW  DVR-105 
   Revision : 1.00

Now run DVRFlash again, from the command prompt, using
one of the device(s) listed above as first parameter

Press the Return key to exit

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:19 PM]#

At this point I know that drive B: is the drive I want (Drive A is presumably the built in Matsushita combo), so I run the software again, this time with the right arguments:

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:19 PM]# ./DVRFlash 
-f PIONEER  R5100004.133 R5100104.133 

DVRFlash v2.0 : Pioneer DVR firmware flasher
by Agent Smith et al.,  July 2004

                       DISCLAIMER

THIS PROGRAM IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.

THE ENTIRE RISK AS TO THE ABILITY OF THIS PROGRAM TO FLASH A
PIONEER OR COMPATIBLE DVR DRIVE IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY
SERVICING, REPAIR OR CORRECTION.

THIS PROGRAM IS NOT ENDORSED BY PIONEER CORPORATION OR ANY
COMPANY RESELLING PIONEER EQUIPMENT AS THEIR OWN BRAND

IF YOU UNDERSTAND THE RISKS ASSOCIATED WITH THIS PROGRAM AND
DISCHARGE BOTH THE AUTHOR AND PIONEER CORPORATION FROM ANY
DAMAGE OCCURING AS THE RESULT OF ITS USE, PLEASE INDICATE SO
BY ANSWERING THE FOLLOWING QUESTION:

Do you understand and agree to the statement above (y/n)?
y

Commandline:
  ./DVRFlash -f PIONEER R5100004.133 R5100104.133 


Drive Information:
  Description    - PIONEER DVD-RW  DVR-105 
  Firmware Rev.  - 1.00
  Firmware Date  - 02/10/10
  Manufacturer   - PIONEER  
Drive is in normal mode.

Are you sure you want to flash this drive (y/n)?
y

Switching drive to Kernel mode:
  Description    - PIONEER DVD-RW  DVR-105 
  Firmware Rev.  - 0000
  Firmware Date  - 00/00/00
  Manufacturer   - PIONEER  
Drive is now in Kernel mode

Now sending the Kernel part...
Now internal Kernel reflashing. Please wait... OK.

Now sending the Normal part:
0%          25%          50%          75%         100%
|============|============|============|============|
Please hold your breath for about 30 seconds...

Now internal reflashing. Please wait... OK.

Updated Information:
  Description    - PIONEER DVD-RW  DVR-105 
  Firmware Rev.  - 1.33
  Firmware Date  - 03/05/26
  Manufacturer   - PIONEER  
Flashing operation successful ;)

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:20 PM]#

So after holding my breath for about 30 seconds, it finished. I went ahead and verified that firmware got updated:

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:20 PM]# ./DVRFlash 

DVRFlash v2.0 : Pioneer DVR firmware flasher
by Agent Smith et al.,  July 2004

Commandline:
  ./DVRFlash 

Device parameter was not given, detecting all DVR drives:

     Device : B:
     Vendor : PIONEER 
      Model : DVD-RW  DVR-105 
   Revision : 1.33

Now run DVRFlash again, from the command prompt, using
one of the device(s) listed above as first parameter

Press the Return key to exit

root@gilva:~/Desktop/DVRFlash_2.0/DVR-105 v1.33 - RPC-1 + 2xDVD-R + 12xRip[05:21 PM]#

Merging a bunch of PDFs together

stany

A couple of days ago one of the questions I asked was for an easy (and preferably command line scriptable) way to merge a bunch of PDF files together. Well, I think I found a way.

MonkeyBread Software makes RealBasic plugins and extensions. I’ll be the first one to say that I don’t know jack about RealBasic, however one of the freely downloadable tools that they provice is Combine PDFs (They even include RealBasic source). It’s a tiny carbon app, that basically does what I want it to do.

It has an interesting “feature” – it seems to get rid of the “Image Unavailable for Copyright Reasons” watermark when dealing with PDF files generated by NPG. So I just get white blocks with occasional capture under the text. But hey, it’s free, so who am I to complain?

One of the tricks I use while using Merge PDFs is to rename a bunch of PDFs into numerically ordered list, something like: 

$ grep pdf index.html| sed regular expression or three go here to result in file list 
 | nl -v100 | awk '{print "mv "$2" "$1".pdf"}' | sh

where I basically use nl(1) to start labeling the lines with 100 and counting onwards.

Then inside Combine PDFs I can just tell it to order files in alphabetical order, and off I go.

Here is what a real run would look like: 

stany@gilva:~/nature/www.nature.com/nature/journal/v435/n7043[06:56 PM]$ 
cat index.html | grep  pdf | sed 's/^.*href.................................//g' | 
sed 's/......$//g' | nl -v100  | head
   100  435713a.pdf
   101  435713b.pdf
   102  435714a.pdf
   103  435716a.pdf
   104  435718a.pdf
   105  435718b.pdf
   106  435720a.pdf
   107  435720b.pdf
   108  435723a.pdf
   109  435723b.pdf
stany@gilva:~/nature/www.nature.com/nature/journal/v435/n7043[06:56 PM]$ 
cat index.html | grep  pdf |  sed 's/^.*href.................................//g' | 
sed 's/......$//g' | nl -v100 | awk '{print "mv pdf/"$2" pdf/"$1".pdf"}' | head
mv pdf/435713a.pdf pdf/100.pdf
mv pdf/435713b.pdf pdf/101.pdf
mv pdf/435714a.pdf pdf/102.pdf
mv pdf/435716a.pdf pdf/103.pdf
mv pdf/435718a.pdf pdf/104.pdf
mv pdf/435718b.pdf pdf/105.pdf
mv pdf/435720a.pdf pdf/106.pdf
mv pdf/435720b.pdf pdf/107.pdf
mv pdf/435723a.pdf pdf/108.pdf
mv pdf/435723b.pdf pdf/109.pdf
stany@gilva:~/nature/www.nature.com/nature/journal/v435/n7043[06:57 PM]$

You get the idea.

Then it’s just drag and drop.

I’ve still not found a free way to delete duplicate pages, however PDFpen looks reasonably good (It has a problem with inability to preview the large page and the thumbnails of the rest of the pages in the file at the same time, and the interface for deleting pages is not obvious, but maybe I should contact the authors). It is 50$ USD for the basic version (And I don’t need form creation either), which is much better then fill Acrobat from Adobe.

I should contact the authors, and see if they will add the features I would like, and if they do, register the software. Hrm….

As my Spanish teacher used to say: necesito ganar dinero.

“Image unavailable for copyright reasons”

stany

Background

For the last few years I’ve been subscribing to Nature Publishing Group’s Nature magazine. It is a scientific, peer-reviewed weekly magazine that publishes some of the best and ground breaking papers in hard sciences (And when I say “science”, I don’t mean “English Literature”) – biology, chemistry, material sciences, physics, occasional psychology and mathematics.

It is a great magazine, and probably the only real competition it has in the field is Science, published by AAAS (American Association for the Advancement of Science). AAAS’s Science is also offered as a digital download (and in fact they offer subscription with a significant discount), however it only offers it in a Zinio format. Zinio, on the other hand, is infamous for having DRM in the downloaded files, giving original content providers ability to expire the document after certain time. So in reality I don’t own content, like I do when I buy a magazine, but rent it, and they can pull the plug at any moment.

As an aside, I am seeing that at least some flavors of PDF can be configured not to open after a certain date, however I wonder how widespread the practice is, and how “portable”such documents are, as, presumably, one would need an extension to open such a PDF. Maybe Zinio is in fact an overgrown extension to PDF documents, with a fancy page turning animation? Here is a thought. Link to the PPT presentation, on page 10 says regarding benefits of PDF: Security. Allows multiple security settings from fully editable to print only access. Files can be set to expire (cannot open past expire date).

One of the benefits of a personal subscription to Nature is ability to download any article that they publish as a PDF file once you authenticate and get a cookie loaded into the browser (If you are a student, you can probably see if your university has a site license for Nature and Science, and if you can access their web sites through your university’s web proxy). As Nature is a weekly publication, 52 issues, each 7 – 8 mm thick, start eating up space on the bookshelf rather rapidly, and weigh a good deal too. As a result, back, when I had more time on my hands, I were actually downloading all the PDF pieces for each issue I’d recieve, glue them together into a single “issue” about 10 megs big, and drop it as a single file on my hard drive. Eventually I’ve stopped doing it, because it was taking a fair bit of my time, and attempts to convince my little brother to do it for me (I were offering up to 2$ per issue) were not getting anywhere.

I still upkeep my subscription, and occasionally do have time to read both Nature Science Update online and leaf through the actual magazine.

Image unavailable for copyright reasons

So I went and looked at the PDFs on Nature site today (first time in a few months), and I’ve noticed something that is new to me. On the PDF versions of some pages, more noticably on the feature articles (which are 2 – 3 page articles that describe in depth a particular aspect of science), some images have been replaced with boxes saying “IMAGE UNAVAILABLE FOR COPYRIGHT REASONS”.

Before, the mail difference between the PDF and printed version was lack of advertisment (well, most of the time, sometimes they would goof, and you’d get to see a half page ad somewhere), as NPG was a firm believer that in order to advertise in PDF version advertiser should pay. Occasionally Nature would retract an article, and then the PDF to it would be removed, and on occasion other pages, where article began and ended, censored too.

Now some images are missing. Here is an example. (480K, sorry, I didn’t feel like cropping it).

Note that this is new – I went and doublechecked archives, and older stories don’t have this “feature”. Of course this can change – maybe they didn’t yet have enough man power to go and look through the back archives.

Why? Why would they do something like that?

Maybe they got nailed by their stock image library. Maybe some photographer took them to court. It’s the larger images that seem to be unavailable, so maybe some kid took the image from some Nature issue, and used it in a high school project.

This kind of crap upsets me.

Spotlight and PDFs

Now, while talking about Spotlight, I started thinking about possibilities…. Indexing entire hard drive is evil (Or, in my case eats too many CPU/IO cycles), so for now I’ve disabled Spotlight in my /etc/hostconfig. However, if I were to create a single subdirectory (Or Partition, or, heck, external drive) for documents, turn off indexing for boot volume, turn off indexing for this new volume/subdirectory/partition (from now on: PDF repository), then, once I’ve added/modified the PDF content sufficiently, tell it to index the contents once (but not continuously) using mdimport(1), I’ll get all the benefits of Spotlight/mdfind(1) with none of the slowdowns for the documents in repository (That I presumably want indexed). So I could have the cake and eat it too.

So I started looking at various options. Dave’s pointer about using wget with cookies seems like the right step forward. I’ll have to make sure that I tell wget to send the same User-Agent string as my browser does, as I recall from older days that folks at Nature actually keep track of that.

I’ve not tried it yet, but I’ll be seeing what I can do.

So here are questions that I don’t know answers for:

What is the easy way to join a bunch of PDFs into a single PDF? Bonus points if it’s something that can be done from command line, maybe as a batch.

Also, is there an easy way to screen out duplicate pages in a PDF, preferably not involving human iunteraction? Under Acrobat 4 (or 3) it was rather simple – I’d just generate previews for each page, and then click on whichever look similar, and kill them, but that requred at least glancing through the PDF document.

Zinio reader uses it’s own format, that is heavily DRMed, and, as far as I can tell, might actually be based on PDF, as they licensed the PDF library from Adobe. So, question I have is: Is there a tool that will strip the DRM, and generate a normal PDF out of Zinio file? Best solution so far that I found was to print each page to PDF individually, and then basically merge pieces together, but this is ugly as sin.

Lastly Any better suggestions on how to deal with spidering sites like Nature’s, and pulling down certain types of content? Maybe I shouldn’t try to re-invent the wheel.

As usual, feel free to post comments (all 3 people and 100 comment spam bots that occasionally look at the site) 🙂

Dual Layer DVD burners in PowerMac G5s

stany

Andy called my “employer” today, and asked us to find out for him what dual layer burners are in PM G5s. So of course the question percolated down to me, without the associated name attached to the question.

Apple ships different burners in different batches of systems, depending on which manufacturer gives Apple a better deal. So new PM G5s can come with either SONY DW-Q28A or Pioneer DVR-A09 (Which is just an Apple branded version of Pioneer DVR-109, and has no functional or firmware differences).

While I can understand why someone might want an Apple Shipped/Apple Supported DVD burner, the benefts of such support are in reality rather slim. Apple will support CD burning on either Apple Shipped or Unsupported DVD burner, as licensing is limited to DVD support. Ditto with booting (Booting is actually something that starts regardless of the OS, as it’s triggered by OpenFirmware. Thus as long as device supports standard ATAPI command set, it can be used for booting). So in reality all one loses is lack of DVD burning from Disk Utility, iTunes and things like iDVD.

What I recommend is buying whatever is the cheapest dual layer burner you can find that has patched firmware available from download from rpc1.org, and then using Patchburn to install a profile, turning the device into “Vendor Supported”, and reenabling burning from iTunes, Disk Utility and iDVD. That coupled with RPC1 firmware and ripping lock removal (That removes the restriction built into most new DVD drives to slow down reading of disks to 2x if a directory VIDEO_TS is detected on disk) makes the drive into a rather useful piece of equipment that OWNER controls.

So you might think that something free, like Patchburn would be slow to release updates for Tiger. You’d be wrong, however, as support for Tiger existed on the day Tiger was released. We will of course see what happens when Leopard comes out.

Patchburn might sound like an inconvinience. One has to go to a germanweb site, download software, click… So let me ask you a question: how often do you burn DVDs using Apple Disk Utility, while waiting for it to create 8.5 gig dmg file? Right. You burn your DVDs using Roxio Toast, aren’t you? And your Roxio Toast supports “Unsupported” drives as well as it does “Apple Shipped”, right? So I don’t see a problem, but please leave a comment and let me know if you don’t agree.

Here is some basic economics: I bought an LG HL-DT-ST GSA-4160b dual layer DVD burner at Best Buy on boxing day 2004 for 120 CAD, with 40 mail in rebate (that I recieved). So in reality after taxes I spent 98 CAD on it. At that time a Pioneer DVR-A09 was selling for 150-170 CAD plus taxes. On the saved money I bought an external enclosure for it, making it mobile.

Don’t get me wrong, Pioneer DVR-109 is a great drive, and I see that Compunation is listing it for just a shade over 100CAD at the time of this writing, but then again, LG burners are ~65CAD now too. Lasers in CD/DVD burners burn out after about the same number of writes, so is paying 40$ extra worth it?

Lastly, I have a DVR-105 at work. I’ve upgraded it to the latest firmware, and tried burning with it. It chokes on cheap silver only DVD-R media (No idea what kind, probably rebranded ritek, or something equally cheap), creating corrupted burns in all tries (I learned the lesson after 3rd attempt to burn). A cheap LG and BenQ burners I have here don’t have an issue with media at all, writing on it at 8x, and passing all the verifications afterwards (Generally it’s a good idea to do verification, just to prevent frustration later). So go figure, cheaper drive reliably burns on cheap media too, so you don’t need to buy expensive Apple branded blanks. I wonder…..

BTW, I am still wondering how to turn MATSHITA CD-RW CW-8123 (Combo drive that shipped with iBook G4) into a region-free drive – I don’t believe that firmware updates for it exist.

OpenSolaris: Sun releases Solaris 10 source

stany

Sun released Solaris source code as part of their OpenSolaris initiative today.

Seems like some things are still binary only (Although fewer then last time Sun showed outsiders their source code, back with Solaris 8), and I didn’t notice the X drivers, but with the source for basic OS (which is what Sun made available), gcc, OpenMotif and X.org‘s drivers it’s probably possible to roll your own Solaris, and the only bit that will be missing with be CDE (Ok, OpenLook would be missing too. But is there anyone out there who actually likes OpenLook, especially since it was depreciated starting with Solaris 9?). Oh, and Display PostScript extension for X would be missing.

*sigh* As weird as it sounds, I miss CDE.

P.S. A mirror of the source code is at: http://www.genunix.org/mirror/index.html plus torrents are available at http://dlc.sun.com/torrents/