Creating dynamic libraries under Mac OS X

So you are compiling some piece of C code, possibly ancient, possibly written for Linux, and you get to the place where a library is about to be created, and you get something like:

stany@gilva:~/src/socks/socks5-v1.0r11/shlib[05:16 AM]$ gcc -o libsocks5_sh.so  
-shared msg.o protocol.o log.o hostname.o confutil.o buffer.o cache.o wrap.o 
wrap_tcp.o wrap_udp.o conf.o libproto.o select.o rld.o null.o addr.o 
upwd.o gss.o   -ldl  
gcc: unrecognized option `-shared'
ld: warning multiple definitions of symbol _gethostbyname2
hostname.o definition of _gethostbyname2 in section (__TEXT,__text)
[....]
ld: Undefined symbols:
_main
stany@gilva:~/src/socks/socks5-v1.0r11/shlib[05:16 AM]$ 

And you get all confuzzled.

Well, unrecognized option `-shared’ warning is not generated by gcc, but by ld, which is the dynamic linker, and is amongst other things in charge of creating dynamic libraries (and static archive files). Of course, dynamic libraries are just collections of functions, and do not need main(). Under Linux, and most other unices (Including Solaris), -shared is what ld wants in order to create a dynamic library. However, Darwin is different, and linker expects -dynamiclib instead.

So:

stany@gilva:~/src/socks/socks5-v1.0r11/shlib[05:16 AM]$ gcc -o libsocks5_sh.so 
-dynamiclib msg.o protocol.o log.o hostname.o confutil.o buffer.o 
cache.o wrap.o wrap_tcp.o wrap_udp.o conf.o libproto.o select.o  rld.o 
null.o addr.o upwd.o gss.o   -ldl  
ld: warning multiple definitions of symbol _gethostbyname2
hostname.o definition of _gethostbyname2 in section (__TEXT,__text)
[...]
stany@gilva:~/src/socks/socks5-v1.0r11/shlib[05:19 AM]$ file libsocks5_sh.so 
libsocks5_sh.so: Mach-O dynamically linked shared library ppc
stany@gilva:~/src/socks/socks5-v1.0r11/shlib[05:20 AM]$ 

However, in spite of it’s magical properties, it doesn’t fix the function name clashes. 🙂

Capturing RTSP streams to file

Introduction

RTSP is Real Time Streaming Protocol that is documented in RFC 2326. Some reasonably good background information is available on the webpage of Henning Schulzrinne. Multicast people also created a specification for a really simple RTSP client with a reasonably good description of a client/server interaction.

My /etc/services lists the following ports as registered with IANA:

stany@gilva:~[01:18 AM]$ grep rtsp /etc/services 
rtsps           322/udp     # RTSPS
rtsps           322/tcp     # RTSPS
rtsp            554/udp     # Real Time Stream Control Protocol
rtsp            554/tcp     # Real Time Stream Control Protocol
rtsp-alt        8554/udp     # RTSP Alternate (see port 554)
rtsp-alt        8554/tcp     # RTSP Alternate (see port 554)
stany@gilva:~[01:18 AM]$ 

RTSP is used by a number of applications, on mbone (where one can watch all sorts of streams, including NASA TV), and on normal IPv4 networks. Vast majority of mbone streams I saw myself encapslate MPEG1 or MP3 data, however MPEG2 support is showing up more and more commonly. People who are not connected to mbone see rtsp URLs as parts of RealAudio/RealVideo streams (Provided, for example, by BBC), and QuickTime streams.

Why would one want to “record” these streams?

There are many cases when this might be desirable – for example one might not have speedy connection to internet, yet want to watch media content provided in high resolution. Occasional high speed connectivity to the internet is a possibility too – for example I can stop by a coffee shop that has wi fi, and have full high speed internet connectivity. So a question arises, how does one capture a stream contents to a file, to watch offline? Sadly, clients that are provided to view/listen to the RTSP streams don’t support capture of data at all (Real Player) or saving of the stream to file requires a “Pro” version of the software and can be disabled by the server (QuickTime Player).

RealMedia Streams

One of the options might be rtspget which is an add-on to xine. I’ve compiled xine under 10.4.1, with just ./configure –prefix=/opt/gnu –with-included-gettext –with-x, however if this is the first time you are building xine, you might need to actually “make install” contrary to instructions on the Ian Collier’s page. In order to buiild, rtspget would want a whole bunch of xine includes that are spread all over, yet get dropped into {PREFIX}/include/xine/ by the install script (In other words you can’t just give -I{path} option to gcc, as the location of includes is different in the tree then in the installed location rtspget expects. And yes, you can edit the source, if you want to). Any way, assumption here is that you know what you are doing, and given some time, and maybe a choice word or three it xine-lib would eventually build, and rtspget would eventually compile.

In my tests, rtspget would be able to negotiate the connection with BBC’s streaming servers, that are Real Media based, and download the streams. Resulting downloads were identified by the file(1) command to indeed be Real media files. No go with the QuickTime streams at all, and the downloaded files were not showing anything in the Real Player 10. On this page (And don’t ask me why engineering at Purdue University insists on HTTPS protocol, as I don’t know either) I saw reference to this, that seemed to indicate the file still plays normaly in xine. So maybe there is something wonky somewhere, and warrants further testing.

Besides this “small” problem with rtspget, a couple of others are present. It sends fixed User-Agent, GUID and StartTime to the servers, and that might cause problems in the future, as servers might be modified to recognize that signature, and deny rtspget access to the media:

stany@gilva:~/src/rtsp[01:59 AM]$ strings rtspget
[...]
rtsp: bad mrl: %s
User-Agent: RealMedia Player Version 6.0.9.1235 (linux-2.0-libc6-i386-gcc2.95)
rtsp: failed to connect to '%s'
CSeq: 1
ClientChallenge: 9e26d33f2984236010ef6253fb1887f7
OPTIONS
PlayerStarttime: [28/03/2003:22:50:23 00:00]
CompanyID: KnKV4M4I/B2FjJ1TToLycw==
GUID: 00000000-0000-0000-0000-000000000000
RegionData: 0
ClientID: Linux_2.4_6.0.9.1235_play32_RN01_EN_586
[...]
stany@gilva:~/src/rtsp[02:00 AM]$

Something to keep in mind. This user agent string is not in the actual rtspget source code but in librtsp in xine source code (from ./xine-lib-1.0.1/src/input/librtsp/rtsp.c):

[...]
  if (user_agent)
    s->user_agent=strdup(user_agent);
  else
    s->user_agent=strdup("User-Agent: RealMedia Player Version 6.0.9.1235 (linux-2.0-libc6-i386-gcc2.95)");
[...]
  s->server_state=RTSP_CONNECTED;

  /* now lets send an options request. */
  rtsp_schedule_field(s, "CSeq: 1");
  rtsp_schedule_field(s, s->user_agent);
  rtsp_schedule_field(s, "ClientChallenge: 9e26d33f2984236010ef6253fb1887f7");
  rtsp_schedule_field(s, "PlayerStarttime: [28/03/2003:22:50:23 00:00]");
  rtsp_schedule_field(s, "CompanyID: KnKV4M4I/B2FjJ1TToLycw==");
  rtsp_schedule_field(s, "GUID: 00000000-0000-0000-0000-000000000000");
  rtsp_schedule_field(s, "RegionData: 0");
  rtsp_schedule_field(s, "ClientID: Linux_2.4_6.0.9.1235_play32_RN01_EN_586");
  /*rtsp_schedule_field(s, "Pragma: initiate-session");*/
  rtsp_request_options(s, NULL);

  return s;
[...]

This leads me to believe that anything linked against xine’s librtsp would have that user agent, etc combination (That someone just captured on the wire while investigating the way RealPlayer establishes a session). Note to myself: Investigate #define RTSP_RECORDING 16 and the rest of server capabilities in xine-lib-1.0.1/src/input/librtsp/rtsp.c)

QuickTime Streams

Another option seems to be vlc. You would want the latest version of it (vlc-0.8.2test2 at the writing time), and you’d want to keep in mind that vlc doesn’t at this time deal with H.264 files properly yet.

Here are the instructions:

  • Locate the stream you want to download. You can generally start watching in QuickTime player, get info, make sure that the stream you are getting is not H.264, but MPEG4, etc, and, if you have a license key for QuickTime Pro version, save the file to disk. If you are lucky, and the content owner didn’t restrict saving, QuickTime Pro would do just that. However, if content owner restricted the saving feature, QT would offer to save the file for you any way. This would be a bit of a misnomer, as that action will generated a small file with an embedded rtsp URL, not actually save the actual file, and you would need internet connection in order watch it again.

    At this point you can use strings on the file to extract the rtsp string. If you don’t have pro license, getting info on the file you are streaming would give you rtsp path as well, however occasionally it seems like it is a path to a container file that contains the actual path, not to the media itself. Warrants further investigation.

  • open VLC
  • File -> Open Network.
  • Select HTTP/FTP/MMS/RTSP
  • Set URL to the rtsp URL you obtained already in the first step
  • Check Advanced output
  • Click Settings
  • Click Browse, choose a name to save the file as
  • Set Encapsulation method to QuickTime. Encapsulation method seems to make a minor difference – when set to quicktime as opposed to the default mpeg_ts, it seemingly started to pixellate less. I don’t know if anything changes in the output of the file.
  • Cleck Play locally if you want to watch while it downloads. Note that MPEG4 files as played in VLC would appear with “blocky” pixellisation. If you manage to save the file, open it in QuickTime, and all that blockiness would go away, as QT does a better job rendering MPEG4
  • Click OK
  • Click OK
  • Wait 🙂

As a test I’ve used a link to Shakira’s full video from Sony’s web site (As Joshie used to say, “Boobies!”).
rtsp://qt.sony-global.speedera.net/qt.sony-global/Epic/Shakira/Shakira_LaTorturaVidFull_300.mov

End result:

stany@gilva:~/tmp[02:19 AM]$ file vlc-output.ts
vlc-output.ts: Apple QuickTime movie file (moov)

Renamed the above file to .mov, opened and played it in QT 7 with no problems, and QT reported the file as mpeg4 video, AAC audio.

Windows Media Files

Lastly, for the sake of completeness, Windows Media files. They don’t use RTSP:// protocol, but something called MMS.
SDP Multimedia has more information and a download client for Windows.
mmsclient deals with files that are not continuous (ie movie clips), however in my tests against the CBC radio streams doesn’t quite work. And yes, I, too, have no idea what “dbl” is, as it’s not a command on either Solaris or Mac OS X. (As an aside, while linking under Solaris, you want to tell gcc to link against libnsl and libsocket: gcc -g -O2 -Wall -o mmsclient client.o -lnsl -lsocket Mac OS X Just Works). CBC radio stream for Ottawa, high quality, is at mms://wm05.nm.cbc.ca/cbcr1-ottawa at the time of this writing. Also note that the version mmsclient broadcast is also hardcoded to Media Player 7 in client.c:

  sprintf (str, "3403NSPlayer/7.0.0.1956; {33715801-BAB3-9D85-24E9-03B903282
70A}; Host: %s",
           host);

This might limit what media it can save, as now a days pretty much everything in Windows world depends on WMP 9.0 and up, as it has DRM support.

MP3/Shoutcast Streams

For that you just want streamripper.

Avenues of further investigation

Darwin Streaming Server, freely available as both source and binary from Apple, has Darwing Streaming Proxy, which, after a fair bit of ignoring wrong documentation, it is possible to get to work. DSS-5.0.3 doesn’t compile out of the box under Tiger, and seems like misses -lcrypt somewhere. However build process uses Xcode (which I adhor), and I have no idea how to convince it to work, so I gave in and downloaded the binary. I am not sure that it’s written in ANSI C, which is also a problem. Idea would be to convince it to fopen() and save the data to the file as it passes through the proxy.

Also, more reading of the simplified RTSP client specification is probably warranted.

stany@iskra:~[03:11am]$  telnet qt.sony-global.speedera.net 554
Trying 209.133.111.201...
Connected to qt.sony-global.speedera.net.
Escape character is '^]'.
DESCRIBE rtsp://qt.sony-global.speedera.net/qt.sony-global/Epic/Shakira/Shakira_LaTorturaVidFull_300.mov RTSP/1.0
Cseq: 1

RTSP/1.0 302 Found
Server: DSS/5.0.3.2 (Build/452.22.3; Platform/Linux; Release/Panther; Update/3GPP; )
Cseq: 1
Location: rtsp://qt.sony-global.speedera.net.central.speedera.net/qt.sony-global/Epic/Shakira/Shakira_LaTorturaVidFull_300.mov

Connection to qt.sony-global.speedera.net closed by foreign host.
stany@iskra:~[03:12am]$ 

Getting Rawr-Endezvous to work with recent Growl framework versions

20050618 Edit: Initially when I wrote this, I were referring to Rawr-Endezvous v0.6b3. In the latest version (0.6b4) the problem I were refering to in this note got fixed. Thank you very much, Jerome. I am impressed that my voice was heard. –stany

I’ve cleanly installed Tiger on my iBook, and then installed latest Growl framework. Sadly rawr-endezvous stopped working. As Adium, etc was happily doing Growl notifications, I figured that the problem is in Rawr-Enedzvous, not in Growl.

It seems that Jeremy Knope basically disappeared off the face of the universe, and didn’t leave forwarding address, so I did a bit of digging, and noticed the following:

stany@gilva:/Applications/extras/Rawr-endezvous.app/Contents/MacOS[06:08 PM]$ otool -L Rawr-endezvous
Rawr-endezvous:
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 8.0.0)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 218.0.0) 
        /Users/jerome/Library/Frameworks/GrowlAppBridge.framework/Versions/A/GrowlAppBridge (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 63.0.0)

Next I checked if GrowlAppBridge existed on my system, and noticed that it is now part of Growl.framework, and not part of GrowlAppBridge.framework:

stany@gilva:/Applications/extras/Rawr-endezvous.app/Contents/MacOS[06:10 PM]$ locate GrowlAppBridge
/Library/Frameworks/Growl.framework/GrowlAppBridge
/Library/Frameworks/Growl.framework/Versions/A/GrowlAppBridge
/Library/Frameworks/Growl.framework/Versions/A/Headers/GrowlAppBridge-Carbon.h
/Library/Frameworks/Growl.framework/Versions/A/Headers/GrowlAppBridge.h

So as an interim fix the following works:

stany@gilva:/Library/Frameworks[06:10 PM]$ mkdir -p GrowlAppBridge.framework/Versions/A/
stany@gilva:/Library/Frameworks[06:11 PM]$ cd GrowlAppBridge.framework/Versions/A/
stany@gilva:/Library/Frameworks/GrowlAppBridge.framework/Versions/A[06:11 PM]$ ln -s ../../../Growl.framework/GrowlAppBridge GrowlAppBridge

followed by restarting rawr-endezvous

Already Seen? or a rant about emerging file types

Already Seen?

Yesterday I’ve encountered a rather interesting problem. I were visiting Alex, and in conversation he complained about a new phenomena in book and magazine distribution – LizardTech djvu file format . He downloaded an issue of “Penthouse”, and wanted to extract a particular page out of it, and print it onto a sticker that he could apply to face of burned CD. However the file he had was in djvu format, and the stand-alone viewer he was using didn’t let him print. I agreed to take a look at the problem for him, and whipped out my trusty iBook….

In any case, Alex was viewing djvu files using OpenDjVu viewer for Windows. While that viewer is a stand-alone (and one can wonder about the purpose of “easy navigation with one hand from keyboard” considering that magazines like “Penthouse” and “Playboy” are distributed in this format). one of the shortcomings that it has is lack of print support. And ofcourse one can’t export the djvu file as anything.

After looking at LizardTech web site, I’ve noticed that the only Mac OS X implementation that they support is a Safari Plugin (For the reference it installs into /Library/Internet Plug-Ins

stany@fiona:/Library/Internet Plug-Ins[01:53 PM]$ ls -lad NPD*
-rwxr-xr-x  1 stany  admin  8966  2 Nov 01:51 NPDJVU
drwxr-xr-x  3 stany  admin   102  2 Nov 01:51 NPDjVu.plugin
stany@fiona:/Library/Internet Plug-Ins[01:53 PM]$ 

).

I subscribe to the orignal Mac software distribution model where all the bits and pieces needed by a particular program should be available inside a single directory tree, making upgrading and pruning file system simple (getting rid of a program should be rm -rf /opt/program_name or rm -rf /Applications/Program Name.app. This method was field tested over and over again on literally tens of Solaris systems I’ve administred, allowing me to compile once, and copy over everywhere, and while somewhat wasteful on disk space, seems to be rather effective), so I were reluctant to install the plug-in.

No problem, there is a djvulibre implementation that is designed to run under X. Downloaded that, looked through INSTALL file. It depends on qt-X11 and libtiff. Compiled libtiff. Remembered that there is such a thing now as qt-mac, and downloaded that. Attempted to follow the installation instructions for qt-mac (uncompress into /Developer/, rename qt-mac-3.x.x.x to /Developer/qt/, run configure, followed by make). Took over an hour and generated tons of object files, with no end in sight. Realized that I don’t really need the fancy graphic utilities that are part of djvu-libre, and all I need is djvups. Compiled that.

Run it on a 5.7 meg .djvu file, redirecting the output into new file, that is supposed to be PostScript. End result have been running for 15 minutes and over a gigabyte, until it run out of disk space. Cursed a lot. Copied the .djvu file to a Solaris system with much more RAm and disk space, and let it run there for a while. Half an hour later I had a 1.6 gigabyte PS file. Fed the resulting PS file into Adobe Distiller 4.0 that came with Adobe Frame Maker 6.0. It run for over 2 hours on a 400 Mhz USII CPU, and used up over 450 megabytes of real RAM. End result was 7.5 megabyte PDF file.

I’ve opened the resulting file, and attempted to zoom onto some text, only to find out that the result is not readable. At this point I suspect that the fault is with Adobe Distiller, and that I should have specified that I want print resolution, and not default 72dpi.

Example of pixilization in 7 meg PDF output at 200% magnification

This is what it looked like at 200% magnification

By this point it’s 2 in the morning, and me and Alex are both cursing a lot.

So I eventually gave in and installed the Safari plugin and restarted Safari. In README for it, there is a little note that says that viewing local djvu files is not supported. No problems, copy the files into Apache directory, access it over HTTP. No workie, Safari is showing a bunch of binary garbage. More cursing. At this point I go into Help-> Installed Plug-ins, and notice that while the djvu plugin is installed and enabled, it is set as a handler for only a number of particular mime types (Subject of another RANT – Who still uses MIME types that are essentially extension based instead of using something similar to file(1) database to identify what kind of file one is handling regardless of extension, resource fork, etc? Get with the program, folks). So a quick edit of httpd.conf to add AddType image/x-dejavu .djvu, and a quick apacehctl restart later, the plugin was displaying djvu files.

At this point printing was working. So I just printed the thing to PDF using normal Panther PDF support. Another 10 minutes later, I had a 215 meg file. Opening it, and zooming in was actually showing the text in a readable way, so I started up Windows File Sharing, and copied it over to his system.

Example of pixilization in 215 meg PDF output at 200% magnification

This is what a 215 meg PDF output generted from Panther looked like at 200% magnification

So good 4 hours later Alex had his 5.7 meg djvu file as a 215 meg PDF file that he could actually use.

Here are the file sizes:

-rwxr--r--   1 stany    staff    5616972 Nov  2 00:37 penthouse_11.djvu
Original
-rw-r--r--   1 stany    staff    1608552487 Nov  2 00:59 penthouse_11.ps 
Generated with djvups with above as input
-rw-r--r--   1 stany    staff    7686172 Nov  2 03:11 penthouse_11.pdf
Generated by Adobe Distiller
-r-xr-xr-x  1 root      wheel  221127535  1 Nov 21:22 penthouse_11.pdf
Output of Safari djvu viewer printed to PDF

So here is the rant part:

If you create a new file format, no matter how cool you think it is, and how new and advanced it is, please consider that there might be people who would want to actually do things to data in the file that you never anticipated. So even if you are 100% sure that your file format is the best thing since chocolate and everyone on the planet should adopt it, please keep in mind that there might be some specialized applications written to solve a particular problem, and that those applications can do things that you never anticipated, and support file formats that are existing standards, and not emergent standards. Thus, if you want your file format to be successful, do take care in writing and making publically available filters that not only convert to your file format, but from it as well. This way people will not perceive that you are attempting to lock them into your design, but instead that your idea has genuine merit, and technological innovation.

It seems that djvu has some benefits, but primarily folks prefer it for distribution of images as it generates smaller files then .PDF. However that is not exactly true. After some research on the internet I believe that djvu generates smaller files only if the source material is a series of images. The moment you run any sort of OCR on the images, and distill that, you will get a significantly smaller PDF file.

So djvu is a file format for folks who either need to distribute high quality image galleries as single files, or who do not have access to OCR software (or do not want to proofread). It is also not searchable or indexable.

Obviously neither the folks at LizardTech nor numerous open and free and libre implementators of the djvu support or even think about such a novel and advanced concept as export to JPG page by page. The resolution that is generated by djvu is obviously high enough that the generated images can be successfully OCRed using commercial off the shelf software. However right now in order to do something like that, one has to first dump djvu to PDF and then convert PDF to jpg page by page, with introduction of noise and compression artifacts at the extra step. Lack of generally available tools to modify the files seems to also hinder djvu adoption as a mainstream file format.

It is well known that VHS adoption was widely won due to efforts of pornographers, who chose simpler if technologically inferior VHS over Beta. It seems that LizadTech is betting on the similar vector, as currently it is the warez community that seems to utilize djvu file format, due to it’s smaller size, and thus lighter load on their servers. While I am not 100% certain that this is an ideal business plan, there of course might be some merit to it.

To quote MasaManiA.com, “You must agree my logical thinking and natural fear” (Note: Potentially NSFW but generally hilarious never the less).

rentzsch.com: fs_usage Intro

rentzsch.com: fs_usage Intro

[excerpts from his blog entry follow]
fs_usage is a command line tool that displays file system activity.

That’s a little better, but still a firehose. You can cut it down by grepping out the CACHE_HIT lines and grep’s own reads:

$ sudo fs_usage -e -f filesystem|grep -v CACHE_HIT|grep -v grep

Now you have a solid base. It’s still a lot of information delivered pretty quickly, but now it’s realistic to — say — start the recording, do your thing, stop the recording and comb through the resulting logs.

Of course, you can further focus the output. For example, discover what files are being opened, as they’re being opened:

$ sudo fs_usage -e -f filesystem|grep -v CACHE_HIT|grep -v grep|grep open

Or, who’s writing to your disk:

$ sudo fs_usage -e -f filesystem|grep -v CACHE_HIT|grep -v grep|grep write

Howto: Mirroring a site that uses cookies which expire at the end of a session

• Open your browser to the apropriate page and login
• Sniff session while loading random pages from the site in the browser
• Leave broswer open to the site
• Stop sniffing the data, and analyze the dump from the site, find and copy the Cookie string
• With wget you add –header “Cookie: cookiename=data; secondcookiename=data” http://username:password@site to the options.
• make sure that you start mirroring from inside the site, ie: beyond the login page.

In my case the final wget string looked like this:

wget –mirror –convert-links –page-requisites –html-extension –cookies=off –header “Cookie:ABC%2ESession=YcnpPdnqLFCfYA6p%2DITgU84eRUVSrKV30; ASPSESSIONIDCCSTCRTT=GCCLDPDDKDIACCHE” http://username:password@site

Note that most of the cookie data has been deleted to keep this example brief.

Linux Tips — Firefox 0.9 remote newtab syntax

When I open a link from another application (gnome-terminal or evolution, for example), I would like it to open in a new tab in Firefox, so I had a script which I called newmoz:
firefox -remote %u201CopenURL($1,new-tab)%u201D
This is my default web browser. With Firefox 0.9, the syntax has changed. You now need:
firefox -a firefox -remote %u201Copenurl($1,new-tab)%u201D
I believe this has something to do with the confusion of the possibility of several related Mozilla applications running all at once, although I%u2019m not entirely convinced. To make it really snazzy, try:
firefox -a firefox -remote %u201Copenurl($1,new-tab)%u201D || firefox $1
This way, if firefox isn%u2019t already running, it will still work.

Adam Rosi-Kessel’s Fair and Balanced Weblog