How to Clean-Install OS X Mountain Lion But Still Have an Upgraded Recovery Partition

I have a 2011 MacBook Air with OS X Lion preinstalled. For the past year I’ve done quite a few projects and tried many things on it. When OS X Mountain Lion came out, I felt it was about time to start over.

It turned out to be more complicated than I had imagined.

What I want is really simple: I want my recovery partition to be also Mountain Lion-based. This will some day be handy when I need to reinstall the system or to repair something.

Mountain Lion’s installer actually upgrades the recovery partition if you do an upgrade install. Well I didn’t want an upgrade install, so I followed the instructions here and made a bootable Mountain Lion partition on an external drive.

I booted into the installer, wiped my (FileVault-encrypted) disk, and installed the new cat… only to find that when it was done, the recovery partition was gone!

There are two issues here. First, wiping out your main volume also wipes out the recovery partition, even if you go to Terminal and use diskutil to reformat you main volume alone. The recovery partition has a different id, but it doesn’t matter: it’s removed once the main volume is reformatted. Second, the Mountain Lion installer on an external drive doesn’t install the recovery partition, and there is no tool that installs the recovery tool after you’ve installed the OS.

I finally found the combination that worked. I had to boot into the Internet recovery system that comes with my MacBook Air and re-install Lion. This installed Lion and the recovery partition. Then I did a upgrade install of Mountain Lion, and booted into the now upgraded recovery partition, wiped out the drive again, then install Mountain Lion from the recovery system. Now I have a clean-installed OS X with a new recovery partition.

The only two cases where you get a new recovery partition seem to be (1) if you upgrade-install and (2) if you clean-install Mountain Lion from the recovery system thet you get after you upgrade-installed! It took me a few hours of re-installing the two versions to figure this out. So I’m writing my experinces down in hopes that it will save your time if you also want a clean-installed OS X with an upgraded recovery system.

"Currying" Objective-C Methods

In this blog post, I’ll show you a way to “curry” Objective-C methods. I’ll first talk about the problem I’m trying to solve, what currying is, and how currying can help solve the problem. Then I’ll discuss how you can use blocks to achieve currying and the issue with that approach. Next, I’ll present a library that can easily “curry” Objective-C methods of your choice. You have probably noticed I put the term in quotes, and I’ll briefly talk about why the approach I present here is not the real currying in a strict sense. I’ll also discuss some other issues in introducing the notion of currying in an imperative, object-oriented programming language like Objective-C.

Please note that the library is written for study purposes, and I didn’t consider its practicality or performance. Nor is it intended for production use.

If you want to dive right into the code, here’s the source, along with some sample showing you how to use it.

Introduction

Let’s say we have a method in SomeClass:

    - (void)doThis:(id)a withThat:(id)b andThat:(id)c;

If we refer only to the method and want to emphasize its class membership, we can also write it as -[SomeClass doThis:withThat:andThat:], although that does not reveal the type information for the parameters and return value.

Then one day you find that, in your project, you have quite a few places that look like this:

    SomeClass *foo = [[SomeClass alloc] init];
    [foo doThis:a withThat:b andThat:c];
    [foo doThis:a withThat:b andThat:d];
    [foo doThis:a withThat:b andThat:e];
    [foo doThis:a withThat:b andThat:f];

The method takes three arguments, but we are always passing the same a and b here. Would it be nice if there’s some way to remember those, so that we only need to pass the argument that actually changes, namely, c, d, e, and f, like this:

    [someMagicFoo blahAndThat:c];
    [someMagicFoo blahAndThat:d];
    [someMagicFoo blahAndThat:e];
    [someMagicFoo blahAndThat:f];        

It turns out that there are quite a few possible solutions to this problem. Before I discuss them, let me first introduce a concept called currying.

Currying

Let’s take the C function for example:

    int f(int x, int y)
    {
        return x + y;
    }

Then say you want to do these computations:

    int z;
    z = f(5, 10);
    z = f(5, 20);
    z = f(5, 30);

Again, we see that we’re always passing 5 for x. One way to save that effort is to write another function:

    int g(int y)
    {
        return f(5, y);
    }

    int z;
    z = g(10);
    z = g(20);
    z = g(30);

In other words, we want to “fix” the first argument of f and we only want to pass the second argument to get the function value. This is called “partial application”, as we only want to apply the function to a smaller set of arguments.

Wouldn’t it be nice if, instead of writing g ourselves, the compiler can help us write that, using the hypothetical construction below:

    int (*g)(int) = fix(f, 5);  // hypothetical construction
    int z;
    z = g(10);
    // ...

The construction above is actually not hypothetical, although it’s not available in the C language. The transformation that allows such construction is called currying.

In C, we declare the prototype of function f as int f(int, int), meaning it’s a function that has two integer parameters and returns an integer value. It can also be written formally as:

    f: int × int -> int

We can also write it as:

    f: (int, int) -> int

Which reads “f is a function that maps a pair of integers to an integer”. The two forms are equivalent, as n-tuples (a pair is a 2-tuple) can be used to represent set cross products.

Now let’s talk about currying with a concrete example. We can “curry a function”, in the sense that we pass a function to a machinery called curry, and it will give us a new function (think of it as someone who writes a new function for us).

So, if we curry f (i.e. passing f to curry), then we get a new function with the following type:

    curry f: int -> (int -> int)

That looks bizarre, so let me explain. The above line reads “curry f (the returned function of calling curry with f) is a function that, when passed an integer, returns a function that takes an integer and returns an integer”.

Sounds familiar?

It should! Remember the C example we just mentioned. Let’s suppose if the C language supports currying, and we can actually obtain a function from that:

    // we need this bizarre type to declare h
    typedef ...SomeBizarreType...;

    SomeBizarreType h = curry(f);

Then, if we call h(5), it will give us the function g that we want!

    int (*g)(int) = h(5);
    int z;
    z = g(10);
    z = g(20);
    z = g(30);

curry is actually a function that takes a form like this:

    curry: ((a, b) -> c) -> (a -> (b -> c))

It can be a bit cumbersome to read out the definition above. Here’s a way to see this: if you call curry(f), it returns function h to you. Then if you call h(5), it returns our good friend g. Here, f is the function of type (a, b) -> c, and h is (a -> (b -> c)), and g is (b -> c). Those parentheses are added to disambiguate, but rest assured that there are many people who can do without them.

This transformation technique is named after Haskell Curry (1900-1982), a mathematician (yes, the language is named after him), who, among other things, discovered a fixed-point combinator called Y combinator (yes, that firm is named after it), a construct which can express recursive functions.

Also note that curry is not confined to functions that have two parameters (i.e. have a pair of parameters):

    curry: ((t_0_, t_1, ..., t_{n-1}) -> t_n) ->
            (t_0 -> (t_1 -> ( ... -> (t_{n-1} -> t_n))))

And there’s uncurry that is the inverse of curry:

    uncurry: (t_0 -> (t_1 -> ( ... -> (t_{n-1} -> t_n)))) ->
            ((t_0_, t_1, ..., t_{n-1}) -> t_n)

Now that we know what currying is, let’s see how it can help us find a solution to the problem we have in the beginning.

Using Objective-C Blocks for Currying

It turns out that there isn’t really a way to write a curry function in C. The problem is that in C, functions are not first-class objects, meaning you can’t manipulate functions like you do data structures. Unlike, say, in Haskell, where curry is a function you can actually use.

Then, there’s a practical issue. Recall the example of f and g above. If we always want to fix the x parameter with a constant, then for each constant we need to write a new g. What if we don’t want to be confined to a constant?

    int g(int y)
    {
        // a is an argument we want to fix
        return f(a, y);
    }

Now the problem is that g has to “remember” what a is. You don’t want to use global variable for that:

    // suppose a is a global variable and a = 5 initially
    z = g(10);  // 15
    z = g(20);  // 20
    foo();      // a gets changed to 0
    z = g(10);  // ugh! it's 10 now

In the example above, we want to have a g that “remembers” to pass 5 for x to f. Using a global variable ruins that.

A construct to “remember” something happening in your program is called closure. A closure “remembers” the local states when it’s created.

In Objective-C, blocks are closures. We can actually use blocks to write the h function (compilable code here):

    typedef int (^int_to_int_t)(int);
    typedef int_to_int_t (^int_to_int_to_int_t)(int);

    int_to_int_to_int_t h = ^(int x) {
        int_to_int_t fx = ^(int y) {
            return f(x, y);  
        };

        return fx;
    };

    int_to_int_t g = h(5);

    int z;
    z = g(10);

So, when the block fx is created, it will “remember” the x argument that’s passed to h. You also noticed that we need to use some typedefs here.

I’m declaring h as a local block because I don’t want to dive into the block copying issue here, which is another complex matter that requires an entire blog post. Just remember that you should return an autoreleased copy of the block if you want to pass it around (yes, I’m essentially telling you that I’m doing it wrong here for the sake of brevity).

In fact, we can go one step further, then we’ll have something that’s very close to the curry function, by passing the function we want to curry using a function pointer (compilable code here):

    // typedefs for int->int, int->(int->int), etc.
    typedef int (^int_to_int_t)(int);
    typedef int_to_int_t (^int_to_int_to_int_t)(int);
    typedef int (*int_x_int_to_int_t)(int, int);
    typedef int_to_int_to_int_t (^curry_t)(int_x_int_to_int_t);

    curry_t curry = ^(int_x_int_to_int_t f) {
        int_to_int_to_int_t h = ^(int x) {
            int_to_int_t g = ^(int y) {
                return f(x, y);  
            };
            return g;        
        };
        return h;
    };

    int_to_int_to_int_t h = curry(f);
    int_to_int_t g = h(5);

    int z;
    z = g(10);

That’s a lot of declarations, not pretty, but it’s something very close to the real thing.

What about Objective-C Methods?

Now that we know what currying is, and how Objective-C blocks can give us some kind of “curried” functions, let’s see how they can help when it comes to Objective-C methods. Recall our first example:

    SomeClass *foo = [[SomeClass alloc] init];
    [foo doThis:a withThat:b andThat:c];
    [foo doThis:a withThat:b andThat:d];
    [foo doThis:a withThat:b andThat:e];
    [foo doThis:a withThat:b andThat:f];

If we want to fix foo, a, and b, we can actually write a simple block like this:

    void (^fooABAndThat)(id) = ^(id x) {
        [foo doThis:a withThat:b andThat:x];
    };

    fooABAndThat(c);
    fooABAndThat(d);
    // ...

If you want to have to flexibility to just fix a then fix b or some other arguments later, however, then you’ll end up with something as messy as the previous “curry” function we have. Namely, if you want to do something like this (provided we always fix foo, to make things simpler):

    ((fooDoThis(a))(b))(c);

Then you have to write something like:

    typedef void (^id_to_void)(id);
    typedef id_to_void (^id_to_id_to_void)(id);
    typedef id_to_id_to_void (^id_to_id_to_id_to_void)(id);
    id_to_id_to_id_to_void fooDoThis = ^(id x) {
        id_to_id_to_void withThat = ^(id y) {
            id_to_void andThat = ^(id z) {
                [foo doThis:x withThat:y andThat:z];
            };
            return andThat;
        };
        return withThat;
    };

    ((fooDoThis(a))(b))(c);

Apparently, that’s not very fun to write.

As I mentioned before, in C, functions are not first class objects. That you have to declare a type for everything doesn’t help, either (hence the messy typedefs above). Unless we have a fancy compiler or language extension that writes those declarations implicitly and automatically for us, this seems to be as good as it can get with blocks.

But wait. Objective-C is a dynamic language. Its runtime API allows us to create classes and methods on the fly. Before the introduction of blocks, a lot of language have actually been built by making use of that property. Is there an easier way to “curry” an Objective-C method, so that we can do something like this?

    [someMagicFoo blahAndThat:c];
    [someMagicFoo blahAndThat:d];
    [someMagicFoo blahAndThat:e];
    [someMagicFoo blahAndThat:f];

With ObjCurry, we can do that.

Using Objective-C Runtime API to “Curry” Methods

ObjCurry is an Objective-C library that allows you to “curry” a wide range of Objective-C methods. Once included in a project, you can do this:

    [SomeClass curry:@selector(doThis:withThat:andThat:)];

Then, you can do things like:

    id fooDoWithA = [foo doThis:a];
    id thenB1 = [foo withThat:b1];
    id thenB2 = [foo withThat:b2];

    // equivalent to [foo doThis:a withThat:b1 andThat:c]
    [thenB1 andThat:c];

    // equivalent to [foo doThis:a withThat:b1 andThat:c]
    [thenB2 andThat:c];

    id someMagicFoo = [[foo doThis:a] withThat:b];
    [someMagicFoo blahAndThat:c];
    [someMagicFoo blahAndThat:d];
    [someMagicFoo blahAndThat:e];
    [someMagicFoo blahAndThat:f];

And, of course, we can curry methods in a Foundation class. For example:

    [[NSMutableDictionary class] curry:@selector(setObject:forKey:)];

Then, say, we have a dictionary, and we want to be able to set the same value to different keys:

    NSMutableDictionary *dict = ...;    
    id dictSetBarWithKey = [dict setObject:@"bar"];
    [dictSetBarWithKey forKey:@"a"];
    [dictSetBarWithKey forKey:@"b"];
    [dictSetBarWithKey forKey:@"c"];

    // dict is now {"a":"bar", "b":"bar", "c":"bar"}

Under the Hood

ObjCurry uses Objective-C runtime API to dynamically create new classes and methods. In the previous example, after you call:

    [SomeClass curry:@selector(doThis:withThat:andThat:)];

A new instance method doThis: will be added to SomeClass. Then, when you call:

    id fooDoWithA = [foo doThis:a];

The returned object, fooDoWithA, is actually a proxy object whose class implements withThat:. That method, in turn, will return another proxy object, whose class (currently the same as the previous proxy object’s) implements andThat:. The last method in the chain, andThat:, will actually send the message doThis:withThat:andThat: to foo, along with the captured three arguments.

The implementation uses NSInvocation, which is the working horse for many Objective-C techniques often collectively called “higher-order messaging”, to capture the arguments along the way. Proxy objects are also often used in tandem with NSInvocation. One of the best example is NSUndoManager, which can give you a proxy object for undoing. Any message you send (i.e. and method invocation) to that proxy object is recorded but not executed. The real invocation is deferred until the user chooses to undo, then the recorded invocation is “played back” (i.e. the real method on the target object is called).

Because methods have to be typed, ObjCurry only supports a limited number of common types, such as id, char, (un)signed short/int/long/long long, and generic pointer (void *). For return types, there is one more supported: void.

It’s possible to add more types — see NSObject+Curry.m to see how to expand the type support. For example, you can add CGRect/NSRect support. The problem is that, unless deeper runtime API hack is found (perhaps some assembly required), for n argument types and m return types, we have to write nm* definitions. Even with the current macro usage it still won’t scale well. If you have better solution, I’d be very happy to hear from you.

Issues and Extensions

You probably noticed that when I say we want a way to “curry” Objective-C methods, I always put the term in quotes.

The reason is that ObjCurry is not the real thing. To understand why, we have to first know how is message sending done in Objective-C.

Let’s say we have an Objective-C instance method of class Foo that adds two NSNumber objects:

    - (NSNumber *)add:(NSNumber *)x to:(NSNumber *)y;

When we call:

    NSNumber *z = [foo add:x to:y];

The compiler actually turns that into a function call:

    z = objc_msgSend(foo, @selector(add:to:), x, y);

(The real code is of course optimized, but this is what the compiler would do from a high-level point of view.)

And the method -[Foo add:to:] is compiled as something like the following (again this is just a sketch). Remember that in Objective-C there is actually only one object type, hence the use of type id all over:

    id Foo_add_to(id obj, SEL selector, id x, id y)
    {
        // ...
    }

Let’s ignore the selector parameter to simplify the matter. Obviously, a method is like a function, but it needs to take an object parameter so that it knows which object to act on. In other words, our Foo_add_to function actually has the type:

    Foo_add_to: id × id × id -> id

Ideally, a curry function should give us something like:

    curry Foo_add_to: id -> (id -> (id -> id))

But because ObjCurry introduces proxy objects in the picture, what we actually have is more like this:

    "curried" Foo_add_to: (id × id) -> (id × id -> id)

That’s why ours is not a real thing in the strict sense.

Another issue that comes with using ObjCurry is that Objective-C compilers give you warnings on previously undeclared methods. This is a nuisance, although there is a workaround. In the main.m that demonstrates how the use ObjCurry, I put a few Objective-C categories that declares the methods as a result of currying. This silences the compiler warnings, but it can be a debugging hazard when you invoke a method that’s not actually created, since the compiler now trusts you that you have an implementation somewhere, even if not statically available at compile or link time.

In addition, there’s the issue of mutability. Although proxies behave like immutable objects (as each time a curried method is called, a new proxy object is created), the target object (i.e. the original object on which the original method is invoked) is not. So while the proxies create an illusion of closures, you are still dealing with an imperative language. So if you intend to try ObjCurry in a multithreaded system, you have been warned.

With the basics in place, it’s possible to extend the fun. For example, each proxy object can actually remember the parameters that are ever passed to them and the returned objects. So if the parameter doesn’t change, the proxy can always returned the “memorized” return value. This is known as “memoization" (some people say it’s just a fancy term for "caching"). Again, in an imperative language like Objective-C, there are all kinds of issues that come with this.

The use of proxy object also brings some overhead and will have some impact on performance in places such as a tight loop. NSInvocation is slow compared to the straightforward message sending, considering the overhead in creating invocation objects and all the house keeping efforts.

Finally, because this is not a feature supported by the compiler, debugging can be tricky if something happened inside the proxy object (especially in the last step that actually invokes the original method).

Related Works

Conclusion

The idea of currying is actually very straightforward, but implementing it with a non-functional programming language incurs many issues, some of which I have discussed above. We have seen how Objective-C blocks can be used to curry functions, and I’ve also introduced the library ObjCurry to “curry” Objective-C methods using the Objective-C runtime API. As I said in the beginning, the project is more of a study and does not consider performance issues. Still, we have in effect created a small language extension of our own. There are still other issues with the implementation, which I’ve also briefly addressed. As Curry is an open source project, you can try it out yourself. And you know a way to generate automatically IMPs for non-id, non-primitive data types such as NSRect/CGRect, I’d be more happy to hear from you.

Krzysztof Kieślowski on Crafting an Experience

I recently re-watched the Three Colors trilogy by the late Polish director Krzysztof Kieślowski (1941-1996). I first watched the three films when I was in college. I think I understood more and saw more in the films than I did when I was younger. Growing up, having traveled, and some knowledge of the history and one of the languages spoken in the film (French) all helped. But even back then I was deeply touched by the films the first time I watched them.

I also appreciated more Kieślowski’s mastery of the film language.

The DVD edition of the trilogy comes with an extra feature called “Krzysztof Kieślowski’s Cinema Lesson”, in which the director talked about the making of one significant scenes in each of the trilogy. In particular, he talked about the filming process and the editing choices involved. This is the auteur dissecting his own work in details.

In Blue, for example, there’s a famous “sugar cube scene”, starting at 52′20″ and lasting about five seconds. Julie, the heroine, is found sitting in a café by Olivier. Julie has recently lost her husband and daughter in a car accident, and she tries hard to leave everything in her life behind. Olivier, who has worked with her husband on a song commissioned by the European Union, becomes a lover and puts an effort in finding her. Julie turns him down in the café. After he leaves, she plays with the sugar cube that comes with her coffee, while a flute player on the street is playing a number that sounds very similar to one of the motives in the commissioned (but yet finished) song.

Here’s Kieślowski talking about how he required the cube to be soaked in coffee in exactly five seconds, and the rationale behind it. Emphasis mine:

This is a sugar cube about to fall in the cup of coffee. What does this obsession with close-ups mean? Simply that we’re trying to show the heroine’s world from her point of view, to show that she sees these little things, things that are near her, by focusing on them, in order to demonstrate that the rest doesn’t matter to her.

It seems easy to film a sugar cube soaking up coffee, sucking it up and turning brown. […] We can start a stopwatch. It should take five and a half seconds or five seconds to be completely soaked. How to make sure that it only takes five seconds? Not so easy. Let’s take a regular sugar cube, like this one, and soak it in my coffee. I’ll start the stopwatch… Eight seconds. That’s three seconds too long. We had to prepare one that would be soaked in five seconds. We decided such a detail shouldn’t last more than five seconds. For half a day, my assistant tested all kinds of sugar cubes, […]

What do we care about a stupid sugar cube sucking up some stupid coffee? Nothing… unless we are, for a moment, in our heroine’s world. She dips a sugar cube in her coffee and focuses on it to reject the offer that the man who loves her just made her. […]

And when you ask me if I think about the viewer, about the viewer’s point of view — I go back to this stupid sugar cube — I try always to keep this in mind. We don’t do previews, test screenings. I believe an audience can take four and a half seconds of sugar soaking, but that eight and a half would be too much.

Cirque du Soleil and Radio Canada International

In the mid-1990s I owned a shortwave radio. It was one of the top-tier models made by Sangean, although I forgot the model number. That was before Internet radio came along, and you could still write to shortwave radio stations, with an attached international reply coupon, to request for their schedule and publications. So I wrote to Radio Canada International (RCI) to ask for their schedule and a booklet on how to install shortwave antennae. Sangean’s manual listed the addresses of the radio stations you could write to.

Because the city is in a basin, shortwave reception in Taipei wasn’t great, so an external antenna made sense. I bought one coupon from a stamp collecting store (that’s the one and only time I did that in my life), sent the letter, and a few weeks later, I received the schedule (which got updated every 6 months — because the atmospheric condition varies by season) and a guide to install various kinds of antennae. My father helped me install a long-wire antenna, the simplest type, on the roof of our apartment building (he also told me how shortwave radio was a restricted item in the 1950s in Taiwan, as the government didn’t want people to have too much access to the information of the outside world). We calculated the needed length and got the insulation right, dropped the connecting cable to my window, and connected to the radio’s external antenna socket. In addition to RCI, we were also able to listen to BBC World Service, Voice of America, Deutsche Welle, Radio Japan (NHK) and a few others with different reception qualities, a great supplement (my father would have said alternative) to ICRT, Taipei’s local English station.

RCI’s broadcast time in East Asia was short, 30 minutes in French and then 30 minutes in English, but the timing was perfect. The French broadcast started at 6 AM Taipei time, then followed by the English broadcast at 6:30. My Sangean radio had both a timer and tape recorder, so I let RCI wake me up every morning, then later on the bus I would play the English news and programs in my Aiwa cassette player. That was a great way to wake up and learn English.

Before a shortwave broadcast begins, the station plays a short sequence to identify itself, called “ident.” The ident of BBC World Service, “This is London calling,” is among the best known (check out this YouTube video, the version I heard is between 1:01 and 1:30). RCI, unsurprisingly, used the first bar of “O Canada,” the national anthem, followed by the bilingual announcement “Radio Canada International / Radio Canada International” (yes, one of them was in French…), and the sequence repeated itself two more times.

For me, one mystery about RCI’s broadcast, at least in East Asia, was that before the ident, they actually played three minutes of a song, in a language that was definitely neither French or English. I hummed the tune to many people, but no one I asked knew what it was. It was also strange, too, because not many shortwave stations did that, a full three minutes of a song before the ident.

I usually erased the tape on which I recorded each morning’s program, but one day I decided to save a tape for the opening song. A few years later I converted the tape into MP3, played the clip to a few more people, but I still hadn’t got any answer. And then I forgot the whole thing.

Just today, out of the blue, I realized I still kept the MP3 file, and now I could use services like Shazam to identify what the song was. And so I did. Turns out that the opening song was “Kumbalawe,” the opening song of a show by Cirque du Soleil. It totally made sense now, as they are Canadian (from Montréal). The song uses an invented language, though. Isn’t it amazing, that with a technology like music identification, we can find out the name of a song whose melody you never forget, after so many years?

On the other hand, I guess I could have written to RCI long ago to ask about the song. Still, it was a happy rediscovery of a song that made so many wonderful early mornings.

Permanently Convert WordPress Blogs to Static Pages

Since I started using Tumblr to host my blogs, I no longer used WordPress. It’s a great blog software, but even if you stopped writing blogs with it, you still need to run its software update from time to time for security. Because of that, plus other various personal reasons, I decided to convert all the WordPress blogs that I hosted on my own to static pages.

After some research online, the best solution I found for this task is Ammon Shepherd’s WPStatic utility. Shepherd wrote extensively about his project (usage, origin, more details). The project has been around for a while, and the version hosted on github works with the latest WordPress. Here’s a quick summary of what I do to “staticize” my blogs.

First, you need to know what WPStatic does:

  • It backs up your WordPress database (as a MySQL dump). It actually makes two backups: One before it makes the changes to your database, and one after.
  • It changes your blog’s permalink structure and close all the comments (for both posts and pages)
  • It runs wget to fetch every static pages linkable from the top page
  • It archives the MySQL dump, the static pages, and the files in wp-content into a .zip file

The script is only one file, called wpstatic. Just place the file in your blog root, then run chmod +x wpstatic to make it executable, and then run ./wpstatic -h for help. Some of the above actions, such as permalink structure modification, can be toggled off.

Then, here’s what I did:

  1. I ran some of the backups on Mac OS X and some on Linux. If you run the script on Linux, you need to change nothing. If you run the script on Mac OS X, you need to do a global replace to change every occurrence of sed -r to sed -E (that’s because the sed that comes with Mac OS X comes from BSD).

  2. Run ./wpstatic but don’t proceed with the wget backup (the actual page fetching). I ran this step to let the script close all the comments for me.

  3. Now this step is important. By default, the script changes your blog’s permalink strcuture to something like http://example.com/year/month/day/post-name. This didn’t work for me since some of my blog posts contain CJK titles, and CJK filenames work badly (a tragedy compounded by both URL encoding and file systems). Plus I have been using the structure like http://example.com/archives/123 for many years, and Google searchs on my blogs point to URLs like that. So, after the script changed the permalink structure, I had to go back to the admin page again, first change the permalink back to /archives/%post_id%, then I added a / at the end. The final permalink structure became /archives/%post_id%/, and this is great, because this actually caused the pages to be saved into something like /archives/123/index.html.

  4. Modifiy wp-includes/default-filters.php and comment out all actions that generate feed links. I turned off (commented out) these lines. You might want to remove more:

    add_action('wp_head','feed_links',2);
    add_action('wp_head','feed_links_extra',3);
    add_action('wp_head','rsd_link');
    add_action('wp_head','wlwmanifest_link');
    add_action('wp_head','rel_canonical');
    add_action('wp_head','wp_shortlink_wp_head',10,0);
    add_action('template_redirect','wp_shortlink_header',11,0);    
    
  5. Modify your theme. In particular, remove the Meta links in the sidebar, the feed links in the header and the footer, and anything (such as the comment feed link in the single post page) that may generate RSS feeds, trackbacks or pingbacks — if you’re converting your blog into static pages, those data is probably going to be meaningless since you are not updating the pages anymore. Search engines will still love your static pages.

  6. Run the script again, this time with wpstatic -p. You probably need to run a few times to make sure that it fetches no more than it should (i.e. no feeds, no trackback links, etc.)

  7. If your blog has database prefix wp_, the generated static page directory will be called wp-static. Inside the script has packed the MySQL dump and other static contents to wp-content.zip. You want to back that file up immediately.

  8. Backup your original blog directory for its scripts and settings.

  9. Remove the wp-content.zip in the generated static folder. Move static folder out. Remove the WordPress folder and rename the static folder to your blog’s directory name.

Your mileage may vary. And most of important of all, backup everything before you perform any of the above conversion steps.

Tim Cook, CEO of Apple, at the company memorial of Steve Jobs:

But there is one more thing he leaves us. He leaves us with each other. Because without him, Apple would have died in the late-90s, and the vast majority of us would have never met.

The works of Apple have touched and will continue to touch many people’s lives. They have certainly touched, and changed, mine. I gave up on programming when I decided to major in the humanities. Many years later, I realized that I still liked software and cared about it. In 2004 I wanted to pick up the skills again, but found the landscape wide and unwieldy. I didn’t know where to start. In the summer of that year, I bought the first Mac that I truly owned — a PowerBook G4 — and found a starting point: an open source project on Traditional Chinese input methods that a couple of good friends and I started. Mac OS X always had a great UNIX foundation, and Mac was already the choice of many open source developers. Friends taught me important things I needed to catch up, like how C++ had evolved during the years I wasn’t paying attention. At the same time I realized that the native app development on Mac OS X had its roots in NeXTSTEP. Years ago, I was once shown the amazing NeXT cubes at an acquaintance’s lab when he was doing his grad study in Hsinchu, Taiwan, and I never dreamt that one day I’d be able to program with the operating system’s software frameworks. It was a joy, almost illuminating, to discover and learn that so many good things had never lost their way. With the PowerBook, I learned how to program once more, and I’ve got to know many great people and made many good friends.

I wrote to Steve Jobs once. That was when Snow Leopard first came out in the late summer of 2009, and there were some issues with the new Traditional Chinese fonts. Those issues made them unusable in formal settings. With friends’ help and encouragement, we compiled the screenshots, filed Radar, and I finally got to draft the email. I didn’t expect him to read it, but the next day, someone at Apple replied, asking if I could send back the attachments that I “emailed Mr. Jobs.” And the issues with the fonts were eventually resolved. It’s not the famous one-liner that people loved to post online, and it’s easy for us to attribute everything that a company does to one person. Still, that is important: there are people at Apple that care.

I appreciate that Apple shared the recording of its company event with the rest of the world. It was solemn but at times lighthearted, comforting and at the same time gracefully cheering. It gave me a way to say goodbye, even though I never met him.

Nobody is an island, and what a great loss it was. Norah Jones’s songs soothed, and Coldplay lifted us up. It is truly, in Apple’s own way, a celebration of Steve’s life.

Goodbye, Steve.

Thank you, Steve.

Flickr OAuth Support in ObjectiveFlickr

I’ve added Flickr OAuth support to ObjectiveFlickr over the weekend.

Flickr announced in June that they started requiring new apps to use the OAuth-based authentication process, and the previous authentication process was deprecated and will be eventually phased out in early 2012. I recommend developers who use ObjectiveFlickr check out the latest version to start the transition.

ObjectiveFlickr is hosted on github.

What’s Added to ObjectiveFlickr

The main class, OFFlickrAPIContext, has two new methods:

  • -fetchOAuthRequestTokenWithCallbackURL: makes a request to Flickr for a request token
  • -fetchOAuthAccessTokenWithRequestToken:verifier: makes a request to Flickr to exchange the user-authorized request token (see below) for an access token

Two new delegate methods are added to the OFFlickrAPIRequestDelegate protocol, which you need to implement:

  • -flickrAPIRequest:didObtainOAuthRequestToken:secret: handles the result of request token fetch
  • -flickrAPIRequest:didObtainOAuthAccessToken:secret: userFullName:userName:userName:userName: handles the result of access token fetch

The class OFFlickrAPIContext now also support properties OAuthToken and OAuthTokenSecret. It also has a new method -userAuthorizationURLWithRequestToken:requestedPermission: that returns the URL for user authorization, which you need to open in a new browser window.

A new sample app, OAuthTransitionMac, is also added to show how to use the new authentication API along with the old. It also shows you how to exchange the existing auth token (from the old API) for an OAuth-based token. OAuthTransitionMac uses ARC, so it also shows how to set up the project (since ObjectiveFlickr still uses manual retain/release).

The iPhone app, SnapAndRun, has also been updated to show how to use OAuth in your iOS apps. Many people have had difficulties in linking the ObjectiveFlickr library externally, particularly on the iOS Simulator. So I simply put everything in the project. I also realized that SnapAndRun is unnecessarily complicated for its purpose — this is one thing you may be able to help improve (see the section “ObjectiveFlickr Needs Your Contribution” below).

What You Need to Do

  1. Read Flickr’s own document to get an overview of the new authentication process. A glimpse of the flow chart should suffice for knowing what needs to be done.

  2. If your app doesn’t have callback URL support, this is the time to add it. Refer to SnapAndRun (which has snapnrun://) and OAuthTransitionMac (oatransdemo://) for how to add URL support. In essence, you need to handle the incoming URL in your app delegate. This applies to both Mac and iOS apps.

  3. When your app requests the user to authenticate, and when the user agrees to proceed, make a call using -fetchOAuthRequestTokenWithCallbackURL: to obtain a request token. Pass your app’s callback URL (e.g. oatransdemo://callback) to the method.

  4. Implement -flickrAPIRequest:didObtainOAuthRequestToken:secret:. This delegate method is called after the request token is successfully obtained, and you need to set your Flickr context’s OAuth and OAuthSecret properties to the returned token and secret. Setting those two properties is required so that the subsequent access token fetch request will be signed correctly.

  5. If any error occurs, the delegate method -flickrAPIRequest:didFailWithError: will still be called. The NSError object passed will have the error code OFFlickrAPIRequestOAuthError, and the userInfo dictionary will have a key-value pair under OFFlickrAPIRequestOAuthErrorUserInfoKey, which is simply the error message that Flickr returns.

  6. In the same delegate method, call -userAuthorizationURLWithRequestToken:requestedPermission: to obtain the user authorization URL. The permission will be something like OFFlickrWritePermission (see ObjectiveFlickr.h for possible values). Then, open the URL in a separate browser. Use -[UIApplication openURL:] on iOS, -[NSWorkspace openURL:] on Mac.

  7. After the user authorized your app to use their data, your app will be activated again. Parse the URL’s query with the OFExtractOAuthCallback function supplied in OFUtilities.h. You should obtain the authorized request token and verifier now.

  8. Obtain the actual access token with -fetchOAuthAccessTokenWithRequestToken:verifier:

  9. Implement the delegate method -flickrAPIRequest:didObtainOAuthAccessToken:secret: userFullName:userName:userName:userName:. This handles the access token fetch, and set your Flickr context’s OAuth and OAuthSecret to the final access token/secret pair that is passed back to the delegate method. Again, any error occurred will be passed to the error handling delegate method.

  10. Done! You can start making API calls and upload photos with your existing code base.

Implementation Details

Flickr’s OAuth uses HMAC-SHA1 signing, which I decided to implement inside ObjectiveFlickr without depending on any other library. At this point our app projects probably have too many dependencies, and I want the changes to OF and to the way you use it to be minimal.

I’ve moved the inline functions out of OFUtilities.h to a new OFUtilities.m. This makes projects that use ARC happier, as inline functions cannot make retain/release calls under ARC. A Base64 encoding routine written by Matt Gallagher is also embedded in OFUtilities.m and is not exposed to the outside (i.e. the functions are declared static). Again this is to minimize dependencies.

To minimize implicit state changes, I decided to require you to set your Flickr context’s OAuthToken and OAuthTokenSecret in your delegate methods. Having OFFlickrRequest do it for you implicit seems to be a bad idea architecturally.

The current version “burns in” Flickr’s auth service endpoints in the code. It may be a good idea to make them properties in OFFlickrContext. Flickr now also has HTTPS endpoints. I’ll be glad to merge your pull requests if you have tried them out and made changes to the framework.

The API calling routines in OFFlickrRequest now has two code paths. If both the OAuthToken and OAuthTokenSecret properties are present (i.e. not nil) in your OFFlickrContext object, then the request object uses the OAuth-based signing to make the call. Otherwise it falls back to use the traditional authToken. If you’re only interested in public data, there’s no need for the authentication.

The OAuth spec allows three different ways of passing the signature. To simplify, I didn’t use the Authorization HTTP header, but chose to put the signature as part of the HTTP POST body or HTTP GET query.

ObjectiveFlickr Needs Your Contribution

Many projects use ObjectiveFlickr, and I need your help.

Please let me know if ObjectiveFlickr works or breaks your project. I no longer work on projects that need to support OSes older than Mac OS X 10.6 or iOS 4.3. I tried to be careful, for example I don’t use fast enumeration (lest it break OS X 10.4 support), but there may be places I use properties instead of old-school getters and setters in the library (not samples — they now only target the latest Mac OS X/iOS). You can help expand the test coverage in this area.

Eventually support for older OS or SDKs will and should be dropped. So much happened during the past two years, and more will come! It is important the shed some skin in order to catch up with those new developments in the Mac/iOS development landscape.

OAuth requires the consumer of an API (your app) to sign their calls and it involves a lot of URI string escaping. Handling improperly and you might see strange errors when you set a fancy title to a photo. The chars that need to be escaped, in addition to the default charset, are defined in the kEscapeChars constant in ObjectiveFlickr.m. I tried a number of common cases, but again the test coverage is very small. This is another area that you may be able to help.

ObjectiveFlickr was first developed in 2006 and was rewritten in early 2009 during the iOS 2.2 days. And it shows: the SnapAndRun iPhone sample app is unnecessarily complicated, and many conventions are out of date. A sample app written from ground up should be able to demo how to use the library more succinctly. The library’s own Xcode project has also needed to be remade so that app projects can again use cross-project reference to ObjectiveFlickr.

Then there’s documentation and code comments. Or synchronous API calls (so that you can make calls in other threads easily) or even NSOperation/block-based request methods. A lot of things can be and need to be done in a framework project like ObjectiveFlickr. Your contribution will benefit the Flickr and Mac/iOS development community a lot.

Conclusion

It’s highly recommended that you start migrating to use Flickr’s OAuth-based authentication, and ObjectiveFlickr now has enough support for it. The work for the transition shouldn’t be too much, as you only need to change your authentication process. Hopefully ObjectiveFlickr continues to serve your apps well. Best of luck with the transition!

The Low-Hanging Fruits and Bumpy Ride into the Future

I bought Tyler Cowen's book “The Great Stagnation" and finished it the same evening. I enjoyed it much like I did from reading books like "Day of Empire”, in that they shed light on understanding the world we live in and where we are going from that. Cowen’s book is entirely about the situation in the US, but much of his analysis should apply to many other places in the world. He also talked about the political implication of such situation, or the “great stagnation”, and his view on China’s catching up, in an interview with The Economist.

Cowen’s main point in the book is that we have exhausted what the technological advances of the past few decades have been able to bring us — hence the first chapter’s name, “The Low-Hanging Fruit We Ate.” New advances are harder to come by, as evidenced by the diminished return in ever-increasing R&D spending. The situation is worsened by two additional factors: The US education system is inadequate, especially the K-12, and the hubris in the financial system, based on the assumption that previous growth trend would continue, finally led to all kinds of troubles that we are still seeing today.

I’m not so sure about Cowen’s view on the Internet. He devotes a whole chapter on it arguing that, while the Internet has brought us many great stuff, it creates few jobs, and people may even spend less because of all the (mostly free) fun they get from the Internet. Certainly eBay (one of his low job creation examples) does not hire many people — not at a scale of the Big Cars — but what about the benefits of disintermediation, which reduces the friction in transaction and makes many sellers reachable to more potential buyers (indie music comes to mind). But Cowen only spends a page to say that Google and Facebook hire less than 30k people combined and doesn’t give us numbers to argue that it’s indeed “not saving the revenue-generating sector of the economy.” What’s worrying about such argument is that, with politicians fixated on job creation (and that’s certainly not confined to the US), when they start pointing fingers at the tech industry that they aren’t creating many jobs, we may start to wonder where this may lead us.

One interesting, if also somber, implication of “the great stagnation” is political polarization, which Cowen doesn’t mention in his book but talked about in the aforementioned interview. This seems a good perspective to understand the political dramas (sometimes ludicrously baffling to foreigners) in the US.

Cowen prescribes bitter remedies: “Raise the social status of scientists” — and this is not easy as it sounds. He warns us that the recession will be long, although Japan shows the US how to cope with the decline as a society to a large extent. There’s a twist to the theme here, because Cowen also warns us to be ready when more low-hanging fruit comes:

[…] because sometimes low-hanging fruit is dangerous. The last time the world had a major dose of low-hanging fruit, a few countries didn’t handle it very well, including the Axis powers, the Soviet Union, and Communist China, among others.

Without the new technologies of the time, the totalitarian mistakes of the twentieth century would not have been possible. […] The record-keeping techniques of mass bureaucracy were used to control and often kill other human beings en masse. Only after bitter experience did fascist ideas become less popular, and social and political norms subsequently evolved to protect electorates against the fascist temptation.

So expect a bumpy ride to the future. I think that’s a good advice to everyone who’s lived the optimistic 1990s. The ending words are worth quoting in full:

In the meantime, we need to be prepared for a recession that could last longer than we are used to. We need to be prepared for the possibility that the growth slowdown could continue once the immediate recession passes. Part of science is coming to terms with its limits. The rate of scientific progress will continue to be uneven, sometimes grossly so. Yet reason and science have never been more important: If nothing else, a more reasonable and more scientific understanding of our predicament can help us cope, both intellectually and emotionally.

Back to the hard problems.

Indeed.

Using OS X’s Built-in ICU Library in Your Own Project

Unicode string is usually not difficult to handle1. The tricky part comes when you need to have knowledge of Unicode on things like:

  • Turning strings like “Café” to upper case or lower case (the toupper() or tolower() in Standard C Library won’t work)
  • Stripping all punctuation symbols. This is not as easy as it seems. How do you also strip various European quotation marks as well as full-width ones used in East Asian text?

Some languages, like Java, already have such Unicode knowledge baked in. For C and C++, the standard library to use is IBM’s ICU.

You can either build ICU from its source code, or use tools like MacPorts to install it for you. But you may also wonder: Since OS X has great internationalization support, perhaps OS X also uses ICU?

It turns out that OS X does include a version of pre-built ICU library2 placed in /usr/library/libicucore.dylib, but for various reasons — and I also assume it has much to do with the fact that ICU is written in C++, and library versioning with C++ is a pain — it does not include the headers for you to use.

Here’s how you can link against the built-in library:

  • Install a copy of the latest ICU
  • In your project file, set header path to include the installed ICU header (e.g. /opt/local/include)
  • Define U_DISABLE_RENAMING in your project (i.e. setting the compiler flag -DU_DISABLE_RENAMING=1)— this turns off ICU’s version renaming scheme so that you’ll be able to link against an older library (the one that comes with OS X in this case) with the latest header
  • Link against libicucore (with the linker flag -licucore)

As the name implies, it only contains the core ICU library, so it lacks quite a few things. For example, C++ methods such as UnicodeString::toUTF8, classes as StringPiece, StringByteSink are missing. A good way to check availability is to use the UNIX tool nm to dump libicucore.dylib. Your best bet is to stick to the C API. On the other hand, this saves the trouble of including your own copy of ICU.


  1. Read Joel Spolsky’s excellent primer on Unicode if it still isn’t to you. 

  2. Many of NS/CFString and NSCharacterSet’s internationalization features use ICU under the hood. I haven’t tested it, but I believe the tips provided here can also be used on iOS. 

Brad Cox’s “Object-Oriented Programming” (1986)

A few weeks ago I had a chance to re-read this particular book by Brad Cox1. It is also the first book that introduced the Objective-C language to the world.

It’s not a heavy book — at 274 pages including index, Cox has outlined his solution for the software reusability problem in a succinct language (yes, pun intended). The book spends the first two chapters on why reusability is essential to mitigate the complexity in building a large system, what Cox calls “software crisis”, and on how the object-oriented programming paradigm, which Cox describes using the analogy of integrated circuits (“Software ICs”), is in turn the way to achieve reusability.

In the following chapters, Cox describes the language that he designed to fulfill his vision of Software IC: Objective-C. A short introduction and evaluation of Smalltalk-80 is given, and comparisons with Ada and then C++ are also provided. Cox likes many aspects of Smalltalk-80: The language is designed for personal computing, with interaction and graphics in mind (remember this is the language that gives the birth to modern graphical user interface), and it has many advanced features in its time: virtual machines with persistent stores (so say “saving files” is an unnecessary feature), a language built on blocks (think lambda) and message passing mechanisms, an IDE, and so on. As for why Smalltalk-80 didn’t become mainstream, Cox’s explanation is that Smalltalk-80 is a language designed for research purposes. To be of practical value, a hybrid language that can work with existing infrastructure will have a better chance:

My own interest in hybrid object-oriented language developed from the dream that organizational computing and personal computing could meet each other halfway, via a bridge over which arbitrary data could flow with minimum programming intervention. The bridge is created by providing the organizational computer with its own object-oriented programming language so that data can be put into object form. Object passivation [persistence] machinery […] can automatically convert this data into a form that Smalltalk can read and vice versa, so that much of the information transfer could be done automatically. This, and C’s ability to access databases maintained on older languages like FORTRAN [Cox also mentions COBOL in previous paragraph], creates a bridge between data locked in the mainframe and the user by relying on the ability of Smalltalk-80 to build user interfaces rapidly.

For the rest of the book, Cox describes the features of Objective-C in detail with an example of using the language to build a visual graph manipulation tool. Some syntactical features are no longer found in today’s Objective-C (e.g. the = ClassName declaration, and the feature that class declaration and definition are one unified compilation unit, much like Java, instead of separate parts). In the first of the last two chapters, Cox outlines the architecture of an “iconic user interface” — the model-view-controller paradigm (MVC), which was also introduced by Smalltalk. The last chapter briefly discusses the runtime environment (such as garbage collection) and possibilities for other higher-level language features, including “virtual object memories”2, concurrency (using coroutines), distributed systems and coordination systems3.

Overall, I find the book full of insights that still resonate, 25 years after its publication. It’s interesting to take a historical perspective on how much progress we have made since then. Objective-C had never been as popular as C++, a “competitor” in some senses, or as Java, some features of which are actually influenced by Objective-C (such as interfaces, or what Objective-C calls protocols). NeXTSTEP, the first platform that heavily used Objective-C as the main language, was never mainstream. It takes Mac OS X, and then iOS, to propel it to one of the most popular programming languages worldwide. And many ideas are still waiting for more refined realizations. Some, like garbage collection, finally become an option on desktop platform. Some, like concurrency, are still things that frustrate programmers, expert or novice alike. Perhaps, like what some who teach functional programming would like to show, we are past the apex of the object-oriented model, and a new paradigm shift awaits.


  1. Cox, Brad J. Object-Oriented Programming: An Evolutionary Approach. Addison-Wesley, 1986. I’m referring to the first edition here. If you know the differences between its first and second editions, I’d appreciate it if you can write something about it. 

  2. We can identify some of Cox’s description of such a “capability-based” addressing system in today’s uniform resource identifiers (URI). 

  3. Although gcc has been supporting gc-capable Objective-C for years, it was not possible to use it in Mac applications due to lack of framework support. Apple eventually came up with its own “mix of curry” of concurrent, generational garbage collector, with runtime and full framework support, in 2006. Objective-C also has a feature called Distributed Objects, which is often used among Mac applications for inter-process communications. Library support for operation-based concurrency became part of the Mac OS X 10.5 framework, again since 2006.