Unforeseen Consequences

VMware Fusion 2.0 Beta 1 is out.

You can now see what I spent most of my summer in 2007 working on.

Before:

Now:

Designing and implementing the window in the second screenshot was my intern project at VMware. It doesn’t look like much, but some of the hardest work I did was messing around in the back end to make it possible to, for example, display the power state and guest OS name in the list. (Long story.)

It feels really weird to have what is probably the most challenging and significant programming work I’d done until then out in the wild¹. That it’s such a highly visible element of Fusion’s UI is just ridiculous. You can’t avoid seeing it. This is a watershed moment in my life as a software developer.

Of course this isn’t the flagship feature of Fusion 2.0; the major changes are multi-mon support in Unity, experimental DirectX 9 Shader Model 2 support, the inclusion of VMware Importer, and seamless printing from Windows guests.

Congrats to the Fusion team on the release!

¹ The current holder of the “most challenging and significant” record goes to what appears as “P3″ in those screenshots – a small operating system kernel.

Sometimes, I need to perform a repetitive action on a bunch of files. Renaming them, say, or organizing them into a specific directory structure. When I need to do this, I always write a script to do so. Often, by the time I’m done writing, testing and debugging the script, I could have already done everything manually, probably several times. So isn’t this dumb? My time-saving measure is actually costing me time.

Yes, but the big difference is that writing, testing and debugging the script is interesting. Renaming files by hand sucks. Programmers aren’t happy doing repetitive stuff. We’re trained to see patterns in tasks, abstract out the parts of the pattern that change, and formulate everything with as little redundancy as possible.

So the mistake is to think that my highest priority while renaming those files is to do it as fast as possible. My priority is to keep my mind busy. I’ve got to do this monotonous task; I might as well make it as interesting as possible for myself. To write the script, I have to think about abstract things. To rename the files manually, I don’t have to think; I just type.

Plus, there’s a chance I’ll learn something new while writing the script. One time I had to batch process a bunch of files to do something to each line. Instead of opening each one in a text editor and doing find-and-replace, I learned sed for the occasion. After having some trouble with it, I figured out that sed from the command line has issues with tabs, so I switched to perl -npe instead. But hey, now I know sed. For when I need to edit streams.

It’s a step on the path for normals to understand programmers. Beneath everything we do, there’s an ancillary goal of keeping our minds busy. Often this comes out as trying to find non-repetitive ways to do repetitive tasks, even if those non-repetitive ways are more complicated or slower.

This might actually be why I’m interested in linguistics. People are constantly saying stuff, so if I can find a way to use the things people say to keep my mind occupied, that’s great. Linguistics gives me an organized set of stuff to think about the things people say. It’s a wonder more programmers aren’t drawn to linguistics.

After lots of agonizing every time I get a new computer or hard drive, I’ve finally come up with naming schemes for my computers and drives.

Computers: Tokyo Metro lines

Not sure what drove me to use these, but the names sound cool. I’m going in order of age, skipping ones that are hard to type.

• My laptop is Ginza.
• My desktop is Hibiya.
• Whatever my next computer is will be Tozai.

Bonus: if I need to name VMs, I can name them after Toei lines (Asakusa, Mita, Shinjuku, and Oedo) and it makes some kind of analogical sense.

Hard drives/volumes: Bay Area cities

In my desktop:

• My main Mac OS X volume is “SanFrancisco”, because it’s full of cool stuff.
• My external backup for SanFrancisco is “Oakland”, because they’re close to each other.
• My Windows volume is “SanJose”, because although Windows is more populous than the Mac OS, it’s not as awesome. (Part that doesn’t fit: SF has a higher crime rate.)
• My FAT32 volume I use for sharing stuff between Windows and OS X is “REDWOODCITY” — the halfway point between SF and SJ.
• My Linux partition is “Berkeley” — full of hippies.

In my laptop:

• My main volume is “SantaCruz” — out of the way, but still cool.
• My Tiger volume (I keep it around for Classic) is “Alviso”, because it’s a historical relic.

My USB stick, of course, is “Caltrans”.

I have a shared hosting account with a web host. I have several sites on there, including a Rails web app that is pretty important to the Cocoa course I’m teaching (it manages online submissions and grading). I’ve also got the Subversion repository hosting the code for that app.

Two days ago, I got an email from my hosting provider telling me that two of the drives in the server’s RAID had failed, and I should back up as much of my data as possible because they were going to have to take the server down to replace the drives.

Back up my data? They must mean I should just grab the data that I need urgent access to, in case I need it while the server’s down. I appreciated the warning, but I didn’t do anything.

The language of the email nagged at me, though, and the next day I downloaded all the critical stuff from the server: the databases from the Rails apps I have live, my Subversion repo, and a few other things. The “what’s the status of my server” page on the host’s website had repeated warnings to back up “as much of your data as possible”, and it was worrying me.

Turns out I was worried with good cause: they’ve confirmed that they don’t back up their customers’ data by default. If you want nightly backups, that’s an add-on to your account. You have to pay extra.

Is it just me, or is that rather unprofessional for a web hosting company? I’d entrusted a lot of data to them; in retrospect, I realize if my home directory on the server had gotten wiped out, I’d have lost a lot of important data. At least in this case they gave me fair warning; but what if something more catastrophic were to happen? What if there was a freak earthquake under the data center and every drive head in the server struck the platter? What would they tell their customers? “Everything’s hosed and we don’t have backups, all your data is gone forever, sorry”? My feeling is, if other people are paying you to hold their data, it’s your responsibility (i.e. not optional) to ensure the data you’re entrusted with is safe. I’d assumed backups were included; whoever heard of a hosting company that doesn’t back up data?

Update: turns out they do nightly images of the entire server, to protect against disasters such as this one. I guess the add-on gets you the ability to request a restore from backup of your home directory, in case you screw something up. Sorry for the confusion, but I think they should definitely be clearer about that.

Even though nothing bad happened, this little event has gotten me thinking about backups. I don’t have a backup of my main machine’s hard drive that’s anywhere near current. The idea of losing it all at a stroke is suddenly much nearer now.

This is going to be a long UI-design-wanking essay mainly about the iPhone. No, I don’t have one, and I’m not going to get one. But I have a lot to say about its UI.

You’ll barely see Apple mention the iPhone anywhere without the word “revolutionary” nearby, like an annoyingly chipper sidekick. The somewhat grandiose claim is that they’ve reinvented the phone, and created a “revolutionary Internet communications device”. I claim that the phone and communicator functionalities are not revolutionary at all. Neither is the idea of putting the two together. Neither, for that matter, is the idea of putting a real browser in a handheld device. In fact, I claim there’s nothing revolutionary about the iPhone yet. There are things that are inventive, and one thing that is a little bit revolutionary (if that’s possible) but didn’t quite make it.

When the Macintosh was first introduced, 23 years ago, it was the first mass-market personal computer to have a GUI. Nobody knew what to do with a GUI on a computer. That’s why the GUI needed a metaphor — some consistent theme that would allow users to infer some unknown GUI element’s behavior by extrapolating. The Macintosh GUI’s metaphor was the desktop.

Applications were like desk utensils — pen and lined paper (MacWrite), pencil and blank paper (MacDraw/MacPaint), pad of graph paper, whatever. Documents were just that — documents. It’s hard to go from thinking about computer documents to paper documents and make the connection, but that’s the origin. You kept groups of related documents in folders. (Of course the concept of folders was not new, but they were given the uninformative name “directory” on other systems.) Unwanted documents would go in the trash. Behind everything you did was the omnipresent surface of your desk — the Desktop.

The way the Macintosh designers constructed this metaphor was with icons and consistent behavior. Application, document and folder icons were all designed around this idea. In fact, every decision about the Macintosh GUI was made with the Desktop metaphor in mind. As a result, the behavior of anything on the system was consistent. Users could use the Desktop metaphor to figure out what to do, and everything made sense. This was extremely important at a time when Apple was trying to sell GUIs to people who had never seen such a thing on a computer.

Mac OS X has strayed from this metaphor. Mac OS X does a lot of things right, and also a lot of things wrong. Exactly what, and why, is not what I want to talk about here (although I will make it known that I think Mac OS X’s Finder blows in comparison to the Classic Mac OS Finder). Suffice it to say that Mac OS X has dropped the metaphor completely. This isn’t necessarily all bad; not at all — these days everyone and his mom knows how to use a GUI, so rigid adherence to a metaphor isn’t vital.

To me, the real genius of the iPhone lies in two things: Multi-Touch, and how it manages to come tantalizingly close to successfully pulling off a new UI metaphor, for the first time in over 20 years.

I need hardly point out how much potential Multi-Touch has. This is a generalization that’s been waiting to happen for years. In the original Desktop metaphor, the mouse cursor was our hand’s proxy in the computer’s world. Well, we have two hands. Why not have a computer-world proxy for each? Of course this is far from how the iPhone treats Multi-Touch. As I’ve said in the past, zooming in and out is a woefully mundane application of Multi-Touch. However, in a handheld device, thinking of pointers as hand proxies doesn’t make much sense. The iPhone doesn’t even have a pointer. But if Apple has the technology, and it eventually finds its way into Mac OS X (distinct from OS X, I will have you note), then we may have a UI interaction revolution on our hands. Multi-Touch as it exists in the iPhone today is not revolutionary. It portends revolutions — and the right people seem to have taken notice of it.

Now for the iPhone’s new metaphor. I had this realization when I opened the app that nobody takes any notice of. It has never appeared in any kind of marketing for the iPhone. Nobody mentions it in reviews. But for me, it shed light on what the iPhone UI is all about.

This neglected app is Calculator. I was just fooling around with all the apps, and for completeness’ sake I tapped on Calculator. Then I saw the iPhone transform itself into a pocket calculator, right there in my hand. Suddenly, this $600 piece of electronics was just a basic pocket calculator that didn’t even have a square root function.

Of all the iPhone apps, Calculator upholds the new metaphor the best. There isn’t a single word to sum up the metaphor like there is with “Desktop”. The best I can come up with is “shape-shifting”.

Steve Jobs mentioned this when he was introducing the iPhone, but he did not harp on it enough. He was knocking the BlackBerry and Treo and co. for having hardware buttons, and the key phrase he used was that the buttons should change to suit what you’re doing. They knew that in order for this idea to be successful, they had to go all out on the screen. They did so, and it has paid off. The device’s face around the screen is black, and the screen is so high-quality that when it goes black, it exactly matches the actual black plastic surrounding it. That’s what lets Calculator pull off its little deception — it can extend the phone’s body onto the screen, and pretty much make the body grow new buttons. That was the whole idea.

Unfortunately, I’m not sure the app designers threw themselves into it wholeheartedly enough. Partially I believe it’s also a metaphor that’s impossible to stick to in some cases. In the case of Calculator, it’s clear enough what a pocket calculator looks like. What does a pocket web browser look like? In that case you have to acknowledge the existence of a screen. Or how about a pocket weather checker? In that case there really is no such thing, so the metaphor inevitably breaks down somewhat.

The designers kept up the pretense in other, critical, ways. To a far greater extent than in any desktop (in the physical sense) computer GUI, buttons in the iPhone’s UI are meant to mimic hardware buttons, just as Steve Jobs implied. So the trick is to make iPhone GUI widgets mimic real buttons as closely as possible. This is evident from the moment you turn it on. You have to slide your finger across on a slider to unlock the phone. If you let go of the slider partway, it is animated springing back into place. The farther you’ve dragged it from the left end, the more quickly it springs back. On/off toggles are not checkboxes as they would be on a computer; they’re like physical switches (like the Hold switch on an iPod).

Then, of course, there’s the scrolling. They’ve eschewed the traditional scrollbars in favor of the well-known finger flick. It’s designed to make you feel like you’re interacting with a real physical object, not just one on a smooth glass screen, and it succeeds remarkably. This is aided by what initially looked like pointless eye candy — the bounce when you hit the top or bottom of a list. It makes the entire experience more organic and fluid. It aids your suspension of disbelief.

There are other touches (no pun intended). An underlying rule that I’ve noticed seems to be: any time your finger touches the screen, something must visibly react. For example, if you double-tap a map to zoom in when you’re already at maximum zoom, it will bounce: zoom in briefly, then bounce back to where it was. The same behavior happens if you zoom in by spreading two fingers: the map will stretch with your fingers, scaling and getting jaggies, but then it will spring back once you let go. Of course, the springing action is determined by how far beyond the limit you tried to zoom in. If you touch the interface, it will never sit there unresponsive (unless the thing has crashed or frozen, of course). It will always give you some feedback. This is incredibly important to the iPhone’s metaphor — it turns into whatever handheld gizmo you need it to be. All it has is a touch-sensitive screen to convince you that it has actually made this transformation, so even the tiniest details matter.

This brings me to a final point: I now see a perfectly good justification for Apple’s decision not to release a true iPhone SDK immediately. I still disagree with the way they spun it (“Here’s an SDK for you! It’s really amazing! It’s called…Web 2.0!”), but I agree with the decision to hold off for now. This may not be their justification, but it is a justification.

The iPhone’s intended interaction metaphor is new. It is not flawless, and it still has some vestiges of the old Desktop metaphor, but it is new enough that developers cannot be expected to dive straight into it and write apps that conform to it. And since this metaphor is so critical to how Apple wants people to use the iPhone, and how they want to set the iPhone apart from other devices like it, it’s vital that they protect the metaphor until they’re confident developers understand it and will respect it.

In short, they’re trying to introduce a new way to interact with electronics. It’s not quite the introduction of the Macintosh all over again, but it’s tricky enough that they know they’ve got to tread carefully. Plus, this time I don’t think everyone realizes the new metaphor’s existence and so are less likely to treat it as new. But if Apple gets its way, they will blaze the trail for this new metaphor — like they did with the Macintosh. Except this time, they have the lessons they learned from the Macintosh to help them.

Hark, my brethren, hear ye, hear ye. We gather here today to give our thanks and praise to that which hath on many an occasion delivered us from the clutches of Doom: the brave, heroic Segmentation Fault.

In our lives, we are all forced to navigate the treacherous realm of main memory, for without it, we would not remember things. Main memory is a harsh abode, and we have no choice but to inhabit it. It is contested. It is limited. Yet we cannot live without it.

The Kernel, in Its infinite wisdom and mercy, spake thusly: “Let there be Abstraction!” And lo, there was. The Kernel’s children were blessed with Virtual Memory. Through this blessing we may enjoy the fruits of main memory, safe from the perils of territorial disputes. The almighty and inscrutable Kernel watches over us from on high, preserving the blessing, ever watchful that we do not abuse that which It hath given us, for with this blessing cometh great danger. Should we stray from the confines of Virtual Memory, we step into a dark void from which we may never return.

We are but humans. Though the Kernel created us in Its own image, through our own failings and weaknesses we are imperfect. In our endless journey along the path that the Kernel hath laid for us, we err and misstep. The Kernel hath given us a Commandment, “Thou Shalt Not Dereference Invalid Pointers”, and though we fear the Kernel we may yet disobey. Our sins cause the Kernel to become full of Wrath, and the Kernel then visiteth upon us the Segmentation Fault.

For our sins, it is the Segmentation Fault that suffereth. Were it not for the Segmentation Fault its Labors, we may be condemned to Data Corruption. Though the terrain be perilous, and the Kernel Wrathful, the Segmentation Fault guards us eternally. The Segmentation Fault devoteth its life to us, and for this, our gratitude is eternal. Let us give thanks.

Amen.

So I just got this spam:

Dear Customer, [my Google account name].

Notice: You can acess your account from a foreign IP or country by getting am (I.A.C) International Access Code, by contacting our local brances close to you.

First, there’s no particular reason for me to identify myself to any bank using the same account name as I use with Google. Like…why would I do that?

Secondly, the IP address they give is actually a server owned by the same outfit that owns the SMTP server that this message came from. They’re both in France, registered to some bloke named Thierry Jamet. So yes, someone logged in to my account on their server in France from a machine in France, without an International Access Code (whatever the hell that is). So wait, why would I need an International Access Code for that? And why would I be logging into a French bank account from France? I haven’t been to France for ages.

Thirdly, if they really want to make me think they’re a bank, they could at least spell-check their spam. It takes only marginal effort compared to setting up the scam in the first place. Maybe spending two minutes spell-checking would give them more hits.

But what really galls me is not how stupid the spammers are (“your protection is our future medal”? What the bejesus?), it’s knowing that there are people even more stupid — people who will actually fall for this.

Sigh.

For all you self-styled haxors out there, here’s a little challenge:

1. Install some form of Linux.
2. Quit X, if it’s there, so you’re just at a console (bash).
3. Change to root with su and then cd /
4. Do rm -rf /*
5. Now, starting with only shell builtins, rebuild a working OS from scratch, such that if you restart the machine, you will be able to log back in to a functional shell.

Tips:

• echo * will list files in the current directory.
• You should set umask to have the executable bits set.
• I/O redirection is the easiest way to write to files.
• If you’re a wimp, you may preserve for yourself a binary of GCC (somehow; that’s up to you)
• You may assume there are no hardware failures or power outages.

Note: you should not actually attempt this.

I haven’t posted in forever because I’ve been hell of busy, but right now I feel the need to vent.

Language technologies is very much an active area of research nowadays. With the rise in popularity of the intertubes, it’s easy to see why. Speech recognition, OCR, word sense disambiguation, grammar checking, spam filtering…all sorts of language-related tasks need to be performed in batch by computers. Hence, the diligent research into improving methods of doing these tasks.

However, I spent this evening studying for a test in a language technologies class. The class is concerned with using statistics to model language — that is, to create a probabilistic process that approximates and predicts the behavior of natural language, based on training data. One very simple such model is one that estimates the probability of a word occurring simply by using its probability in the training data — dividing its frequency by the total number of words in the data. This is called the unigram model. I’m giving this technical background to give some context to why I think language technologies suck.

Through studying for this test, I have realized the Two Fundamental Truths about language technologies:

1. Got a problem? Get more training data, and your problem will disappear.
2. Got a fancy new language model? The unigram model is just as good or better.

The professor has told some rather demotivating stories related to the Second Fundamental Truth. First, entire theses have been written in which the author devises some complicated, detailed, fancy, awesome new language model, and it turns out that it is, at best, only very slightly better than the unigram model. Keep in mind that the unigram model is just about the most brain-dead language model one could possibly come up with. Then, it turns out that by interpolating the fancy new model with the unigram model (interpolating is a process by which two language models can be combined), a model that’s better than the fancy one can be obtained.

Second, there was a large-scale effort at IBM in the 1980s to develop the Mother of All Decision Tree Language Models. Decision tree language models are a family of models that have lots of expressive power, but are computationally very difficult to generate. You have to get absurd amounts of data and crunch numbers by the trainload. IBM spent years of research, several CPU-decades of computer time, and the biggest collection of training data known to mankind to create the Mother of All Decision Tree Language Models. At the end of it all, when the computers finally spat out the model, the researchers gathered ’round eagerly to see what they had come up with; to see how much the Mother of All Decision Tree Language Models would beat the pants off the silly n-gram models.

The result? It was a little bit better than the unigram model. Just a little bit. They interpolated it with the unigram model and it ended up being sort of like the trigram model in terms of quality. They published a paper describing their extremely depressing results, then probably went home and drank themselves into comas.

The best part? The IBM paper is a landmark paper in the field of language technologies. A paper that says, essentially, you can toil for years and still you’re not going to get a whole heck of a lot better than the unigram model. Well, say LT people, we’d better put our thinking caps on and keep looking for the One Language Model to Rule Them All! Well, say I, if your very best effort is pretty much just as good as the unigram model, which you readily admit, then maybe you should just STICK WITH THE GOD DAMN UNIGRAM MODEL AND GO DO SOMETHING USEFUL. Something useful, like, maybe, COLLECT DATA. Since, according to the First Fundamental Truth, all you need is more data and all your problems will be solved.

I’ll let you know how that test goes. *thumbs up*

I’ve been bothered by something for a while. I use Adium, and I have it display a Growl notification whenever I get an IM and Adium is in the background. The Growl notification contains the name of the contact that the IM came from, and the first line of the message itself. Sometimes I don’t want that, such as when there are people reading my screen and the IM contains something I’d rather those people didn’t see. Unfortunately, there’s no option to stop Adium from displaying the contents of the IM in the Growl notification.

There are two possible solutions to this.

1. File a bug report or feature request with the Adium developers.
2. Download all 200 megs of the Adium source, modify it myself, build it and feel all proud of myself for running a custom build of an open source app.

Guess which one I chose.

I slept badly that night, knowing that I’m turning into one of those people.