Why you should reinvent the wheel

This applies to pretty much anything but I’m going to apply it specifically to programming since that’s my field. Have you ever heard someone ask, “Why reinvent the wheel?” as they begin to instruct a new developer on why they shouldn’t undertake a new project that’s been done a thousand times before by other people? That’s always bothered me. When I was learning to program I used to “reinvent the wheel” all the time! Honestly, I hated just using someone else’s code without having taken a crack at it myself.

I always thought I could do it better somehow. I thought I could make it more efficient, add some new feature or somehow simplify it. If nothing else, at least I would know how it worked and could fix bugs myself. I was stubborn for a while after jQuery came about and wouldn’t use it because I always wanted to write my own library to do something like that. I hated using code that wasn’t mine. None of my code from those years ever saw the light of day. I’ve still got some of it sitting around though. It’s interesting to take a look back at it sometimes and see what I was thinking or see how I did certain things. Frankly, I would probably be embarrassed to show the code to anyone now. The libraries that are out there have been done so much better.

If you’re not particularly passionate about programming it’s tempting to just take someone’s black box library and just use their API and assume it runs on magic more powerful than your own! It’s not like you NEED to understand how it works to use it, right? It’s less work you have to do and it gets the job done.

Problem is, if that’s your mentality you won’t go far. I always insisted on seeing inside the black box and figuring out how it works, then re-creating it. Of course, when I would re-create it I would always try to improve it. Heck, back when I was 10 years old trying to learn HTML I was constantly viewing the source on websites across the internet because I had to figure out how they did what they did. Then I would go and try to do it myself. “Reinventing the wheel” was absolutely critical for me in those early years because that’s how I learned to program.

And, that’s how innovation is done every day! You can’t improve something if you don’t even understand how it works. Once you do, you create it again, but better!

For me, it’s paid off. I’m 24 with an awesome job working on a pretty advanced web application with cutting edge technologies and pioneering new solutions in the web application arena. I’m lucky to work for a company that lets me do this. A great example is what’s happening at work right now. We originally tried using SlickGrid in our application and had to pour over the source code and write lots of little hacks to make it do what we needed. Unfortunately, it just can’t handle what we’re throwing at it. Now, we’re reinventing it - but waaay better this time!

We’re actually about to open source the realtime JavaScript framework we’ve been building on top of Backbone. It includes many examples of innovation and lessons learned from other frameworks and libraries we’ve studied or worked with in the process of developing our product. We could have just gone with what was out there at the time we started (which wasn’t much, to be honest1), but that would have defined and limited what we could achieve. Instead, we’ve now created a system unlike any out there to use in the newsroom at KSL and Deseret News. I’m excited to launch the framework we built it on publicly. Sorry about the little tangent there :)

We need people that challenge the assumptions made and the patterns used in the things we take for granted in our every day lives. That goes for any industry. How will we ever have a better wheel if no one tries to reinvent it, or worse, if no one even understands how it works? Who says a wheel is even the best tool for the job?

So why reinvent the wheel?

1. To learn
2. To make a better wheel
3. Because the wheel you have doesn’t solve your needs or wants

The End.

---

1. At the time we started our framework Backbone was all the rage. AngularJS and Ember.js were just starting and had no community. SproutCore was on it’s way out. ExtJS had been around for a while but that was slow, had bugs and didn’t allow us the flexibility we wanted - it’s a monolithic framework. While our system bears some interesting conceptual similarities in a few parts with Meteor, it didn’t even exist at the time. ↩

Exceptions and resource leaks in Javascript and Node.js

A while ago I read a comment in the Node.js documentation on uncaughtException mentioning that it may be removed (no longer true), that it is a crude mechanism for exception handling and shouldn’t be used and that domains should be used instead. So I made a mental note and figured I’d look into domains when I got the chance. This past week I started looking into it a bit and noticed some more concerning comments in the documentation for domains:

By the very nature of how throw works in JavaScript, there is almost never any way to safely “pick up where you left off”, without leaking references, or creating some other sort of undefined brittle state.

In the comment in the code block a little below that, where it’s demonstrating a bad way to use domains as a global catch-all like uncaughtException it states:

The error won’t crash the process, but what it does is worse! Though we’ve prevented abrupt process restarting, we are leaking resources like crazy if this ever happens.

They recommend that when you catch exceptions in a global way like this you always shut down and restart the process. My first reaction was something like “Woah! Hold on there Skippy!” I’ve written a server that handles a lot of persistent websocket connections and I can’t just shutdown because I didn’t handle some exception thrown while handling a misbehaving client. And what the heck is it talking about with this whole “leaking resources like crazy” thing anyway?

So naturally I did some Googling to see if I could figure out what it’s talking about. Unfortunately I couldn’t find anything useful. I posted a question on Stack Overflow which was helpful though. While the answers there do seem kind of obvious in retrospect they weren’t obvious to me at the time because the resources (open files and sockets) mentioned in the answers don’t apply to the server I’ve written (another reason why it’s bad to blindly tell everyone they’re leaking resources).

My purpose in writing this post is mostly to help answer the question I had about what resources might be leaking and how to write code taking this into account. The Node.js documentation is misleading and non-descriptive. The only time it’s definitely bad to continue after an exception is if you literally have no idea where it was thrown and what resources might have been in use (e.g. global exception handlers).

When an exception is thrown while you have an open stream (file, socket, etc…) the code that follows which normally should finish with the resource and close it, never runs. That means it’s left open and you are now leaking resources. When you throw an exception in a block of code where resources are open you are introducing leaks.

The only way to make sure you aren’t leaking resources is on a case-by-case basis… sort of.

The problem with exceptions can be handled pretty well with domains. However, you run into the exact same problem if you simply forget to close a resource or return too early, while it’s still open.

Some Solutions

Ideally you shouldn’t have unhandled exceptions. We don’t live in an ideal world, but it’s still good to try to handle as many as you can. If you catch the exception close enough to where it occurred that you know what resources are in use and can clean them up properly then it’s alright to continue. If you write code that throws exceptions, keep in mind what resources are open at the time you throw the exception and try to clean them up before throwing.

You can use domains in Node.js to try to have some context and specificity to your error handling. For example, if you were writing an HTTP server, you could use domains to handle the case where an exception is thrown while processing the request (even if it’s doing asynchronous stuff) and then at least shut down the open socket from the client and clean up the request a bit. Otherwise, since an exception was thrown, no response would be sent and the request would be left open until it timed out.

If you are writing a server or process that can fork or spawn workers (such as a request handler for a web server or something else) then the problem can be largely mitigated by simply shutting down the worker (hopefully gracefully) and spawning new workers as needed.

An Ideal Solution

Though it doesn’t exist in JavaScript, an ideal solution would be something like Python’s context managers. They are a powerful tool for managing resources. As mentioned here:

The advantage of using a with statement is that it is guaranteed to close the file no matter how the nested block exits. If an exception occurs before the end of the block, it will close the file before the exception is caught by an outer exception handler. If the nested block were to contain a return statement, or a continue or break statement, the with statement would automatically close the file in those cases, too.

Using the with statement in Python (for those unfamiliar) looks like this:

with open('/path/to/file', 'w') as f:
    f.write('some text')

That’s it! There is no need to bother closing the file. Anything that needs to be done with the file is done within the with block. When it exits the scope the context manager closes the file. And the best part is that you can write your own context manager, just like open() in this example! You could write one for handling sockets, including sockets that are already open.

Though there is nothing like this in JavaScript I think it’d definitely be a nice addition. Even if it were never to become a part of JavaScript itself, it would be a particularly nice addition to Node.js. Unlike JavaScript in the web browser, JavaScript on the server handles a lot more resources like this (open files and sockets) so it is much more important to be able to handle resources in an elegant and clean way without having to shutdown every time an unhandled exception occurs.

That said, I don’t know exactly how they would implement something like context managers with the asynchronicity of JavaScript. I suppose when there are no more references to a resource it’s essentially “out of scope” and the exit on a context manager could be called to close the resource. In that case it would have to run as part of the garbage collection process and you really cannot know when it will run which could produce other unpredictable and/or undesirable behavior.

Conclusion

Keep in mind that this post only really addresses concerns relating to leaking resources, not the brittle state of the application after an unhandled exception that is being caught for the first time globally, and thus without context. It is difficult then to be able to know the state of the application. Whatever function threw the exception (which you don’t know since you have no context) could have modified any shared state variables and not finished its work, leaving it in an unknown state. Global exception handlers really should only be used for logging uncaught exceptions so you can fix them and to shutdown gracefully. Though it’s not impossible, especially in smaller systems, to be able to verify that the system is in a sane state from a global exception handler.

For an interesting and brief explanation of things to consider with exception handling, check out this explanation from an answer to my SO question. Trust me, you want to click that link!

Closures in loops - Javascript Gotchas

When you do enough Javascript programming, it’s difficult to avoid learning about this sooner or later, but I still feel like it’s not well known enough. If you learn about it from a blog post like this, you’re one of the lucky ones. Once upon a time I spent hours going crazy thinking the world didn’t make sense anymore - that or every Javascript implementation had a serious bug. I was trying to figure out why it was that a closure within a for loop seemed to have the same value for every iteration of the loop - it just wasn’t changing even though the loop was iterating properly. I was baffled. Isn’t a closure supposed to capture the current value of variables within it’s accessible scope?

Here’s a simple piece of code you can run to see what I’m talking about:

for(var i = 0; i < 3; i++) {
    setTimeout(function() {
        console.log(i);
    }, 0);
}

If you run that in a Javascript console, it’ll output something like this:

3
3
3

Wait! What? Ok, so maybe I gave it away by wrapping it in a call to setTimeout. What’s happening here? It turns out JavaScript is single threaded so it keeps an event queue where it queues up things to do. The closure created in each loop iteration is queued to run as soon as the rest of the current execution context finishes and CPU time is returned to the event loop. setTimeout here serves to simply defer the execution of each closure until after the loop finishes running. By that point the final value of i is ‘3’. Keep in mind that i was declared before the loop started. It’s scope is external to the loop.

Ok, that makes sense, but what if I do something like this:

for(var i = 0; i < 3; i++) {
    var j = i;
    setTimeout(function() {
        console.log(j);
    });
}

Now we’re declaring a variable j inside the loop that is a copy of i. Well, it turns out that the scope of j here is also external to the loop. It will continue to exist outside the loop and thus only ever has one value which changes as the loop iterates. This in turn means that when a deferred closure runs, the latest value will be the one that all the closures from this loop end up with. In Javascript it is very common practice to pass callbacks around that are deferred until something completes or some event occurs. So like I said, sooner or later you’re likely to run into this problem. It’s gotten me more than once, even though I knew about it.

So what can I do then to make my closures work right in a loop?

I’m glad you asked! There are a few different ways you can handle it, but the basic concept is that you have to capture the value of i in each iteration of the loop. By far the cleanest solution is to just use an iterator function like _.each or $.each from the Underscore or jQuery library:

_.each(_.range(3), function(i) {
    setTimeout(function() {
        console.log(i);
    });
});

If you already have a loop that you don’t want to convert to use an iterator function, all you have to do is wrap your closure in a closure in which you define new variables which capture the current value of the variables that change on each iteration. Got that? The trick to capturing the variables is making sure your outer closure executes immediately during the current iteration of the loop. You can use one of these two similar approaches:

Method 1:

for(var i = 0; i < 3; i++) {
    (function() {
        var num = i;
        setTimeout(function() {
            console.log(num);
        });
    })();
}

Method 2:

for(var i = 0; i < 3; i++) {
    (function(i) {
        setTimeout(function() {
            console.log(i);
        });
    })(i);
}

And that’s it! The world makes sense again! Happy coding :)


UPDATE - 10/7/2013

There is now another simpler solution to this problem since the let keyword is now supported in both Firefox and Chrome. The let keyword is used in place of var to scope variables to the block it’s used in. Similar to one of the above examples that does not work, you can swap var for let and it works magically, like so:

for(var i = 0; i < 3; i++) {
    let j = i;
    setTimeout(function() {
        console.log(j);
    });
}

This works because we declare a new variable, j, which we set to the value of i which is captured by the closure within the loop, but it doesn’t continue to exist beyond the end of one iteration of the loop since it’s scoped locally.