Skip to main content

Iterating faster

Today I spent a few minutes writing a Makefile. I think they may turn out to be some of the most effective minutes I've ever spent.

At work we build using SCons. When I change a single source file, rebuilding takes between 50 and 65 seconds. That's more than long enough to lose my focus and get distracted, so to stay productive I have to work in larger batches. I code in longer bursts and recompile less often, so that the ratio of productive time to down time stays up; unfortunately that also means fewer opportunities to test my changes.

That's not ideal, but it's manageable much of the time. Not at the moment though. Right now I'm developing an algorithm where I need to be able to quickly try out different approaches for each of the steps. In this situation I really want to see the effect of each change in isolation, so I can't batch up my coding. All of a sudden those 50 second build times are a real problem.

I spent a little while looking into ways to speed SCons up and managed to get the rebuild time down to around 45 seconds. Still nowhere near fast enough.

Fortunately I'm working in a very self-contained part of our code-base, so I could limit the amount of files I had to consider. Re-running SCons with verbose output turned on gave me the exact command being used to compile each of them, so I saved those to a shell script. Rebuild time with this shell script was down to about 15 seconds: a huge improvement, but still not quite quick enough. This script was recompiling every file each time I ran it though - surely I could do something a bit smarter than that...?

Enter make. It knows how to check for changed files & I've used it enough that I can put together a Makefile pretty quickly. Factoring out the commands from the shell script into rules and adding enough dependency information to be useful, I had it set up in a couple of minutes. The result: most rebuilds are now around 3 seconds. Perfect.

So now, not only am I saving almost a whole minute of wasted time with each rebuild, I've all but removed the chance to get distracted and also opened the door on a more effective way of working. I can make small changes and rebuild to see the effect of each in isolation. I can develop my algorithm a lot faster now, because I can iterate faster; and that's why I think they'll be some of the most effective minutes I've ever spent.


Post a Comment

Popular posts from this blog

How to outperform std::vector in 1 easy step

Everyone who's familiar with C++ knows that you should avoid resizing a std::vector inside a loop wherever possible. The reasoning's pretty obvious: the memory allocated for the vector doubles in size each time it fills up and that doubling is a costly operation. Have you ever wondered why it's so costly though?

It's tempting to assume that because implementations of the STL have been around for so long that they must be pretty efficient. It turns out that's a bad assumption because the problem, in this case, is the standard itself: specifically, the allocator interface.

The allocator interface provides two methods that obtain and release memory:

allocate allocates uninitialized storage
(public member function)deallocate deallocates storage
(public member function)

(taken from this page).

What's missing is away of growing an existing memory allocation in place. In C this is provided by the realloc function, but there's no equivalent in the std::allocator interfa…

OpenGL ES and occlusion queries

This is a follow-up to my earlier post "WebGL doesn't have query objects".

Since I wrote that post, the situation has changed a bit. It's still true to say that WebGL doesn't have query objects, but the underlying reason - that OpenGL ES doesn't - is no longer true.

For OpenGL ES 2.0, there's an extension which provides basic query functionality: EXT_occlusion_query_boolean (which seems to have been based on ARB_occlusion_query2 from regular OpenGL). For OpenGL ES 3.0, the functionality from that extension appears to have been adopted into the standard. The extension provides two query types, both of which set a boolean value to indicate whether any pixels passed the depth and stencil tests.

While this is progress, unfortunately it's still not sufficient to implement the pixel accurate collision detection method I described in an earlier post. For that purpose it's not enough to know whether any pixels passed the tests; you want to know whether al…

Octree node identifiers

Let's say we have an octree and we want to come up with a unique integer that can identify any node in the tree - including interior nodes, not just leaf nodes. Let's also say that the octree has a maximum depth no greater than 9 levels, i.e. the level containing the leaf nodes divides space into 512 parts along each axis.

The encoding The morton encoding of a node's i,j,k coordinates within the tree lets us identify a node uniquely if we already know it's depth. Without knowing the depth, there's no way to differentiate between cells at different depths in the tree. For example, the node at depth 1 with coords 0,0,0 has exactly the same morton encoding as the node at depth 2 with coords 0,0,0.

We can fix this by appending the depth of the node to the morton encoding. If we have an octree of depth 9 then we need up to 27 bits for the morton encoding and 4 bits for the depth, which still fits nicely into a 32-bit integer. We'll shift the morton code up so that i…