The original title for this post was “The problem with Rake”. The issue still exists but I have learned enough to mitigate it’s effect. But first things first.

The original problem is that Rake will not re-evaluate the dependencies of a task after one of the dependencies executes.

This simply means that there is no way to dynamically add dependencies to a task from one of it’s dependencies.

To illustrate this consider the following rakefile

The output is

when, if the dependency tree was built dynamically, ‘c’ should pop up before ‘a’:

Why would you need this you will ask?

I believe that a development environment should not impose structure but rather should evolve it’s structure in tandem with the architecture of the system you are trying to build. Empirically it is a much more effective approach when the thought patterns formulated in the architecture are reflected in the structure of your workspace and the tools just support this. This is a variation of the “convention vs. configuration” mantra for when you are actually building something from scratch.

So in a layered, component based C application I would like to write a single method that compiles a component and then have a task that creates as many tasks as there are components, associate them with the sources and be done with it.

Unfortunately, since Rake locks the dependencies when the task is executed, the obvious solution was to create the tasks while Rake instantiates it’s application.

def create_tasks
  task :compile => find_components
end

create_tasks

task :compile

This proves very ineffective and performance degrades even further the more the project grows.

Some of the fault lies actually with the approach: I approached the problem from the source side, trying to determine which sources belong to a component and creating a task for each component, which essentially forces us to create every task and then the dependency chains. The more components, e.g. code, the longer this procedure takes.

Coupled with the fact that this happens “on load”, this adds a perceptible delay in task invocation for an added psychological effect of slowness.

The solution was to change tack and approach the problem from the results side: Each component corresponds to a library, an artifact of the build process. It is trivial to determine which libraries are needed for each program/task and from there determine which components we have to build.

So we want to build an application that depends on libraries. These are file tasks

file "app.exe" => ["1.lib","2.lib"]

We still have to construct the application file task “on load” and associate it with the library file tasks and then associate the library file tasks with the source files but the scope is drastically reduced and performance is not affected (meaning the time to define the task dependencies is negligible compared with the compilation time of a single component).

Now compilation can be expressed with rules

rule '.lib' do |r|
 build_the_library(r)
end

The nice thing about Rake rules is that they can be regular expressions. With a couple of rules like the following you can use the same compilation code for multiple platforms

/out\/pc\/.+\/.*\.lib/ do |r|
 build_the_library(r,'pc')
end

/out\/target\/.+\/.*\.a/ do |r|
 build_the_library(r,'target')
end

For all of this to work it goes hand in hand with naming conventions and a rather strict directory structure. It’s a balance between explicit rules expressed in code and implicit rules expressed in the conventions used that gives us a lot of flexibility - but this is a subject for a later post.