Python templating

by leonardo maffi
November 11 2006

[Go back to the index]

Python code plus some examples: templat.zip

In the book "Game programming gems", edited by Mark DeLoura, at pages 20-35 there is a chapter by Pete Isensee that talks about a complex use of C++ templates to do some computations at compile time. Template programming is Turing-complete, and it can be used to compute some constant values, to unroll loops and so on, that can be useful to speed up the code at runtime, without the need to put the unrolled code inside the source itself. The idea is neat, but the syntax of templates is far from being nice, and some C++ compilers are unable to do their work correctly. This is a simple example that computes Fibonacci number at complile time:

// Template Factorial

// Copyright  2000 Pete Isensee (PKIsensee@msn.com).
// All rights reserved worldwide.
template< unsigned N > struct Fact
{
   enum { Val = N * Fact< N - 1 >::Val };
};

// Specialization for base cases (termination conditions)
template <> struct Fact< 0 > { enum { Val = 1 }; };
template <> struct Fact< 1 > { enum { Val = 1 }; };

// Make the template appear like a function
#define FactT( n ) Fact< n >::Val

This is a more complex example, that creates identity matrixes at compile time:

// Identity matrix
// Copyright  2000 Pete Isensee (PKIsensee@msn.com).
// All rights reserved worldwide.

#pragma warning( disable: 4127 ) // ignore conditional expression is constant
#pragma inline_depth( 255 )
#pragma inline_recursion( on )

// N is matrix size
template< class Mtx, unsigned N > struct IdMtx
{
   static inline void eval( Mtx& mtx )
   {
      IdMtxImpl< Mtx, N, 0, 0, 0 >::eval( mtx );
   }
};

// Assigns each element of the matrix
template< class Mtx, unsigned N, unsigned C, unsigned R, unsigned I >
struct IdMtxImpl
{
   enum
   {
      NxtI = I + 1,          // Counter
      NxtR = NxtI % N,       // Row (inner loop)
      NxtC = NxtI / N % N    // Column (outer loop)
   };
   static inline void eval( Mtx& mtx )
   {
      mtx[ C ][ R ] = ( C == R ) ? 1.0 : 0.0;
      IdMtxImpl< Mtx, N, NxtC, NxtR, NxtI >::eval( mtx );
   }
};

// Specialize for 3x3 and 4x4 matrix 
template<> struct IdMtxImpl< matrix33, 3, 0, 0, 3*3 >
{
   static inline void eval( matrix33& ) {}
};
template<> struct IdMtxImpl< matrix44, 4, 0, 0, 4*4 >
{
   static inline void eval( matrix44& ) {}
};

// Make the template appear like a function
#define IdentityMtxT( MtxType, Mtx, N ) IdMtx< MtxType, N >::eval( Mtx )


Lisp programmers are probably smiling at such pieces of code. Instead of using a language with awful syntax, that some compilers can't execute well, why don't we use the language itself and its compler/interpreter to create the templates? So we have a better syntax and a better compiler.

So I have chosen Python as the language to express the templates, it's simple, and the template parser is little more than an exec of the template blocks. The Python parser can work on all kind of sourcecode, Python, C, C++, ObjectPascal, Java, etc.

How to identify the template blocks? A syntax has to be invented. For templates inside Python source code I have chosen {{ ... }}. Python syntax doesn't allow a dict as a dict key, so this syntax is surely absent from Python sources. {( ... )} can't be used, because {(1):(1)} is acceptable syntax. {{{ ... }}} is a too much long syntax. I have chosen to keep the scope of the names defined into a block inside the scopes of all the following blocks in the same source code.

For simpler situations an expression can be useful too, so I have defined the {[ ... ]} syntax too, that return the repr(eval(...)).

This is the (working) core of this program, templating with Python is really easy:

import glob, re, sys, cStringIO, textwrap

def repl(mobj): # uses "scope" global name
    indent, block, expression = mobj.groups()
    if block:
        sys.stdout, oldstdout = cStringIO.StringIO(), sys.stdout
        exec textwrap.dedent(block.rstrip())+"\n" in scope
        result, sys.stdout = sys.stdout.getvalue(), oldstdout
        return "\n".join(indent + line for line in result.splitlines())
    else: return repr(eval(expression, scope))

finder = re.compile(r"([ \t]*)  {{ (.*?) }}  |  {\[ (.*?) \]}", re.DOTALL|re.VERBOSE)
for filename_in in glob.glob("*.pyt") + glob.glob("*.ct"):
    scope = {}
    code_in = file(filename_in).read()
    file(filename_in[:-1], "w").write( finder.sub(repl, code_in) )

The complete code is longer and it has some safes, some speeds up, looks for templated files recursively, etc. As you can see, the output of the prints contained inside the blocks to be executed is grabbed inside a cStringIO, and the output file has the same name of the input file minus the last "t" of the suffix (t is for template).

Python is really good for such kind of jobs, I have created the core of this program in a short time, about two hours. This is a small example input of templated Python:

{{from math import sin

identity = lambda n: [[[0,1][i==j] for i in range(n)] for j in range(n)]
# With Python2.5 you can use  if i==j else 0   instead of  [0,1][i==j]
}}

print {[ sin(5.5) ]}
{{print "print", repr(sin(5.5)) }}

{{
for i in xrange(0, 20, 6):
    print "a%d = %d" % (i, i)
 }}# Extra space to test

m = {[ [[[0,1][i==j] for i in range(4)] for j in range(4)] ]}
n = {[ identity(3) ]}
The resulting Python code (maybe minus few newlines):
print sin(5.5)

print -0.70554032557039192

print -0.70554032557039192

a0 = 0
a6 = 6
a12 = 12
a18 = 18# Extra space to test

m = [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]
n = [[1, 0, 0], [0, 1, 0], [0, 0, 1]]

More explanations are inside the Python source.

Python code plus some examples: templat.zip

[Go back to the index]