Language Transformations and You: Syntactic Preprocessing

Alright, time for another exciting post about language transformations and believe me, today’s post is incredibly exciting! You see, as I explained in Language Transformations and You: Lexical Preprocessing, Lexical Preprocessing is great for including source code…but doesn’t really cut it in many other situations. Syntactic Preprocessing, however, is always exciting! You see, Syntactic Preprocessing goes further than lexical preprocessing. Instead of parsing for compiler specific preprocessor directives, it actually parses the code in search of text that matches patterns that you yourself define within macros.

What this means is that depending on the language, you can add new syntax and operators, change existing syntax and operators and even add new keywords! This kind of customization is huge with languages like Nim, Lisp or Scheme, but unfortunately hasn’t quite caught on with many of today’s main languages. Sure, you could use the C preprocessor and a Lexical Preprocessor to substitute bits of text as stated here, but I would have to agree with that answer…a simple code substitution isn’t needed, since you can use functions to do the same thing. Syntactic Preprocessing isn’t just a simple swap of a keyword for some code. Instead, it allows you to completely change how you write the language through fully descriptive patterns that can compile into whatever bit of code that you need.

Why is Syntactic Preprocessing really important? Mainly, it allows you to create your own domain specific language, that can result in cleaner code that is easier to maintain. Remember our object hierarchy fun? Instead of having a mess of multiple for loops that you constantly have to code that follow the exact same format, you can instead write a macro that takes cleaner code like parent#child#child#>callback. The compiler transforms that pattern into the messier actual code that your application needs to use in order to function (pardon the pun).

Here’s another example. Javascript will soon have the spread operator with the arrival of ES6. Unfortunately, it could take months or even years before browsers implement all of ES6 (looking at you, Safari and Opera…you’re behind IE’s tech preview!). In the mean time, you can create macros to ease the pain and give you access to some of the features of the spread operator:

macro (..el) {
  // Flatten arrays.
  rule {
    // We're going to use recursion here to ensure that
    // elements that are arrays are also spread.
    [$x:invoke(..el) (,) ...]
  } => {
    $x (,) ...
  // Simply return anything that isn't an array.
  rule {
  } => {

macro (..>) {
  // For an array, iterate over each element in the
  // array and use the ..el macro on it.
  rule {
    [$x:invoke(..el) (,) ...]
  } => {
    [$x (,) ...]

macro (~) {
  // For a function call after the ~ macro, allow a
  // single array to be spread as the function's parameters.
  rule {
  } => {
    $x.apply(null, $y)

// var num = ~some(..>app)
// Outputs: var num = some.apply(null, app);

// var arr = ..>[1,2,3,[1,[2, [3, [foo]]]]]
// Outputs: var arr = [1,2,3,1,2,3,foo];

Here’s a gist of the above macro!

Right? Macros are awesome! They give you so much control over the language. Some developers may not like having a domain language or even using macros, but the way I see it, you need to use the best tool for the job. Creating a domain language that ensures compliance to your application’s needs, can ensure conformance to proper design patterns and still compiles down to the code that you want run in production is a highly effective tool.

Macros sound pretty great so far…but what if you don’t want to use Scheme or Lisp? Luckily there are options for many of the modern languages out there:

  • C,C++, C#, Java, and Objective-C – Unfortunately, they only have the #define preprocessor, which does a simple code substitution and cannot provide as much customization. However, you can change their compilers.
  • Go – No macros here either, but they have a way to generate code, which is just as awesome!
  • Javascript – sweetjs – This is what I used in the above spread operator example, and can be incorporated with Grunt or Gulp.
  • Perl – macro
  • PHP – No macros or even a #define directive. You’re on your own here…
  • Python – macropy
  • Rust – included in language – In fact, the macros in Rust are the same as sweet.js, since both come from Mozilla.

I know what you’re thinking…you could go even further than a domain language and create an entirely new language that transpiles into another language…or even several languages, allowing your team to develop in one language and get native code for desktops, mobile OS’s and even web applications. Of course, you would be right…but I’m not going to discuss that further until my next post Language Transformations and You: Transpiling!

2 thoughts on “Language Transformations and You: Syntactic Preprocessing

  1. Pingback: Language Transformations and You: Lexical Preprocessing | The Code Lion

  2. Pingback: Language Transformations and You! | The Code Lion

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s