Language Transformations and You: Lexical Preprocessing

As I stated in Language Transformations and You!, I will be spending the next few weeks discussing how you can improve your skill set with various Language Transformation techniques. This week’s big topic will be Lexical Preprocessing, which is a method of changing your source code based on specific compiler recognized tokens. Is it worth your time? Let’s take a deeper look at it and I’ll let you decide.

Source Code Importing

The most pervasive example of Lexical Preprocessing is importing source code from one file into another. This is one of the most important tools in a developer’s repertoire because it allows you to organize your source code. As the old saying goes, “A place for everything, and everything in its place.” Now, I could probably rant about the importance of organizing your files into small reusable modules until I’m blue in the face, but I will give you the benefit of the doubt and assume you yourself rant about the very same thing. ūüôā

Anyways, when source code is being interpreted and compiled, the compiler needs to be able to find the references of all of the code that you used within your files. The way it handles that process is during Lexical Preprocessing by loading source from any file that you have imported with of course an import, includes or using statement (or whatever other keyword token your language of choice uses).

Unfortunately…there is a downside (there always is with technology): mocking out imported files for unit testing is a huge pain with normal Source Code Importing. In fact, with many languages, it can’t be done at all without some real hacky and evil code. The solution? Dependency Injection. Since this is a topic about Lexical Preprocessing, I won’t get into the gory details…suffice to say, you definitely need to learn about Dependency Injection. I would start here, here and also in a TLDR¬†post here.

That one con aside, you’ve probably used this technique so often that you barely notice it anymore. It’s become a part of your routine. Add a file, include a file, use the imported¬†contents within your new file. You’ve already begun transforming your source code into being a more readable and well organized project through the use of Lexical Preprocessing. Take a look in the mirror and high five yourself. Yeah, you’re pretty awesome. Now clean off your hand print before anyone realizes you just high fived your reflection. Don’t worry, your secret is safe with me.

“So”, you ask as you grab that bottle of window cleaner, “what else can Lexical Preprocessing do?”

I’m glad you asked.

Source Code Substitution

You see, Lexical Preprocessing isn’t just about importing source code. It’s also about changing your source code based on static, compiler recognized tokens. Let’s say you have a utility or application base code file that you use to define various settings, objects and functions that you use throughout your application. You might have database connection objects, file paths and boolean settings that change the behavior of your application. What if you need those to change based on whether or not the application is running in production? One common approach is to use Source Code Substitution in order to change how compiler output based on various compiler and environmental states.

Remember our database connection logging example in the previous article?

using(SqlConnection con = new SqlConnection(connectionString)){
	// Set logging
	loggingLevel = LoggingLevels.Debug;
	loggingLevel = LoggingLevels.Production;

	//Do some SQL Stuff

// Output for Debug:
using(SqlConnection con = new SqlConnection(connectionString)){
	// Set logging
	loggingLevel = LoggingLevels.Debug;

	//Do some SQL Stuff

//Output for Prod:
using(SqlConnection con = new SqlConnection(connectionString)){
	// Set logging
	loggingLevel = LoggingLevels.Production;

	//Do some SQL Stuff

With the use of the #if, #else and #endif preprocesser directives, we can change the output of our source code to fit our application’s environment. This tells the compiler that if DEBUG is true, that it should only include the line where we set the loggingLevel to Debug.¬†For anything else, it should be set to Production. That’s a pretty nifty¬†trick to use in order to ensure that our application runs exactly as we need it to in a developer environment or a production environment.

The only problem…is that it can clutter up your code if you use it too heavily. Remember your utility file? What if most of the settings in there change depending on the application’s environment? It would be a sloppy mess of preprocessor directives mixed in with your code, and determining the finished output of a source file would be a nightmare. Of course, if kept in one file, it can allow your entire application to behave differently as needed and keep the mess contained.

A better solution? Put your settings into separate configuration files based on your different application environments. Then you can¬†load whichever one you need based upon the environment. It’s cleaner, more organized and easier to maintain. In fact, I would argue that in any situation where you feel you need to use preprocessor directives, you probably don’t. By writing well organized, modular, functional code that utilizes configuration files, proper source code importing and even Dependency Injection, you can make your application behave differently in almost any environment that you care to configure.

There are even some compilers that allow you to configure or code them for source code importing (like Grunt or Gulp with Javascript), so you can tell the compiler to import entirely different files based on your environment. Having a compiler that powerful will allow you to not only separate behavior into different files, but will also allow you to use that behavior for your environment without adding a bunch of compiler jargon to your code files. How awesome is that?

So, to sum up, you most likely already use Lexical Preprocessing often through source code importing. It has a fairly minor con, seeing as you can still write incredible, well organized code. Some languages even allow you to write meaningful unit tests alongside source code importing, without the use of Dependency Injection.

The other main use, source code substitution, should be used sparingly, as there are better ways to make your application behave differently in various environments than by substituting code.

Stay tuned next week for my favorite part of Language Transformations: Syntactic Preprocessing!


One thought on “Language Transformations and You: Lexical Preprocessing

  1. Pingback: Language Transformations and You: Syntactic Preprocessing | The Code Lion

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s