Into the rapids: more basic

As we remember from the second jsonic tuto­rial, a syntax colorer is a lexer called by DrRacket that reads in source code, matches the code to a list of rules, and then returns the matched strings with syntax-coloring anno­ta­tions.

Each anno­ta­tion contains five values: the string to be colored, a coloring cate­gory, a paren­thesis shape, and the starting and ending posi­tion of the coloring. The valid coloring cate­gories are 'error, 'comment, 'sexp-comment, 'white-space, 'constant, 'string, 'no-color, 'parenthesis, 'hash-colon-keyword, 'symbol, 'eof, or 'other. The valid paren­thesis shapes are ()[]{} or #f.

Right now, our coloring for our sample program looks like this, using the default Racket-language colorer and DrRacket’s default color scheme:

Not every­thing is wrong—the strings, numbers, and iden­ti­fiers look right, because they happen to be written the same way in Racket. But the rem comments in lines 30 and 70 aren’t colored as comments. The value 'three' on line 60 is not colored as a string. And on line 10, the part of the line after the ; is formatted as a comment—because it would be, if this were Racket code—but in BASIC, it just repre­sents extra argu­ments to print, and these should be colored like any other values.

In jsonic, we wrote a sepa­rate lexer for the syntax colorer. This time, our colorer will rely on the lexer we already wrote for the main language inter­preter:

This is a conve­nient idea, because later, if we update this main lexer, our colorer will auto­mat­i­cally pick up the changes.

Designing the colorer

Here’s the plan. We’ll read tokens from the lexer, each of which will be an instance of a srcloc-token struc­ture. We’ll return one syntax-coloring anno­ta­tion for each token. To make each anno­ta­tion, we’ll read values from inside the token and use them to figure out what coloring cate­gory applies and what paren­thesis shape we should use (if any). We’ll also read the neces­sary source-loca­tion values from inside the token, and put those into the coloring anno­ta­tion.

There’s one addi­tional wrinkle. When the lexer gets a char­acter that it can’t match to any lexing rule, it raises an error. When we’re trying to run a program, that’s the right behavior, because nothing can happen until the error is corrected.

But in a syntax colorer, it’s the wrong behavior. Why? Because the syntax colorer is called after every keystroke, not just when the user clicks Run. It’s likely that after any given keystroke, the program can’t be lexed without errors. So we want to handle those errors when they arise, so that the user’s code editing can continue peace­fully.

Updating the main module

As we did in jsonic, we make our colorer avail­able to DrRacket by adding a get-info func­tion to our "main.rkt" module:

When get-info gets the color-lexer key, it uses dynamic-require to import the basic-colorer func­tion from the basic/colorer module. We’ll set up this module next.

The colorer module

Let’s start our new "basic/colorer.rkt" module. We import "lexer.rkt" to get our main lexer, and set up a new func­tion called basic-colorer:

DrRacket passes our colorer func­tion a port argu­ment. To start, we just pass this port through to basic-lexer, which will return an instance of srcloc-token that we’ll use to construct our coloring anno­ta­tion.

We know, however, that basic-lexer might raise an error. We want to catch this error. As we’ve done before, we wrap our call to basic-lexer in a with-handlers expres­sion:

As we would find out from reading the docs for lexer-srcloc, when it can’t match any rule, it raises an excep­tion of type exn:fail:read. There­fore, our with-handlers expres­sion invokes the related pred­i­cate, exn:fail:read?, and passes any matching excep­tion to our helper func­tion handle-lexer-error.

To patch over the error, our helper func­tion needs to return an instance of srcloc-token. The srcloc-token constructor func­tion takes two argu­ments: a plain token value, and a srcloc struc­ture.

An excep­tion of type exn:fail:read has a srclocs field that contains a list of srcloc struc­tures related to the error.  + Different excep­tion types have different fields that are useful in handling errors. The docu­men­ta­tion has details for each type. We use exn:fail:read-srclocs to store these in excn-srclocs and then use car to get the topmost one.

As for the token value, we just make a token struc­ture with type 'ERROR.

Taken together, our srcloc-tok vari­able will either get its value from basic-lexer, or if an error occurs, from handle-lexer-error.

Playing with matches

The rest of our colorer will read fields out of the srcloc-tok value and use them to create coloring anno­ta­tions. To make this easy, we’ll use two new func­tions: match and match-define.

match is one of Racket’s secret weapons—an extremely clever feature that doesn’t exist in other program­ming languages. Just as regular expres­sions let us decon­struct strings, or syntax patterns let us decon­struct syntax objects, match allows us to decon­struct any Racket value.

The basic match form works like cond: it takes a value as input, plus a series of branches. If the pattern on the left side matches the value, then the value on the right side of the branch is returned. An optional else branch handles every­thing else:

In this example, we see a few of the options for left-hand patterns.  + Many more can be found in the docs for match. A literal value like "foo" is matched exactly. A pred­i­cate like number? can be matched by wrap­ping it in the ? oper­ator. When list is used as a match pattern, it not only matches the input but also assigns the pieces to new iden­ti­fiers—in this case, extracting the middle element b into a new list. Simi­larly, any struc­ture type can be used as a pattern, and its values matched by posi­tion.

The match-define form uses the same pattern-matching vocab­u­lary as match to directly create new vari­ables by “reverse engi­neering” existing values:

In other words, when we use list (or thing) on the left side of a match-define, we’re not creating a list (or instance of the thing struc­ture type) but rather using it as a template for decom­posing the value on the right.

We’ll use match and match-define to help finish the colorer.

Making coloring anno­ta­tions

The rest of the colorer will proceed by matching against srcloc-tok. As usual, we need to create a special rule to handle the eof case:

We use eof-object? wrapped in ? as a left-hand match pattern. On the right, we return a color anno­ta­tion with srcloc-tok as the value and 'eof as the cate­gory. The other fields will be ignored.

Then we can move on to the more inter­esting cases. First we use match-define to decom­pose our srcloc-tok into fields and assign them to vari­ables:

We know each srcloc-tok is an instance of srcloc-token with a token-struct as the first value and a srcloc as the second. So our match pattern combines all three, letting us directly define vari­ables to hold the fields we need to make our coloring anno­ta­tion: type and val from the token, plus the posn and span from the source loca­tion.(Each _ in the pattern means “ignore this field”.)

Now we need to assemble the five values we need for our coloring anno­ta­tion: the token value, the coloring cate­gory, the paren­thesis shape, and the start and end loca­tions. We already stored the token value in val. The start loca­tion is just posn, and the end loca­tion is (+ start span). Then we use another match-define to figure out the cat and paren values:

Each branch of our match-define for (list cat paren) will return a list with a cate­gory value and paren­thesis shape. This list will be assigned to cat and paren, which can then be inserted into the final values expres­sion that returns the finished coloring anno­ta­tion, along with the already calcu­lated val, start, and end.

Within the match-define, we use two instances of match. The first will match against the type. A token of type 'STRING gets colored as a 'string, a 'REM token gets colored as a 'comment, and an 'ERROR token gets colored as an 'error. (In all cases, there is no paren­thesis shape.)  + Astute observers may have noticed that we gave our 'ERROR tokens a type, but not a value, which defaults to #f. DrRacket primarily relies on the start and end posi­tions for coloring. So as long as those are correct, the colorer will work, despite the missing token value.

If we don’t get a match for type, in its else branch we have a nested match on val. Tokens that are number? are each colored as a 'constant; those that are symbol? are each colored as a 'symbol. We use literal "(" and ")" matches to catch any paren­theses (this time our coloring anno­ta­tion has both a 'parenthesis cate­gory and the correct paren­thesis shape).  + Because paren­theses already serve as list delim­iters in Racket, a paren­thesis used as a literal symbol has to be escaped with vertical bars: thus '|)| rather than '). Finally, anything else gets a 'no-color anno­ta­tion.

And that’s the finished colorer. Keep in mind that these coloring deci­sions are arbi­trary. We’re just trying to figure out the most intu­itive mapping from things in our language to DrRacket’s coloring cate­gories. But DrRacket neither knows and nor cares whether, for instance, some­thing colored as a 'comment is in fact a comment. Syntax coloring doesn’t affect how the language works. Just how it looks.

Testing the colorer

Recall how our sample program looked before we wrote our colorer:

Some elements were colored correctly. But the rem comments in lines 30 and 70 were not colored as comments. The value 'three' on line 60 was not colored as a string. And the right half of line 10 should appear normally.

Let’s open our "sample.rkt" file in DrRacket. For faster perfor­mance, DrRacket caches the result of get-info for each language. We have to force a refresh. If we’re using Racket v6.9 or later, we select Racket → Reload #lang Exten­sions, which reloads our get-info func­tion and our new colorer. If not, we quit and restart DrRacket, which has the same effect. After a moment, the code will look like this:

Notice that the rem comments are now colored correctly, 'three' is correctly colored as a string value, and line 10 is colored normally.

If we type some nonsense char­ac­ters on the last line that are unrec­og­nized by the lexer, they’re colored as error char­ac­ters, but they don’t inter­rupt our editing:

If we go back and put quote marks around the nonsense, we convert it to a valid string, which is recol­ored accord­ingly:

Thought exper­i­ment

What are the limits of sharing the main language lexer with the syntax colorer? For instance, what if we wanted to do some­thing slick, like apply a different color to the line numbers?

In prin­ciple it’s possible. But we can only apply different colors to things that can be differ­en­ti­ated from within the lexer (i.e., with different rules and token types).

For instance, right now a line number is deliv­ered inside a token of type 'INTEGER. So it can’t be differ­en­ti­ated from an ordi­nary number that’s also deliv­ered inside an 'INTEGER token. It’s true that these can be differ­en­ti­ated by the parser—a line number is in a specific posi­tion in the grammar—but we can’t use a parser here.  + Why not? Because we can’t parse some­thing unless it can first be lexed and tokenized. As we noticed at the begin­ning, the code is often in a non-lexable state during editing.

To apply a sepa­rate color to line numbers, we’d have to add a rule to the lexer to specially match line numbers and wrap them in a new token type, say 'LINE-NUMBER. Then we’d be able to apply a color to 'LINE-NUMBER tokens that’s different from 'INTEGER tokens.

So what is this new lexer rule? Because every program line has to be on a sepa­rate line of the file, we could imagine a lexer rule that captured a sequence of a newline and an integer as a 'LINE-NUMBER. But once we change how newlines are tokenized, we’d also have to adjust our parser grammar, as well as the source loca­tion of a 'LINE-NUMBER token to account for the leading white­space.

In short: it could be done. But it might be more trouble than it’s worth.

The alter­na­tive is to write a sepa­rate lexer for the syntax colorer, as we did for jsonic. This would give us the freedom to lex the source code what­ever way we want to produce coloring effects. Of course, because the syntax colorer is strictly cosmetic, this new lexer wouldn’t affect the behavior of the language (nor require changes to the parser, etc.)

Both approaches are reason­able. We shouldn’t forget that even if we don’t share the main lexer with the syntax colorer, we can still share a set of lexer abbre­vi­a­tions (by exporting them from one lexer and importing them into the other). These can be written so they contain core patterns for the lexer rules. This makes both lexers simpler to write.