Level up: jsonic revisited

First, when we say “syntax coloring” it has nothing to do with syntax in the sense of a syntax object or syntax pattern. In this case, “syntax” just refers to the displayed source code. “Source-code coloring” would prob­ably have been the more accu­rate term. But “syntax coloring” is what Racket calls it.

DrRacket uses a syntax colorer to handle the coloring for a language. Unless told other­wise, DrRacket uses the default Racket colorer. But we can also option­ally write a custom colorer for our language, attaching it with the get-info func­tion we just learned about in DrRacket inte­gra­tion.

In addi­tion to coloring, DrRacket also gets infor­ma­tion from the syntax colorer about the loca­tion of delim­iters. It uses this infor­ma­tion to support certain GUI conve­niences, such as selecting a whole delim­ited expres­sion when we click at its boundary.

Coloring prob­lems in jsonic

Right now, the syntax coloring for jsonic is mediocre. Here’s our current test file as it appears in DrRacket:

Because DrRacket uses the Racket colorer by default, the embedded Racket expres­sions are colored accu­rately. But not the jsonic syntax:

Not only are the delim­iters colored incor­rectly, but they also don’t behave as delim­iters in the GUI. For instance, if we click in front of the (* 6 7) expres­sion, DrRacket will select the whole expres­sion:

Whereas if we click in front of the jsonic delim­iter to the left, nothing happens:

Ideally, this click would high­light the whole embedded expres­sion, from the @$ on the left to the matching $@ on the right.

So let’s do better. We can write a custom syntax colorer for jsonic that will contain more precise instruc­tions for how DrRacket should color jsonic source code.

Our syntax colorer will work much like our tokenizer. With the tokenizer, Racket gave us a port containing the source code, and we used a lexer to iden­tify the smallest mean­ingful chunks.

In this case, DrRacket will give our coloring func­tion a new port containing the source code (plus a couple more argu­ments to track the current coloring state). But this time, instead of using a lexer to create tokens, we’ll use a lexer to asso­ciate each part of the code with a coloring anno­ta­tion. DrRacket will rely on these anno­ta­tions to color the code.  + It’s possible to use one lexer for both roles. And we’ll do it that way in a later tuto­rial. But for now, it’s simpler to use two.

Coloring anno­ta­tions

In most cases, our syntax-coloring anno­ta­tions will contain five values:

In special cases where we want to track the state of the colorer or lexer—as we do with jsonic, because we’re going to be alter­nating between two syntax colorers—each syntax-coloring anno­ta­tion contains two supple­men­tary values:

For more about the colorer inter­face, see the docs for start-colorer.

By the way, coloring anno­ta­tions are strictly cosmetic. They don’t change how the language works. If we goof them up, nothing bad will happen.

Designing the colorer

Our goal for our syntax colorer is to map the syntactic elements of our language onto the coloring cate­gories avail­able in DrRacket. How deep we go is up to us. For instance, it wouldn’t be wrong to make a syntax colorer that just anno­tates all the code with the 'string coloring cate­gory.

But that would be boring. So we’ll make a slightly more ambi­tious coloring plan for jsonic:

Writing the syntax colorer

Our syntax coloring will be handled by the color-jsonic func­tion in "jsonic/colorer". Let’s start that module now:

As we did in the tokenizer, we import brag/support to get lexer and other helper func­tions. We also import syntax-color/racket-lexer to get racket-lexer, Racket’s default syntax colorer, which we’ll use later.

We have two func­tions in this module:

The lexer

First, let’s write the coloring rules in jsonic-lexer. As before, we always need a rule that handles the eof signal that our port will emit when it reaches the end of the file. Previ­ously, we relied on the lexer’s default behavior to handle eof. This time, because we need a special return value, we write an explicit eof rule:

Each of these lexer rules needs to return a multiple-valued coloring anno­ta­tion. To do this, we use values, which is a way of returning multiple values from a func­tion, and how DrRacket expects the anno­ta­tions to be pack­aged.  + For more on func­tions returning multiple values, see func­tions. Following our spec­i­fi­ca­tion above, the anno­ta­tion has five values: the lexeme, the coloring cate­gory 'eof, and then the paren­thesis shape and two source loca­tions are all #f.

Next we handle our delim­iters:

We assign both delim­iters to the 'parenthesis coloring cate­gory. For the opening delim­iter, we use a paren­thesis shape of '|(|, and for the closing delim­iter, '|)| (Don’t be alarmed by the funny nota­tion—because of its special role in the Racket language, we have to escape a paren­thesis within vertical bars when we use it as a symbol.) As we learned before, our start and end posi­tions can be found in (pos lexeme-start) and (pos lexeme-end).

Our line comment is straight­for­ward—we can reuse the comment-matching rule from our tokenizer and assign it to the 'comment cate­gory. And as before, we use an any-char rule to catch every­thing else, assigning it to the 'string cate­gory:

The lexer toggler

With the lexer in place, we can write color-jsonic, which keeps track of whether we should be using the racket-lexer or the jsonic-lexer.

color-jsonic gets three argu­ments from DrRacket: an input port containing the source code, an offset value (that we ignore for now) and what­ever value has been set as the current coloring mode, which for us will be a Boolean indi­cating the racket-coloring-mode?. When the colorer starts, this argu­ment defaults to #f. We use a cond expres­sion to deter­mine which lexer to use:

The first branch of the cond invokes the jsonic-lexer in two situ­a­tions:

Once we get the five-value coloring anno­ta­tion from jsonic-lexer, we append two more values. First, a 0 for the backup distance (because we’re not using backup distance in this project, but we have to pass a default value). Second, a value that indi­cates whether we’re switching to Racket mode. The test is simple: if the str value returned from jsonic-lexer is our opening delim­iter @$, we switch, because that means we’re entering an embedded Racket expres­sion. On the next call to color-jsonic, this value will become the value passed as the racket-coloring-mode? argu­ment.

The Racket-mode branch is simpler:

Here, we just get the five basic color-anno­ta­tion values from calling racket-lexer, append another default backup distance of 0, and then #t to indi­cate that we should stay in Racket coloring mode.

Visu­ally testing the syntax colorer

A visual check isn’t a substi­tute for unit tests. But we’ll get to those in a minute. For now, let’s see if our syntax coloring does what we expect. Recall what we started with:

To see our colorer in action, we have to connect it to our get-info func­tion in "main.rkt". We do that by uncom­menting the branch we previ­ously commented out:

Now we can reopen our "jsonic-test.rkt" file in DrRacket. For faster perfor­mance, DrRacket caches the result of get-info for each language. We have to force a refresh. If we’re using Racket v6.9 or later, we select Racket → Reload #lang Exten­sions, which reloads our get-info func­tion and our new colorer. If not, we quit and restart DrRacket, which has the same effect. After a moment, the code will look like this:

We can verify that we’ve accom­plished the syntax-coloring improve­ments we wanted:

Let’s also check that our jsonic delim­iters are recog­nized as delim­iters in the GUI. If we click in front of the (* 6 7) expres­sion again, DrRacket still selects the whole Racket expres­sion:

But now, if we click in front of the jsonic delim­iter to the left, DrRacket will select the whole expres­sion, from the @$ on the left to the matching $@ on the right:

DrRacket is just following our instruc­tions from the syntax colorer, where we anno­tated the @$ and $@ with the 'parenthesis coloring class and the appro­priate paren­thesis shape.

Adding a contract

In the real world, we would’ve fitted our func­tion with contracts and unit tests before we wrote the middle parts. Here in tuto­rial world, better to cover the new mate­rial first.

But we’ve done that. So let’s now write a contract for color-jsonic. For input, we know it takes an input-port?, an offset value, which is an exact-nonnegative-integer?, and a mode value, which for us is a boolean?:

What about a contract for the return value: the color anno­ta­tion? Each color anno­ta­tion has seven values inside. So we can think about this as seven smaller contracts pack­aged inside a values contract combi­nator (just as we used values to package the indi­vidual values):

As we learned at the top of the page, these seven values are as follows: a lexeme (which is a string? or an eof-object?), a coloring cate­gory (symbol?), a paren­thesis shape (symbol? or #f), a starting and ending source posi­tion (both either exact-positive-integer? or #f), a backup value (another exact-nonnegative-integer?) and a boolean? mode. So the whole contract looks like this:

In our code, we add racket/contract to our imports and add our contract to our provide expres­sion using contract-out, so our module so far looks like this:

Adding a unit test

Let’s consider a simple test case, where we use open-input-string to convert the string "x" into an input port that can be passed to color-jsonic, along with a default offset argu­ment of 0 and a default mode argu­ment of #f:

What result do we expect? The matched string should be "x", the coloring cate­gory should be 'string, the paren­thesis shape is #f, the starting and ending posi­tions are 1 and 2, the backup distance is 0, and the mode remains #f. If we enter the above expres­sion at the REPL, we see that this is so:

Now we want to encap­su­late this in an auto­mated rackunit test. Testing a func­tion that returns multiple values requires a little special handling. We saw how rackunit provides func­tions like check-equal? to test the return value of an expres­sion. But these func­tions only take one value as input. They won’t work here, because color-jsonic returns seven values:

We wrap our test expres­sion in values->list, which will repackage our five return values as a five-element list (which only counts as one return value). We can then check this list against a list containing our expected results:

Our "jsonic/colorer.rkt" ends up like this:

Of course, in prac­tice we’d want to write (a lot) more than one unit test. But this suffices to intro­duce the basic pattern for how we’d write tests for color-jsonic.