Level up: jsonic revisited

Before we move into DrRacket inte­gra­tion, we need to take a short detour to prepare our language.

Intro­ducing source loca­tions

A source loca­tion is a set of fields that pinpoint where an S-expres­sion (or other code) came from within a source file. Source loca­tions are used throughout Racket (as they are in other languages) for various tasks. For instance, error reports usually have a source loca­tion:

The message unsaved-editor:3:0 tells us that the error occurred in line 3, column 0.

Source loca­tions are typi­cally tracked as a set of five fields, which can each be #f if the value is unknown, or a number:

One small gotcha: not every Racket func­tion that uses source loca­tions prints all the source-loca­tion fields, or prints them in the same order. As we saw above, rackunit only prints the line and column from the source loca­tion.

Racket has a stan­dard struc­ture type called a srcloc that holds all five of these values. Func­tions that handle source loca­tions often use srcloc struc­tures for input or output.

Why are source loca­tions impor­tant?

As we learned in stacker, one of the big bene­fits of imple­menting a language in Racket is that it can use all of Racket’s existing libraries and tools. This works because every Racket-imple­mented language is really a source-to-source compiler that trans­lates the new language into a Racket program.

This means that even a graph­ical tool like DrRacket can handle languages that don’t look anything like Racket. For instance, source loca­tions are used by DrRacket to handle error-high­lighting effects and other GUI conve­niences. All we have to do is attach the orig­inal source loca­tions to the new Racket code. Suppose our new language lets us define a vari­able like so:

And it’s trans­lated into Racket code that looks like this:

When the divide-by-zero error occurs, DrRacket will be able to use the orig­inal source loca­tion to high­light the error in the orig­inal code.  + We will learn how to actu­ally do this in the basic tuto­rial.

But the cost of this coop­er­a­tion is a little extra house­keeping so our language provides the infor­ma­tion needed to support other Racket tools. Including source loca­tions.

As with contracts and unit tests, using source loca­tions is optional. In earlier tuto­rials, we didn’t worry about source loca­tions because we wanted to keep the focus on learning the core mechanics of making a language. Nothing bad happened.

But since we now want to inte­grate our language with DrRacket, we need to take this detour.

Where are source loca­tions stored?

We’ve already worked with syntax objects as a way of pack­aging a refer­ence to literal Racket code with certain meta­data fields. One field we’ve mentioned is lexical context, which is a list of vari­ables visible to the code.

Source loca­tion is another field that’s stored in a syntax object. For instance, when we use the #' prefix to make a syntax object from a datum, its source loca­tion will auto­mat­i­cally be stored in the resulting syntax object. In turn, these source-loca­tion fields can be read with syntax-line, syntax-column, etc.:

As with lexical context, a syntax object retains a refer­ence to its orig­inal source loca­tion unless these fields are explic­itly changed.

How are source loca­tions tracked?

As we might guess, source loca­tions are derived from the orig­inal source file that holds the code. There­fore, what­ever func­tion reads the source code is respon­sible for collecting source loca­tions.

After that, func­tions that handle the code need to preserve the source loca­tions. If we don’t manip­u­late the syntax objects much, this is easy. But more complex manip­u­la­tions can incur a little extra effort to make sure source loca­tions stay where they should.

In jsonic, the func­tion that first reads in the source code is make-tokenizer. Right now, make-tokenizer doesn’t collect source loca­tions. We can see this if we feed make-tokenizer a char­acter:

Our lexer is using the helper func­tion token to create an instance of token-struct named CHAR-TOK with a value of "x". But the next four fields are the source-loca­tion fields. They’re set to #f because no source-loca­tion infor­ma­tion has been collected. The fifth #f signals whether the parser should ignore the token. (We’ll do more with this in a later tuto­rial.)

Gath­ering the source-loca­tion data

For complete source loca­tions, we need four pieces of data: posi­tion, line, column, and span. We’ll collect this data, then embed it within the token struc­tures emitted by make-tokenizer. After that, as long as our parse and read-syntax func­tions preserve the source loca­tions, the source loca­tions will be avail­able to other tools, like DrRacket.

Let’s open "tokenizer.rkt" so we can add the neces­sary code to make-tokenizer.

Line count and column count are avail­able from the input port. But by default, an input port doesn’t track this infor­ma­tion. So first, we acti­vate line and column counting for our port by adding a call to port-count-lines!:

We won’t attach source-loca­tion data to our line-comment rule, because it doesn’t produce a token. But we will add it to our SEXP-TOK and CHAR-TOK rules.

Let’s start with CHAR-TOK because it’s a little easier:

Just as lexer creates a special vari­able called lexeme that holds the matched char­ac­ters, it also creates lexeme-start and lexeme-end, special vari­ables that hold the posi­tion, line, and column for the start and end of the lexeme. We retrieve these values from lexeme-start or lexeme-end with the helper func­tions pos, line, and col. Because the span is a rela­tive measure­ment, we calcu­late it by subtracting the start posi­tion from the end posi­tion. We then pass these values to token using its corre­sponding keyword argu­ments—#:position, #:line, #:column, and #:span.

Let’s use the REPL to see how this changes the result from make-tokenizer. If we run "tokenizer.rkt" now, we’ll get errors because our unit tests will fail. Don’t panic—we’ll fix those in a minute. Let’s jump down to the REPL and enter the sample expres­sion we tried earlier:

Last time, our source-loca­tion fields were all #f, because we hadn’t filled them in. This time, they show the source loca­tion we embedded for "x": it’s in posi­tion 1 of the input, line 1, column-offset 0, and has a span of 1.

Now we’ll handle the SEXP-TOK rule:

The basic idea is the same. But we need to adjust our source-loca­tion fields because we’re trim­ming two char­ac­ters from each end of the lexeme, but the source-loca­tion data comes from the untrimmed lexeme. Because each delim­iter is two char­ac­ters, we add 2 to both the posi­tion and column, and subtract 4 from the overall span.

That’s every­thing we need to change within make-tokenizer. We can see, perhaps, why not every project needs to track source loca­tions—it requires some extra house­keeping to keep the source-loca­tion data in sync, espe­cially if we’re performing other processing on our lexemes. On the other hand, we only have to do it once, and then we can enjoy the bene­fits of source loca­tions throughout our language, including DrRacket.

Updating our unit tests

Now that we’ve improved make-tokenizer, we have to update its unit tests to reflect the new behavior. (We didn’t change anything about jsonic-token?, so its tests will remain the same.)

We’ll rewrite our tests to use token and its keyword argu­ments to generate test tokens with the right source-loca­tion data. We can do this by just counting char­ac­ters. This was our first test case that got broken:

According to our lexer rules, this will become an SEXP-TOK token. Its lexeme will have two char­ac­ters trimmed from the begin­ning and end. For the trimmed lexeme, the posi­tion is two more than the orig­inal 1 = 3. The line number will remain 1. The column offset is two more than the orig­inal 0 = 2. And the span is four less than the orig­inal 13 = 9. So we’ll expect a list with one token that looks like this:

We do the same for our other test case:

According to our lexer rules, this will become two CHAR-TOK tokens. They will both be on line 1 and both have a span of 1. The first will be in posi­tion 1 and column 0. The second will be in posi­tion 2 and column 1:

Substi­tuting these new test results, the completed module will look like this:

When we run "tokenizer.rkt" again, the unit-testing errors will be gone.

Checking source loca­tions in our reader

Earlier, we noted that once we’ve captured the source loca­tions, our parse and read-syntax func­tions need to leave them intact so they’ll be avail­able to tools like DrRacket. Now that we’ve updated make-tokenizer to collect the source loca­tions, we should verify that they’re making it all the way through.

Let’s open "reader.rkt". Our read-syntax func­tion relies on parse. So it should suffice to pass some source code to read-syntax and see if the source loca­tions come through correctly (because that will also imply they’re being handled correctly by parse).

Let’s try a simple test case on the REPL:

read-syntax usually takes two argu­ments: a path to a source file and an input port that points at that file. But when we’re making test cases, we can also use it with a source string rather than a file. For the first argu­ment, we’ll just pass #f. For the second argu­ment, instead of a file port, we can convert a string into an input port with open-input-string. The string we’re using, "//x\ny\nz", is equiv­a­lent to this source:

The x is inside a line comment, so it should disap­pear. The y and z (and inter­vening newline) should appear in the parse tree, with source loca­tions on line 2 and 3, respec­tively.

When we run this expres­sion on the REPL, the result will look like this:

But in DrRacket, the little arrow on the left end of the line will be click­able. Click on it, and DrRacket will reveal a panel that we can use to explore the syntax object returned by read-syntax:

The left side of the panel shows the literal code inside the syntax object. We can see that it’s what we expected—the first line //x disap­pears, and the other char­ac­ters go into the parse tree.

DrRacket also lets us click on items of interest within the syntax object. The prop­er­ties of each item will be shown in the right-hand panel labeled Syntax Info. Again, click the triangle to the left of the panel’s name to reveal the whole panel. If we then click the "y" in the parse tree (as shown below) its source loca­tion will be revealed: posi­tion 5, line 2, column 0, and span 1.

This is exactly right. Like­wise, if we click "z", we’ll see posi­tion 7, line 3, column 0, and span 1.

Thus, we’ve veri­fied that parse and read-syntax are preserving source loca­tions correctly. It would also be possible to write some unit tests that auto­mat­i­cally verify this. But that would be a detour from the current detour. Let’s press onward.

Recap

What’s neat about source loca­tions in Racket is that they’re a prop­erty of the orig­inal source code that’s carried along with the code wher­ever it goes. So even if our language imple­men­ta­tion arranges the source code into a parse tree, then slices & dices it with any number of macros—the source loca­tions remain attached. (Or if we need to replace the code entirely, we can take its source loca­tion and attach it to the new item.) This, in turn, is possible because Racket handles code as recur­sively anno­tated syntax objects rather than plain strings.

Just as there’s more than one way to read a source file—we’ve been using a combi­na­tion of a tokenizer and parser, but that’s just one possi­bility—there’s more than one way to collect source-loca­tion data. For instance, we’ll learn a sleeker way of collecting source loca­tions in the basic tuto­rial.

The essen­tial idea always remains the same. As we read the source file, we also capture the source-loca­tion fields. We then anno­tate the parsed expres­sions with this infor­ma­tion.

Our process here was simple because lexer auto­mat­i­cally gathers source-loca­tion data—we just needed to start using it—and the parse func­tion created by #lang brag already knows how to preserve it.