Extend a data format: jsonic

Recap of the expander’s role

Along with the reader, the expander is the other essen­tial compo­nent of every Racket-imple­mented language:

For an iden­ti­fier to have a meaning, it needs a binding. Conversely, an iden­ti­fier without a binding has no meaning.

For instance, if we try to run this program:

We get this error:

Why? When we run the program, Racket tries to eval­uate the iden­ti­fier x. But it can’t, because x doesn’t yet have a binding. Hence the “unbound iden­ti­fier” error.

We can bind x by assigning it a value with define:

More broadly, every iden­ti­fier in a Racket program needs a binding before the program can run. define is just one way we can bind an iden­ti­fier. We can also import a binding from another module with require, if it was exported from that module with provide:

This is why Rack­e­teers tend to speak of “binding iden­ti­fiers” rather than “assigning values”. The latter phrase narrowly suggests the use of define. But there are many ways to bind an iden­ti­fier.

The main job of an expander is to bind the iden­ti­fiers in a parse tree, even though no define expres­sions appear in the parse tree itself. This is why the reader delivers code without bind­ings—so the expander can start with a blank slate. The expander does this job by exporting a binding (using provide) for each iden­ti­fier in the parse tree.

When the reader returns its module expres­sion, it embeds a refer­ence to the expander. For instance, in the read-syntax func­tion for jsonic, inside module-datum, we see the refer­ence to jsonic/expander:

When the code repre­sented by module-datum is eval­u­ated, jsonic/expander is imported and provides the initial bind­ings. In other words, module behaves as if there were an implicit (require jsonic/expander) on the inside. So we don’t need an explicit require.

Again, this is mostly a recap. If the details are hazy, here’s a review.

Designing the expander

Racket starts the expander for a lan­guage by invok­ing a macro called #%module-begin. There­fore, as in previous tuto­rials, our jsonic expander will provide a #%module-begin macro.

Beyond that, we just saw how the parse tree produced by the jsonic parser follows the produc­tion rules of the grammar. In turn, we can use these produc­tion rules to orga­nize the rest of our expander. Let’s review the tech­nique we first learned in bf:

The phrase “macro or func­tion” doesn’t imply that we’ll always have a choice. Certain trans­for­ma­tions will require macros. Other­wise, we should prefer func­tions. But here in the jsonic expander, we’ll opt for macros only, just to get some more expe­ri­ence writing them.

With that in mind, let’s take another look at the grammar for jsonic:

This grammar implies that our expander should have three more macros (in addi­tion to #%module-begin):

We also know from our language spec­i­fi­ca­tion that we need to do some house­keeping in terms of converting results to strings, and making sure we’re gener­ating valid JSON. We’ll deal with those tasks as they arise.

That gives us enough of a roadmap to get going.

Starting the expander

Let’s create a new expander module called "expander.rkt". Our expander must have a #%module-begin macro, so we’ll start there. The shell of the macro looks like this:

Following our existing habit, we define the macro as jsonic-mb and then use rename-out in the provide expres­sion, so that it doesn’t conflict with the other #%module-begin imported from br/quicklang. The input to this macro is our PARSE-TREE.

Now let’s fill in the blanks. A major require­ment of our language is that every jsonic program results in valid JSON. This implies two things: that the result of the program has to be a string, and that the string has to be vali­dated against the JSON stan­dard.

Let’s assume for a moment that we can convert our parse tree into a string (and we will, with our next set of macros). Since jsonic-mb is the top-level macro in our program, we can use it to vali­date our new string as valid JSON, and return the result:

We’ll assume for now that the expan­sion of PARSE-TREE results in a string, which we assign to result-string.

To help us out, we import Racket’s json library. A major benefit of making languages in Racket is that all its libraries are avail­able for every DSL, so we can be opti­mally lazy.

In this case, json exports a func­tion called string->jsexpr. This func­tion converts a string to a special kind of S-expres­sion called a JS-expres­sion. A JS-expres­sion holds a repre­sen­ta­tion of JSON data. If string->jsexpr succeeds, it returns a JS-expres­sion; other­wise it raises an error.

Thus, we can confirm that the result-string of our program is valid JSON by calling string->jsexpr. For clarity, we assign its return value to validated-jsexpr. This is optional, since we ignore this result—we only care about the vali­da­tion side effect. Then we just display our result-string.

Expanding the parse tree

Now we’ll add the macros that corre­spond to the produc­tion rules in our jsonic grammar.

We’ll start with jsonic-char, because it’s easy:

This macro is defined with a pattern that matches one item, which is the value from a CHAR-TOK token, repre­senting a char­acter in a JSON string. All we need to do in this case is pass through this string, which means converting it into a syntax object. 

By the way, there’s nothing special about the name of the pattern vari­able, in this case CHAR-TOK-VALUE. It has nothing to do with the token name used in the grammar. And the macro would work the same way with any pattern-vari­able name.

Then we’ll move to jsonic-program. It’s almost as easy:

This macro is defined with a pattern that matches any number of items, each of which is either a jsonic-sexp or jsonic-char. In turn, each of these is going to be a macro that emits code that produces a string. As the top-level macro in the parse tree, jsonic-program needs to combine these little strings into one big string (which will become the input to string->jsexpr in jsonic-mb).

We can do this with string-append and, for tidi­ness, string-trim (which will lop off any leading or trailing white­space). (As above, there’s nothing special about the pattern-vari­able name SEXP-OR-JSON-STR. It’s just meant to be descrip­tive.)

That leaves the jsonic-sexp macro, which is only slightly less easy:

The input is the string value from a SEXP-TOK token, repre­senting a Racket expres­sion. In our macro syntax pattern, we call this SEXP-STR. We need to manu­ally convert this string into a new datum that we can insert back into the program, as if it had been there all along.

with-pattern lets us intro­duce new pattern vari­ables that we can then use inside syntax templates. In this case, we want to create a new SEXP-DATUM.

To make this datum, we pass our SEXP-STR input argu­ment to format-datum. We used this func­tion before, in the stacker reader, to convert a source string into an S-expres­sion. We’re using it for the same purpose here.

A point about nota­tion. In the syntax pattern defining the macro, we call the pattern vari­able SEXP-STR, but within format-datum, we write #'SEXP-STR. In both cases, the name of the pattern vari­able is SEXP-STR. But within the macro, we can’t use a naked pattern vari­able—we have to put it inside a syntax template. So when we say #'SEXP-STR, we’re creating the world’s smallest syntax template that contains only the pattern vari­able SEXP-STR.

Finally, we need to return the result of our Racket expres­sion as a JSON string. In jsonic-mb, we used string->jsexpr from the json library. This library also provides the inverse func­tion, jsexpr->string, to convert a Racket JS-expres­sion to a JSON string. So in our syntax template, we just pass our SEXP-DATUM to this func­tion.

As we saw before, one benefit of relying on Racket’s json library is that we don’t have to work as hard. But even better, we add a layer of error checking to our DSL for free. When jsexpr->string runs, it will confirm that our Racket expres­sion repre­sents a valid JS-expres­sion, and raise an error if it doesn’t.

Of course, the cost of relying on the json library is that we have to abide by its rules. For instance, for a hash table to count as a valid JS-expres­sion, it has to use only symbols for keys. Also, the JSON value null has to be repre­sented as the Racket symbol 'null. If we wanted jsonic to be more lenient with input, we could insert a helper func­tion that would convert a wider range of Racket values to the narrower set accepted as JS-expres­sions.

Checking the expander

Before we test our expander, let’s check that every­thing is in the right place:

Now that we have a reader and expander for jsonic, let’s try using the language.