Extend a data format: jsonic

We remember from our language spec­i­fi­ca­tion that every valid JSON file should be a valid jsonic program. Let’s test that propo­si­tion by running our orig­inal sample JSON under #lang jsonic:

Very good. Now let’s change the 42 to 3/5—which is an invalid JSON number—and see what happens.

We could imagine a friend­lier error message. The number 14 signals that the problem is at the 14th char­acter of the source code. But jsonic is behaving correctly. It’s raising an error because it can’t produce a valid JSON result.

Now let’s replace our JSON values with embedded Racket expres­sions. Taken together, they should produce the same result:

That’s also correct. The white­space is formatted differ­ently. The keys inside the JSON object have a different order. But those differ­ences don’t change the meaning of the resulting JSON.

As we did above, let’s break the source code. This time we replace the Racket expres­sion (* 6 7) with (/ 3 5), which produces an invalid number:

Again, we could imagine a friend­lier error message. But jsonic is doing what we hoped. It raises an error when we try to embed an unco­op­er­a­tive S-expres­sion.

We’ve met our orig­inal require­ments, so jsonic is complete.

Though not fool­proof

One lurking bug we should acknowl­edge. Below, we can see that (symbol->string '$@) is a legit Racket expres­sion that ought to eval­uate to the string "$@". So why doesn’t it?

The problem is that the $@ char­acter sequence is being used both as the right-hand delim­iter of the embedded Racket expres­sion, but also as literal code within that expres­sion. Our jsonic language has no way of distin­guishing between these two uses. It reads the shorter code frag­ment @$ (symbol->string '$@ as a single token because this matches one of the lexing rules. But once it trims the (apparent) delim­iters, it can’t figure out what to do with the resulting malformed Racket expres­sion (symbol->string '. Hence the error.

Unfor­tu­nately, this syntactic problem is always with us. In any language, we need to attach special nota­tional meaning to certain strings. But often, we still want to be able to use those strings as literal data. For instance, these programs won’t work either, for a similar reason:

Now that we’ve built a lexer, we can appre­ciate why languages some­times need escape char­ac­ters—extra char­ac­ters added to the source to signal “read these char­ac­ters liter­ally, not as special nota­tion”. Escape char­ac­ters remove ambi­guity by making the intended meaning explicit in the program syntax.

For instance, we would fix the two programs above by adding the appro­priate escape char­ac­ters:

Like­wise, we could define escape sequences for the $@ and @$ delim­iters in jsonic, so that they could be used in embedded Racket expres­sions. We could accom­plish this by the usual process of refining our lexing and parsing rules to recog­nize the new language elements. But having noted the flaw, we won’t digress further. The details are left as an exer­cise for the curious.