Closing the loop: basic

Our project in this section will be to improve the REPL to be able to under­stand & eval­uate expres­sions written in BASIC.

REPL recap

We haven’t said much about the REPL, though we’ve been using it throughout this book. The REPL—an acronym for read–eval­uate–print loop, pronounced repple—is where Racket prints the results of programs:

On the command line, the REPL starts auto­mat­i­cally when we run racket. In DrRacket, the lower panel of each window contains the REPL. The DrRacket REPL is more capable than the command-line version, however. For a surprise, try this:

The REPL is used for input as well as output. Expres­sions can be entered directly on the REPL, and eval­u­ated imme­di­ately:

The REPL is not an inde­pen­dent eval­u­a­tion envi­ron­ment. Rather, expres­sions entered on the REPL are treated as if they lived in the top level of the currently running program. There­fore, REPL expres­sions can access all the bind­ings avail­able in the program, even if they’re not exported with provide. For instance, this program:

Prints an initial result:

But we can move to the REPL and eval­uate new expres­sions using f and x:

In a way, we can think of the REPL as performing a smaller-scale version of the processing steps that happen in a full Racket-imple­mented language. When we make a language, the source code is sent to our reader, which turns it into a module expres­sion, which is passed to our expander to be expanded and eval­u­ated, and the results are printed. Like­wise, as its name implies, the REPL reads an expres­sion, then expands, eval­u­ates, and prints it.  + In Racket, perhaps “REEPL” would be the more accu­rate acronym, but REPL is the tradi­tional name among Lisps.

The key differ­ence between a language and the REPL is the behavior during the reader phase. In a language, the input (= the lines of source code) is read as a module, which then behaves as a self-contained program. Whereas on the REPL, the input (= a single line) is read as a single expres­sion, which then behaves as a frag­ment of a larger program.

Limits of the REPL

Those who have been fiddling with the REPL during our tuto­rials may have noticed some­thing strange: no matter what our source language looks like, the REPL only accepts Racket-style S-expres­sions as input.

For instance, here’s a BASIC program similar to our earlier example that exports f and x:

When we try to enter a BASIC-style expres­sion on the REPL, we get an error:  + The REPL reads f(x) as two expres­sions: f followed by (x), which becomes (21) and trig­gers an error, becase 21 is “not a proce­dure”.

But if we use Racket-style syntax, it works:

“But we wrote a top-notch BASIC reader. Why doesn’t the REPL use that auto­mat­i­cally?” Because it won’t work. As we learned above, a language reader produces a module that repre­sents a self-contained program. Whereas a REPL reader needs to produce a single expres­sion. There­fore, by default, the REPL doesn’t rely on the reader of the current language. Instead, it relies on the usual S-expres­sion reader.

This isn’t completely useless. It just means that if we want to use the REPL, we need to manu­ally convert BASIC expres­sions to what they’d look like in the parse tree produced by the main language reader.

We just saw one example: we passed x as an argu­ment to the f func­tion on the REPL by using the S-expres­sion func­tion nota­tion (f x) rather than the BASIC func­tion nota­tion f(x).

Here’s another example. Even though we can’t eval­uate x + x * x directly:

We can eval­uate the parse-tree version of this expres­sion:

Not completely useless. But still a chore and a bore. It would defi­nitely be nicer to be able to enter BASIC expres­sions on the REPL with the usual BASIC nota­tion.

Improving the REPL

The good news is that we can configure the REPL to work with BASIC expres­sions. To do this, we have two tasks:

The second part is easy. Our BASIC-aware REPL will only be invoked when a BASIC module is run directly (vs. being imported into another module). There­fore, as we learned in exports, this is a job that can be handled by the configure-runtime submodule. We’ll update our do-setup! func­tion to set the REPL to use the new expres­sion reader.

So how about the first part—where do we get this expres­sion reader? Clearly, we’d prefer to avoid starting from scratch. Ideally, we’d like to derive our expres­sion reader from our current lexer & parser. In partic­ular, we’ve already written a bunch of parser rules for handling BASIC expres­sions—it would be nice to be able to just reach in and reuse them.

As it turns out, we can. So far, we’ve used #lang brag to generate a parse func­tion that starts from the first rule in the grammar. The starting rule ends up serving as the root node of the resulting parse tree. Since we use parse to handle whole programs, we’ve usually named this first rule langname-program.

But any module written in brag also provides a func­tion called make-rule-parser that lets us generate a parser for any rule within the grammar. So rather than starting with the first rule, we can start anywhere.

Let’s see how this works. Suppose we want to parse the string we used above—x + x * x—as a BASIC expres­sion. We convert the string into a port with open-input-string. Then, as we do in our reader, we pass the port to our tokenizer, and then a parser. But this time, rather than using parse, we generate a parser from the b-sum rule in our grammar:

The result, instead of a whole BASIC program, is a parse tree that starts at a b-sum node:

If we remove the trivial one-argu­ment expres­sions, we’re left with this:

This is the same as the parse-tree version of this expres­sion we calcu­lated by hand earlier. Best of all, we didn’t have to write anything new—we just reused our existing tokenizer and brag modules.

As a language reader, this parser wouldn’t be useful, because it can’t handle a whole program—just a sum expres­sion. But as a REPL reader, it’s getting closer to what we need.

Parser updates

Now that we know how to generate a parser for any rule, we have a choice:

That’s the more flex­ible approach, so we’ll do it that way. At the bottom of "parser.rkt", let’s add a new rule called b-repl:

This new REPL rule will accept BASIC state­ments (like print 24 * x) or raw BASIC expres­sions (like 24 * x). Following the conven­tion in BASIC lines, we allow multiple inputs on the REPL to be chained together with : (which we express with a spliced, recur­sive refer­ence to the b-repl rule).

Would it make sense to carve out certain state­ments from our REPL pattern? Our REPL expres­sions will be eval­u­ated as if they were in the top level of our language module. They will not, however, be eval­u­ated as if they were happening during the run loop. That means that certain state­ments on the REPL—like goto and gosub—will be invalid, because they don’t mean anything outside of the context of the run loop.

But if we carve them out here, then they’ll trigger a parsing error. That might be confusing to a user. Instead, we let them all through now, and handle the invalid ones in the expander, where we can raise a more infor­ma­tive error.

Setup module updates

In "setup.rkt", we attach our new parser to the REPL. Just as we do in our main reader module, we start by importing "parser.rkt" and "tokenizer.rkt". We also define a repl-parse func­tion based on our new b-repl parser rule:

To do its job, the REPL relies on a special para­meter called current-read-interaction. This para­meter holds a read-inter­ac­tion func­tion that’s respon­sible for taking the input from the REPL and chunking it into syntax objects to eval­uate. The REPL calls this read-inter­ac­tion func­tion repeat­edly to get new syntax objects. When the read-inter­ac­tion func­tion returns eof, the REPL stops (meaning, control returns to the user at the REPL prompt).

The read-inter­ac­tion func­tion takes two argu­ments: a port, which holds the text input from the REPL, and an origin value (that osten­sibly describes where the port came from, but we shouldn’t rely on it to contain useful infor­ma­tion). Because each user command on the REPL ends with a newline, the contents of the port will be a newline-termi­nated string. So if the user enters this at the REPL prompt:

The port passed to the read-inter­ac­tion func­tion will contain this:

When­ever we want our REPL to behave differ­ently, we need to set the current-read-interaction para­meter to use a new read-inter­ac­tion func­tion. In this case, we call our new func­tion read-one-line:

When the REPL calls read-one-line the first time, it will read the first (and only) newline-termi­nated string in the port using read-line. We convert this string into a new port with open-input-string, pass this port to make-tokenizer, and this tokenizer to repl-parse, which will turn it into a parsed syntax object.

When the REPL calls read-one-line a second time, we get an eof-object? from read-line—because there’s only going to be one newline-termi­nated string in port—and we return eof, which will stop the REPL.

This little eof fandango may seem silly. But the REPL doesn’t make any assump­tions about how input corre­sponds to syntax objects. That’s why we have to do this by hand. Actu­ally, if we wanted, we could imple­ment read-one-line using lexer:

Both versions will do the same thing. But we’ll stick with the first one—in this case, read-line takes care of the house­keeping more neatly than lexer.

By the way, unlike our main read-syntax func­tion, we don’t need to pass around an extra path argu­ment to the tokenizer and parser. We only needed that when we wanted to generate source loca­tions. In the REPL, we can ignore them.

To complete the REPL setup, we go inside do-setup! and reset the current-read-interaction to use our new read-one-line func­tion. We remember that do-setup! is called by our configure-runtime submodule. So when we run a BASIC program directly, the configure-runtime submodule will be trig­gered, and our REPL will be repro­grammed to use our read-one-line func­tion.

The finished "setup.rkt" module looks like this:

Expander updates

Because we chose to create a new b-repl rule in the parser, we also have to imple­ment it in the expander. We add a b-repl macro to "misc.rkt":

In the b-repl macro, we’re intro­ducing some syntax-pattern nota­tion we haven’t used before. The dot . is like a rest argu­ment for syntax patterns. It means “collect the rest of the syntax elements in this pattern vari­able”. It’s an alter­na­tive to an ellipsis ... for matching multiple items. So these two patterns are the same:

One differ­ence is that like a rest argu­ment, the dot can only be used at the tail of a pattern. An ellipsis can be used anywhere.

Another differ­ence comes when we use the resulting pattern vari­able in a syntax template. A pattern vari­able created with a dot, like ALL-INPUTS, repre­sents a self-contained list of syntax objects. If we want to combine it with another element, it has to appear at the end of a paren­the­sized expres­sion in the syntax template, and we use another dot . as a connector.

EACH-INPUT, by contrast, repre­sents the first element of a list of syntax objects, and the accom­pa­nying ... repre­sents the rest. So these syntax templates are equiv­a­lent:

A limi­ta­tion of the dot is that it doesn’t offer the step-and-repeat behavior that the ellipsis does. For instance, this expres­sion will multiply EACH-INPUT by 42:

But in macros where we’re not using this behavior, the dot can make our templates and patterns tidier. It’s an option, not a require­ment. For further details, see syntax patterns.

We still need to fill in the inte­rior of the b-repl macro. When we made the b-repl parser rule, we decided to permit every kind of state­ment and expres­sion as input, even the ones that didn’t make sense in a REPL context, and sort through them later.

That time has come. The main work of the b-repl macro will be to take a list of ALL-INPUTS and winnow it to a list of valid INPUTS.

So which inputs make sense on the REPL? Prob­ably just these:

All the other state­ments—including goto, gosub, and for / next—only make sense in the context of a running program. So we just raise an error if we see any of these.

To process our list of ALL-INPUTS, we use a syntax helper func­tion called pattern-case-filter. This is like a hybrid of cond and filter. It tries to match each element in a list of syntax objects to one or more pattern clauses. If a pattern matches, the syntax template on the right side is returned. If no pattern matches, or the right-side result is #f, the element is omitted. The result of pattern-case-filter is a new (and possibly shorter) list of syntax objects.

Let’s start our pattern-case-filter with a rule to match print state­ments. We want to just pass these through. So our syntax pattern will match input items that begin with b-print, and just emit a result that’s the same as the input:

Next are BASIC expres­sions. These will come through as b-expr items. We want to print these, so we wrap them in a b-print expres­sion. In other words, every raw BASIC expres­sion ex will behave as if we typed print ex.

But wait—there’s a trap for the unwary here. We know that an UPPERCASE pattern vari­able is a wild­card that matches any value; anything else in the syntax pattern is matched liter­ally. When a pattern contains an iden­ti­fier like b-expr, the notion of “liter­ally” means not just the iden­ti­fier’s name but also its binding.  + Racket jocks might know this as equality in the sense of free-identifier=?.

Let’s see an example. Here, zeta-stx is a syntax object that refers to zeta, a binding from the math library. But when we try to match zeta-stx to the pattern zeta, we get an empty list:

The problem here is that the literal iden­ti­fier zeta in the syntax pattern has no binding. So even though the iden­ti­fier names are the same, their bind­ings are not. There­fore, the pattern doesn’t match, and the result is the empty list.

If we want this pattern to work, we have to import math so that the syntax pattern in pattern-case-filter will refer to the same zeta:

Coming back to b-expr—when we see a parsed element starting with b-expr, it’s not just any b-expr, but the b-expr intro­duced from our "expr.rkt" module. So if we want to match this partic­ular b-expr, we need to import "expr.rkt" into this module:  + We didn’t have the same possible problem with b-print because it’s defined in this module.

Moving on to let and def state­ments. We want to support these because it’s useful to be able to create vari­ables and func­tions on the REPL. But we recall that in our language imple­men­ta­tion, our vari­ables and func­tions are hoisted to the top of the module and initial­ized to 0. Then the corre­sponding b-let and b-def macros use set! to change their values. This won’t work on the REPL—we can’t set! a vari­able that hasn’t been created.

So on the REPL, when we come across a b-let or b-def expres­sion, we rewrite them as ordi­nary define expres­sions:

Anything else should be an error. So we add one more case to our pattern-case-filter, pop it in the b-repl macro, and the finished "misc.rkt" looks like this:

Testing the REPL

Now we’re ready to try out our new REPL. Let’s make a little program that defines a func­tion and a vari­able:

We should be able to eval­uate these directly on the REPL:

How about some BASIC-style expres­sions:

Let’s also try using explicit print state­ments:

And colon-sepa­rated chains of state­ments and expres­sions:

Now a def and let:

And finally some bad input:

That took a while, but only because we have such high stan­dards. As with syntax coloring and indenting, our language will work fine even if we never improve the REPL. But the REPL is an intrinsic part of Racket. A small amount of extra effort—adapting the tokenizer and parser we’ve already made—pays off in a better language inter­face.