Thank you for your comment

Beau­tiful Racket / tuto­rials

Closing the loop: basic

We’ll end this tuto­rial by adding support for command-line argu­ments. Let’s be clear: no one will ever write a shell script in BASIC. But it’ll give us a look at how Racket handles command-line argu­ments. We’ll also learn a few things about macro hygiene.

Best of all, we only have to update one module. Really.

Command-line argu­ments

In exports, we learned about the para­meter called current-output-port, which controls where printed output goes; in the REPL, we learned about the current-read-interaction para­meter that controls how the REPL parses its input.

Simi­larly, Racket makes its command-line argu­ments avail­able through a para­meter called current-command-line-arguments, which holds a vector containing the command line argu­ments that were passed during the current session. For instance, let’s make a new "args.rkt" module:

basic/args.rkt
#lang br
(current-command-line-arguments)
#lang br
(current-command-line-argu­ments)
copy to clipboard

If we run this file in DrRacket, the vector of command-line argu­ments is empty—as we’d expect, since we’re not running it from the command line:

'#()
'#()
copy to clipboard

But if we go to the command line, we can ask racket to start with a certain module by using the -l flag and the module name. Subse­quent argu­ments are treated as command-line argu­ments for that module:

racket -l basic/args "foo" 42 --bar "zam"
> racket -l basic/args "foo" 42–bar "zam"
copy to clipboard

This time, "args.rkt" prints the command-line argu­ments:

'#("foo" "42" "--bar" "zam")
'#("foo" "42" "–bar" "zam")
copy to clipboard

Beyond that, what we do with these argu­ments is up to us. Racket’s command-line func­tion, for instance, provides a high-level inter­face for parsing command-line argu­ments, including flags.

Cracking the shell

For BASIC, we won’t get that fancy. Instead, we’ll just make the first 10 command-line argu­ments avail­able within our BASIC program under the special names arg0, arg1, ... arg9. That means we have two tasks:

  1. Create the arg0, arg1, ... arg9 vari­ables.

  2. Set the value of each of these vari­ables to the corre­sponding value read from the command line. (If there are fewer than 10 command-line argu­ments, we’ll set the remaining vari­ables to 0.) 

One small limi­ta­tion is that we can’t match the number of arg* vari­ables to the number of command-line argu­ments. We’ll be relying on syntax trans­for­ma­tion to create the vari­ables at compile time. But the actual command-line argu­ments won’t be avail­able until run time. So the number of vari­ables has to be fixed in advance. That said, 10 is arbi­trary—we could make 25 or 100. The code would work the same way.

Once we do this, we should be able to use these vari­ables like any other. If we make a sample "report-args.rkt" module (using basic-demo-3 just for illus­tra­tion):

basic/report-args.rkt
#lang basic-demo-3
10 print "arg0 is " ; arg0
20 print "arg1 + arg1 is " ; arg1 + arg1
40 print "arg3 is " ; arg3
50 print "arg4 is " ; arg4
℘
10 print "arg0 is " ; arg0
20 print "arg1 + arg1 is " ; arg1 + arg1
40 print "arg3 is " ; arg3
50 print "arg4 is " ; arg4
copy to clipboard

And invoke it from the command line like so:

racket -l basic/report-args "foo" 42 --bar "zam"
> racket -l basic/report-args "foo" 42–bar "zam"
copy to clipboard

We get this report of the command-line argu­ments. We omit arg2 just to show that we don’t have to use them all. Since there is no arg4 passed to the program, it behaves like an ordi­nary BASIC vari­able initial­ized to 0):

arg0 is foo
arg1 + arg1 is 84
arg3 is zam
arg4 is 0
arg0 is foo
arg1 + arg1 is 84
arg3 is zam
arg4 is 0
copy to clipboard

Seems straight­for­ward. But it puts us on a colli­sion course with an issue that confounds every writer of Racket macros at least once—how to circum­vent macro hygiene.

So before we get to the code, a small detour.

Intro­ducing hygiene

Hygiene is the orga­nizing policy of Racket’s macro system. It holds that iden­ti­fiers intro­duced by a macro should get their bind­ings from the place where the macro was defined, not the place where the macro is used. The effect is to keep macro-created iden­ti­fiers in a name­space sepa­rate from that of others. In Racket, these name­spaces are called lexical contexts.

“If hygiene is so impor­tant, then why haven’t we been hearing more about it?” There’s been no need. The nice thing about hygiene is that it usually conforms to our intu­itions about how macro-created code should work, and cuts down on house­keeping. (Hygiene is so named because of the “clean­li­ness” it enforces between lexical contexts.)

For instance, in our macro adven­tures so far, we haven’t had to worry about macro-intro­duced iden­ti­fiers conflicting with others. For this, we can thank hygiene. In this example, we have a begin block that defines a vari­able x, and then a make-x macro that produces the same begin block with a different value for x. When we run both of these blocks in the same module, do the two x defi­n­i­tions conflict?

(begin (define x 42) x)

(define-macro (make-x)
  #'(begin (define x 'macro-id) x))

(make-x)
(begin (define x 42) x)

(define-macro (make-x)
  #'(begin (define x 'macro-id) x))

(make-x)
copy to clipboard

No, they do not:

42
'macro-id
42
'macro-id
copy to clipboard

What’s happening here? When Racket creates an iden­ti­fier during compile time, it asso­ciates it with a partic­ular lexical context. Later, when Racket needs to resolve the binding of that iden­ti­fier, it relies on its lexical context. To Racket, the two x vari­ables in the above example are completely different.

Under the hood, Racket attaches the lexical context to the syntax object that contains the iden­ti­fier. Just like the other meta­data attached to the syntax object—e.g., source loca­tion or syntax prop­er­ties—lexical context is preserved as the iden­tifer travels through the various macro trans­for­ma­tions that happen during compile time.

Having multiple vari­ables with the same name shouldn’t be alarming. Consider an anal­o­gous example: with let, we’re accus­tomed to intro­ducing local bind­ings that are only valid within the body of the expres­sion, that may shadow existing bind­ings. In this example, we have two defi­n­i­tions of a vari­able called z, but we know the local binding of z won’t “escape” its surrounding let and collide with the binding on the outside:

(define z 42)
z ; 42
(let ()
  (define z 'let-id)
  z) ; 'let-id
(define z 42)
z ; 42
(let ()
  (define z 'let-id)
  z) ; 'let-id
copy to clipboard

Hygiene is a different mech­a­nism, but has a similar effect—creating lexical sepa­ra­tion between vari­ables.

So what happens if we remove hygiene? We can find out by changing our define-macro to define-unhygienic-macro, which will make our macro write its code into the existing lexical context, rather than keeping its code sepa­rate:  + “You’re doing it wrong” is the lowest form of program­ming advice. But please don’t use define-unhygienic-macro in your own code. It exists only for demon­stra­tion purposes. Other­wise, it is a bringer of dark­ness and despair.

(begin (define x 42) x)

(define-unhygienic-macro (make-x)
  #'(begin (define x 'macro-id) x))

(make-x)
(begin (define x 42) x)

(define-unhy­gienic-macro (make-x)
  #'(begin (define x 'macro-id) x))

(make-x)
copy to clipboard

This time, our two x vari­ables will occupy the same lexical context, so the result is different:

module: duplicate definition for identifier in: x
module: dupli­cate defi­n­i­tion for iden­ti­fier in: x
copy to clipboard

This illus­trates why hygiene is an essen­tial ingre­dient in a compos­able—meaning, “plays nicely with others”—macro system. Once we make a macro, we want to be able to call it from any code, and have it behave the same way (and not break things). Without hygiene, we’d always have the risk that an iden­ti­fier in the macro could collide with one already defined at the calling site. Hygiene guar­an­tees this will never happen.

The limits of hygiene

In most cases, hygiene makes writing macros easier. But occa­sion­ally it conflicts with legit­i­mate goals, like using a macro as a substi­tute for define.

Here’s an example that trips up every Rack­e­teer at least once (and usually more). Suppose a program repeat­edly defines three vari­ables:

(define a 1)
(define b 2)
(define c 3)
(+ a b c) ; 6
(define a 1)
(define b 2)
(define c 3)
(+ a b c) ; 6
copy to clipboard

Before long, a bright idea arises. “I know—I’ll make a macro!”

(define-macro (define-three-vars)
  #'(begin
      (define a 1)
      (define b 2)
      (define c 3)))
(define-three-vars)
(+ a b c)
(define-macro (define-three-vars)
  #'(begin
      (define a 1)
      (define b 2)
      (define c 3)))
(define-three-vars)
(+ a b c)
copy to clipboard

But then disap­point­ment ensues:

a: unbound identifier in module in: a
a: unbound iden­ti­fier in module in: a
copy to clipboard

What’s the problem here? Same as the problem with make-x—hygiene guar­an­tees that the iden­ti­fiers intro­duced by the macro remain in a sepa­rate lexical context. So a b and c are not acces­sible at the calling site after the macro is invoked.

In a situ­a­tion like this, we have two choices:

  1. We can rewrite the macro hygien­i­cally by passing the iden­ti­fiers as argu­ments:

    (define-macro (define-three-vars ID1 ID2 ID3)
      #'(begin
          (define ID1 1)
          (define ID2 2)
          (define ID3 3)))
    (define-three-vars a b c)
    (+ a b c) ; 6
    (define-macro (define-three-vars ID1 ID2 ID3)
  #'(begin
      (define ID1 1)
      (define ID2 2)
      (define ID3 3)))
(define-three-vars a b c)
(+ a b c) ; 6
    copy to clipboard

    When we rewrite the macro this way, the iden­ti­fiers a b and c are no longer being intro­duced by the macro. Rather, they’re being intro­duced on the outside, and passed to the macro as argu­ments. There­fore, rather than being siloed in the lexical context of the macro, they retain their lexical context from the calling site.

    This is why we didn’t encounter any hygiene-related diffi­cul­ties when we added vari­ables to BASIC. All the iden­ti­fiers for our vari­ables came from the orig­inal source code. Even though our expander picked them up and moved them around, they retained their orig­inal lexical context.

  2. Or, if we don’t want to pass the iden­ti­fiers as argu­ments, we can manu­ally create iden­ti­fiers connected to the lexical context of the calling site. Because they circum­vent hygiene, these are known as unhy­gienic iden­ti­fiers:

    (define-macro (define-three-vars)
      (with-pattern ([ID1 (datum->syntax caller-stx 'a)]
                     [ID2 (datum->syntax caller-stx 'b)]
                     [ID3 (datum->syntax caller-stx 'c)])
        #'(begin
            (define ID1 1)
            (define ID2 2)
            (define ID3 3))))
    (define-three-vars)
    (+ a b c) ; 6
    (define-macro (define-three-vars)
  (with-pattern ([ID1 (datum->syntax caller-stx 'a)]
                 [ID2 (datum->syntax caller-stx 'b)]
                 [ID3 (datum->syntax caller-stx 'c)])
    #'(begin
        (define ID1 1)
        (define ID2 2)
        (define ID3 3))))
(define-three-vars)
(+ a b c) ; 6
    copy to clipboard

    Here, we use datum->syntax to make each iden­ti­fier, which takes a lexical context as its first argu­ment, and a datum as the second. The result is a new syntax object asso­ci­ated with the given lexical context.  + We first used datum->syntax when we made the reader for stacker. Back then, we used #f as the lexical context, because we delib­er­ately wanted the resulting syntax object to have no bind­ings.

    For the lexical-context argu­ment, we pass a syntax object with the lexical context we want to “borrow”. In this case, it’s the macro vari­able caller-stx, which is the syntax object repre­senting the orig­inal call to the macro, and thus carries the lexical context of the calling site.

In terms of Racket idiom, unhy­gienic iden­ti­fiers occupy the same space as muta­tion or para­me­ters: we avoid them when we can, because they go against the natural grain of the language. But when they’re the right tool for the job, we’re not finicky about using them.

For instance, when we want to create a new struc­ture type, we use struct, which works in part by intro­ducing unhy­gienic iden­ti­fiers at the calling site (constructor, pred­i­cate, getters, setters):

(struct thing (left right) #:mutable)
(define t (thing 'foo 'bar))
(thing? t) ; #t
(thing-left t) ; 'foo
(thing-right t) ; 'bar
(set-thing-left! t 'zam)
(thing-left t) ; 'zam
(struct thing (left right) #:mutable)
(define t (thing 'foo 'bar))
(thing? t) ; #t
(thing-left t) ; 'foo
(thing-right t) ; 'bar
(set-thing-left! t 'zam)
(thing-left t) ; 'zam
copy to clipboard

For this reason, struct will fail if any of these unhy­gienic names conflict with existing iden­ti­fiers at the calling site:

(define (thing? x) #t)
(struct thing (left right) #:mutable)
(define (thing? x) #t)
(struct thing (left right) #:mutable)
copy to clipboard
module: duplicate definition for identifier in: thing?
module: dupli­cate defi­n­i­tion for iden­ti­fier in: thing?
copy to clipboard

By the way, datum->syntax is just one option for creating unhy­gienic iden­ti­fiers. Helper func­tions like prefix-id, suffix-id, and format-id can also create iden­ti­fiers in a different lexical context. Further­more, though it’s most common for unhy­gienic iden­ti­fiers to be inserted in the lexical context of the macro calling site, we can put them—or any other syntax items—inside any lexical context.

Having completed our detour about hygiene, we’re ready to imple­ment command-line argu­ments in BASIC.

Expander updates

Our task is to get the argu­ments out of current-command-line-arguments and assign them to the indexed vari­ables arg0 through arg9.

We only need to update "expander.rkt", but in a few places.

In our begin-for-syntax block, we add make-shell-ids-and-idxs, a helper func­tion for making our indexed iden­ti­fiers:

(define (make-shell-ids-and-idxs ctxt)
    (define arg-count 10)
    (for/list ([idx (in-range arg-count)])
      (list (suffix-id #'arg idx #:context ctxt) idx)))
(define (make-shell-ids-and-idxs ctxt)
    (define arg-count 10)
    (for/list ([idx (in-range arg-count)])
      (list (suffix-id #'arg idx #:context ctxt) idx)))
copy to clipboard

This func­tion takes one argu­ment, ctxt, which is the lexical context where we want to place the iden­ti­fiers. It will return a list of syntax objects, each of which contains an iden­ti­fier and the index of the command-line argu­ment it corre­sponds to.  + We could change the arg-count value here to generate more than 10 arg* iden­ti­fiers.

For later use, we refactor our find-property func­tion to extract a helper func­tion called unique-ids, which removes dupli­cate iden­ti­fiers from a list:

(define (unique-ids stxs)
  (remove-duplicates stxs #:key syntax->datum))

(define (find-property which line-stxs)
  (unique-ids
   (for/list ([stx (in-list (stx-flatten line-stxs))]
              #:when (syntax-property stx which))
     stx)))
(define (unique-ids stxs)
  (remove-dupli­cates stxs #:key syntax->datum))

(define (find-prop­erty which line-stxs)
  (unique-ids
   (for/list ([stx (in-list (stx-flatten line-stxs))]
              #:when (syntax-prop­erty stx which))
     stx)))
copy to clipboard

In our b-module-begin macro, we make a list of SHELL-ID and SHELL-IDX values by calling make-shell-ids-and-idxs. We pass caller-stx as the lexical-context argu­ment, since we want these new iden­ti­fiers to behave as if they came from the calling site.

[((SHELL-ID SHELL-IDX) ...)
 (make-shell-ids-and-idxs caller-stx)]
[((SHELL-ID SHELL-IDX) ...)
 (make-shell-ids-and-idxs caller-stx)]
copy to clipboard

If any of our indexed iden­ti­fiers arg0 through arg9 already appear in the code, they’ll already be part of the VAR-ID ... list. We can’t define any vari­able twice. So we need to generate a list of unique iden­ti­fiers. We do this by concate­nating VAR-ID ... and SHELL-ID ... into one list and running this list through unique-ids.

[(UNIQUE-ID ...)
 (unique-ids
  (syntax->list #'(VAR-ID ... SHELL-ID ...)))]
[(UNIQUE-ID ...)
 (unique-ids
  (syntax->list #'(VAR-ID ... SHELL-ID ...)))]
copy to clipboard

After that, we just need to define each UNIQUE-ID:

(define UNIQUE-ID 0) ...
(define UNIQUE-ID 0) ...
copy to clipboard

We also set! every SHELL-ID to its corre­sponding command-line argu­ment, using SHELL-IDX as the index into the vector:

(let ([clargs (current-command-line-arguments)])
  (set! SHELL-ID (get-clarg clargs SHELL-IDX)) ...)
(let ([clargs (current-command-line-argu­ments)])
  (set! SHELL-ID (get-clarg clargs SHELL-IDX)) ...)
copy to clipboard

Because we prefer to mini­mize the amount of code inside the body of our macro, we move the get-clarg helper func­tion to the outside:

(define (get-clarg clargs idx)
  (if (<= (vector-length clargs) idx)
      0
      (let ([val (vector-ref clargs idx)])
        (or (string->number val) val))))
(define (get-clarg clargs idx)
  (if (<= (vector-length clargs) idx)
      0
      (let ([val (vector-ref clargs idx)])
        (or (string->number val) val))))
copy to clipboard

The helper func­tion takes two argu­ments: the vector of command-line argu­ments, and a vector index. If the idx exceeds the number of values avail­able in clargs, we return 0. Other­wise, the command-line argu­ment will be a string, or a string that repre­sents a number. We try converting it with string->number, but if it returns #f, we return the argu­ment itself.

When we put all the pieces together, our "expander.rkt" looks like this:

basic/expander.rkt
#lang br/quicklang
(require "struct.rkt" "run.rkt" "elements.rkt" "setup.rkt")
(provide (rename-out [b-module-begin #%module-begin])
         (all-from-out "elements.rkt"))

(define-macro (b-module-begin (b-program LINE ...))
  (with-pattern
      ([((b-line NUM STMT ...) ...) #'(LINE ...)]
       [(LINE-FUNC ...) (prefix-id "line-" #'(NUM ...))]
       [(VAR-ID ...) (find-property 'b-id #'(LINE ...))]
       [(IMPORT-NAME ...)
        (find-property 'b-import-name #'(LINE ...))]
       [(EXPORT-NAME ...)
        (find-property 'b-export-name #'(LINE ...))]
       [((SHELL-ID SHELL-IDX) ...)
        (make-shell-ids-and-idxs caller-stx)]
       [(UNIQUE-ID ...)
        (unique-ids
         (syntax->list #'(VAR-ID ... SHELL-ID ...)))])
    #'(#%module-begin
       (module configure-runtime br
         (require basic/setup)
         (do-setup!))
       (require IMPORT-NAME) ...
       (provide EXPORT-NAME ...)
       (define UNIQUE-ID 0) ...
       (let ([clargs (current-command-line-arguments)])
         (set! SHELL-ID (get-clarg clargs SHELL-IDX)) ...)
       LINE ...
       (define line-table
         (apply hasheqv (append (list NUM LINE-FUNC) ...)))
       (parameterize
           ([current-output-port (basic-output-port)])
         (void (run line-table))))))

(define (get-clarg clargs idx)
  (if (<= (vector-length clargs) idx)
      0
      (let ([val (vector-ref clargs idx)])
        (or (string->number val) val))))

(begin-for-syntax
  (require racket/list)
  
  (define (unique-ids stxs)
    (remove-duplicates stxs #:key syntax->datum))

  (define (find-property which line-stxs)
    (unique-ids
     (for/list ([stx (in-list (stx-flatten line-stxs))]
                #:when (syntax-property stx which))
       stx)))

  (define (make-shell-ids-and-idxs ctxt)
    (define arg-count 10)
    (for/list ([idx (in-range arg-count)])
      (list (suffix-id #'arg idx #:context ctxt) idx))))
#lang br/quick­lang
(require "struct.rkt" "run.rkt" "elements.rkt" "setup.rkt")
(provide (rename-out [b-module-begin #%module-begin])
         (all-from-out "elements.rkt"))

(define-macro (b-module-begin (b-program LINE ...))
  (with-pattern
      ([((b-line NUM STMT ...) ...) #'(LINE ...)]
       [(LINE-FUNC ...) (prefix-id "line-" #'(NUM ...))]
       [(VAR-ID ...) (find-prop­erty 'b-id #'(LINE ...))]
       [(IMPORT-NAME ...)
        (find-prop­erty 'b-import-name #'(LINE ...))]
       [(EXPORT-NAME ...)
        (find-prop­erty 'b-export-name #'(LINE ...))]
       [((SHELL-ID SHELL-IDX) ...)
        (make-shell-ids-and-idxs caller-stx)] 
       [(UNIQUE-ID ...)
        (unique-ids
         (syntax->list #'(VAR-ID ... SHELL-ID ...)))])
    #'(#%module-begin
       (module configure-runtime br
         (require basic/setup)
         (do-setup!))
       (require IMPORT-NAME) ...
       (provide EXPORT-NAME ...)
       (define UNIQUE-ID 0) ...
       (let ([clargs (current-command-line-argu­ments)])
         (set! SHELL-ID (get-clarg clargs SHELL-IDX)) ...)
       LINE ...
       (define line-table
         (apply hasheqv (append (list NUM LINE-FUNC) ...)))
       (para­me­terize
           ([current-output-port (basic-output-port)])
         (void (run line-table))))))

(define (get-clarg clargs idx)
  (if (<= (vector-length clargs) idx)
      0
      (let ([val (vector-ref clargs idx)])
        (or (string->number val) val))))

(begin-for-syntax
  (require racket/list)

  (define (unique-ids stxs)
    (remove-dupli­cates stxs #:key syntax->datum))

  (define (find-prop­erty which line-stxs)
    (unique-ids
     (for/list ([stx (in-list (stx-flatten line-stxs))]
                #:when (syntax-prop­erty stx which))
       stx)))

  (define (make-shell-ids-and-idxs ctxt)
    (define arg-count 10)
    (for/list ([idx (in-range arg-count)])
      (list (suffix-id #'arg idx #:context ctxt) idx))))
copy to clipboard

Testing our command-line argu­ments

We should now be able to run our "report-args.rkt" module under #lang basic:

basic/report-args.rkt
#lang basic
10 print "arg0 is " ; arg0
20 print "arg1 + arg1 is " ; arg1 + arg1
40 print "arg3 is " ; arg3
50 print "arg4 is " ; arg4
℘
10 print "arg0 is " ; arg0
20 print "arg1 + arg1 is " ; arg1 + arg1
40 print "arg3 is " ; arg3
50 print "arg4 is " ; arg4
copy to clipboard

And invoke it from the command line like so:

racket -l basic/report-args "foo" 42 --bar "zam"
> racket -l basic/report-args "foo" 42–bar "zam"
copy to clipboard

We should see this report of the command-line argu­ments:

arg0 is foo
arg1 + arg1 is 84
arg3 is zam
arg4 is 0
arg0 is foo
arg1 + arg1 is 84
arg3 is zam
arg4 is 0
copy to clipboard

This shows that all the pieces are working. In partic­ular, it shows that we’ve success­fully created unhy­gienic iden­ti­fiers that behave as if they were defined at the calling site (that is, in the BASIC source).

To appre­ciate the impor­tance of breaking hygiene, we can try removing the #:context ctxt argu­ment from the call to suffix-id. If we try invoking the same shell command:

racket -l basic/report-args "foo" 42 --bar "zam"
> racket -l basic/report-args "foo" 42–bar "zam"
copy to clipboard

This time, we get an error:

set!: unbound identifier in module
  in: arg0
  context...:
   standard-module-name-resolver
set!: unbound iden­ti­fier in module
  in: arg0
  context...:
   stan­dard-module-name-resolver
copy to clipboard

This error arises because there are two mismatched arg0 iden­ti­fiers in the code. One is attached to the lexical context of the BASIC source, and appears in the VAR-ID ... list. This one is used for the define. The other is attached to the lexical context of make-shell-ids-and-values, and appears in the SHELL-ID ... list. This one is used for the set!. Hence the error: the macro-intro­duced arg0 used with set! has not been defined yet.

One gotcha about working with unhy­gienic iden­ti­fiers is that error messages can be ambiguous. For instance, the error above says that arg0 is unbound. But it doesn’t tell us which arg0 is the problem. In this case, we’re using set! only with SHELL-ID, so we can deduce the culprit. But in general, it’s another reason to use unhy­gieinic iden­ti­fiers spar­ingly.

← prev next →