Thank you for your comment

Beau­tiful Racket / explainers

Syntax objects

A syntax object is a data struc­ture native to Racket that holds every­thing you’d want to know about a piece of source code. Syntax objects are used exten­sively within Racket’s macro system. For instance, every macro takes a syntax object as input and returns another syntax object as output.

A syntax object is not an “object” in the sense of object-oriented program­ming. Rather, it’s just a data object, similar to a hash table.

At minimum, a syntax object consists of:

  1. A datum that repre­sents the literal code as it would appear in a source file. A syntax object can be flat­tened into a datum with syntax->datum:

    (syntax->datum #'foo) ; 'foo
    (syntax->datum #'(+ 1 2 3)) ; '(+ 1 2 3)
    1
2
    (syntax->datum #'foo) ; 'foo
(syntax->datum #'(+ 1 2 3)) ; '(+ 1 2 3)
    copy to clipboard

  2. Meta­data about the code, most impor­tantly its lexical context and source loca­tion. Most of these meta­data fields have accessor func­tions:

    (define stx #'foo)
    (syntax-line stx) ; 2
    (syntax-column stx) ; 14
    (syntax-span stx) ; 3
    (syntax-srcloc stx) ; (srcloc 'unsaved-editor 2 14 24 3)
    1
2
3
4
5
    (define stx #'foo)
(syntax-line stx) ; 2
(syntax-column stx) ; 14
(syntax-span stx) ; 3
(syntax-srcloc stx) ; (srcloc 'unsaved-editor 2 14 24 3)
    copy to clipboard

Option­ally, a syntax object can also contain:

  1. Syntax prop­er­ties that are arbi­trary key–value pairs, which are written and read with syntax-property:

    (define stx #'foo)
    (define stx+prop (syntax-property stx 'hello "world"))
    (syntax? stx+prop) ; #t
    (syntax-property stx+prop 'hello) ; "world"
    1
2
3
4
    (define stx #'foo)
(define stx+prop (syntax-property stx 'hello "world"))
(syntax? stx+prop) ; #t
(syntax-property stx+prop 'hello) ; "world"
    copy to clipboard

  2. Other syntax objects, in place of any subdatum. These nested syntax objects retain their orig­inal syntax prop­er­ties and meta­data, including lexical context.

    For instance, this macro assigns its argu­ment to the pattern vari­able OUTER-OP, which then holds a refer­ence to a syntax object containing the argu­ment. The macro then uses OUTER-OP within a larger syntax object. This syntax object ends up with two vari­ables named +: one that’s defined inside of the macro, and the other (repre­sented by OUTER-OP) that’s defined outside. But they can coexist peace­fully because they retain sepa­rate lexical contexts:

    (define-macro (nonplussed OUTER-OP)
      #'(begin
          (define + -)
          (list (+ 21 21) (OUTER-OP 21 21))))
    (syntax->datum (expand-once #'(nonplussed +)))
    ; '(begin (define + -) (list (+ 21 21) (+ 21 21)))
    (nonplussed +) ; '(0 42)
    1
2
3
4
5
6
7
    (define-macro (nonplussed OUTER-OP)
  #'(begin
      (define + -)
      (list (+ 21 21) (OUTER-OP 21 21))))
(syntax->datum (expand-once #'(nonplussed +)))
; '(begin (define + -) (list (+ 21 21) (+ 21 21)))
(nonplussed +) ; '(0 42)
    copy to clipboard

    Be cautious of using syntax->datum indis­crim­i­nately, as it discards all the meta­data and prop­er­ties from any nested syntax objects. Usually, it’s wiser to preserve this infor­ma­tion.

Creating syntax objects

To create a syntax object in the current lexical context, simply wrap a datum with syntax, or equiv­a­lently, prefix the datum with #'. For instance, these syntax objects are the same (except for their source loca­tion):

(syntax (+ 1 2))
#'(+ 1 2)
1
2
(syntax (+ 1 2))
#'(+ 1 2)
copy to clipboard

The syntax prefix #' corre­sponds logi­cally to the ' prefix used to create a datum. Just as a syntax object is a datum with meta­data attached, the syntax prefix #' is the datum prefix with an extra char­acter attached.

'(+ 1 2) ; datum '(+ 1 2)
#'(+ 1 2) ; syntax object with datum '(+ 1 2) inside
(syntax->datum #'(+ 1 2)) ; '(+ 1 2)
1
2
3
'(+ 1 2) ; datum '(+ 1 2)
#'(+ 1 2) ; syntax object with datum '(+ 1 2) inside
(syntax->datum #'(+ 1 2)) ; '(+ 1 2)
copy to clipboard

To create a syntax object within a different lexical context, use datum->syntax. The first argu­ment is a syntax object that carries the target lexical context, and the second argu­ment is a datum. The result is a syntax object that will behave as if it were created in the other lexical context.

This is a common way of circum­venting macro hygiene. Below, make-z will create an iden­ti­fier z with the lexical context of OUTSIDE-ID, so z behaves as if it had been defined in the same lexical context as OUTSIDE-ID (= outside the macro):

(define-macro (make-z OUTSIDE-ID)
  (with-pattern ([Z-ID (datum->syntax #'OUTSIDE-ID 'z)])
    #'(define Z-ID 42)))
(make-z out-here)
z ; 42
1
2
3
4
5
(define-macro (make-z OUTSIDE-ID)
  (with-pattern ([Z-ID (datum->syntax #'OUTSIDE-ID 'z)])
    #'(define Z-ID 42)))
(make-z out-here)
z ; 42
copy to clipboard

This macro combines two syntax objects: the iden­ti­fier z returned by datum->syntax, which is then matched to the pattern vari­able Z-ID and used inside the larger syntax template #'(define Z-ID 42).

Without this maneuver, z is created in the macro’s lexical context (which is sepa­rate) and thus is not visible outside, due to the protec­tions of hygiene:

(define-macro (make-z OUTSIDE-ID)
  #'(define z 42))
(make-z out-here)
z ; error: unbound identifier
1
2
3
4
(define-macro (make-z OUTSIDE-ID)
  #'(define z 42))
(make-z out-here)
z ; error: unbound identifier
copy to clipboard

Inspecting syntax objects

Any syntax object printed to the DrRacket REPL will be displayed with an arrow on the left. Clicking the arrow will reveal a box with the syntax object datum on the left, and meta­data and prop­er­ties on the right. Clicking on code within the datum on the left will reveal infor­ma­tion about nested syntax objects. For instance, try running this in DrRacket:

(define-macro (nonplussed OUTER-+)
  #'(begin
      (define + -)
      (list (+ 21 21) (OUTER-+ 21 21))))
(expand-once #'(nonplussed +))
1
2
3
4
5
(define-macro (nonplussed OUTER-+)
  #'(begin
      (define + -)
      (list (+ 21 21) (OUTER-+ 21 21))))
(expand-once #'(nonplussed +))
copy to clipboard

expand-once will reveal the syntax object returned by the (nonplussed +) macro and print it on the REPL. Click on the arrow to reveal the details:

Click each of the two + signs in the third row of the datum and notice that they have different line numbers (because the second retains its source loca­tion from outside the nonplussed macro).

Editing syntax objects

The easiest way to decon­struct a syntax object is with a syntax pattern, which offers regexp-style matching to break a syntax object into pieces. Forms like define-macro and define-macro-cases use syntax patterns to match the input argu­ments. Forms like with-pattern can be used inside a macro to further disas­semble syntax objects and rearrange them:

(define-macro (rearrange XS)
  (with-pattern ([(1ST 2ND 3RD) #'XS])
    #'(list 3RD 2ND 1ST 3RD)))
(rearrange (10 20 30)) ; '(30 20 10 30)
1
2
3
4
(define-macro (rearrange XS)
  (with-pattern ([(1ST 2ND 3RD) #'XS])
    #'(list 3RD 2ND 1ST 3RD)))
(rearrange (10 20 30)) ; '(30 20 10 30)
copy to clipboard

By the way, when a pattern vari­able appears in a syntax pattern, like XS in (rearrange XS) or 1ST or 2ND or 3RD, it doesn’t have a #' prefix, because it doesn’t yet repre­sent a stand­alone syntax object. But when that pattern vari­able appears in a syntax template on the right side of a pattern clause or in the body, like #'XS, it is acting as a syntax object, so it needs the syntax prefix (and the XS is implic­itly replaced with the matched value).

For finer control, syntax objects can be disman­tled manu­ally, much like lists, though some extra house­keeping is neces­sary to use them with stan­dard list oper­a­tions. syntax->list will turn a single syntax object with a list-shaped datum into a list of syntax objects:

(syntax->list #'(10 20 30))
; '(#<syntax:2:17 10> #<syntax:2:20 20> #<syntax:2:23 30>)
1
2
(syntax->list #'(10 20 30))
; '(#<syntax:2:17 10> #<syntax:2:20 20> #<syntax:2:23 30>)
copy to clipboard

These smaller syntax objects can then be manip­u­lated with ordi­nary list oper­a­tions to build new syntax objects. Below, we use reverse and list to reverse our input:

(define-macro (rev ARGS)
  (define arg-stxs (syntax->list #'ARGS))
  (with-pattern ([(REVERSED-ARG ...) (reverse arg-stxs)])
    #'(list REVERSED-ARG ...)))
(rev (10 20 30)) ; '(30 20 10)
1
2
3
4
5
(define-macro (rev ARGS)
  (define arg-stxs (syntax->list #'ARGS))
  (with-pattern ([(REVERSED-ARG ...) (reverse arg-stxs)])
    #'(list REVERSED-ARG ...)))
(rev (10 20 30)) ; '(30 20 10)
copy to clipboard

Just as the syntax prefix #' is the syntaxed version of the basic ' prefix, the quasiquote oper­a­tors (` , and ,@) also have quasisyntax equiv­a­lents (#` #, and #,@) that can build syntax objects in an anal­o­gous way:  + See Lists for an intro­duc­tion to quasiquote.

(define num 1)
(define nums '(2 3))
`(+ ,num ,@nums) ; '(+ 1 2 3)
#`(+ #,num #,@nums) ; #<syntax:8:2 (+ 1 2 3)>
1
2
3
4
(define num 1)
(define nums '(2 3))
`(+ ,num ,@nums) ; '(+ 1 2 3)
#`(+ #,num #,@nums) ; #<syntax:8:2 (+ 1 2 3)>
copy to clipboard

Syntax prop­er­ties

Syntax prop­er­ties are used to attach extra meta­data fields to a syntax object. For instance, the read-syntax func­tion in #lang br attaches a 'paren-shape prop­erty to indi­cate whether the source code used square or curly brackets to delimit a list:

(define stx1 (read-syntax #f (open-input-string "(1 2 3)")))
(syntax-property stx1 'paren-shape) ; #f
(define stx2 (read-syntax #f (open-input-string "[1 2 3]")))
(syntax-property stx2 'paren-shape) ; #\[
(define stx3 (read-syntax #f (open-input-string "{1 2 3}")))
(syntax-property stx3 'paren-shape) ; #\{
1
2
3
4
5
6
(define stx1 (read-syntax #f (open-input-string "(1 2 3)")))
(syntax-property stx1 'paren-shape) ; #f
(define stx2 (read-syntax #f (open-input-string "[1 2 3]")))
(syntax-property stx2 'paren-shape) ; #\[
(define stx3 (read-syntax #f (open-input-string "{1 2 3}")))
(syntax-property stx3 'paren-shape) ; #\{
copy to clipboard

Why? Because paren­theses, brackets, and braces are syntac­ti­cally equiv­a­lent. With a syntax prop­erty, we can preserve the extra infor­ma­tion without affecting the meaning of the syntax object. In this case, if we wanted to convert the syntax back to a string, we could still recover the orig­inal shapes:

(require syntax/to-string)
(define stx1 (read-syntax #f (open-input-string "((1))")))
(syntax->string stx1) ; "(1)"
(define stx2 (read-syntax #f (open-input-string "([2])")))
(syntax->string stx2) ; "[2]"
(define stx3 (read-syntax #f (open-input-string "({3})")))
(syntax->string stx3) ; "{3}"
1
2
3
4
5
6
7
(require syntax/to-string)
(define stx1 (read-syntax #f (open-input-string "((1))")))
(syntax->string stx1) ; "(1)"
(define stx2 (read-syntax #f (open-input-string "([2])")))
(syntax->string stx2) ; "[2]"
(define stx3 (read-syntax #f (open-input-string "({3})")))
(syntax->string stx3) ; "{3}"
copy to clipboard

More broadly, syntax prop­er­ties are useful because macros have a limited inter­face: they take one syntax object and return another syntax object. Syntax prop­er­ties can be used to stash other infor­ma­tion so that macros can commu­ni­cate with each other without affecting the input and output. In this example, a base macro echo is used by the twice and thrice macros with a 'count argu­ment stashed as a syntax prop­erty, which echo uses to change its result:

(require (for-syntax racket/list))
(define-macro (echo ARG)
  (define how-many (or (syntax-property #'ARG 'count) 1))
  (with-pattern ([ARGS (make-list how-many #'ARG)]
                 [ARGLIST (cons #'list #'ARGS)])
    #'ARGLIST))
(echo "foo") ; '("foo")

(define-macro (twice ARG)
  (with-pattern ([ARGPLUS (syntax-property #'ARG 'count 2)])
    #'(echo ARGPLUS)))
(twice "bar") ; '("bar" "bar")

(define-macro (thrice ARG)
  ; quasisyntax notation, equivalent to above
  #`(echo #,(syntax-property #'ARG 'count 3)))
(thrice "zam") ; '("zam" "zam" "zam")
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
(require (for-syntax racket/list))
(define-macro (echo ARG)
  (define how-many (or (syntax-property #'ARG 'count) 1))
  (with-pattern ([ARGS (make-list how-many #'ARG)]
                 [ARGLIST (cons #'list #'ARGS)])
    #'ARGLIST))
(echo "foo") ; '("foo")

(define-macro (twice ARG)
  (with-pattern ([ARGPLUS (syntax-property #'ARG 'count 2)])
    #'(echo ARGPLUS)))
(twice "bar") ; '("bar" "bar")

(define-macro (thrice ARG)
  ; quasisyntax notation, equivalent to above
  #`(echo #,(syntax-property #'ARG 'count 3)))
(thrice "zam") ; '("zam" "zam" "zam")
copy to clipboard

Why not just use strings?

Syntax objects are the foun­da­tion of hygiene in Racket, which is what makes its sophis­ti­cated macro system possible. Hygiene is the idea that a code frag­ment floats within a bubble of bind­ings—provided by its lexical context—thereby permit­ting clean, predictable inter­ac­tions between those frag­ments. The syntax object creates that bubble by asso­ci­ating a lexical context with a code frag­ment. Strings, by contrast, can capture the code frag­ment, but not the lexical context.

“If syntax objects are so great, why don’t other languages have them?” Those languages don’t have hygienic macros, so there’s no need. Instead, many languages have an eval or exec func­tion that will treat a string as source code. For instance Python:

>>> eval("6 + 7 * len('this string')")
83
1
2
>>> eval("6 + 7 * len('this string')")
83
copy to clipboard

You can perform an anal­o­gous oper­a­tion on a Racket string with eval, using make-base-namespace to set up the default bind­ings:

(define str
  "(+ 6 (* 7 (string-length \"this string\")))")
(eval (format-datum '~a str)
      (make-base-namespace)) ; 83
1
2
3
4
(define str
  "(+ 6 (* 7 (string-length \"this string\")))")
(eval (format-datum '~a str)
      (make-base-namespace)) ; 83
copy to clipboard

But a syntax object will always be more idiomatic and compact:

(define stx #'(+ 6 (* 7 (string-length "this string"))))
(eval stx) ; 83
1
2
(define stx #'(+ 6 (* 7 (string-length "this string"))))
(eval stx) ; 83
copy to clipboard

Further

← prev next →