Thank you for your comment

Beau­tiful Racket / racket school 2019

Day 2

Powerful points

  1. The reader converts the source code into S-expres­sions. It runs before the expander. Every reader must provide a func­tion named read-syntax.

  2. read-syntax must accept two input argu­ments (= source path & input port). It must return a module expres­sion as a syntax object without bind­ings. This module expres­sion must include a refer­ence to the expander that will provide the initial set of bind­ings when the module code is eval­u­ated.

  3. Often, a reader uses a tokenizer to convert the source code into tokens. A token is a contiguous sequence of char­ac­ters that affects the meaning of the program.

    Ques­tion: how many tokens are in this trigram program?

    12 & AB
    1
    12 & AB
    copy to clipboard

    How about this math expres­sion?

    12+97
    1
    12+97
    copy to clipboard

    How about this Racket program?

    ;; hello world!
    (* 62 719)
    1
2
    ;; hello world!
(* 62 719)
    copy to clipboard

    How about this Python program? (Count care­fully.)

    if True:
        print "hooray"
    1
2
    if True:
    print "hooray"
    copy to clipboard

  4. Often, a reader uses a parser to arrange these tokens into a hier­ar­chical struc­ture, called a parse tree.

    Python if condi­tional:

    if y > 0:
        x / y
    else:
        print("nope")
    1
2
3
4
    if y > 0:
    x / y
else:
    print("nope")
    copy to clipboard

    As a chart:

    As an S-expres­sion:

    '(if (> y 0)
         :
         (/ x y)
         else:
         (print "nope"))
    1
2
3
4
5
    '(if (> y 0)
     :
     (/ x y)
     else:
     (print "nope"))
    copy to clipboard

    Racket if condi­tional:

    (if (> y 0)
      (/ x y)
      (print "nope"))
    1
2
3
    (if (> y 0)
  (/ x y)
  (print "nope"))
    copy to clipboard

  5. The parser consumes a sequence of tokens, and the resulting parse tree becomes the program embedded in the module expres­sion. Pseudocodishly:

    (define (read-syntax src ip)
      (define toks (tokenize-all ip))
      (define parse-tree (parse src toks))
      (strip-bindings
       (with-syntax ([PT parse-tree])
         #'(module mod-name expander-mod
             PT))))
    1
2
3
4
5
6
7
    (define (read-syntax src ip)
  (define toks (tokenize-all ip))
  (define parse-tree (parse src toks))
  (strip-bindings
   (with-syntax ([PT parse-tree])
     #'(module mod-name expander-mod
         PT))))
    copy to clipboard

  6. If you use a parser gener­ator like brag to make your parse func­tion (and we will, later today) it can also accept a zero-arity func­tion (aka thunk) that produces one token (which will be called repeat­edly, until eof). Pseudocodishly:

    (define (read-syntax src ip)
      (define token-thunk (λ () (tokenize-one ip)))
      (define parse-tree (parse src token-thunk))
      (strip-bindings
       (with-syntax ([PT parse-tree])
         #'(module mod-name expander-mod
             PT))))
    1
2
3
4
5
6
7
    (define (read-syntax src ip)
  (define token-thunk (λ () (tokenize-one ip)))
  (define parse-tree (parse src token-thunk))
  (strip-bindings
   (with-syntax ([PT parse-tree])
     #'(module mod-name expander-mod
         PT))))
    copy to clipboard

    Ques­tion: In what situ­a­tions would the second approach poten­tially be wiser?

  7. module+ is a variant of module that lets us create a submodule that auto­mat­i­cally includes the surrounding bind­ings, and possibly adds new ones.

    (module reader br
      (provide read-syntax)
      (define (read-syntax src ip)
        ···))
    1
2
3
4
    (module reader br
  (provide read-syntax)
  (define (read-syntax src ip)
    ···))
    copy to clipboard
    (module+ reader
      (provide read-syntax))

    (define (read-syntax src ip)
      ···)
    1
2
3
4
5
    (module+ reader
  (provide read-syntax))

(define (read-syntax src ip)
  ···)
    copy to clipboard

    Ques­tion: How might this be useful?

Mission

Make a language called taco-compiler that takes any #lang racket program made of ASCII char­ac­ters as input and prints it—the program, not its result—in nota­tion that only uses the iden­ti­fier taco and paren­theses. If you need math oper­a­tions, look them up in the Racket docs.

Easier option: imple­ment the sample nota­tion shown below.

Harder option: devise your own.

Example:

taco-compiler/test.rkt
#lang taco-compiler

"hello world"
(+ 1 (* 2 (- x)))
1
2
3
4
#lang taco-compiler

"hello world"
(+ 1 (* 2 (- x)))
copy to clipboard

One possible result:

(() taco () taco () () ())
(() taco () taco () () ())
(() taco () () () taco ())
(() () () taco () taco taco)
(taco () taco () () taco taco)
(() () taco taco () taco taco)
(() () taco taco () taco taco)
(taco taco taco taco () taco taco)
(() () () () () taco ())
(taco taco taco () taco taco taco)
(taco taco taco taco () taco taco)
(() taco () () taco taco taco)
(() () taco taco () taco taco)
(() () taco () () taco taco)
(() taco () () () taco ())
(() taco () taco () () ())
(() () () taco () taco ())
(taco taco () taco () taco ())
(() () () () () taco ())
(taco () () () taco taco ())
(() () () () () taco ())
(() () () taco () taco ())
(() taco () taco () taco ())
(() () () () () taco ())
(() taco () () taco taco ())
(() () () () () taco ())
(() () () taco () taco ())
(taco () taco taco () taco ())
(() () () () () taco ())
(() () () taco taco taco taco)
(taco () () taco () taco ())
(taco () () taco () taco ())
(taco () () taco () taco ())
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
(() taco () taco () () ())
(() taco () taco () () ())
(() taco () () () taco ())
(() () () taco () taco taco)
(taco () taco () () taco taco)
(() () taco taco () taco taco)
(() () taco taco () taco taco)
(taco taco taco taco () taco taco)
(() () () () () taco ())
(taco taco taco () taco taco taco)
(taco taco taco taco () taco taco)
(() taco () () taco taco taco)
(() () taco taco () taco taco)
(() () taco () () taco taco)
(() taco () () () taco ())
(() taco () taco () () ())
(() () () taco () taco ())
(taco taco () taco () taco ())
(() () () () () taco ())
(taco () () () taco taco ())
(() () () () () taco ())
(() () () taco () taco ())
(() taco () taco () taco ())
(() () () () () taco ())
(() taco () () taco taco ())
(() () () () () taco ())
(() () () taco () taco ())
(taco () taco taco () taco ())
(() () () () () taco ())
(() () () taco taco taco taco)
(taco () () taco () taco ())
(taco () () taco () taco ())
(taco () () taco () taco ())
copy to clipboard

Hint: A char is a Racket datatype that repre­sents one Unicode char­acter.

Hint: read-char takes an input port as an argu­ment and returns the next char­acter in the port. So tokenize, in this case, returns a list of char­ac­ters. "hello" would become (list #\h #\e #\l #\l #\o).

Hint: You can use char->integer and integer->char to convert to and from an integer value.

Hint: The sample nota­tion relies on a certain fact about the numer­ical repre­sen­ta­tion of ASCII char­ac­ters.

Hint: Here’s some code to get you started:

taco-compiler/main.rkt
#lang br/quicklang

(module+ reader
  (provide read-syntax))

(define (tokenize ip)
  (for/list ([tok (in-port read-char ip)])
    tok))

(define (parse src toks)
  ;; ···
  )

(define (read-syntax src ip)
  (define toks (tokenize ip))
  (define parse-tree (parse src toks))
  (strip-bindings
   (with-syntax ([PT parse-tree])
     #'(module tacofied taco-compiler
         PT))))

(define-macro (mb PT)
  #'(#%module-begin
     (for-each displayln 'PT)))
(provide (rename-out [mb #%module-begin]))
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#lang br/quicklang

(module+ reader
  (provide read-syntax))

(define (tokenize ip)
  (for/list ([tok (in-port read-char ip)])
    tok))

(define (parse src toks)
  ;; ···
  )

(define (read-syntax src ip)
  (define toks (tokenize ip))
  (define parse-tree (parse src toks))
  (strip-bindings
   (with-syntax ([PT parse-tree])
     #'(module tacofied taco-compiler
         PT))))

(define-macro (mb PT)
  #'(#%module-begin
     (for-each displayln 'PT)))
(provide (rename-out [mb #%module-begin]))
copy to clipboard

Summary

It’s tacos all the way down.

← prev next →