Thank you for your comment

Beau­tiful Racket / racket school 2019

Day 2

Setup

Take your taco-compiler result and run it under #lang tacopocalypse-prep:

#lang tacopocalypse-prep
(() taco () taco () () ())
(() taco () taco () () ())
(() taco () () () taco ())
(() () () taco () taco taco)
(taco () taco () () taco taco)
(() () taco taco () taco taco)
(() () taco taco () taco taco)
(taco taco taco taco () taco taco)
(() () () () () taco ())
(taco taco taco () taco taco taco)
(taco taco taco taco () taco taco)
(() taco () () taco taco taco)
(() () taco taco () taco taco)
(() () taco () () taco taco)
(() taco () () () taco ())
(() taco () taco () () ())
(() () () taco () taco ())
(taco taco () taco () taco ())
(() () () () () taco ())
(taco () () () taco taco ())
(() () () () () taco ())
(() () () taco () taco ())
(() taco () taco () taco ())
(() () () () () taco ())
(() taco () () taco taco ())
(() () () () () taco ())
(() () () taco () taco ())
(taco () taco taco () taco ())
(() () () () () taco ())
(() () () taco taco taco taco)
(taco () () taco () taco ())
(taco () () taco () taco ())
(taco () () taco () taco ())
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#lang tacopocalypse-prep
(() taco () taco () () ())
(() taco () taco () () ())
(() taco () () () taco ())
(() () () taco () taco taco)
(taco () taco () () taco taco)
(() () taco taco () taco taco)
(() () taco taco () taco taco)
(taco taco taco taco () taco taco)
(() () () () () taco ())
(taco taco taco () taco taco taco)
(taco taco taco taco () taco taco)
(() taco () () taco taco taco)
(() () taco taco () taco taco)
(() () taco () () taco taco)
(() taco () () () taco ())
(() taco () taco () () ())
(() () () taco () taco ())
(taco taco () taco () taco ())
(() () () () () taco ())
(taco () () () taco taco ())
(() () () () () taco ())
(() () () taco () taco ())
(() taco () taco () taco ())
(() () () () () taco ())
(() taco () () taco taco ())
(() () () () () taco ())
(() () () taco () taco ())
(taco () taco taco () taco ())
(() () () () () taco ())
(() () () taco taco taco taco)
(taco () () taco () taco ())
(taco () () taco () taco ())
(taco () () taco () taco ())
copy to clipboard

You’ll get some­thing like:

##$%#$%#$#$#$$##$%#$%#$#$#$$##$%#$#$#$%#$$##$#$#$%#$%%$#%#$%#$#$%%$##$#$%%#$%%$##$#$%%#$%%$#%%%%#$%%$##$#$#$#$#$%#$$#%%%#$%%%$#%%%%#$%%$##$%#$#$%%%$##$#$%%#$%%$##$#$%#$#$%%$##$%#$#$#$%#$$##$%#$%#$#$#$$##$#$#$%#$%#$$#%%#$%#$%#$$##$#$#$#$#$%#$$#%#$#$#$%%#$$##$#$#$#$#$%#$$##$#$#$%#$%#$$##$%#$%#$%#$$##$#$#$#$#$%#$$##$%#$#$%%#$$##$#$#$#$#$%#$$##$#$#$%#$%#$$#%#$%%#$%#$$##$#$#$#$#$%#$$##$#$#$%%%%$#%#$#$%#$%#$$#%#$#$%#$%#$$#%#$#$%#$%#$$
1
##$%#$%#$#$#$$##$%#$%#$#$#$$##$%#$#$#$%#$$##$#$#$%#$%%$#%#$%#$#$%%$##$#$%%#$%%$##$#$%%#$%%$#%%%%#$%%$##$#$#$#$#$%#$$#%%%#$%%%$#%%%%#$%%$##$%#$#$%%%$##$#$%%#$%%$##$#$%#$#$%%$##$%#$#$#$%#$$##$%#$%#$#$#$$##$#$#$%#$%#$$#%%#$%#$%#$$##$#$#$#$#$%#$$#%#$#$#$%%#$$##$#$#$#$#$%#$$##$#$#$%#$%#$$##$%#$%#$%#$$##$#$#$#$#$%#$$##$%#$#$%%#$$##$#$#$#$#$%#$$##$#$#$%#$%#$$#%#$%%#$%#$$##$#$#$#$#$%#$$##$#$#$%%%%$#%#$#$%#$%#$$#%#$#$%#$%#$$#%#$#$%#$%#$$
copy to clipboard

Hooray!

Powerful points

  1. A lexer is the power tool for tokenizing. It contains a series of clauses with a regex-like pattern on the left. It matches the longest pattern (not the first pattern). Then it returns the value on the right.

  2. lexer creates a func­tion that takes one input port as its argu­ment. Thus, it can be used in the same posi­tions as read or read-char.

  3. Special iden­ti­fiers avail­able within lexer rules: lexeme (matched item), input-port (current port being consumed).

  4. Ques­tion: without running the code, what will be the result of each tokeniza­tion?

    (require brag/support) ; imports `lexer`

    (define (apply-port-proc proc str)
      (define ip (open-input-string str))
      (for/list ([tok (in-port proc ip)])
        tok))

    (define lex (lexer
                 ["fo" lexeme]
                 [(:: "f" (:+ "o")) 42]
                 [any-char (lex input-port)]))

    (apply-port-proc read "foobar")
    (apply-port-proc read-char "foobar")
    (apply-port-proc lex "foobar")
    (apply-port-proc lex "fobar")
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
    (require brag/support) ; imports `lexer`

(define (apply-port-proc proc str)
  (define ip (open-input-string str))
  (for/list ([tok (in-port proc ip)])
    tok))

(define lex (lexer
             ["fo" lexeme]
             [(:: "f" (:+ "o")) 42]
             [any-char (lex input-port)]))

(apply-port-proc read "foobar")
(apply-port-proc read-char "foobar")
(apply-port-proc lex "foobar")
(apply-port-proc lex "fobar")
    copy to clipboard

  5. Ques­tion (hard): why not just tokenize using cond and regular expres­sions?

  6. Lexers are also used for source-code coloring in DrRacket.

Mission

Make a language called tacopocalypse that takes the output you gener­ated above and converts it back into the orig­inal Racket source code.

Example:

tacopocalypse/test.rkt
#lang tacopocalypse
##$%#$%#$#$#$$##$%#$%#$#$#$$##$%#$#$#$%#$$##$#$#$%#$%%$#%#$%#$#$%%$##$#$%%#$%%$##$#$%%#$%%$#%%%%#$%%$##$#$#$#$#$%#$$#%%%#$%%%$#%%%%#$%%$##$%#$#$%%%$##$#$%%#$%%$##$#$%#$#$%%$##$%#$#$#$%#$$##$%#$%#$#$#$$##$#$#$%#$%#$$#%%#$%#$%#$$##$#$#$#$#$%#$$#%#$#$#$%%#$$##$#$#$#$#$%#$$##$#$#$%#$%#$$##$%#$%#$%#$$##$#$#$#$#$%#$$##$%#$#$%%#$$##$#$#$#$#$%#$$##$#$#$%#$%#$$#%#$%%#$%#$$##$#$#$#$#$%#$$##$#$#$%%%%$#%#$#$%#$%#$$#%#$#$%#$%#$$#%#$#$%#$%#$$
1
2
#lang tacopocalypse
##$%#$%#$#$#$$##$%#$%#$#$#$$##$%#$#$#$%#$$##$#$#$%#$%%$#%#$%#$#$%%$##$#$%%#$%%$##$#$%%#$%%$#%%%%#$%%$##$#$#$#$#$%#$$#%%%#$%%%$#%%%%#$%%$##$%#$#$%%%$##$#$%%#$%%$##$#$%#$#$%%$##$%#$#$#$%#$$##$%#$%#$#$#$$##$#$#$%#$%#$$#%%#$%#$%#$$##$#$#$#$#$%#$$#%#$#$#$%%#$$##$#$#$#$#$%#$$##$#$#$%#$%#$$##$%#$%#$%#$$##$#$#$#$#$%#$$##$%#$#$%%#$$##$#$#$#$#$%#$$##$#$#$%#$%#$$#%#$%%#$%#$$##$#$#$#$#$%#$$##$#$#$%%%%$#%#$#$%#$%#$$#%#$#$%#$%#$$#%#$#$%#$%#$$
copy to clipboard

Result (as before, blank newlines are delib­erate):

 
 
"hello world"
(+ 1 (* 2 (- x)))
1
2
3
4
 
 
"hello world"
(+ 1 (* 2 (- x)))
copy to clipboard

Hint: Here’s some starter code:

tacopocalypse/main.rkt
#lang br/quicklang
(require brag/support) ; imports `lexer`

(module+ reader
  (provide read-syntax))

(define lex
  (lexer
   ;; ···
   ))

(define (tokenize ip)
  (for/list ([tok (in-port lex ip)])
    tok))

(define (parse src toks)
  ;; ···
  )

(define (read-syntax src ip)
  (define toks (tokenize ip))
  (define parse-tree (parse src toks))
  (strip-bindings
   (with-syntax ([PT parse-tree])
     #'(module taco-mod tacopocalypse
         PT))))

(define-macro (my-module-begin PT)
  #'(#%module-begin
     (display (list->string 'PT))))
(provide (rename-out [my-module-begin #%module-begin]))
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#lang br/quicklang
(require brag/support) ; imports `lexer`

(module+ reader
  (provide read-syntax))

(define lex
  (lexer
   ;; ···
   ))

(define (tokenize ip)
  (for/list ([tok (in-port lex ip)])
    tok))

(define (parse src toks)
  ;; ···
  )

(define (read-syntax src ip)
  (define toks (tokenize ip))
  (define parse-tree (parse src toks))
  (strip-bindings
   (with-syntax ([PT parse-tree])
     #'(module taco-mod tacopocalypse
         PT))))

(define-macro (my-module-begin PT)
  #'(#%module-begin
     (display (list->string 'PT))))
(provide (rename-out [my-module-begin #%module-begin]))
copy to clipboard

Hint: you need at least three lexing rules. You don’t neces­sarily have to lex every char­acter. Perhaps some are redun­dant?

Hint: in parse, you might find it useful to deal with sublists of tokens. The func­tions take and drop could be helpful.

Hint: no, I didn’t tell you how the nota­tion works. There’s already a DSL mentioned on this page that you can use to figure it out.

Summary

Don’t panic.

← prev next →