Beau­tiful Racket / appendix

1. 

   Within your path/to/source, create a project direc­tory for the language called dsl. Switch into the new direc­tory and use raco pkg install to add it to your local Racket instal­la­tion as a package.

   cd path/to/source
   mkdir dsl
   cd dsl
   raco pkg install

   1 2 3 4

   > cd path/to/source > mkdir dsl > cd dsl > raco pkg install

   
2. 

   Alter­na­tively: at the command line, switch to your path/to/source and use raco pkg new dsl, which will create the dsl subdi­rec­tory and stub out other files that will be useful in the package (including docu­men­ta­tion and config­u­ra­tion).

   cd path/to/source
   raco pkg new dsl
   cd dsl
   raco pkg install

   1 2 3 4

   > cd path/to/source > raco pkg new dsl > cd dsl > raco pkg install

   

• 

  If your language is invoked as #lang dsl, its boot module will be "dsl/main.rkt".

• 

  If your language is invoked as #lang dsl/dialect, its boot module will be "dsl/dialect.rkt". (Caution: the boot module will not be "dsl/dialect/main.rkt".)

• 

  A language can have any number of dialects living in the same project direc­tory, with the boot modules following this naming conven­tion.

• 

  As the first step in running any program, Racket invokes the reader for the language. The reader converts source code (= a string of char­ac­ters in a source file) into S-expres­sions (= Racket-style syntactic forms).

• 

  To start the reader, Racket calls the read-syntax func­tion for the language. Racket looks for this func­tion in the reader submodule of the boot module of the #lang, which must provide it. For instance:

  dsl/main.rkt

  #lang br
  (module reader br
    (provide read-syntax)
    ···)

  1 2 3 4

  #lang br (module reader br (provide read-syntax) ···)

  
  

  read-syntax can either be defined within the reader submodule:

  dsl/main.rkt

  #lang br
  (module reader br
    (provide read-syntax)
    (define (read-syntax name port)
      ···))

  1 2 3 4 5

  #lang br (module reader br (provide read-syntax) (define (read-syntax name port) ···))

  
  

  Or imported from else­where:

  dsl/main.rkt

  #lang br
  (module reader br
    (provide read-syntax)
    (require module/that/exports/read-syntax))

  1 2 3 4

  #lang br (module reader br (provide read-syntax) (require module/that/exports/read-syntax))

  
• 

  read-syntax must accept two input argu­ments: a source name (that holds the loca­tion of the source—e.g., for file input, the name would be a path) and an input port (that points at the source file). read-syntax should read its data from the port.  + In prin­ciple, it’s possible to read directly from a source name when it’s a path. But this is unre­li­able, because it assumes the input source is a file. Maybe it’s not. By contrast, the port argu­ment always contains the source data, regard­less of the under­lying input type. read-syntax should consume all the data from the port (that is, until the port returns eof). In pseudocode:

  dsl/main.rkt

  #lang br
  (module reader br
    (provide read-syntax)
    (define (read-syntax name port)
      (define s-exprs (read-code-from port))
      ···))

  1 2 3 4 5 6

  #lang br (module reader br (provide read-syntax) (define (read-syntax name port) (define s-exprs (read-code-from port)) ···))

  
• 

  read-syntax must return one value: code for a module expres­sion, repre­sented as a syntax object. Typi­cally, the converted S-expres­sions from the source file are inserted into this syntax object. This syntax object must have no iden­ti­fier bind­ings. This module code must include a refer­ence to the expander that will provide the initial set of bind­ings when the module code is eval­u­ated. In pseudocode:

  dsl/main.rkt

  #lang br
  (module reader br
    (provide read-syntax)
    (define (read-syntax name port)
      (define s-exprs (read-code-from port))
      (strip-bindings
       #`(module dsl-mod-name dsl/expander
           #,@s-exprs))))

  1 2 3 4 5 6 7 8

  #lang br (module reader br (provide read-syntax) (define (read-syntax name port) (define s-exprs (read-code-from port)) (strip-bindings #`(module dsl-mod-name dsl/expander #,@s-exprs))))

  
• 

  Racket takes the module expres­sion returned from read-syntax and uses it to replace the code in the source file. So a source file that looks like this:

  source-file.rkt

  #lang dsl
  dsl source code;
  ···

  1 2 3

  #lang dsl dsl source code; ···

  
  

  Becomes this:

  source-file.rkt

  (module dsl-mod-name dsl/expander
    dsl-sexprs ···)

  1 2	(module dsl-mod-name dsl/expander dsl-sexprs ···)

  
• 

  Eval­u­a­tion of this module expres­sion continues from here, starting by importing bind­ings from the expander.

• 

  The tokenizer reads char­ac­ters from the input port and converts them to tokens, which are the smallest mean­ingful units of source code.

  

  You can write a tokenizer by hand. You can also use a helper func­tion called a lexer that uses regexp-style rules to convert source code into tokens.

• 

  These tokens are passed to the parser, which arranges them into a hier­ar­chical S-expres­sion called a parse tree.

  

  You can write a parser by hand. You can also use a grammar-based parser gener­ator, like brag, to make a parser func­tion.

dsl/main.rkt

#lang br
(module reader br
  (require module/that/provides/parse
           module/that/provides/tokenize)
  (provide read-syntax)
  (define (read-syntax name port)
    (define the-parse-tree (parse (tokenize port)))
    (strip-bindings
     #`(module dsl-mod-name dsl/expander
         #,the-parse-tree))))

1 2 3 4 5 6 7 8 9 10

#lang br (module reader br (require module/that/provides/parse module/that/provides/tokenize) (provide read-syntax) (define (read-syntax name port) (define the-parse-tree (parse (tokenize port))) (strip-bindings #`(module dsl-mod-name dsl/expander #,the-parse-tree))))

• 

  Once the reader finishes, the expander starts. At the end of the reader phase, the source code has been converted into a module expres­sion that contains S-expres­sions, but has no bind­ings, e.g.—

  source-file.rkt

  (module dsl-mod-name dsl/expander
    dsl-sexprs ···)

  1 2	(module dsl-mod-name dsl/expander dsl-sexprs ···)

  
• 

  The expander provides the initial set of bind­ings for the module expres­sion returned by the reader, thereby deter­mining the meaning of the iden­ti­fiers within the S-expres­sions. This, in turn, allows the expres­sions to be eval­u­ated as Racket code.

• 

  Any module (that meets the other require­ments below) can be used as the expander for a language. It’s invoked by the first line of the module code returned by the reader. For instance, this module expres­sion will rely on dsl/expander as its expander:

  source-file.rkt

  (module dsl-mod-name dsl/expander
    dsl-sexprs ···)

  1 2	(module dsl-mod-name dsl/expander dsl-sexprs ···)

  
  

  In essence, the expander name is like an implied require at the begin­ning of the module:

  (module dsl-mod-name dsl/expander
    (require dsl/expander)
    dsl-sexprs ···)

  1 2 3

  (module dsl-mod-name dsl/expander (require dsl/expander) dsl-sexprs ···)

  
  

  Most Racket languages have their own custom expander module. But it’s not manda­tory.

• 

  Every expander must provide a #%module-begin macro, which will be the first thing invoked in the expander. #%module-begin must accept as input all the expres­sions that appear in the body of the module expres­sion that read-syntax makes. There­fore, it’s not a bad idea to build your #%module-begin around a syntax pattern that accepts any number of input argu­ments.

• 

  To eval­uate the module expres­sion returned by the reader, Racket imports the #%module-begin from the expander spec­i­fied in the module expres­sion. It replaces the module expres­sion with a call to this #%module-begin, passing it all the parsed expres­sions that are in the body of the module expres­sion. So this:

  source-file.rkt

  (module dsl-mod-name dsl/expander
    dsl-sexprs ···)

  1 2	(module dsl-mod-name dsl/expander dsl-sexprs ···)

  
  

  Becomes this:

  source-file.rkt

  (module dsl-mod-name dsl/expander
    (#%module-begin ;; imported from `dsl/expander`
      dsl-sexprs ···))

  1 2 3

  (module dsl-mod-name dsl/expander (#%module-begin ;; imported from `dsl/expander` dsl-sexprs ···))

  
• 

  Often, the #%module-begin for a language will perform some language-specific processing on the parse tree, and call the #%module-begin in the imple­men­ta­tion language. To prevent name­space colli­sions between the two #%module-begin macros, use rename-out. In pseudocode:

  dsl/expander.rkt

  #lang br
  (define-macro (dsl-module-begin PARSED-EXPR ...)
    #'(#%module-begin ;; from `br`
       PARSED-EXPR ...))
  (provide (rename-out [dsl-module-begin #%module-begin]))

  1 2 3 4 5

  #lang br (define-macro (dsl-module-begin PARSED-EXPR ...) #'(#%module-begin ;; from `br` PARSED-EXPR ...)) (provide (rename-out [dsl-module-begin #%module-begin]))

  
• 

  Option­ally, an expander can provide certain inter­po­si­tion points:

  • 

    #%top-interaction is used to acti­vate the REPL.

  • 

    #%app adds support for func­tion calls.

  • 

    #%datum adds support for self-eval­u­ating values (like numbers and strings).

  • 

    #%top adds support for missing iden­ti­fiers.

• 

  Writing unit tests.

• 

  Inte­grating the language with DrRacket, including syntax coloring and indenting.

• 

  Adding Scribble docu­men­ta­tion.

• 

  Adding an "info.rkt" file.

• 

  Making the language avail­able through the Racket package server.