Thank you for your comment

Beau­tiful Racket / tuto­rials

Go with the flow: basic

We already did most of the heavy lifting in the lexer, so our tokenizer module can be simpler than before. We can use the jsonic tokenizer as a template. But instead of having the lexer embedded inside, we import basic-lexer from our "basic/lexer.rkt" module:

basic/tokenizer.rkt
#lang br
(require "lexer.rkt" brag/support)

(define (make-tokenizer ip)
  (port-count-lines! ip)
  (define (next-token) (basic-lexer ip))
  next-token)

(provide make-tokenizer)
1
2
3
4
5
6
7
8
9
#lang br
(require "lexer.rkt" brag/support)

(define (make-tokenizer ip)
  (port-count-lines! ip)
  (define (next-token) (basic-lexer ip))
  next-token)

(provide make-tokenizer)
copy to clipboard

But since we’re now relying on lexer-srcloc to gather source loca­tions for us, we make one update. One of the fields tracked in a source-loca­tion struc­ture is the path to the file containing the source code. But our basic-lexer only gets an input port as an argu­ment, not a path. Without that extra path argu­ment, its source loca­tions will be incom­plete.

We can fix this in two steps:

basic/tokenizer.rkt
#lang br
(require "lexer.rkt" brag/support)

(define (make-tokenizer ip [path #f])
  (port-count-lines! ip)
  (lexer-file-path path)
  (define (next-token) (basic-lexer ip))
  next-token)

(provide make-tokenizer)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#lang br
(require "lexer.rkt" brag/support)

(define (make-tokenizer ip [path #f])
  (port-count-lines! ip)
  (lexer-file-path path)
  (define (next-token) (basic-lexer ip))
  next-token)

(provide make-tokenizer)
copy to clipboard

  1. We modify make-tokenizer to take an optional path argu­ment. Later, when we invoke the tokenizer from the reader, we pass it both an input port and a path. For testing purposes, we might not want to always use a path, so we have a default argu­ment of #f for the path.

  2. We set the lexer-file-path para­meter to this value. A para­meter is a special kind of Racket value that approx­i­mates a global vari­able. Unlike a global vari­able, which is a simple value, a para­meter is a func­tion that is used to store and retrieve a value. So what­ever argu­ment we pass to lexer-file-path “sets” it, and it can then be retrieved by other func­tions that rely on it, including lexer-srcloc.  + True, para­me­ters go some­what against the grain of func­tional program­ming. Thus, they are used spar­ingly in Racket. But some­times they’re the cleanest way to do things.

With those changes, we’re done with the tokenizer.

← prev next →