Thank you for your comment

Beau­tiful Racket / appendix

Why language-oriented programming? Why Racket?

This is a sequel to my earlier piece Why Racket? Why Lisp?, which I wrote a year or so after I discov­ered Racket. As someone new to Lisp languages, I had trudged through bogs of Lisp flat­tery, never sure what to make of the often hand­wavy claims. For instance, that Lisp even­tu­ally induces “profound enlight­en­ment”. Sure—what­ever you say, bro.

I had a simpler ques­tion: what’s in it for me, now? My earlier piece tried to answer that ques­tion. I summa­rized why someone who hadn’t looked into a Lisp language—or Racket in partic­ular—might want to.

I included a list of the nine language features that were most valu­able to me as a Racket noob. Feature #5 was “create new program­ming languages”. This tech­nique also goes by the name language-oriented program­ming, or LOP.

In the years since, language-oriented program­ming has risen to become my favorite part of Racket. Along the way, I converted my enthu­siasm into action. In addi­tion to making languages, I wrote this online book—Beau­tiful Racket—that teaches LOP as a tech­nique, and Racket as a tool.

In my own work, one example is Pollen, a text-based language I wrote to make my online books Prac­tical Typog­raphy and Beau­tiful Racket possible.  + Pollen is based on Racket’s docu­men­ta­tion language Scribble, which handles most of the heavy lifting. In Pollen, the para­graph above is programmed like so:

#lang pollen

I included a list of the ◊link["why-racket-why-lisp.html#so-really-whats-in-it-for-me-now"]{nine language features} that were most valuable to me as a Racket noob. Feature #5 was "create new programming languages". This technique also goes by the name ◊em{language-oriented programming}, or ◊em{LOP}.
1
2
3
#lang pollen

I included a list of the ◊link["why-racket-why-lisp.html#so-really-whats-in-it-for-me-now"]{nine language features} that were most valuable to me as a Racket noob. Feature #5 was "create new programming languages". This technique also goes by the name ◊em{language-oriented programming}, or ◊em{LOP}.
copy to clipboard

Another example is brag, a parser gener­ator (in the lex/yacc style) that takes a BNF grammar as its source code. A simple example for the bf language:

#lang brag

bf-program : (bf-op | bf-loop)*
bf-op : ">" | "<" | "+" | "-" | "." | ","
bf-loop : "[" (bf-op | bf-loop)* "]"
1
2
3
4
5
#lang brag

bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"
copy to clipboard

Both these languages are imple­mented in Racket, and can be run with the usual Racket inter­preter, or inside the Racket IDE (called DrRacket).

The big ques­tions

And yet. Though this book has started thou­sands of people on their LOP journey, I some­times fear that I’ve fallen into the same quick­sand as the Lisp advo­cates I once crit­i­cized.

If LOP is so great, then you shouldn’t need to spend a few days working through the tuto­rials in this book. Right? I should be able to explain it concisely, with minimum hand­waving. I should be able to answer two simple ques­tions:

  1. What prob­lems are best suited to language-oriented program­ming?

  2. Why is Racket best suited for the task of making languages?

The second ques­tion is easy. The first ques­tion isn’t. I’ve been asked it many times. I’ve often resorted to the answer made famous by Justice Potter Stewart: you’ll know it when you see it. That answer is good enough for those already LOP-curious. But not for those still on the side­lines who want more of a prac­tical justi­fi­ca­tion.

So here’s a better attempt. Bear in mind that I’m not a computer-science professor. I’m not going to be talking about program­ming languages the way they might. Rather, I use Racket and DSLs for prac­tical purposes—my daily work depends on them. My goal is to frame the issue in a way that will be useful for other prac­tical users. (If I haven’t, you’re welcome to click in the left margin and send me a comment.)

The short answer

  1. Language-oriented program­ming is really an inter­face-design tech­nique. It’s unbeat­able for tasks that demand minimum nota­tion while preserving maximum preci­sion. Minimum nota­tion = the only nota­tion is what you permit. There is nothing extra­neous. Maximum preci­sion = the meaning of that nota­tion is exactly what you say. There is no scaf­folding or boil­er­plate. LOP gets to the point like nothing else.

    (The impa­tient can jump ahead to these cate­gories of tasks that are likely to benefit from LOP.)

  2. Racket is ideal for LOP because of its macro system. Macros are indis­pens­able for making languages because they make compiler-style code trans­for­ma­tions easy. Racket’s macro system is better than any other.

About half the readers of this piece are now departing to post anony­mous internet comments disputing the above claims. Before you leave, please know: I’m hedged either way. LOP and Racket have been an incred­ible force multi­plier on my program­ming produc­tivity. I’m happy to share this knowl­edge with you, so that you too may benefit. But I’m equally happy for these tools to remain my secret weapons, so I can keep producing work that is more ambi­tious, more impres­sive, and more prof­itable than the other 99.9%.

The choice, however, is yours.

The long answer

I finally started to unravel the Big Ques­tions by consid­ering a metaque­s­tion: why does it seem hard to explain the bene­fits of language-oriented program­ming?

Perhaps because when we speak of program­ming languages—or just languages, short­hand that I’ll use inter­change­ably—the term is freighted with expec­ta­tions about what a language is and what it does. As long as we’re standing inside that box, it’s more diffi­cult to see the value of language-oriented program­ming.

But if we zoom out and look at languages as part of the larger cate­gory of human–computer inter­faces, then it becomes easier to see the special bene­fits of LOP.

So let’s do that.

Domain-specific vs. general-purpose languages

First, some termi­nology. Language-oriented program­ming (aka LOP) is the idea of solving a program­ming problem by making a new program­ming language, and then writing a program with the new language. Often, these “little languages” are known as domain-specific languages or DSLs.

As the name implies, a domain-specific language is one tailored to the needs of a partic­ular set of prob­lems. For instance, Post­Script, SQL, make, regular expres­sions, .htaccess, and HTML qualify as domain-specific languages. They don’t try to do every­thing. Rather, they focus on doing one thing well.

At the other end of the language spec­trum are what we’ll call general-purpose languages. Out here we find the usual suspects—C, Pascal, Perl, Java, Python, Ruby, Racket, etc. Why aren’t these languages DSLs? Because they hold them­selves out as being suit­able for a wide range of computing tasks.

In prac­tice, general-purpose languages often have certain jobs that they do better than others, depending on the condi­tions of their birth. For instance, C excels at systems program­ming. Perl excels as a sysadmin scripting language. Python excels as a language for begin­ners. Racket excels at LOP. In each case, because that’s what the language was designed to do.

The distinc­tion between domain-specific and general-purpose languages is perme­able. For instance, Ruby started as a general-purpose language, but became popular largely as a web-appli­ca­tion DSL through its asso­ci­a­tion with Ruby on Rails. Going the other direc­tion, JavaScript was orig­i­nally a DSL for web-browser scripting. Like a mutating virus, it has since grown far beyond.

What is a language?

If all these things on the spec­trum between domain-specific languages and general-purpose languages deserve to be called languages, then what are the defining features of a language?

I know what you’re thinking: “Well, that’s where you’re wrong. HTML isn’t a language. It’s just markup. It can’t express algo­rithms.” Or maybe: “Regular expres­sions aren’t a language. They can’t stand on their own. They’re just syntax within another language.”

I once felt that way too. But the closer I looked, the more these distinc­tions seemed vaporous. Thus, my first major claim (of three): a program­ming language is inher­ently a medium of exchange—a nota­tion system mutu­ally intel­li­gible to humans and computers.

“Nota­tion” connotes that a language has syntax; “intel­li­gible” connotes that a language has meaning (or seman­tics, to use the fancier word) attached to the syntax. This defi­n­i­tion covers all general-purpose program­ming languages. And all DSLs. (But not every data stream—about which more later.)

(By the way, even though “program­ming” and “language” are words idiomat­i­cally used together, these languages are not used strictly by humans to program computers. Some­times they’re used by computers to commu­ni­cate with us;  + For instance: S-expres­sions. some­times with each other.  + For instance: XML, JSON, HTTP. Defi­n­i­tion­ally, it seems wrong to exclude these possi­bil­i­ties. But in prac­tice, yes—what we’re usually doing with a program­ming language is, you know, program­ming.)

Consider HTML. It’s a way of telling a computer—in partic­ular, a web browser—how to draw a web page. It’s a nota­tion system (= angle brackets, tags, attrib­utes, and so on) intel­li­gible to the human and computer (= the charset meta defines the char­acter encoding, p tag contains a para­graph, and so on).

Here’s a little HTML page:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>My web page</title>
  </head>
  <body>
    <p>Hello <strong>world</strong></p>
  </body>
<html>
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>My web page</title>
  </head>
  <body>
    <p>Hello <strong>world</strong></p>
  </body>
<html>
copy to clipboard

Suppose you dispute that HTML is a program­ming language. Fine. We’ll output our page with Python. That’s a real program­ming language, right?

print("<!DOCTYPE html>")
print("<html>")
print("<head>")
print("<meta charset=\"UTF-8\">")
print("<title>My web page</title>")
print("</head>")
print("<body>")
print("<p>Hello <strong>world</strong></p>")
print("</body>")
print("<html>")
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
print("<!DOCTYPE html>")
print("<html>")
print("<head>")
print("<meta charset=\"UTF-8\">")
print("<title>My web page</title>")
print("</head>")
print("<body>")
print("<p>Hello <strong>world</strong></p>")
print("</body>")
print("<html>")
copy to clipboard

If Python is a program­ming language and HTML is not, then it must be true that this Python sample is a program, and the HTML sample is not.

Plainly, this distinc­tion is tortured. Here, the Pythoniza­tion adds nothing but complexity and boil­er­plate. More piquantly, the only inter­esting semantic content in the Python program—in terms of control­ling the web browser—is what’s embedded in the HTML.  + Arguably, HTML tags—like DOCTYPE and meta and strong—are nothing more than func­tions that take argu­ments. My DSL Pollen builds on this corre­spon­dence. Consis­tency compels us to conclude that HTML, though simpler and less flex­ible than Python, is never­the­less a program­ming language too.

Embedded languages

Our HTML example was contrived. But this pattern—a DSL nested within another language—is perva­sive. Languages used in this way are called embedded languages. They repre­sent the most common form of language-oriented program­ming. As a programmer, you’ve relied on LOP for years, even if you didn’t know its name.

For instance, regular expres­sions.  + Other exam­ples: printf format­ting strings, CLDR date/time patterns, SQL. We may not think of regular-expres­sion nota­tion as an inde­pen­dent language. But every programmer knows what this means:

^fo+(bar)*$
1
^fo+(bar)*$
copy to clipboard

More­over, you can prob­ably type this regular expres­sion into your favorite program­ming language and it will just work. This consis­tent behavior is only possible because regular-expres­sion nota­tion is an embedded language, exter­nally defined (by POSIX).

As with HTML, we could write the equiv­a­lent string-matching compu­ta­tion in the nota­tion of the host language. For instance, Racket supports Scheme Regular Expres­sions (SREs), which are regular expres­sions that use S-expres­sion nota­tion. The above pattern would be written like so:

(seq bos "f" (+ "o") (* (submatch "bar")) eos)
1
(seq bos "f" (+ "o") (* (submatch "bar")) eos)
copy to clipboard

But Racket program­mers rarely use SREs. They’re too long and too hard to remember.

Another ubiq­ui­tous example of an embedded DSL: math expres­sions. Every programmer knows what this means:

(1 + 2) * (3 / 4) - 5
1
(1 + 2) * (3 / 4) - 5
copy to clipboard

On their own, math expres­sions don’t make inter­esting programs. We need to combine them with other language constructs. But as with regular expres­sions, that’s an ergonomic and prac­tical consid­er­a­tion. Math expres­sions have their own nota­tion and meaning, intel­li­gible by both humans and computers, and thus qualify as a sepa­rate embedded language.

You can’t really be saying that HTML is program­ming

Indeed I am. I’m claiming that HTML (and regular expres­sions and math expres­sions) qualify as rudi­men­tary program­ming languages. This neces­sarily implies that writing HTML (or regular expres­sions or math expres­sions) qual­i­fies as rudi­men­tary program­ming.

Please—don’t panic. We can agree that someone who billed them­selves as a “programmer” on LinkedIn while only knowing HTML and arith­metic would be consid­ered delu­sional.  + And next week, will prob­ably have a $180K soft­ware-engi­neering job in Menlo Park. But that spills into a sepa­rate issue about what the desig­na­tion “programmer” typi­cally denotes in the job market. That’s not our concern here.

The Turing-complete­ness tarpit

If this defi­n­i­tion of program­ming languages still rankles, maybe it’s because you think a real program­ming language needs to be capable of expressing all possible algo­rithms—that is, it must be Turing complete.

I see why that might be part of the intu­ition. Every general-purpose program­ming language is Turing complete.

The problem? Turing complete­ness is a low bar. It’s a tech­nical measure­ment that doesn’t tell us anything inter­esting about the language in the real world.  + “Beware of the Turing tarpit in which every­thing is possible but nothing of interest is easy.“—Alan Perlis For instance, regular expres­sions aren’t Turing complete, but they’re valu­able because they express a lot of compu­ta­tion with minimal nota­tion. HTML is also not Turing complete, but it’s a useful way to control a browser. By contrast, the bf language is Turing complete, but even the most banal tasks require acres of impen­e­trable code.

The limits of language

Does my defi­n­i­tion of a program­ming language include every­thing? No.

Languages as inter­faces

Above, I described a program­ming language as a “medium of exchange” between humans and computers. In that way, languages fit among the broader cate­gory of things we call inter­faces.

This brings me to my second major claim (of three): that language-oriented program­ming is funda­men­tally an inter­face-design tech­nique. If you like thinking about inter­faces, you’ll love LOP. If you don’t, you may still love LOP, because it enables certain inter­faces that are other­wise unat­tain­able.

One of my favorite exam­ples of language as inter­face is brag, a parser-gener­ator language made with Racket. If you’ve ever used the lex/yacc tool­chain, you know the goal is often to generate a parser from a BNF grammar. For instance, the BNF grammar for the bf language looks like this:

bf-program : (bf-op | bf-loop)*
bf-op : ">" | "<" | "+" | "-" | "." | ","
bf-loop : "[" (bf-op | bf-loop)* "]"
1
2
3
bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"
copy to clipboard

To make a parser in a general-purpose language, we’d have to trans­late this grammar into a pile of native code. Even with the help of a parser gener­ator, it’s a tedious job. And point­less—haven’t we already notated the grammar? Why do it again?

With brag, however, our wish comes true. To make a parser, we simply add #lang brag to the file, which magi­cally converts our BNF grammar into brag source code:

#lang brag
bf-program : (bf-op | bf-loop)*
bf-op : ">" | "<" | "+" | "-" | "." | ","
bf-loop : "[" (bf-op | bf-loop)* "]"
1
2
3
4
#lang brag
bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"
copy to clipboard

That’s it! When compiled, this file will export a func­tion called parse that imple­ments this BNF grammar.

This is one of my favorite exam­ples of LOP because it’s unas­sail­ably supe­rior to the long-winded alter­na­tive. More­over, with a general-purpose language, this kind of inter­face is essen­tially impos­sible.

But the language-oriented programmer does it all the time.

Where languages excel as inter­faces

This brings me to my third and final major claim, which is that among inter­faces, languages have unique strengths. The cate­gories below are not exhaus­tive or exclu­sive, of course. But I’ve found that LOP has a lot to offer in these situ­a­tions:

  1. When you want to create an inter­face usable by less skilled program­mers, or nonpro­gram­mers, or lazy program­mers (don’t under­es­ti­mate the size of that last cate­gory).

    For instance, Racket has an elab­o­rate web-appli­ca­tion library. But it’s also possible to spin up a simple web server quickly with the web-server/insta language:

    #lang web-server/insta
    (define (start request)
      (response/xexpr
       '(html (body "Hello LOP World"))))
    1
2
3
4
    #lang web-server/insta
(define (start request)
  (response/xexpr
   '(html (body "Hello LOP World"))))
    copy to clipboard

    Matthew Flatt’s article Creating Languages in Racket demon­strates a language that gener­ates playable text adven­tures. As with brag, it looks more like a spec­i­fi­ca­tion than a program, but it works:

    #lang txtadv

    ===VERBS===

    north, n
     "go north"

    south, s
     "go south"

    get _, grab _, take _
     "get"

    ===THINGS===

    ---cactus---
    get
      "Ouch!"


    ===PLACES===

    ---desert---
    "You're in a desert. There is nothing for miles around."
    [cactus, key]

    north
      meadow

    south
      desert 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    #lang txtadv

===VERBS===

north, n
 "go north"

south, s
 "go south"

get _, grab _, take _
 "get"

===THINGS===

---cactus---
get
  "Ouch!"


===PLACES===

---desert---
"You're in a desert. There is nothing for miles around."
[cactus, key]

north
  meadow

south
  desert
    copy to clipboard

  2. When you want to simplify the nota­tion. Regular expres­sions are one example. Another example is my DSL Pollen, a language for making online books (including this one). Pollen is like Racket, except that you start in plain-text mode, and use a special char­acter to denote Racket commands, which are eval­u­ated and spliced into the content.  + Pollen is based on Racket’s docu­men­ta­tion language Scribble, which handles most of the heavy lifting. So the first part of this para­graph is programmed like so:

    #lang pollen

    When you want to simplify the notation. Regular expressions are one example. Another example is my DSL ◊link["https://pollenpub.com"]{Pollen}, a language for making online books (including this one).
    1
2
3
    #lang pollen

When you want to simplify the notation. Regular expressions are one example. Another example is my DSL ◊link["https://pollenpub.com"]{Pollen}, a language for making online books (including this one).
    copy to clipboard

    Pollen takes care of inserting all the neces­sary tags and converting it to infal­lible HTML. I get the bene­fits of manual markup (I still have total control of what ends up in the page) but none of the costs (I cannot, say, acci­den­tally omit a closing tag).

    Another example of simpli­fied nota­tion is lindenmayer, a language for gener­ating and drawing frac­tals called Linden­mayer systems, like this one:

    In ordi­nary Racket, a Linden­mayer program might look like this:

    #lang racket/base
    (require lindenmayer/simple/compile)
    (define (finish val) (newline))
    (define (A value) (display 'A))
    (define (B value) (display 'B))
    (lindenmayer-system
     (void)
     finish
     3
     (A)
     (A -> A B)
     (B -> A))
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    #lang racket/base
(require lindenmayer/simple/compile)
(define (finish val) (newline))
(define (A value) (display 'A))
(define (B value) (display 'B))
(lindenmayer-system
 (void)
 finish
 3
 (A)
 (A -> A B)
 (B -> A))
    copy to clipboard

    But one can use the simpli­fied nota­tion merely by changing the #lang desig­na­tion at the top of the file:

    #lang lindenmayer/simple
    ## axiom ##
    A
    ## rules ##
    A -> AB
    B -> A
    ## variables ##
    n=3
    1
2
3
4
5
6
7
8
    #lang lindenmayer/simple
## axiom ##
A
## rules ##
A -> AB
B -> A
## variables ##
n=3
    copy to clipboard

    The language presumes you already know some­thing about Linden­mayer systems. But the simpli­fied nota­tion makes it easy to trans­late what you know into a program that does what you want.

  3. When you want to build on existing nota­tion. Above, we saw brag, a DSL that uses a BNF grammar as source code.

    #lang brag
    bf-program : (bf-op | bf-loop)*
    bf-op : ">" | "<" | "+" | "-" | "." | ","
    bf-loop : "[" (bf-op | bf-loop)* "]"
    1
2
3
4
    #lang brag
bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"
    copy to clipboard

    Another example—early on, people who tried Pollen said “that’s cool dude, but I prefer Mark­down.” Wish granted. pollen/markdown is a Pollen dialect that offers the seman­tics of Pollen but accepts ordi­nary Mark­down nota­tion:

    #lang pollen/markdown

    When you want to simplify the notation. Regular expressions are one example.
    My DSL [Pollen]("https://pollenpub.com") is a language for making online books.
    1
2
3
4
    #lang pollen/markdown

When you want to simplify the notation. Regular expressions are one example.
My DSL [Pollen]("https://pollenpub.com") is a language for making online books.
    copy to clipboard

    The nicest part? It took me about one hour to create this dialect by combining a Mark­down parser with my existing code.

  4. When you want to create an inter­me­diate target for other languages. JSON, YAML, S-expres­sions, and XML are all DSLs that define data formats meant to be machine-writable and -read­able.

    In Beau­tiful Racket, one tuto­rial language is called jsonic. It lets us embed Racket expres­sions in JSON, thereby making JSON program­mable. So a source file like this:

    #lang jsonic
    // a line comment
    [
      @$ 'null $@,
      @$ (* 6 7) $@,
      @$ (= 2 (+ 1 1)) $@,
      @$ (list "array" "of" "strings") $@,
      @$ (hash 'key-1 'null
               'key-2 (even? 3)
               'key-3 (hash 'subkey 21)) $@
    ]
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    #lang jsonic
// a line comment
[
  @$ 'null $@,
  @$ (* 6 7) $@,
  @$ (= 2 (+ 1 1)) $@,
  @$ (list "array" "of" "strings") $@,
  @$ (hash 'key-1 'null
           'key-2 (even? 3)
           'key-3 (hash 'subkey 21)) $@
]
    copy to clipboard

    Compiles to an ordi­nary JSON result:

    [
      null,
      42,
      true,
      ["array","of","strings"],
      {"key-1":null,"key-3":{"subkey":21},"key-2":false}
    ]
    1
2
3
4
5
6
7
    [
  null,
  42,
  true,
  ["array","of","strings"],
  {"key-1":null,"key-3":{"subkey":21},"key-2":false}
]
    copy to clipboard

  5. When the bulk of the program is config­u­ra­tional. Dotfiles, for instance, can be char­ac­ter­ized as DSLs. A more sophis­ti­cated example in Racket is Riposte by Jesse Alama, a language for testing JSON-based HTTP APIs:

    #lang riposte

    $productId := 41966
    $qty := 5
    $campaignId := 1

    $payload := {
      "product_id": $productId,
      "campaign_id": $campaignId,
      "qty": $qty
    }

    POST $payload cart/{uuid}/items responds with 200

    $itemId := /items/0/cart_item_id

    GET cart responds with 200
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    #lang riposte

$productId := 41966
$qty := 5
$campaignId := 1

$payload := {
  "product_id": $productId,
  "campaign_id": $campaignId,
  "qty": $qty
}

POST $payload cart/{uuid}/items responds with 200

$itemId := /items/0/cart_item_id

GET cart responds with 200
    copy to clipboard

    As a minia­ture scripting language, Riposte is a lot smarter than the average dotfile. It hides all the inter­me­diate code required for HTTP trans­ac­tions, and lets the language user focus on writing tests. It’s still house­keeping. But at least you can focus on the house­keeping that you care about.

Why Racket?

A common critique of LOP is “why make a domain-specific language? Isn’t that more work than writing a native library?”

Not if you have the right tool. Racket is unusual: it’s been designed from the ground up to support LOP. Thus, imple­menting a DSL in Racket is faster, cheaper, and easier than the alter­na­tives. For instance, in the first tuto­rial in this book, I show how you can make a language in one hour—even if you’ve never used Racket.

Under the hood, every DSL in Racket is actu­ally a source-to-source compiler that converts the nota­tion and seman­tics of the DSL into the equiv­a­lent Racket program. For this reason, a Racket DSL isn’t going to run as fast as one written in hand-carved C. But it also makes all of Racket’s tooling and libraries acces­sible in every Racket DSL. So what you lose in perfor­mance, you get back many times over in conve­nience. And when DSLs are conve­nient and cheap, they become a real­istic option for a much wider range of prob­lems.

Thus, to answer the critique—no, a DSL is not neces­sarily more work than a native library. More­over, as we’ve already seen, as an inter­face, a language can do things that a native library cannot.

Why macros?

Because Racket DSLs compile to Racket, a language-oriented programmer using Racket needs to write some syntax trans­formers that convert the DSL nota­tion into native Racket. These syntax trans­formers are known as macros. Indeed, macros can be char­ac­ter­ized as exten­sions to the Racket compiler.

Racket’s macro system is vast, elegant, and unde­ni­ably its crown jewel. But a lot of this book is about the joy of Racket macros. That mate­rial is ready when you are. For now, the two standout features:

  1. Racket has a special­ized data struc­ture called a syntax object that is the medium of exchange among macros. Unlike a string, which can only contain raw code, a Racket syntax object pack­ages the code in a way that preserves its hier­achical struc­ture, plus meta­data like lexical context and source loca­tions, and arbi­trary fields called syntax prop­er­ties. This meta­data stays attached to the code during its various trans­for­ma­tions. (See syntax objects for the details.)

  2. Racket macros are hygienic, which means that by default, the code produced by a macro retains the lexical context from where the macro was defined. In prac­tice, this elim­i­nates a huge amount of the house­keeping that would ordi­narily be required to make DSLs work. (See hygiene for the details.)

Is it possible to imple­ment a DSL in, say, Python? Sure. In fact, I wrote my first DSL in Python—one that I still use to help with my type-design work. Yikes. Once was enough. Ever since, I’ve used Racket.

Conclu­sion: winning with LOP

At this point, you may be having one of two reac­tions:

  1. “LOP seems inter­esting, but I don’t know what I’d do with it.” Sure—I wasn’t neces­sarily expecting to imme­di­ately recruit you into the LOP army. Rather, my goal has been to torque your thinking in a produc­tive way. You’ve now learned about a previ­ously unfa­miliar tool. Today, you might not see a use for it. But someday, you’ll confront a problem that exceeds the limits of your current favorite language. On that day, LOP will have its opening.

  2. “OK, you’ve convinced me, but there’s no way I can get LOP or Racket into my work­place.” Jesse Alama’s story of how he intro­duced his DSL Riposte is a great example of winning over colleagues with LOP (emphasis mine below):

    [One can try] to get permis­sion upfront to do some­thing in Racket. That’s less likely to succeed, I think, than just making some­thing great and explaining its bene­fits ... In my case, that meant talking with co-workers about their work and asking: “How can we model the proposed change in the API and be sure that we’ve really succeeded?” The implied answer being: “Write a Riposte script.” That’s the moment where it becomes clear that [the DSL] I made has real bene­fits. I don’t even “push” Racket. I just intro­duce the DSL and show them how it helps them.

    Jesse talked more about Riposte at Rack­etCon 2020.

At the end of Why Racket? Why Lisp?, I said that a Lisp language “offers you the chance to dis­cover your poten­tial as a pro­gram­mer and a thinker, and thereby raise your expec­ta­tions for what you can accom­plish”.

LOP offers a similar oppor­tu­nity: to raise our expec­ta­tions for what program­ming languages can do for us. Languages are not black boxes. They are inter­faces that we can design. In so doing, we open up new possi­bil­i­ties for what we can accom­plish with programs.

If you can find a better program­ming tech­nique, use it. Now that I have LOP & Racket, I’m never going back.

Further reading

← prev next →