Beautiful Racket: Why language-oriented programming? Why Racket?

Beautiful Racket / appendix

#lang pollen

I included a list of the ◊link["why-racket-why-lisp.html#so-really-whats-in-it-for-me-now"]{nine language features} that were most valuable to me as a Racket noob. Feature #5 was "create new programming languages". This technique also goes by the name ◊em{language-oriented programming}, or ◊em{LOP}.


#lang pollen

I included a list of the ◊link["why-racket-why-lisp.html#so-really-whats-in-it-for-me-now"]{nine language features} that were most valu­able to me as a Racket noob. Feature #5 was "create new program­ming languages". This tech­nique also goes by the name ◊em{language-oriented program­ming}, or ◊em{LOP}.

#lang brag

bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"


#lang brag

bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"


What problems are best suited to language-oriented programming?

Why is Racket best suited for the task of making languages?


Language-oriented programming is really an interface-design technique. It’s unbeatable for tasks that demand minimum notation while preserving maximum precision. Minimum notation = the only notation is what you permit. There is nothing extraneous. Maximum precision = the meaning of that notation is exactly what you say. There is no scaffolding or boilerplate. LOP gets to the point like nothing else.

(The impatient can jump ahead to these categories of tasks that are likely to benefit from LOP.)

Racket is ideal for LOP because of its macro system. Macros are indispensable for making languages because they make compiler-style code transformations easy. Racket’s macro system is better than any other.

About half the readers of this piece are now departing to post anonymous internet comments disputing the above claims. Before you leave, please know: I’m hedged either way. LOP and Racket have been an incredible force multiplier on my programming productivity. I’m happy to share this knowledge with you, so that you too may benefit. But I’m equally happy for these tools to remain my secret weapons, so I can keep producing work that is more ambitious, more impressive, and more profitable than the other 99.9%.

(By the way, even though “programming” and “language” are words idiomatically used together, these languages are not used strictly by humans to program computers. Sometimes they’re used by computers to communicate with us; + For instance: S-expressions. sometimes with each other. + For instance: XML, JSON, HTTP. Definitionally, it seems wrong to exclude these possibilities. But in practice, yes—what we’re usually doing with a programming language is, you know, programming.)

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>My web page</title>
  </head>
  <body>
    <p>Hello <strong>world</strong></p>
  </body>
<html>


<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>My web page</title>
  </head>
  <body>
    <p>Hello <strong>world</strong></p>
  </body>
<html>

print("<!DOCTYPE html>")
print("<html>")
print("<head>")
print("<meta charset=\"UTF-8\">")
print("<title>My web page</title>")
print("</head>")
print("<body>")
print("<p>Hello <strong>world</strong></p>")
print("</body>")
print("<html>")


print("<!DOCTYPE html>")
print("<html>")
print("<head>")
print("<meta charset=\"UTF-8\">")
print("<title>My web page</title>")
print("</head>")
print("<body>")
print("<p>Hello <strong>world</strong></p>")
print("</body>")
print("<html>")

Plainly, this distinction is tortured. Here, the Pythonization adds nothing but complexity and boilerplate. More piquantly, the only interesting semantic content in the Python program—in terms of controlling the web browser—is what’s embedded in the HTML. + Arguably, HTML tags—like DOCTYPE and meta and strong—are nothing more than functions that take arguments. My DSL Pollen builds on this correspondence. Consistency compels us to conclude that HTML, though simpler and less flexible than Python, is nevertheless a programming language too.

The problem? Turing completeness is a low bar. It’s a technical measurement that doesn’t tell us anything interesting about the language in the real world. + “Beware of the Turing tarpit in which everything is possible but nothing of interest is easy.“—Alan Perlis For instance, regular expressions aren’t Turing complete, but they’re valuable because they express a lot of computation with minimal notation. HTML is also not Turing complete, but it’s a useful way to control a browser. By contrast, the bf language is Turing complete, but even the most banal tasks require acres of impenetrable code.


Binary data formats don’t qualify as languages. For instance, a jpeg file. Though a computer can understand it, a human cannot. Or a PDF: if you crack one open, you’ll find some parts that seem human readable. But that’s incidental to how PDF works. There’s no sense in which a human is intended to notate ideas using PDF constructs.

Plain text files are not languages. Suppose we have a file containing Homer’s Iliad. We humans can read and understand this file. Though a computer can trivially process the file—say, by printing its contents—the text within the file is unintelligible to the computer.

Graphical user interfaces are not languages. Yes, they’re notation systems (that rely on text and image). But they’re only intelligible to humans. Computers draw GUIs, but they’re not intelligible to the computer. + But see Racket’s 2d language, an embedded language that makes ASCII-art boxes intelligible.

bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"


bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"

#lang brag
bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"


#lang brag
bf-program : (bf-op | bf-loop)*
bf-op      : ">" | "<" | "+" | "-" | "." | ","
bf-loop    : "[" (bf-op | bf-loop)* "]"


When you want to create an interface usable by less skilled programmers, or nonprogrammers, or lazy programmers (don’t underestimate the size of that last category).

For instance, Racket has an elaborate web-application library. But it’s also possible to spin up a simple web server quickly with the web-server/insta language:

#lang web-server/insta
(define (start request)
(response/xexpr
'(html (body "Hello LOP World"))))

#lang web-server/insta (define (start request) (response/xexpr '(html (body "Hello LOP World"))))

Matthew Flatt’s article Creating Languages in Racket demonstrates a language that generates playable text adventures. As with brag, it looks more like a specification than a program, but it works:

#lang txtadv

===VERBS===

north, n
"go north"

south, s
"go south"

get _, grab _, take _
"get"

===THINGS===

---cactus---
get
  "Ouch!"

===PLACES===

---desert---
"You're in a desert. There is nothing for miles around."
[cactus, key]

north
  meadow

south
  desert

#lang txtadv ===VERBS=== north, n "go north" south, s "go south" get _, grab _, take _ "get" ===THINGS===—cactus—get "Ouch!" ===PLACES===—desert—"You're in a desert. There is nothing for miles around." [cactus, key] north meadow south desert

When you want to simplify the notation. Regular expressions are one example. Another example is my DSL Pollen, a language for making online books (including this one). Pollen is like Racket, except that you start in plain-text mode, and use a special character to denote Racket commands, which are evaluated and spliced into the content. + Pollen is based on Racket’s documentation language Scribble, which handles most of the heavy lifting. So the first part of this paragraph is programmed like so:

#lang pollen

When you want to simplify the notation. Regular expressions are one example. Another example is my DSL ◊link["https://pollenpub.com"]{Pollen}, a language for making online books (including this one).

#lang pollen When you want to simplify the notation. Regular expressions are one example. Another example is my DSL ◊link["https://pollenpub.com"]{Pollen}, a language for making online books (including this one).

Pollen takes care of inserting all the necessary tags and converting it to infallible HTML. I get the benefits of manual markup (I still have total control of what ends up in the page) but none of the costs (I cannot, say, accidentally omit a closing tag).

Another example of simplified notation is lindenmayer, a language for generating and drawing fractals called Lindenmayer systems, like this one:


In ordinary Racket, a Lindenmayer program might look like this:

#lang racket/base
(require lindenmayer/simple/compile)
(define (finish val) (newline))
(define (A value) (display 'A))
(define (B value) (display 'B))
(lindenmayer-system
(void)
finish
3
(A)
(A -> A B)
(B -> A))

#lang racket/base (require lindenmayer/simple/compile) (define (finish val) (newline)) (define (A value) (display 'A)) (define (B value) (display 'B)) (lindenmayer-system (void) finish 3 (A) (A -> A B) (B -> A))

But one can use the simplified notation merely by changing the #lang designation at the top of the file:

#lang lindenmayer/simple
## axiom ##
A
## rules ##
A -> AB
B -> A
## variables ##
n=3

#lang lindenmayer/simple ## axiom ## A ## rules ## A -> AB B -> A ## variables ## n=3

The language presumes you already know something about Lindenmayer systems. But the simplified notation makes it easy to translate what you know into a program that does what you want.

When you want to build on existing notation. Above, we saw brag, a DSL that uses a BNF grammar as source code.

#lang brag
bf-program : (bf-op | bf-loop)*
bf-op : ">" | "<" | "+" | "-" | "." | ","
bf-loop : "[" (bf-op | bf-loop)* "]"

#lang brag bf-program : (bf-op | bf-loop)* bf-op : ">" | "<" | "+" | "-" | "." | "," bf-loop : "[" (bf-op | bf-loop)* "]"

Another example—early on, people who tried Pollen said “that’s cool dude, but I prefer Markdown.” Wish granted. pollen/markdown is a Pollen dialect that offers the semantics of Pollen but accepts ordinary Markdown notation:

#lang pollen/markdown

When you want to simplify the notation. Regular expressions are one example.
My DSL [Pollen]("https://pollenpub.com") is a language for making online books.

#lang pollen/markdown When you want to simplify the notation. Regular expressions are one example. My DSL [Pollen]("https://pollenpub.com") is a language for making online books.

The nicest part? It took me about one hour to create this dialect by combining a Markdown parser with my existing code.

When you want to create an intermediate target for other languages. JSON, YAML, S-expressions, and XML are all DSLs that define data formats meant to be machine-writable and -readable.

In Beautiful Racket, one tutorial language is called jsonic. It lets us embed Racket expressions in JSON, thereby making JSON programmable. So a source file like this:

#lang jsonic
// a line comment
[
  @$ 'null $@,
  @$ (* 6 7) $@,
  @$ (= 2 (+ 1 1)) $@,
  @$ (list "array" "of" "strings") $@,
  @$ (hash 'key-1 'null
           'key-2 (even? 3)
           'key-3 (hash 'subkey 21)) $@
]

#lang jsonic // a line comment [ @$ 'null $@, @$ (* 6 7) $@, @$ (= 2 (+ 1 1)) $@, @$ (list "array" "of" "strings") $@, @$ (hash 'key-1 'null 'key-2 (even? 3) 'key-3 (hash 'subkey 21)) $@ ]

Compiles to an ordinary JSON result:

[
  null,
  42,
  true,
  ["array","of","strings"],
  {"key-1":null,"key-3":{"subkey":21},"key-2":false}
]

[ null, 42, true, ["array","of","strings"], {"key-1":null,"key-3":{"subkey":21},"key-2":false} ]

When the bulk of the program is configurational. Dotfiles, for instance, can be characterized as DSLs. A more sophisticated example in Racket is Riposte by Jesse Alama, a language for testing JSON-based HTTP APIs:

#lang riposte

$productId := 41966
$qty := 5
$campaignId := 1

$payload := {
  "product_id": $productId,
  "campaign_id": $campaignId,
  "qty": $qty
}

POST $payload cart/{uuid}/items responds with 200

$itemId := /items/0/cart_item_id

GET cart responds with 200

#lang riposte $productId := 41966 $qty := 5 $campaignId := 1 $payload := { "product_id": $productId, "campaign_id": $campaignId, "qty": $qty } POST $payload cart/{uuid}/items responds with 200 $itemId := /items/0/cart_item_id GET cart responds with 200

As a miniature scripting language, Riposte is a lot smarter than the average dotfile. It hides all the intermediate code required for HTTP transactions, and lets the language user focus on writing tests. It’s still housekeeping. But at least you can focus on the housekeeping that you care about.

Under the hood, every DSL in Racket is actually a source-to-source compiler that converts the notation and semantics of the DSL into the equivalent Racket program. For this reason, a Racket DSL isn’t going to run as fast as one written in hand-carved C. But it also makes all of Racket’s tooling and libraries accessible in every Racket DSL. So what you lose in performance, you get back many times over in convenience. And when DSLs are convenient and cheap, they become a realistic option for a much wider range of problems.


Racket has a specialized data structure called a syntax object that is the medium of exchange among macros. Unlike a string, which can only contain raw code, a Racket syntax object packages the code in a way that preserves its hierachical structure, plus metadata like lexical context and source locations, and arbitrary fields called syntax properties. This metadata stays attached to the code during its various transformations. (See syntax objects for the details.)

Racket macros are hygienic, which means that by default, the code produced by a macro retains the lexical context from where the macro was defined. In practice, this eliminates a huge amount of the housekeeping that would ordinarily be required to make DSLs work. (See hygiene for the details.)


“LOP seems interesting, but I don’t know what I’d do with it.” Sure—I wasn’t necessarily expecting to immediately recruit you into the LOP army. Rather, my goal has been to torque your thinking in a productive way. You’ve now learned about a previously unfamiliar tool. Today, you might not see a use for it. But someday, you’ll confront a problem that exceeds the limits of your current favorite language. On that day, LOP will have its opening.

“OK, you’ve convinced me, but there’s no way I can get LOP or Racket into my workplace.” Jesse Alama’s story of how he introduced his DSL Riposte is a great example of winning over colleagues with LOP (emphasis mine below):

[One can try] to get permission upfront to do something in Racket. That’s less likely to succeed, I think, than just making something great and explaining its benefits ... In my case, that meant talking with co-workers about their work and asking: “How can we model the proposed change in the API and be sure that we’ve really succeeded?” The implied answer being: “Write a Riposte script.” That’s the moment where it becomes clear that [the DSL] I made has real benefits. I don’t even “push” Racket. I just introduce the DSL and show them how it helps them.

Jesse talked more about Riposte at RacketCon 2020.


Beautiful Racket by me, Matthew Butterick. The online book that you’re currently visiting. But you’re in the appendix. The main part of the book is a progressive series of LOP tutorials, all relying on Racket and its fantastic macro system.

Language-Oriented Programming in Racket: A Cultural Anthropology by Jesse Alama. An imposing title, but inside is a friendly and readable set of interviews that Jesse has conducted with numerous Racketeers (me included), with contrasting perspectives on how LOP fits into our work.

Creating Languages in Racket by Matthew Flatt. Matthew is the lead architect of Racket (and wrote the foreword for this book). This brief article nicely demonstrates the increasing levels of sophistication in DSL design, using a clever game-creation DSL.

More examples of Racket-implemented DSLs, and even more.

Beautiful Racket / appendix

Why language-oriented programming? Why Racket?

The big questions

The short answer

The long answer

Domain-specific vs. general-purpose languages

What is a language?

Embedded languages

You can’t really be saying that HTML is programming

The Turing-completeness tarpit

The limits of language

Languages as interfaces

Where languages excel as interfaces

Why Racket?

Why macros?

Conclusion: winning with LOP

Further reading

Beau­tiful Racket / appendix

Why language-oriented programming? Why Racket?

Beautiful Racket / appendix