Chez Scheme is a well-respected implementation of the Scheme programming language that was bought by Cisco in 2012 and released under an open-source license for the first time in 2016. The following year, the Racket team announced that Racket would abandon its own C-based foundation in favor of Chez Scheme. That is, the parts of Racket that were written in C would be rewritten in Racket (where possible) or Chez Scheme (where necessary). In so doing, Racket would essentially itself become a DSL running on Chez Scheme. In 2021, that project was meaningfully finished, as the new Racket based on Chez Scheme (working title “Racket CS”) became the default version of Racket in release 8.0.
Though Chez Scheme is one of the fastest Scheme implementations, the primary motivation for the move was not performance so much as maintainability. Instead of relying on a custom pile of C code, and the ongoing maintenance costs thereof, Racket planned to hop on the back of Chez Scheme and float merrily down the river. Furthermore, because Chez Scheme is owned by Cisco—and employs Chez Scheme’s principal architect, Kent Dybvig—there’s been a sense that Chez Scheme’s long-term survival is already assured. So by lashing Racket to Chez Scheme, Racket would indirectly benefit from this institutional underpinning.
When I first heard about this “Racket CS” project in 2017, I didn’t spend too long thinking about it. I trusted that the Racket team had good reasons for making the switch. Whatever adjustments that would be needed down the road from Racket developers would be worthwhile.
In hindsight, I misunderstood the project. I thought the goal was to move Racket onto Chez Scheme and accept whatever losses to backward compatibility might arise, on the idea that the future benefits were so terrific. This is a tradeoff that happens all the time in programming.
But the actual project undertaken by the Racket team was to preserve the existing behavior of Racket and all its libraries. + Except for the C API and a few other outliers. On the one hand, this is ostensibly a more ambitious & accommodating goal. On the other hand, 4+ years later, it has left the Racket team in a puddle of unintended consequences.
Foremost among these: though the project is notionally complete, the main mission was arguably never accomplished, because Racket doesn’t run on the main branch of Chez Scheme. Rather, it runs on a special Racket-specific fork of Chez Scheme that’s had certain patches applied to preserve compatibility with everything already in Racket.
Let’s rewind for a moment. It was always apparent to the Racket team that certain discrepancies between the Racket view of the world and the Chez Scheme view would need to be smoothed over. How? By patching the main branch of Chez Scheme through pull requests, they predicted. The Racket team got some of their pull requests accepted into the main branch of Chez Scheme, because these patches were broadly useful.
But not all. Some patches that Racket needed were detrimental (or unworkable or inappropriate) for Chez Scheme at large. In an “experience report” near the end of the project, the Racket team admitted that “[t]he least predictable part of our effort was how big the differences between Racket and Chez Scheme would turn out to be”. And instead of capitulating to the Chez Scheme way—say, by removing or changing certain features of Racket—these Racket-compatibility patches became the basis of the Racket-specific fork of Chez Scheme.
“But that’s close enough, right?” As a Racket developer, I hope that turns out to be true. But I have doubts that it will. The main point of moving to Chez Scheme was to harvest certain maintainability benefits. It stands to reason—if your project is based on Chez Scheme, then as Chez Scheme is improved, your project is thereby also improved.
To be fair, the Racket team has described its fork more as a “set of patches” that otherwise stays in sync with the main branch of Chez Scheme. Still, as time passes, the two will naturally tend to diverge further. Whether one calls it a “fork” or “set of patches”, the result is the same: some nonzero maintenance cost will persist, and likely grow.
In their experience report, the Racket team is blasé about the risks of this divergence: “we are willing to maintain Chez Scheme and any patches needed”. Maybe that’s fair. Even with the divergence factored in, maybe the benefits of using a fork of Chez Scheme is better than no Chez Scheme at all.
Still, as a Racket developer, it seems like an opportunity has been missed. I thought the main objective was to permanently combine the greatest assets of Chez Scheme (= its speed and stability) with the greatest assets of Racket (= its language-oriented programming facilities, including its macro system and module system), while forever exempting the Racket team from a certain level of back-end maintenance. + At the start of the project, this was explicitly the expectation. This didn’t happen. + At the end of the project, this was explicitly admitted to be the “least predictable” difficulty. As the Racket team turns its attention to other projects, it doesn’t look like it ever will. + At the end of the project, this was explicitly conceded—“we are willing to maintain Chez Scheme and any patches needed”.
“Why do you care if the Racket team assumes the burden of maintaining (patches for) Chez Scheme?” Because they’re people with full-time jobs who seem spread too thin already, and their available time is better spent on the things that make Racket special. For cultural reasons as well, I think that adopting Chez Scheme in toto would’ve had a beneficial stabilizing effect on the Racket community.
In any case, this is all water under the bridge. I’m recounting how it appeared to me, as a Racket developer. I never had a vote. As I said above, I hope the choice works out.
The Racket CS effort has also had the perhaps unintended consequence of encouraging me to take a closer look at Chez Scheme for my own future projects.
Interestingly, both Racket and Chez Scheme began as academic projects. Both have been continuously developed over a long time—Racket since 1995; Chez Scheme since 1984. (There’s also Gambit Scheme, a Scheme-to-C compiler with academic roots, developed since 1988.)
At times, the Racket team has claimed to want more users outside of academia. After eight years working with Racket, it’s hard to take them seriously on that point. (See also: the ongoing struggles with the Rhombus project. See also: the relicensing project to make Racket friendlier for industrial users, which inexplicably dragged out for nearly three years. See also: one member of the Racket team has a habit of openly shaming and terrorizing users of Racket outside academia, including me.)
I still like the language. But despite occasional gestures otherwise, it was and is a project led by five computer-science professors. Their development priorities are set largely by their research objectives. To be clear, there’s nothing wrong with this: it’s their project and they can do whatever they want.
But Chez Scheme has the much stronger story beyond academia. In 2012, Kent Dybvig left his position as a computer-science professor to work at Cisco, which (reputedly) uses Chez Scheme as an embedded language for networking equipment. + For that reason, the Racket team’s contention that Chez Scheme isn’t suited for “constrained, embedded settings” seems possibly wrong. That is, Chez Scheme meaningfully transitioned from an academic project to an industrial project. To be fair, Cisco’s patronage isn’t necessarily permanent. Big tech companies are famously capable of pivoting to new technologies and dumping the earlier ones with zero sentimentality.
Racket has a larger & richer set of libraries than Chez Scheme, which makes getting things done in Racket easier at times. But that’s largely because Racket has been open source since the beginning. Chez Scheme has only been open source since 2016. Given its technical superiority as a Scheme implementation, it seems inevitable that its library ecosystem will grow.
Will an industrial user be better off investing in Racket or Chez Scheme? As one of those users, I’m not sure yet. In terms of engineering, the Racket team has conceded that Chez Scheme is superior. In terms of orientation to industrial users, Chez Scheme has also been superior (keeping in mind that before it was open-sourced, it was licensed as proprietary software to customers largely outside academia). Racket has barely any use outside academia, and no demonstrated interest in getting there. As for the libraries, though Racket has an advantage for now, it seems likely that many of these libraries could be ported to Chez Scheme if needed—Racket & Chez Scheme are under compatible open-source licenses—though this will require more study.
In sum, I think the possibilities of Chez Scheme are interesting enough that I plan to spend more time experimenting with it, and using it where possible.
15 July 2021: Improved the description of Racket’s fork of Chez Scheme. Added my view about why avoiding Chez Scheme maintenance entirely would’ve been useful. Added various supporting hyperlinks and facts to existing points.