Shoes For the Cobbler’s Children

The cobbler’s children go unshod.

- Venerable English saying

The YACC school of compiler-compilers, like so much of our professional tooling, is of a certain vintage. It predates syntax highlighting, for example, by the simple expedient of predating color monitors: while it doesn’t strictly predate monitors entirely, it was unlikely to have been developed on one¹.

There is a distinction drawn in manufacturing, between a machinist and a tool maker. They operate the same kinds of equipment, broadly speaking, but a machinist does so on the shop floor, as part of production, and a tool maker does so in the toolroom, to support production: the machinist makes product, the tool maker, tooling.

A program like Yacc, (and therefore, Lemon, (and therefore, Zitron)) is for the tool makers of software. Zitron is only of interest to someone who wants to define a parser, and someone who wants to do that is, generally, making software for other developers to write programs in.

Programs in some sense, at least: if we can broaden our concept of a program to include Pic and Mermaid diagrams, LaTeX documents, and so on: and we should, because the core audience for that kind of little language is, and remains, programmers.

But say they’re languages, and not programming languages: they remain tools. We deal in bits², not steel and swarf. Our field is softer, less sharply defined, more accessible to the amateur and the civilian. When we say tooling, we mean something different. But not unrecognizably so!

Most tools, in both contexts, are dual use, but some are more or less proper to the tool maker. In software, parser generators are the most prominent example of this, as suggested by their whimsical alternate name, a “compiler compiler”: if you’re compiling a compiler, then you’re making a tool.

The use of parser generators has fallen out of fashion over time, and there are a number of reasons for this. One is that new custom languages have become rare: I expect this to change.

Some of it is the quality of the product a compiler-compiler produces. There’s some truth to this, and some folklore as well. It is said, especially, that a hand-rolled recursive-descent parser can give better error messages, and do better error recovery. Although the former claim is true of Zitron at the moment, it’s actually a solved problem, and one I intend to port in the relatively near future. Error recovery is maybe not solved, but there’s solid work in that direction, and a case can be made that it matters less now than it did in the past: computers are so fast that running the front end once per syntax error is not much of a burden.³

Less discussed, but important, is that these tools are subjectively unpleasant to use. We’ve grown accustomed to our angry fruit salad syntax highlighting, after all, and by now the IDE concept has pervaded text editing via the language server protocol. A custom language which embeds another forgoes all these affordances, both for itself and for that which it embeds.

Which is why the very first thing I did after releasing Zitron, was to write a tree-sitter grammar for it. This at least avoids editing in black and white; one at least begins to feel at home.

Now I am chuffed to announce: we have a language server too.

Tools for the Tool God! Servers for the Server Throne!

The nature of these tools is simple, and unchanged for many years. They have a DSL which defines a grammar as a set of productions, along with code to execute when those productions reduce. There are miscellaneous directives to configure everything which isn’t a rule, and a way to add more arbitrary code, and that’s it.

This source file is parsed: if that goes well, it will try to make a grammar out of what it gets. If that goes well, it takes a template file and uses it to spit out code which will parse that grammar, executing those actions when those reductions occur.

This gives rise to our first problem: we’re compiling twice, not once, and the second compiler doesn’t know about the first one. C happens to have a balm for this pain, and yacc is the reason why: the #line directive.

The first C compilers were written in assembly language, reimplemented for each new platform Unix ran on. yacc is actually older than C: Stephen Johnson wrote it to add an XOR operator to Richie’s B compiler, itself the middle stage between BCPL and C. yacc in turn was used to implement the first C compiler in C, the Portable C Compiler. While source mapping was, perhaps, less urgent, when both sources were sitting on a table in fan-fold paper, the problem has existed as long a C has, and C’s remedy, the #line directive, for nearly as long.

The #line directive is a funny one. Syntactically, it’s a preprocessor instruction, and the preprocessor is nominally a macro-expander: like #define, like #include, the output of the preprocessor is source code. But think about it: there’s no way to implement that functionality as transformation of the token stream. It’s actually a side channel, however it’s implemented⁴.

Zig has no preprocessor at all, which I consider, on the whole, to be a strength. It also has no other mechanism which could serve as a line directive, and that’s a bit unfortunate, being as it is otherwise a fairly nice compiler target.

It will not surprise you to learn that Lemon emits #line directives, just as Yacc does, and for the same reason. Since Zitron began as a complete port of Lemon, there was line directive code at the time when I began adapting it for Zig, and rather than remove it entirely, I made them into comments: // #line. It seemed like the conservative thing to do: there was already a flag to disable them, and it’s somewhat useful in the event that one has to dig into the generated code, for whatever reason.

In any event, Zig growing the ability to use line directives would solve less of the mismatch today than it did in the 20th century. Yes, compile errors would have more useful stack traces, but while this used to be our main feedback loop, we’ve grown accustomed to richer, tighter ones.

Adding a tree-sitter for Zitron gets familiar color on the screen, including the Zig code (which I considerably improved on a Zig tree-sitter fork), but working with it felt strangely inert.

Zig is lucky to have ZLS, which was a good language server when I started using the language, and is now a great one. The keystrokes for hover, go to definition, go to reference, autocompletion, are all deep in my muscle memory, and I’d type them working on a Zitron file and (of course) nothing happened.

Zitron is very fast. Milliseconds fast. Lemon is fast, and the port of Lemon to Zig is just enough faster (on the order of 2-3%) that I can be confident that it isn’t measurement error and wishful thinking: Zitron can no longer be compared like-for-like, but having done nothing to make it slow — it’s fast.

So I started thinking: what about just calling Zitron after every keystroke? Then giving the Zig to ZLS? Use the line comments to map between them, and write a middleware: to the editor, it’s a server, to ZLS, a client.

TL;DR, it works. I eventually started using ZLS in library form, rather than interprocess, to shave off a few cycles and add a trick or two; but Zitron is still called as a separate process, there being little reason not to.

The Build

Felicitously, the ZLS team packages the raw LSP-wielding library as a separate module. Loris has used this to make language servers for his ‘super’ variants of Markdown and HTML, so there was a solid proof of concept, along with ZLS itself.

That means the hard part was already done: I have both a library for the protocol, and a complete example of how to use it. I’m grateful to the ZLS team for those things, not to mention an excellent tool which I use for many hours of many days. Thanks.

The architecture is straightforward enough: maintain a Tree-sitter parse of the Zitron document, and, in the original implementation, run a ZLS client for Zig. Get a change, patch the document, patch the Tree-sitter, send it off to Zitron to get the Zig, diff the Zig to get the changes, send those to ZLS as edits.

Diffing the Zig proved to be a mistake. Muad’iff is fast, for diffing, but it’s still diffing. One pathological case which dogged me for some time is the very line numbers I’m using to map between the two source files, because adding or removing a single line, anywhere, changes all the downstream numbers, which then have to be diffed.

That problem I fixed by replacing the numbers with XXX. I had to find the numbers anyway to update the translation map, and the comments do not contribute to the language server in any way, so this was free. But there was another: any change to the grammar itself results in completely different tables, the numbers move around quite substantially, and this is a worst-case scenario for diffing.

The silly part is that this was purely wasted effort. It made sense to me that handing ZLS a minimum edit would be ‘efficient’, but it’s not: ZLS uses Zig itself to parse and reckon with code, and the Zig parser doesn’t know about deltas either, like Zitron, it’s just very fast.

Ultimately the solution was to use ZLS as a module, not as a standalone program, and just hand it the new version of the Zig code and let it do its thing. There’s a still a need to queue up work, and stall if edits are coming in very hot and heavy, but it’s never been a bottleneck.

zitron-ls works something like this:

Adding features was easy: the agent can just do that. But efficiency was not. That was a matter of continually riding herd, and insisting that the correct thing, rather than the easy, obvious thing, be done. This got slightly easier over time: LLMs are just pattern-matching, so as several examples of, say, binary search made it into the middleware layer, it stopped doing the next search as a naïve linear probe.

The central challenge is maintaining the translation map, updating and using it efficiently. Every update to the source code needs a traversal of the generated Zig, looking for the line numbers, to establish the baseline of the map, but there’s also column skew to be considered.

The %include and %code blocks can get quite large, and get to be special-cased, because they’re included into the generated Zig exactly. Most code isn’t: aliases or special $$ tokens need to be translated into the form they take in the parser, and that means diffing. They tend to be individually small, but not so small that zitron-ls can afford to rebuild them from scratch unless it absolutely has to.

It took a couple design iterations to find an approach which could work, and several passes at implementation before I actually got what I needed. pikchr.zy, which I translated specifically to stress-test the language server, is about 250KiB, and has more than a hundred diffable code blocks.

The biggest challenge by far is semantic tokens, which arrive in a format which is efficient on-the-wire, but uses relative coördinates in a way which is quite fiendish to map over from one side to the other. It doesn’t help that ZLS has no truck with ranges of semantic tokens, it literally sends all of them every time. It can get away with this, because both ZLS and the Zig parser are very fast, and while it might be nice to offer this mode to clients⁵, I’ve found ZLS plenty snappy in practice, so there’s no particular reason to do it.

But this does mean that zitron-ls, which does support semantic ranges for pure survival, needs to figure out what the client wants, ask ZLS for everything, find the part of “everything” which it actually needs, and then translate all of those over to Zitron coördinates and send them on.

Getting this snappy requires everything to work right: doing the Zitron-side updates in parallel with ZLS doing its thing, only ever re-diffing when changes demand it, pervasive binary search, and cursors both within the translation matrix and each diff, recording the position of the last request for information.

In the end I was able to get round-trip averages down to about 40ms, on an M1 Macbook. Which is a speedy and capable machine, but no longer cutting-edge. That’s on pikchr.zy, which is a single-file program and rather on the large side of what might be expected.

This response time is equivalent to five words per second, or 300 WPM. It’ll do. There’s probably some slack in there still, I may be able to get some milliseconds back from Zitron by making it into a library and retaining the intern pool between runs. One can always shave a few more cycles, in my experience; it’s more a question of when to stop.

For a tool like this, though, there’s a threshold effect: the goal is to be able to edit a Zitron file and not notice the language server. It’s just there, doing its thing. Depending on how much experience you may have using language servers for different languages, some of them achieve this (ZLS does, broadly speaking), some of them do not.

Computer-Aided Programming

I would be delighted if anyone other than myself were to use Zitron, and the tools I’ve written to make that experience pleasant. That might not happen. Parsers are pretty niche, after all, and at the moment, so is Zig: I used to think the latter would inevitably take its place in the mainstream of computer programming, and I’m not so confident of that anymore. It’s not like there are better options.

Even if it’s just me using it, that’s fine: I have plans for Zitron, and more that, I’m in it for the love of the game. My growing dismay at the behavior of Zig’s leadership doesn’t change the fact that it’s the best language I know of for the kind of program which needs a parser, and I like to write those.

I’m a tool maker by nature, and by profession when I can get away with it. I like parsers, and parsing engines, and the kind of developer tool which parses its inputs: that’s most of them, especially so when weighted by complexity.

zitron-ls would simply not exist if ‘agents’⁶ were not available. When I started the Zitron project, they weren’t; I knew I’d write a Tree-sitter for it⁷, but a language server was strictly in the “wouldn’t that be cool” bucket.

It’s the sheer tedium of the thing. Poring over the specification, reading and re-reading ZLS and the LSP library, writing basically boilerplate code, fixing the mistakes in it: I do this for fun, and that’s not the fun part of programming, not for me. It would take a long time, during which my spare capacity for writing source code, debugging, and generally thinking very hard about what I’m doing, would be largely taken up by the task. I’m more than capable of it, the opportunity cost is just too high.

But having had a couple month’s experience with Codex by that point, it was clear to me that these tools are capable enough that I could get it done, without pissing myself off, burning out, or spending an excessive amount of my precious time making it happen.

Somewhere in the process of working on it, I started to see the task as a sort of statement as well. Specifically I was stuck and frustrated: latency was unacceptably high, and the first attempt to fix that a) didn’t work and b) was simply the reversion of regressed code back to the state I had originally insisted that it be in. One of those pessimistic moments, where I wonder why I even bother.

Machinist was, at one time, a widespread, skilled, and well-paid profession. It still is, but it used to be as well. CNC, and CAM generally, have totally transformed the profession. It’s just not economically valuable, in most labor markets, to run a lathe for an eight-hour shift, making the same part every time.

That’s not really the job anymore: operating a CNC mill is also skilled, also widespread (although many of those jobs have moved), and still pays well, but per part manufactured, there aren’t nearly so many machinists. I’ve known a few retired machinists, who lived through that transition, and it wasn’t fun. Many of those lathes and mills have ended up in private shops, some were converted to CNC, but many were just scrapped.

It was a good job, and machinists are proud of their ability to operate their machines, justly so. Many were no longer needed, and plenty of those kept didn’t like their new job babysitting a robot.

Many were also missing a finger by the time they retired. CNC does offer new and exciting opportunities for workplace injury, but it’s safer, more efficient in materials and labor, and more productive. It’s a win for capital, a win for the consumer, a win for the CNC companies, and a win for the labor which made the transition. But it wasn’t a win for the workers who didn’t, and it hasn’t been kind to makers of fine manual machine tools, either.

The impact on the toolroom was different. CAM was adopted, of course, with all its advantages, but this was not a quick transformation, and is by no means complete. As a rule, CAM calls for more tool makers, rather than less, and it doesn’t change the nature of the profession: a tool maker is the automation, jobs are one-off or small, and the robot which can build and run the factory has yet to be built.

A tool maker is also rather more likely to run a manual mill or lathe than is a machinist on the shop floor. They frequently need just one of something, and it is often easier to make one piece directly, rather than designing it in CAD and running gcode.

There are still a lot more CNC machinists on the shop floor than there are tool makers in the toolroom. The analogy to what’s happening in software development isn’t perfect by any means, but that isn’t, in general, how analogies work. I expect this much of the analogy to programming will hold: software ain’t gonna write itself, a statement I’m not going to try and defend here. Perhaps another post.

Zitron, and zitron-ls, are a statement of confidence in the future. We’re going to need languages, and we’re going to need to navigate, alter, understand, and even *gasp* write software. I’m finding language servers more critical than ever: more of my time is spent reading, navigating, renaming, and otherwise relying on IDE features, and yes, less time writing code.

I don’t see that going away, and I see this as the best time in decades to write DSLs and other grammar-driven software. It isn’t obvious that CNC has resulted in fewer machinists⁸, but it has clearly lead to more machined stuff, including plenty of products which simply would not exist with only manual tools available. It has also meant that once-important specializations like lathe operator have become fairly scarce.

It happens that I like writing software with agents. It plays to my strengths, and that’s just my good luck, it won’t be true for everyone. To make another prediction which I won’t defend today, I expect the productivity gains which agents bring will fill latent demand for software, leading to more professionals working in the field than ever, rather than less. But people are going to lose their jobs, niches which have been cultivated over years and decades are going to lose value, and some programmers are going to go from doing something they like to something they don’t.

This has been the nature of automation since well before the first valve, and we’re in the automation business. It’s seldom fun, being on the receiving end of it, but speaking for myself, I’m grateful to live in an industrial civilization, which is nothing if not the sum of automation over time.

The phrases “computer programming” and “software development” have been synonymous, or nearly so. Perhaps that’s changing. Writing zitron-ls, if that is the verb, is certainly software development. It seems reasonable to also describe it as computer programming: there’s a program, and it runs on a computer, and it isn’t the product of spontaneous generation. But it’s also reasonable to reserve that word for the act of writing source code directly, a distinction (and practice!) which is well worth preserving, so I just don’t know.

I’m not entirely sure how to finish this post. When, against my better judgment, I find myself on social media, I am thunderstruck by the sheer intensity of the discourse surrounding this topic. Either computers have achieved perpetual motion and software just grows on trees now, or large language models are a crime against humanity, and I feel with great sincerity and conviction that anyone in either of those camps is a crazy person.

If your fingers are balling into fists right now, while your cheeks turn red, and you hold your breath, please: consider that you might be a little crazy. The Internet can do that! It’s not your fault: computers may not have been a mistake, even the Internet might not be a mistake, but social media, surely social media was a mistake.

I see no irony at all in writing a language server for a grammar DSL, using an agent to do so. If I thought either of software development or computer programming, as distinguished above, is or will be a thing of the past, then this would be futile. If I felt that the sad reactionary cult fighting a rearguard action against their very existence could possibly win, it would be unwise.

These are new and powerful tools, which change quickly, and which we barely know how to use. It’s no surprise that many get bad results from them. ‘Many’ includes me. Some sessions are just wasted, and I’m left scratching my head and wondering what I need to do differently.

It’s not always true that one needs to have experience using something in order to have informed opinions about it. You do not, in fact, have to smoke PCP, to conclude that smoking PCP is a bad idea. I do think that agent-driven development is a topic where inexperienced opinions are generally bad ones. I see too many ‘antis’ saying things which simply aren’t true anymore. That undermines the impact of any trenchant critique which is offered next to it, of which there remain many.

The state of the art is still moving rapidly, and it’s clearly the case that mountains of slop are showing up everywhere and clogging the gears of open source society, which has already been under strain for as long as I can remember. I find this simple to explain: it’s a new skill, using new tools which themselves change quickly, and the failure mode is different. The reason we see mountains of bad output is that mostly, we’re bad at using these tools, because we lack the experience to be good with them. This includes the poor social interactions, some of which I chalk up to investment-driven pushing to adopt new processes faster than wisdom would suggest, and most of which is just that humans are always bad at something until they aren’t.

zitron-ls is new. There are a few things it doesn’t do, yet, and if there are no bugs in it, then that would be the first time I ever released bug-free software and I will be appropriately astonished. I’m proud of it. I know of nothing like it. Those who insist that the result of agent whispering cannot be good software are wrong: they are wrong today, and they will not get less wrong over time. I hope they left some room to change their minds.

If you like writing parsers, and agree with me that Zig is the closest thing we have to an ideal language for them, give Zitron a try. I’m not done with it by any means, but it now has a complete modern toolchain: compiler, syntax highlighter, and language server.

I do. Like writing parsers, I mean. And parsing engines, and tools which are interacted with through a parser, ‘languages’ as we like to call them, and this perhaps less of an analogy than many English words we hackers borrow and transform into jargon. I genuinely think there may never have been a better time to practice this art, but that deserves its own post.