As often happens with a project of this nature, this blog has been ‘almost’ ready to publish for a couple of months. Just sitting here on my laptop, legible to no one but myself.
So I’m brushing off the last of the sawdust, filling out the parts I’ve been putting off, and kicking it out the door.
The main thing I left out of the initial release of zitron-ls was a formatter. This wasn’t an accident, grammar files have a particular sort of conventional layout and it took time to come up with a rule-based system which would look right.
This even affects the Zig code: for instance, it’s not uncommon to put two or three short actions on a single line following a rule, here’s an example from pikchr:
boolproperty ::= LARROW. { p.cur.larrow = 1; p.cur.rarrow = 0; }
Formatting the Zig AST wants to put those two on their own lines: good for Zig, not as good for Zitron.
Once I had worked out a vocabulary and decision criteria for the formatting, Astra had it working in an hour: ten minutes of asking me questions, 25 minutes of planning, 33 minutes of implementation.
Followed by several rounds of tweaking, of course. I called it a day when I liked how pikchr.zy is laid out after formatting, more than I liked it before. YMMV, of course, in which case:
// zitron fmt: off
Knock yourself out!
The format functionality is implemented as a standalone-ish module, which uses the Zitron tree-sitter. It seems less than traditional for formatting to be tied directly to a language server, so I wanted to keep my options open.
In classic yacc, one can use characters directly as tokens: '+' is a valid token name. Bison extends this with strings, so you can do ":=" if you’d like.
These are accomplished with hacks, basically. The character one ain’t so bad, at least back in the day: '+' is just a way of saying 0x2b, depending on locale naturally, and since it’s just a number, a lexer can return it as such.
This kind of sucks anyway, because the parser table wants to be dense: 0 for the accept action, then terminals, then nonterminals, without skipping anything. So a tax is paid: yacc translates these oh-so-convenient character tokens down into the numbers which its parser needs to see.
The Bison strings are even worse, because there’s no obvious way to turn ":=" into a number, and it doesn’t provide a non-obvious way. Instead, you get to tell it what you mean: %token ASSIGN ":=", so that lexer can read an enum named ASSIGN and return it when appropriate. Window-dressing, basically.
Lemon has never supported this. Terminals are given numbers in the order in which they appear, and are referred to using the CONSTANT_CASE which is good C style for an enum. As such there’s no translation, no epicycles where every glyph still needs a name and now you have two ways to refer to it.
The %token directive in Lemon gives the author control over the order of token assignment, which has some value: related tokens grouped into a sequence can be tested for as a range, for example.
It was the natural choice for Zitron to follow this example, and make a Zig enum out of the tokens. This does deviate from the official style: the token ASSIGN ends up as the enum literal .ASSIGN, when the style guide would prefer .assign. This is ok in my book, and comes with certain advantages: a token WHILE can be the enum .WHILE, but in “good style” would have to be .@"while". The grammar’s grammar is case sensitive, I entirely refuse to make nonterminals capitalized, and the tortuous mechanism which I would need, in order to end up with a lowercased enum, is a perversity which I refuse to countenance.
That’s the thing though: it’s Zig, .@":=" is a cromulent enum literal, so why not support string tokens as well?
Well, now we do. My suggestion is to use them only for strings which zig fmt would not un-at, so to speak. Zitron will accept and use "smol_token" as a token, and refer to it as .@"smol_token" - However, this would be formatted into just smol_token by the formatter, and I have put no effort into preserving this quirk in the language server.
This little nicety was always intended to obviate the need to invent names for glyphs. A production like
expr ::= LP expr RP.
Is better written
expr ::= "(" expr ")".
As now it can be.
Single-quoted terminals are not supported. There is no equivalent enum literal form, so this would be a mere complication, without any gain in expressive power.
The Zitron parser has a more tolerant concept of a string than the Zig parser happens to. Of note, newlines and tabs may be directly embedded in a string, this will be passed directly to the enum, and the emitted Zig will not be valid. The new formatter would replace the direct tab with four spaces, as it happens; this may or may not improve the situation.
The original parser would have interpreted the sequence "\"" as a string containing \, followed by a (presumably unterminated) second string. This is no longer the case. So long as you stick to the subset of this string syntax which Zig will understand, the expected will result.