Issues With The Lojban Formal Grammar
It is my opinion, and that of others in the Lojban community,
that the available grammars for Lojban are not sufficiently
formalized. Please note that this page makes essentially no
reference to the YACC grammar because of its unreadability; the YACC
and BNF grammars are assumed to be equivalent.
Points we are concerned about:
- The BNF used is non-standard. In places it is very
non-standard. This hinders attempts to formally analyze
- The morphology is not formalized at all. All of the
terminal productions are hand-waved and, in practice, handled by
code which thus far has been separately written by everyone to
produce a program that parses Lojban.
- The elidable terminators are not formalized at all. The
elidable terminators, again, have been handled by code which
thus far has been separately written by everyone to produce a
program that parses Lojban.
I, for one, feel that it is extremely misleading to say
that Lojban is formally parseable while this state of affairs
exists, so I'm trying to fix it.
In order of relevance; older stuff is farther down.
- camxes, the Rats!
based PEG parser itself, as a Java JAR file. Please do not ask
me for help on running it; I'm very bad with Java.
- The by-hand modified PEG grammar
for camxes, which is mostly my work. See the "Changes Made To
The PEG Grammar" section for what I've done to it over the
automatically generated version. The current version of the
morphology to go with this, which is mostly xorxes' work, can be
the BPFK morphology page.
Note that the morphology does not have to be in a separate file
and, in fact, the two files are merged before processing, but
all of the things in the morphology file are done first to make
the grammar itself easier to read. There's a bit of interface
code between the main grammar and the morphology; some of it is
in the main grammar itself, and some is in the morphology header file.
- An old version of the by-hand modified PEG morphology
- The folder for the Rats! parser
generated from the PEG above. Please do not ask me how to make
it run; I am very, very bad with Java. The Howto.cook file explains how to build
it; anything not referenced there is probably irrelevant. The
command line I use to run it from that directory is
"/usr/local/java/bin/java -Xss64m -jar lojban_peg_parser.jar",
with the sentence to parse passed on standard in. I suggest
piping it to a pager such as "less" or "more" as it is currently
set in debug mode and hence produces a lot of output.
Help making the java portions less stupid would be
- I have recently updated the PEG
to Rats! converter to have it produce a Rats! grammar that
automatically generate a (very ugly) parse tree. The Perl code
is even uglier than the parse tree. It doesn't deal well with
anything that could be considered lexing; all such productions
should either have "NORATS" at the start of the line or be after
the line "; --- NORATS ---" in the grammar.
- A set of test sentences for
testing changes to the PEG grammar (by comparison to the
official parser and jbofihe). Contains (in order) a bunch of
test sentences I used for testing pre-processor tokens, all the
example sentences from the refgram, all of Alice (one paragraph
per line) and all the IRC logs as of the end of March 2004.
Total is 34 thousand lines. Lines marked with "-- GOOD" are
those that I have carefully examined and determined to be valid
Lojban; those marked with "-- BAD" are invalid Lojban. These
two are generally only used when one or more parsers gets it
- The automatically generated
PEG version. This version is programatically generated from
the expanded ABNF version.
- The perl program that did the BNF
to ABNF conversion.
- An expanded ABNF version. The
only differences between this and the file above are the
addition of productions for the various selma'o and a few
example brivla and cmene.
- An ABNF version of the grammar file.
All changes were made programmatically, and should make no
- The perl program that did the ABNF
to PEG conversion.
- The original bnf.300 grammar file.
- A discussion between me and John Cowan
about the elidable terminators issue. As of 10 Feb 2004, no
real conclusions were reached.
- A link to the ABNF standard,
AKA RFC 2234. ABNF is very widely used in various RFC
I've decided CFGs (and hence BNF in any form, let alone yacc) are
simply not the right formalism for Lojban. See "Old Approach" at the
bottom of this page for an explanation as to why. I thought we were
stuck with them, though, and that the only other option for clean,
elegant formalism was a full-on context-sensitive grammar
I was wrong. I recently found Parsing
Expression Grammars, which seems perfect for what Lojban needs.
Note that that's "((Parsing Expression) Grammars)", not "(Parsing
(Expression Grammars))". I am currently working on a PEG for
Lojban. The inital version, which is just an automated conversion
of the BNF with some morphological information added, already parses
most of Lojban!
Please note that while I am not aiming for "bug-for-bug"
compatilibilty with either the current official parser or
the grammar definition in grammar.300, I am trying to make sure that
differences only occur in areas covered by the preprocessor section
of grammar.300, which was very much limited by what YACC was able to
handle, rather than how the Reference Grammar said the language
worked or what a listener or speaker would expect.
I've tried to do as much as I can programatically, because makes
it easier to convince people that what I produce is equivalent to
the original grammar. The BNF is converted to ABNF, some very
simple productions are added, and the result is converted to
Unfortunately, a fair amount of by-hand work is then required.
For one thing, the PEG grammar requires writing productions to
lexically break up the input. For another, PEG grammars are
sensitive to the order of elements, preferring earlier options, and
the BNF has several places where taking the earliest option that
matches is guaranteed to fail later.
Actual testing is done by converting the PEG grammar to a form
suitable for use by a PEG parser generator (of which I am aware of
which generates Haskell parsers, and Rats!, which
generates Java parsers). I've been using Rats!, due to having
problems with Pappy.
- 'si' handling is quite different in the way it interacts
with "lo'u" and "zo". Current rule: 'si' is ignored in
"lo'u...le'u", but a string of 'si' *afterwards* is honored one
word per 'si', including 'si' and 'zo', because neither have any
special power in the lo'u clause. As an interesting side
effect, "lo'u mi le'u si lo'u mi le'u" works. It means "lo'u mi
lo'u mi le'u".
- BU handling actually works. It doesn't in the official parser as of 29
Mar 2004 ("bu bu broda" passes, and "ky bu bu broda" fails).
- '.y.' is completely ignored; the only way it can interact with
the rest of the grammar is as "zo .y." or as ".y. bu". It can
actually have more than one y in a row (i.e. ".yyyyy."), because
I at least often use it that way on IRC.
- Multiple BAhE in a row are allowed. It is assumed this will
be used for emphasis. Note that the current official parser
also accepts this, even though it should not according to
- "su" can be backed out of with "si", allowing a speaker to
save themselves from a potentially crushing mistake.
- "!" and "?" are treated as white space. Probably other
things should be added to that list (q and w?).
- Groups of "si" and everything up to a "sa" are both erased
at the beginning of a string. This may or may not be
justifiable according to grammer.300; no-one's really
sure. This means that sentences like "si si si" and "sa"
are legal, as well as sentences like "le broda sa .i mi cusku".
- SA and SI now interact in a more obvious fashion. For
example, "le broda brode brodi .y. sa le si la broda brode
brodi" is equivalent to "la broda brode brodi". Just using "sa"
would not work because "le" and "lo" are in different
Interactions between ZOI, SI, and SA are much
richer. The goal is to achieve something more like what a user
would 'expect', given the basic definitions of those words.
- The first SI after the close of a ZOI clause erases the
closing delimiter, allowing one to add to the protected
text. "zoi gy weeble gy si bob gy" is equivalent to "zoi gy
weeble bob gy".
Two consecutive SI after the close of a
ZOI erases the non-Lojban text itself; while it would
theoretcially be possible to have consecutive SI after the
close of a ZOI erase individual words inside the ZOI
protected text, this is a bad idea because (for example)
breaking up a bird call into words makes very little
So, for example, "zoi gy da da da gy si si de gy" is
equivalent to "zoi gy de gy".
- The interaction of these two features leads to a
somewhat strange, but very minor, side effect: It is
impossible to add to the protected text inside a
zoi clause (i.e. using a single SI after the closing
delimiter) any text that starts with "si" (unless it then
goes on to be something that looks like a Lojban brivla or
cmene), because it will be interpreted as two SI, causing
erasure of the entire protected text.
- Three consecutive SI after the close of a ZOI erases
everything but the ZOI itself, so that, for example, "zoi gy
da da da gy si si si dy weeble dy" is equivalent to "zoi dy
- Four consecutive SI after the close of a ZOI erases the
entire ZOI clause, including the ZOI.
- Similarily, after ZO+word, a single SI deletes the word,
causing the next word to be caught by the ZO, but two SI
delets both the word and ZO.
- Because of the SA and SI interaction enhancements, the
fast way to delete and accidental ZOI is to close the
delimiter and say "sa zoi si", and then continue on. For
example, "broda zoi gy da da da da gy sa zoi si da" is
equivalent to "broda da".
- Multiple zei are handeled in a different order;
historically, "broda zei zei broda" was "(broda zei zei) broda"
and "zei zei broda" was invalid. In my parser, it's "broda (zei
zei broda)", and "zei zei broda" is "zei type-of broda". This
was accidental at first, but it was pointed out that with the
old way it was essentially impossible to say "zei type-of
lujvo", which this fixes.
- None of si, sa, su, y, or zei are allowed as zoi
delimiters, since delimiters are not scarce so it doesn't make
sense to block the more useful interpretation.
- Multiple sa in a row delete back to further previous
instances of that selma'o. For example, "le le broda cu brode
sa sa le brodi" is the same as "le brodi".
- Y is ignored anywhere except in front of BU.
- As a very special case, ZOI SA clauses can accept arbitrary
strings, to handle things like "zoi foo booz foo co si si WEEB!
foo dysa zoi bar baz bar" (which, in case you're wondering is
equivalent to "zoi bar baz bar").
- Allows things like "byfy doi mark cu broda", whereas before
a boi would have been required after "byfy".
- Allows "free" in many more places.
- Allows things like ".i fi'o broda bo mi klama".
- Allows "lo broda joi lo broda" without a ku before the joi.
- Error reporting is essentially non-existant. This may not
By-Hand Changes Made To The PEG Grammar
This section enumerates the changes that were made to the PEG
grammar starting from the automatically generated version.
If an entry looks like "* [number][letter] -- [stuff]", the
number and letter are a reference to a section of the pre-processing
guide in grammar.300. It means that that section is intended to
implement (or help implement) that rule.
Fixing up of things like (NAI+)? into NAI*, and removing extraneous (...).
Change of = to
3 -- Fixed the 'text' production to ignore everything after fa'o
Left-factored rp-expression (this is a syntactic change only).
NOTE: I'm not completely certain I did this correctly,
so please take a look if you're into this sort of thing. The
rp-expression <- rp-operand rp-operand operator
rp-operand <- operand / rp-expression
rp-expression <- (operand / operand rp-operand operator) rp-operand operator
rp-operand <- operand / rp-expression
Re-ordered some selma'o productions due to how PEGs work. For
FA <- "fa" / "fe" / "fi" / "fo" / "fu" / "fai" / "fi'a"
won't work because the 'fa' will match first, even if the word is actually
'fai'. Same with 'fi' and "fi'a". So it was re-ordered to:
FA <- "fai" / "fa" / "fe" / "fo" / "fu" / "fi'a" / "fi"
Added productions 'Spacing' and 'Spaces', for handling
4c -- Added 'post-cmavo', which goes after every cmavo string in every
selma'o. post-cmavo only accepts a string if it is not followed
by a member of BU. It also requires the string to be followed
by 'post-cmavo-spacing', which was also added.
post-cmavo-spacing allows an optional trailing '.' and either
some spaces or another cmavo. This is probably an approximation,
and will likely need to be reviewed when stricter morphology is
Added some basic morphology constructions, as follows. Please
note that these are preliminary and know imperfect. For
example, "la fo''''o" is perfectly acceptable to these
- consonant and vowel
- other-letter, which is ['y,.]
- lojban-letter, whichi s any of the above
- cmene-letter, which is lojban-letter plus upper-case
CMAVO (which contained just a list of all selma'o) was moved
to 'known-cmavo'. CMAVO became either known-cmavo or a
consonant or '.', followed by a vowel, followed by any
number of single-quote vowel pairs or vowels or both,
followed by cmavo-spacing.
CMENE is now an optional ".", one or more cmene-letters, a
consonant, and spacing.
BRIVLA is now basically a tester for consonant in the first
5 characters and ending with a vowel.
4e -- Added (UI NAI?)+ to the end of 'spaces'. This allows any word,
basically, to be followed by any number of UI or UI-NAI pairs.
2a, not working -- removed the ZOI production from sumti-6, as
this can't be made to work without a pre-processor.
2b -- In the ABNF version I changed 'any-word' to be a brivla or
a cmene or a cmavo. This, along with the alread extant "ZO
any-word" rule, handles zo.
2c -- added lohu-tail to handle lo'u...[first le'u]
2e -- Added 'si-clause' to 'spaces', which takes any nesting
"word si" pairs. Also tweaked ZO and LOhU so that they refuse
to process SI in this way. This required making a copy of
'any-word' that wouldn't handle SI clauses, and a fair bit of
tweaking of LOhU. Also added si-clause as on option to the
beginning of text. 'si' seems to be working with lo'u.
-- incomple; "zo si si mi".
- 4c -- Re-ordered sumti-6 to start with ZO, LOhU, and LU productions, in
that order. This fixed BU interaction with LOhU.
- 4c -- Re-ordered fragment to put 'terms' out in front, to fix ".abu"
(and probably other BU problems)
- 4c -- Re-ordered sumti-6 to put BU just before LU.
- 2b,2e -- Added a SI clause to zo, allowing things like "zo si .y. si
fi" to do the thing a human would expect.
- 4e -- Added Y to all the spaces functions, so that the example above
*actually* works. Created "absorb-indicators" to do this. Scattered "Y*"
liberally throughout the grammar, so it's ignored basically
- 4c -- Made "indicator" not work before BU. Allowed Y
without BU at the beginning of text as a free token.
- Added '.' to the spaces functions, so it's treated just like
' '. Much easier. Removed the leading '.' from relevant cmavo.
This necessitated changes to the CMAVO and CMENE productions.
- 4e -- Added a special case to allow "zo y" and "y bu" to
work ("y" is ignored everywhere else).
- 4e -- added "NAI CAI?", DAhO, FUhO and FUhE to
- 4a,others? -- Added a second option to 'sentence', which
contains only 'bridi-tail', so ZEI will work properly in cases
where the first word could be a sumti.
- 4a -- moved ZEI productions to the front of
- 4b -- Added 'pre-cmavo' to all selma'o except BAhE, SI and
BU. Put BAhE in pre-cmavo. Aded a special case for "BAhE
- 2g -- Added "su-clause" to the beginning of text, to handles
starting SU clauses.
- 2g -- Added "su-clause" after NIhO and TUhE; LU and TO
contain 'text' already.
- 2f -- Added "[selma'o]-sa-clause" to *every* selma'o, along
with "[selmaho]-no-SA-handling", and any-word-no-SA-handling and
- Reordered indicators & free in text to have "indicators
free+" be first.
- Added 'text-1' and 'paragraphs?' to text-1 to match the
YACC grammar (bug in the BNF).
- Added a second clause to statement-2 and
statement-3 so that sentences with statement clauses would be
- Fixed a parenthesis error in sumti-6.
- Made the morphology a bit more sane.
- Reordered space-interval to make the longest option come
- Added an option to text without the CMENE eater, so that
"bab zei bab", for example, works at the start of text, instead
of having the first 'bab' eaten by 'text' itself.
- More morphology fixes.
- Fixed up reverse polish notation again; seems to actually work
- Re-ordered vocative to handles "coi doi".
- Re-ordered 'text' massively to have joiks not
necessarily get eaten. Also made the 'paragraphs' in
text-1 not optional, but made text-1 optional in 'text' (which
was the same thing).
- Added "!gek" to "term" to allow things like "mi pu gi
[stuff] gi [stuff]" to not try to treat the second term as the
start of a tensed sumti.
- Added "!(stag? BO) !(stag? KE)" to bridi-tail-1 to have it
not eat giheks at every possible opportunity.
- Added "!MOI" to quantifier to allow "pamoi" and such to
- Fixed BRIVLA to not end in 'y' and CMENE to not stop just
before a 'y'.
- Added a check to not match cmavo if they are *immediately*
followed by a string of non-spaces ending in a consonant.
- Explained to the two occurences of lerfu-string that aren't
followed by MOI that if they are followed by MOI they
- Fixed a translation bug; at some point the optionality got
dropped from the first "NUhU free*" in termset.
- Fixed a bug in text-1: instead of allowing ijek [text-1]
xor paragraphs, it required both.
- Allowed ! and ? as spaces.
- Prioritized sumti-tail to not have a quantifier, so "le pa
roi broda" would work.
- Added su-clause to NIhO cases that didn't have it.
- Added a special case to allow 'su' to be the first word in a
- Expanded text and text-1 to more accurately match
grammar.300, and to have better preferential behaviour.
- Re-ordered simple-tense-modal to prefer ((time space? /
space time?) CAhA).
- Tried to clean up the morphology, once again, by moving
morphology checks to before words.
- Minor, cosmetic changes to make things work better with
- Implementation of 'si' and 'sa' at the beginning of
- Fixed zo so that zo + any-word is itself a possible outcome
of any-word, so that "zo irk zei broda" works, for example.
- ZOI handling added using semantic predicates and parser
actions (and a few changes to peg2rats.pl).
- Added EOF and not cases to "text". There might be a better
way of handing the 12 (!) productions that "text" now has, but I
don't know what it is. Broke out parts of "text" as
sub-productions as part of this.
- Implemented multiple "sa" handling, such that each extra sa
takes things back to an early instance of the following selma'o.
As a side effect, there can be any number of sa at the beginning
of a string.
- Cleaned up space handling a bit.
- Re-ordered "fragment" to maximally prefer prenexes.
- Various tweaking of magic word handling. BU handling is now
inline with BPFK decisions as of 16 Jun 2004.
- Various re-ordering, re-naming and addition of tags to help
with automated implementation of various parser features.
- Fixed a bug where "lu na jo li'u" would be read as "lu
fragment( na ) [elidede li'u] joik-jek [errore]". This only
affected "NA JA" inside a quoting structure that referred back
- Fixed a bug where "la cmen zei [anything]" would fail due to
"la cmen" being preferred.
- Moved indicators around so they would be visible without -b.
- Fixed a bug with ba'e and indicators.
- Change (BOI free*)? in sumti-6 to BOI? free*. This allows
things like "byfy doi mark cu broda", whereas before a boi would
have been required after "byfy". In the case of conflict, "MAI"
- Added "!BU" after BRIVLA, so the bare sentence "slaka bu"
- Stopped ZEI from absorbing indicators.
- Fixed a bug in lerfu-string + MAI handling where "le me
bypyfyky moi" would break into "(le mi bypyfy) (ky moi)", rather
than lerfu-string being greedy (thus causing failure).
- Change (BOI free*)? to BOI? free* in the other few places it
- Fixed FUhE to not absorb indicators (doesn't work too well
- Added indictor absorbtion to bu-clause.
- In tanru-unit-2, changed
"ME free* sumti (MEhU free*)? (MOI free*)?"
"ME free* (sumti / lerfu-string) (MEhU free*)? (MOI free*)?"
because that seems to have been the intention, and it was broken
by me scattering !MOI in various lerfu-string situations.
Changed all things like (TERM free*)? to TERM? free*.
This should allow free in many more places. It seems to have
made no changes WRT the test suite, but is still a somewhat
experimental change. This change took place is RCS version
1.26. Note that it has now been thoroughly tested by comparing
the tree output of the two versions for everything in
test-sentences.txt; no changes were found except for in
sentences exploiting the new stag functionality (see below).
- Also made stag equivalent to tag, which allows things like
".i fi'o broda bo mi klama". This is rather less experimental;
we understand pretty well what it does. Same change number.
- Fixed up a very subtle interaction bug between normal ZOI
and ZOI in a si clause.
- Seperated morphology into a separate file,
- Many changes to cause Magic Words handling to conform to the
current BPFK proposal (as of 30 Nov 2004).
- Added !selbri-1 after tag in term to stop "mi bai klama"
from being read as "mi bai ku klama". -- NOT SUFFICIENT; breaks
"mi broda lo nu brodo da ca brode".
- Discovered unnecessary complexity in statement-2;
streamlined to match the BNF after no reason for it could be
- Turned all (ek / joik) into joik-ek.
- Broke mex-forethought and fore-operands out of mex-2 for
clearer parse trees.
For a while I was trying to adjust the BNF grammar (after
conversion to ABNF) to do The Right Thing with respect to elidable
terminators, because that's the hard part. I have since come to the
conclusion that elidable terminators can merely be made optional if
longest-match disambiguation is used, but that puts us in the realm
of specifying the parser again, which is what I was trying to
I still think that one could probably get BNF to do The Right
Thing with respect to elidable terminators, but it would be Very,
VERY hard. I would be surprised if it could be done
without expanding the grammar by a factor of 20 or so. No, that's
not an exaggeration or a joke.
Update 20 Sep 2005: I am no longer so sure that it's possible,
but I don't have any good reason; just a change in my gut feeling. I
also think a 20 times size increase is probably conservative.
To give you a sense of what I mean, consider fixing 'kei'. This
requires having the grammar descending from a NU clause to eat all
brivla it sees until the next kei. Because BNF is inherently
ambiguous, forcing this requires that every place where two brivla
could occur next to each other be re-written`to only form two
separate selbri when there is a kei between them, but only inside a
NU clause. If this is possible in BNF/CFGs, and I'm not totally
certain it is, it requires nearly doubling the size of the grammar
because you have to have everything under 'subsentence' copied into
a "[foo]_during_NU" form, or whatever.
When you're done with that, try another big elidable terminator,
like 'ku'. This will require the same thing, but the ku additions
to the grammar and the nu additions to the grammar must work nested,
in either order. That's two more complete sets, not including the
'ku' or 'kei' sets. You now have a grammar on the order of four
times the original size, and you've fixed only two elidable
Good luck; let me know when you're done.