Eclector: A highly configurable Common Lisp reader

Introduction

Motivation

The Common Lisp reader

• The function cl:read, special variables, readtable, reader macros
• Typically used for a REPL, the file compiler or de-serializing data

Other uses cases for a reader add new requirements:

• Parsing an editor buffer
  • Recovering from errors
  • Incremental operation
  • Controlling side effects, interning, literal creation, etc.
  • Source location tracking and reporting of skipped material
• Reading target code for cross compilation
  • Avoid creating/interning target packages/symbols in the host
  • Difference in representation between host and target, e.g. of floating point numbers

Existing Common Lisp Readers

• Often designed for
  • use in a REPL, file-compiler, de-serializing data
  • high performance
  • fixed set of extensions
• Generally not designed to
  • recover from any error in a fully automatic way (which makes sense for the evaluator and file-compiler use cases)
  • track source locations and make it accessible to clients
  • provide skipped material such as comments, reader conditionals, read-time evaluated expressions to the client

Eclector

Project Origin and History

• Started by Robert Strandh as the reader for SICL
  • Initial goal: conforming Common Lisp reader implementation
• Extracted from SICL (and named) by Robert Strandh
  • New goal: portable and extensible reader implementation
  • The name: Eclector → Common Lisp, Eclector → reader
• Developed over the last 10 years by Jan Moringen
  • Completed conforming reader
  • Conditions and error recovery
  • Optimized
  • Made extensible
  • Documented

Project Goals

Create a reader library that

1. is written in portable Common Lisp
2. conforms to the Common Lisp specification
3. provides distinct condition types with translatable error messages for all possible syntax errors
4. can recover from all possible syntax errors while representing the invalid input as closely as possible
5. is highly extensible and customizable beyond what is possible with the standard special variables and readtable
6. supports source location tracking and parse result construction, including the representation of skipped material
7. achieves acceptable performance for most clients and use cases

Project Structure

Eclector is a portable Common Lisp reader library

License

2-clause BSD license

Systems and Dependencies

• eclector ⟶ alexandria, acclimation, closer-mop
• eclector-concrete-syntax-tree ⟶ concrete-syntax-tree
• eclector.syntax-extensions ⟶ no external dependencies

Size

About 5,000 lines of non-test code + 5,000 lines of test code

Tests

• Unit tests and random tests with close to 100 % coverage
• Work-in-progress integration with ansi-tests

Documentation

texinfo-based reference manual

Features

Specific Error Conditions

At the character level, there are many possible ways of violating the specified Common Lisp syntax:

• (1 2 3 While reading list, expected the character ) when input ended.   ⟝   ECLECTOR.READER:UNTERMINATED-LIST
• ::A symbol token must not start with two package markers as in ::name.foo   ⟝   ECLECTOR.READER:TWO-PACKAGE-MARKERS-MUST-NOT-BE-FIRST
• #\HyperUnrecognized character name: "Hyper"   ⟝   ECLECTOR.READTABLE:UNKNOWN-MACRO-SUB-CHARACTER
• #10R89bThe character b is not a digit in base 10.76   ⟝   ECLECTOR.READER:DIGIT-EXPECTED
• `(:foo ,)An object must follow a unquote.   ⟝   ECLECTOR.READER:OBJECT-MUST-FOLLOW-UNQUOTE

To produce good error messages for as many of those as possible, Eclector can detect around 115 specific kinds of syntax errors for which it is has corresponding condition types.

Error Recovery

Applications like editors, IDEs and static analyzers must incorrect and incomplete source code:

(defun foo (x y)
  (+ x #b002The character 2 is not a digit in base 2.0101 (code-char #\RetrnUnrecognized character name: "Retrn")	Avoid tab.'(,Unquote not inside backquote.(frob::barDo not use unexported symbols. y)) #1Reference to undefined label #1#.#)
 While reading list, expected the character ) when input ended.

The restart and function eclector.reader:recover can be used to recover and continue reading after (hopefully) all syntax errors:

(handler-bind ((error (lambda (condition)
                        (let ((restart (find-restart 'eclector.reader:recover)))
                          (format t "Recovering from~20T~A~%Using~20T~A~2%"
                                  condition restart))
                        (eclector.reader:recover))))
  (print (eclector.reader:read-from-string "`(::foo ,)")))

Recovering from     A symbol token must not start with two package markers as
		       in ::name.
Using               Treat the character as if it had been escaped.

Recovering from     An object must follow a unquote.
Using               Use NIL in place of the missing object.


(ECLECTOR.READER:QUASIQUOTE (:FOO (ECLECTOR.READER:UNQUOTE NIL)))

Source Location Information

• Eclector can track source locations in a customizable way
• Default representation: stream position; clients can customize this to e.g. line and column
• Source location information is used
  • in error conditions signaled by Eclector
  • by clients which receive parse results

(let ((client (make-instance 'eclector.parse-result.test::list-result-clientDo not use unexported symbols.)))
  (eclector.parse-result:read-from-string client "(1 #|foo|# \"foo\" 2)"))

(:RESULT (1 #1="foo" 2) :CHILDREN
 ((:RESULT 1 :CHILDREN NIL :SOURCE (1 . 2))
  (:REASON :BLOCK-COMMENT :CHILDREN NIL :SOURCE (3 . 10))
  (:RESULT #1# :CHILDREN NIL :SOURCE (11 . 16))
  (:RESULT 2 :CHILDREN NIL :SOURCE (17 . 18)))
 :SOURCE (0 . 19))
19
NIL

Parse Results and Skipped Input

Consider an input with some skipped material:

;; foo bar
(#+bar baz 1 :foo)

Resulting parse results (concrete syntax tree):

#<SEMICOLON-COMMENT-WAD abs:0[0],0 -> 1,0>
├─#<WORD-WAD rel:0[0],3 -> 0,6>
└─#<WORD-WAD rel:0[0],7 -> 0,10>
#<CONS-WAD-WITH-EXTRA-CHILDREN abs:1[1],0 -> 1,18>
├─#<SKIPPED-POSITIVE-CONDITIONAL-WAD rel:0[1],1 -> 0,10>
│ ├─#<ATOM-WAD-WITH-EXTRA-CHILDREN rel:0[1],3 -> 0,6 raw: #<INCREMENTALIST:EXISTING-SYMBOL-TOKEN [#1=KEYWORD]BAR {120AF8F753}>>
│ │ └─#<WORD-WAD rel:0[1],3 -> 0,6>
│ └─#<READ-SUPPRESS-WAD rel:0[1],7 -> 0,10>
├─#<ATOM-WAD rel:0[1],11 -> 0,12 raw: 1>
├─#<ATOM-WAD-WITH-EXTRA-CHILDREN rel:0[1],13 -> 0,17 raw: #<INCREMENTALIST:EXISTING-SYMBOL-TOKEN [#1#]:FOO {120BF8F6E3}>>
│ ├─#<PUNCTUATION-WAD rel:0[1],13 -> 0,14>
│ └─#<WORD-WAD rel:0[1],14 -> 0,17>
└─#<ERROR-WAD rel:0[1],11 -> 0,12 condition: INVALID-SYNTAX-ERROR>

The example uses the incrementalist incremental parsing library which is based on Eclector and the technique described in I. A. Durand and R. Strandh. (2018) Incremental Parsing of Common Lisp Code.

Protocols

The Client Parameter

Operations performed by the reader are expressed as protocols in which generic functions accept a client parameter:

Reader Behavior Protocol

Protocols control the behavior of the reader, including aspects that are not customizable with standard Common Lisp readers. Examples:

• Handling of whitespace and skipped material

  (defgeneric eclector.reader:read-maybe-nothing
      (client input-stream eof-error-p eof-value))
  
  (defgeneric eclector.reader:note-skipped-input
      (client input-stream reason))
  

• Interpretation of tokens and in particular symbols

  (defgeneric eclector.reader:interpret-token
      (client input-stream token escape-ranges))
  
  (defgeneric eclector.reader:interpret-symbol-token
      (client input-stream token package-marker-1 package-marker-2))
  

• Read-time evaluation

  (defgeneric eclector.reader:evaluate-feature-expression
      (client feature-expression))
  

Reader State Protocol

Allows clients to control aspects of the reader state

Examples of reader state aspects:

• current package (cl:*package*), current radix (cl:*read-base*)
• current readtable (cl:*readtable*)
• consing dot allowed (non-standard)
• backquote state (non-standard)

Example of generic functions in the reader state protocol:

(defgeneric eclector.reader:state-value (client aspect))

(defgeneric (setf eclector.reader:state-value) (new-value client aspect))

Use cases

• Incremental parsing, static analysis
• Cross compilation

Labeled Objects Protocol

The labeled objects protocol controls the processing of labeled object definitions (#1=…) and references (#1Reference to undefined label #1#.#). It consists of two parts:

1. A state machine for labeled objects

   

2. Generic functions for fixing up circular objects with the least amount of work
   • Enables customization to support e.g. fixup in hash-tables

Applications

Clients

SICL and Clasp

use Eclector as the Common Lisp reader

Coalton language

uses Eclector a the basis for a source-tracking reader

Staple documentation system

uses Eclector for example source highlighting and cross-referencing

formgrep

S-expression-aware searching

Mallet

Linting

Second Climacs

uses Eclector a the basis for incremental parsing, syntax highlighting, expression-based editing, paredit, syntax checking, basis for static analysis etc.

common-lisp-jupyter

uses Eclector for completion

Syntax Highlighting

Performance

Performance: Results

Versions: flexi-streams-20241012-git, screamer-20210807-git

Conclusion

Future Work

• Object lookup and construction protocol: packages, symbols, (floating point) numbers, etc.
  • Use quaviver for number construction
• Finer-grained parse results
• Customizable quasiquotation
• Customizable character traits
• Performance improvements
• Syntax extensions
  • Extended package prefix (::(EXPRESSION)) ✅
  • S-expression comment (#2; SKIPPED₁ SKIPPED₂ EXPRESSION …) ✅
  • Rational float syntax (1.234R2)
  • Thousand separator (1_000_000)
  • Hash-table literals (#H(:test …)(key₁ value₁ …), #{key₁ value₁ …})
  • Unicode character names
  • Extended array literals

Thank You for Your Attention!

Eclector Resources

Code

https://github.com/s-expressionists/eclector

Documentation

https://s-expressionists.github.io/Eclector/

This presentation

https://s-expressionists.github.io/Eclector/european-lisp-symposium-2026-presentation-slides/slides.html

IRC

• Libera IRC network, channels #sicl and #commonlisp
• Jan Moringen is scymtym and Robert Strandh is beach