commander

0.1.0

About Commander

Commander is a system implementing a server and a client component that allows you to remotely command a lisp system, query its state, manage debuggers, and so on.

If you're familiar with Emacs' Slime/Swank, this is the same idea, but based on a standardised and versioned protocol, and using standard portability libraries to deal with various implementations, rather than ad-hoc re/implementing all the needed features like Swank does.

Using as a Server

To run a server, load commander/server and launch a server instance: (org.shirakumo.commander.server:start T). The server will by default listen on localhost and port 4004. There's nothing else you need to do, the server will simply sit in the background and wait for remote connections.

By default the server will modify parts of the global environment, to be able to capture output and input from global streams, redirect debugger invocations, and so on. If you would like to prevent that from happening and "contain" the server, you can instead use org.shirakumo.commander.server:with-server.

By default also the server will listen to an arbitrary number of clients and not stop or quit when the last client exits. For a more typical editor environment, you will probably instead want to set org.shirakumo.commander.server:if-no-clients to :quit to make the implementation exit when the last client disconnects.

Using as a Client

To run a client, load commander/client. Often when you implement a client, you will want fine-grained control over how streams from the server are handled, what messages are sent, how results are displayed, and so on. As a result, the default org.shirakumo.commander.client:client does not handle most messages that the server can send, and won't send anything on its own other than the occasional ping to keep the connection alive.

If you want to run a headless client to command a remote server as part of a program however, you can make use of org.shirakumo.commander.client:headless-client. This will by default redirect all stream output to local streams, and give you a convenient org.shirakumo.commander.client:with-remote-eval to evaluate expressions on the server, but see the results on the client side:

(org.shirakumo.commander.client:with-client (client 'org.shirakumo.commander.client:headless-client)
  (org.shirakumo.commander.client:with-remote-eval (client)
    (format T "Hello there!~%")
    (finish-output)
    (sleep 1)
    (+ 1 1)))

Writing Your Own Client

If you would like to write your own client, you should subclass org.shirakumo.commander.client:client and implement message handlers via org.shirakumo.commander.client:define-handler. In the very least you should implement handlers for all the stream messages in the V1 protocol to ensure the input and output go where they need to in your interface. You can send messages to the server explicitly with org.shirakumo.commander.client:query, which will also take care of waiting for the reply from the server for you.

An example client that implements a basic REPL and debugger can be found in the commander/example system and org.shirakumo.commander.client.example package. It's a single file with a rather rudimentary REPL loop. It should however give you enough of an idea on how to start structuring your own client implementation.

Protocol Specification

The Commander wire protocol is versioned, but defines a couple of base concepts that all versions of the protocol inherit from.

The protocol is made up of "base types" and "structure types". The base types correspond to the following Common Lisp types:

string
Note that the element-type of strings does not need to be transmitted.
number
Note that this includes all numbers including bignums, complexes, and ratios.
character
null
keyword
cons
vector
As for strings, the element-type does not have to be transmitted, but obviously it is highly advantageous for it to be, to provide for efficient transmission. A client may impose an upper limit on the vector length before it is instead translated to a pointer (see below).

Objects of those types can be transmitted with any protocol version, and behave as one would expect. Note that general symbols are missing from this list. Instead of general symbols, the identifier type is introduced, which holds a name, package name, and a context identifier. This is done to avoid issues around interning and package management on either side of the wire. The identifier is a subtype of wireable, which identifies all "structure types".

Any structure type has a fixed set of typed slots, and is what forms the basis of all messages and other types that the protocol can transmit. Aside from identifier the following core protocol types exist:

message
Base for "top level" messages exchanged by peers. Only includes an id field, which is used to locally (on the sender side) uniquely identify the messages and associate them with a reply.
pointer
An opaque wrapper to identify complex objects on a remote peer. Only includes an id field, which is used to locally (on the creator side) uniquely identify the object. Note that the existence of a pointer does not guarantee that the object is kept live, as we cannot know the lifetime of pointers on peers.
reference
A pointer with extra display information for the user.

The protocol also defines the following terminology and behaviour:

"peer" A participant in the network
"server" The lisp process that is being remotely controlled
"client" A peer that connects to a server in order to issue commands to it.
"command" A message that requests the receiving peer to perform some kind of action.
"reply" A message that a peer sends in response to a command it received from another peer.

And that's about it for specifications about the general protocol. Everything else is protocol version dependent, including the wire format.

V1 (Version 1)

This protocol defines an octet-stream of messages. Every type of object on the wire is preceded by a single octet which serves as the type tag. The contents after the type tag are type dependent, and for wireables are simply the slots in order as defined by the slots of the type.

Particularly notable is that this version serialises strings as null-terminated UTF-8, and the following types using their packed binary representations, as well as vectors specialised on those types:

(unsigned-byte 64)
(unsigned-byte 32)
(unsigned-byte 16)
(unsigned-byte 8)
(signed-byte 64)
(signed-byte 32)
(signed-byte 16)
(signed-byte 8)
single-float
double-float

In general the V1 protocol should be reasonably efficient in terms of wire representation without the need for compression algorithms. For the particular type tags and layout of each type, please see the source files in commander/protocol/.

The V1 protocol also defines the following additional wire types:

restart
Represents a restart active during a debugger session.
frame
Represents a stack frame active during a debugger session.
thread
Represents a native thread.
compiler-note
Represents a single note resulting from a compile operation.
compile-result
Represents the results of a compile operation.
definition
Represents a global object definition for cross references and so on.

For messages, V1 mandates that every message be either derived from reply in which case it can only be sent in response to a message not derived from reply (a "command"). Every command must be answered with one of the following reply subtypes:

ok
The command succeeded.
failure
The command failed for some reason.
return
The command succeeded and produced a set of return values.

Each reply contains the original command's id in its reply-to field such that the peer can identify replies that appear out of order.

As an example, the following would be a valid sequence of exchange:

p1 -> p2 (v1:ping :id 1)
p2 -> p1 (v1:ok :id 0 :reply-to 1)

Connection Establishment

When a client first connects to a server, it must send a v1:connect command to the server. The server must then acknowledge the message, and subsequently send its own v1:connect command.

Once both v1:connects have been exchanged, the server and client may start sending commands to each other until either side sends a v1:disconnect command. The receiving side of a v1:disconnect must reply with an v1:ok, but both sides may already start tearing down the connection at that stage and do not have to wait for further messages or acknowledgement.

All peers must also implement timeout logic in order to automatically clean up broken connections to other peers.

Commands

The following commands may be sent by any peer:

ping
Just ask the target to respond with ok to keep the connection a live.
connect
Sent on first connection to verify each peer's version, name, and so on.
disconnect
Sent to terminate the connection.
eval
Sent to ask the peer to evaluate a lisp form. This may be rejected.
If successful, the response should be a return with the values returned by the evaluation.
compile
Sent to ask the peer to compile (and load) a lisp form or file. This may be rejected.
The response should be a return with a compile-result object.

The following commands may be sent only from a client to the server:

abort
Asks the server to try and interrupt another command that is currently in flight.
If successful replies with ok.
definitions
Ask the server to search for definitions matching the identifier, and return the results.
eval-in-frame
(only during a debugging session)
Asks the server to evaluate a form in the environment of a specific frame of the current stack.
If successful returns the values returned by the evaluation.
invoke-restart
(only during a debugging session)
Asks the server to invoke a specific restart.
If successful replies with ok before invoking the restart. This will likely also end the debugger session (via exit-debugger).
make-repl
Asks the server to create a new REPL context. The return will give the thread handle necessary to direct other commands to that REPL.
quit
Asks the server to stop and terminate entirely.
restart-frame
(only during a debugging session)
Asks the server to restart evaluation at the given stack frame.
If successful replies with ok before restarting the frame.
restarts
(only during a debugging session)
Asks the server to return a list of active restarts.
return-from-frame
(only during a debugging session)
Asks the server to return the specified values from the given stack frame.
If successful replies with ok before restarting the frame.
stack
(only during a debugging session)
Asks the server to return a list of stack frames.
stop
Asks the server to stop listening to other connections and exit, but leave its Lisp process alive.
threads
Asks the server to return a list of currently active threads.