how to hit index for packages?

[jurov]

Hello, it appears encoder does try to look up the package in the index, but I cannot get it to work. I intend to use cl-conspack for RPC and it would be nice if package is not transmitted all the time with the symbols. Perhaps passing package object instead of its name in #'encode-symbol might be a bug.

CL-USER> (cpk:encode 'a)
#(130 64 1 65 129 64 16 67 79 77 77 79 78 45 76 73 83 80 45 85 83 69 82)
CL-USER> (cpk:with-index (a) (cpk:encode 'a))
#(176) 
;; This is fine!!! 
;; But trying to index package, no cigar:
CL-USER> (cpk:with-index (common-lisp-user) (cpk:encode 'a))
#(130 64 1 65 129 64 16 67 79 77 77 79 78 45 76 73 83 80 45 85 83 69 82)
CL-USER> (cpk:with-index (:common-lisp-user) (cpk:encode 'a))
#(130 64 1 65 129 64 16 67 79 77 77 79 78 45 76 73 83 80 45 85 83 69 82)
CL-USER> (cpk:with-index ((symbol-package 'a)) (cpk:encode 'a))
#(130 64 1 65 129 64 16 67 79 77 77 79 78 45 76 73 83 80 45 85 83 69 82)

;;Even tried to paste the object using SLIME to no avail:
CL-USER> (cpk:with-index ( #<PACKAGE "COMMON-LISP-USER"> ) (cpk:encode 'a))
#(130 64 1 65 129 64 16 67 79 77 77 79 78 45 76 73 83 80 45 85 83 69 82)

[rpav]

This is due more to package-name not actually being a symbol:

> (package-name (find-package 'cl-user))
"COMMON-LISP-USER"
>

But strings (or packages) aren't indexed; only symbols.

There are probably a few ways to handle what you want to do. Adding strings to indexes is probably not the best idea, since it'd be ambiguous about how to handle equality, decoding, etc. First, note that conspack isn't a format, but an encoding: you don't necessarily have to have a 1:1 correspondence between CL types and your format. So you could:

• Use keywords, uninterned symbols, or even strings, if your package can always be predicted anyway.
• Negotiate an index, and even define a protocol for such.
• Use TRACKING-REFS, which will minimize but not eliminate package names.

The first option is a quick and easy solution; if you don't need packages, you don't save anything by sending encoded symbols. If you can't always predict your package, then you need to communicate the package somehow anyway, so even a predefined index of packages doesn't really solve the problem.

With the second option, you could actually provide a query protocol that would send an enumeration of symbols you could then use as an index. This could be as dynamic as you want, e.g. sending a protocol version in each message, and replying with protocol-is-outdated message should something change etc etc. If you're going to send tons of otherwise-small messages, this is possibly ideal for minimizing size. A few small messages or very large messages, and the package name is probably not going to make much difference.

The final option is the simplest and most dynamic, of course.

[jurov]

Thanks for quick and thorough reply. As I wrote, it's for RPC so all used packages can be predicted in advance. I came up with this patch which seems to work fine - if symbol has a package, its name is interned as a keyword and put into the index as such. I can then use (cpk:with-index (:common-lisp-user) ...) No change is needed at decoder. If it is acceptable (or as an option) will make pull request.

index 42e30c0..18ed7d3 100644
--- a/src/encode.lisp
+++ b/src/encode.lisp
@@ -163,7 +163,10 @@
   (encode-header (symbol-header value) buffer fixed-header)
   (encode-ref-or-value (symbol-name value) buffer)
   (unless (keywordp value)
-    (encode-ref-or-value (symbol-package value) buffer)))
+    (encode-ref-or-value
+     (and (symbol-package value)
+         (intern (package-name (symbol-package value)) "KEYWORD"))
+     buffer)))
 
 (defun encode-character (value buffer &optional fixed-header)
   (let ((bytes (trivial-utf-8:string-to-utf-8-bytes (string value))))

[rpav]

Sorry for the late response, this is probably fine, I'll change this along with amending SPEC to make package take a string designator for package, much like CL itself. Most package names end up interned as keywords at some point anyway, I think. I suppose one could check to see if it's already interned, though.