Feedback for basic table operations

[lluchs]

Hi!

As a followup to #659, I ended up doing some basic JSON work using Rye tables (though not what I originally added to-json\lines for). Here's the script I wrote. It takes some users from a JSON file and adds them to a Pocket ID server via its API.

members: Read-all Open %members.json |parse-json |to-table
	|columns? { 'name 'firstname 'lastname 'mail }
	|add-column 'username { firstname lastname } {
		concat3
			firstname
			""
			substring lastname 0 1
	}

users: ref members
users .rename-column! "name" "displayName"
users .rename-column! "firstname" "firstName"
users .rename-column! "lastname" "lastName"
users .rename-column! "mail" "email"

; users |display

api-key: ""

pocket-id-request: fn { path method body } {
	join { "https://id.example.com/api/" path }
	|to-uri .Request method body
	|Header! 'X-API-KEY api-key
}

; existing-users: pocket-id-request "users" 'GET ""
; 	|Call |Read-body |parse-json |-> "data" |to-table
; 
; existing-users |display

; email-diff: difference
; 	users .deref .column? 'email
; 	existing-users .column? 'email
; 
; email-diff |display
; users .deref .where-in 'email email-diff |display

for users {
	::user ,
	pocket-id-request "users" 'POST to-json user
		|Call |Read-body |print
}

Things that I did not find in the documentation while writing the script:

• I somehow missed ++ for concatenating strings and URLs. The documentation for + should point to ++ (and is wrong in suggesting that + works for strings etc. anyways).
• concat and concat3 are somehow in different parts of the documenation, making me miss concat for two strings.
• It took me a while to understand that the IO functions operate on the special path values / URLs.
• I had to grep in the source code to find to-uri for building the request URLs. I now see that ++ would be the easier solution for that.

Table API:

• Why does columns? end with ?? Isn't that more for a "getter" type functionality?
• I wish renaming columns was easier. Maybe columns? could learn some tricks from Julia DataFrame select(). Something like tbl .columns? { 'name :displayName 'firstname :firstName } to filter and (optionally) rename in one go?
• In the commented-out part I tried to filter out users that already have an account. The where-in unfortunately didn't work (don't remember exactly why, probably it doesn't accept a list?) and I would have needed where-not-in anyways.
• Is there no plain where that takes a code block like a normal filter operation?

I did have fun writing the script, though, so I'll probably use Rye again in the future :)

One more thing that came up while I was working on #659: I was confused regarding which functions take blocks verbatim (e.g., list, dict) and which function evaluate the items in the block (e.g., join). I'd argue that list and dict should behave like join, because I don't see why one would want a list of words rather than the values named by these words?

[refaktor]

Hi, cool script and great observations. Just some quick responses:

• ++ is quite new. It's part of tightening up type rules, + (add) is for math and ++ (concat) is for collections. This came with other type-related changes, such as if now accepting only booleans.
  You must be explicit if is-empty list { } or if is-zero number { } instead of relying on not always obvious truthy/falsy rules (related: https://www.reddit.com/r/ryelang/comments/1k50j4v/add_vs_concat_and_constants_by_default/).
  But yes, there is still + in many examples. I went through README, front page, Meet Rye so far and incorporated this and other changes (uploaded today). Next is Cookbook and examples/ folder.
• I see, concat is under collections as it accepts strings, blocks and lists, but concat3 currently just accepts strings. I will unify that, thanks.
• There is a lot of IO functions, some accept file-schema (url) - like REBOL, some accept reader interface or are more inspired by underlying Go functions ... this is area where I did a lot of exprimenting so far and I will
  at some point try to clear the picture and make things more uniform / consistent.
• to-type is Rebol's pattern, there is still dilemma if we would use to-type, for example to-string (converter) or just string (noun - type) as sort of constructor. that accepts multiple types.
  I see these are still not displayed in reference, so I understand it was harder to find. Concat (or ++) accepts many Rye types and returns same type so if you concat to a Uri it you get a Uri back yes.

Table API:

• noun? at the end is a convention for get-noun (https://ryelang.org/meet_rye/soft_values/conventions/)
• rename! is just low level imperative solution for mutating direct columns. What you propose is more in line in what I want and actually could be implemented right as you proposed it, so Nice :) We already have a conversion dialect
  which uses (left-set-words) for similar meaning. This will get implemented.
• where-in for now accepts just blocks yes, It would make sense for it to also accept lists so it will be added. I added where-not-in yesterday
• General where shold be there, but I also don't find it in code right now. There is example of it in cookbook (https://ryelang.org/cookbook/working-with/tables/#filtering-rows - last line of code). I will bring it back.

Other:
join/list/dict/... , you noticed well. This is (or was) a dilemma yes, which functions should evaluate expressions in block and which take it as it is requiring vals (reduce in Rebol). But one change from this year changed this.
Now { } are literal (pasive) blocks, but [ ] evaluates expressions. So now it will probably make sense that no function evaluates by itself (except vals) because you can just use [ ] instead of { } for the same effect.

name: "Jim"
; all these are the same
a: vals { 1 + 2 name }
b: [ 1 + 2 name ]
c: { 3 "Jim" }
probe [ a = b b = c a = c ]
; prints
;  [Block: ^[Boolean: true] [Boolean: true] [Boolean: true] ]

I will do some updates and also try to fine tune your example. Thanks for giving me material for thought and good proposals.

[lluchs]

noun? at the end is a convention for get-noun

Right, that's a good convention - I just somehow didn't think of the columns? operation as a "get" operation, since it's a function that somewhat transforms one table to another. But I see that it does also "get" a subset of the table, so it's probably fine.

Now { } are literal (pasive) blocks, but [ ] evaluates expressions

Aha, that is very useful. I noticed that [ ] also creates blocks, but did not understand the difference. The examples for list and dict should probably use [ ] by default then since that is more likely what people want.

[refaktor]

I was mostly working on adding MQTT, but I've also upgraded columns? as you proposed. I wanted to test your code in concrete example but I saw Pocket is closing down. This would be your code now ... haven't been able to test it all together, but partially.

Read %members.json |parse-json .to-table
|columns? { 
    'name :displayName 'firstname :firstName 
    'lastname :lastName 'mail :email 
}
|add-column 'username { firstName lastName } {   
    firstName ++ first lastName 
} :users

api-key: "****"

; active argument is in Rye first if possible
pocket-id-request: fn { body path method } {
	https://id.example.com/api/ ++ path
	|Request method body
	|Header! 'X-API-KEY api-key
}

for users {
    .to-json .pocket-id-request "users" 'POST
	|Call .Read-body 
    |print
}

Only change for this is upgraded columns?, it's already pushed to main, tomorrow I will make a binaries release of latest changes.

Thanks so far for this ... any more feedback or discussion or examples of what you would like to do are welcome.