As a data consumer, I would like to get aggregate statistics about the entire data set so that I can make more informed comparisons

[shawnbot]

One need that we have for the Ed project is to get aggregate statistics, e.g. the median graduation rate for all schools. I'm imagining an API endpoint that you could access like so:

/:endpoint/stats?fields=graduation_rate

When you need more than one field or statistic, as we will in our project, you could just comma-separate them:

/:endpoint/stats?fields=graduation_rate,net_cost

[shawnbot]

The Elasticsearch "stats" aggregation looks promising!

[shawnbot]

As a bonus, it'd be nice to be able to specify which aggregations get calculated:

/:endpoint/stats?fields=x&metrics=min,max

[siruguri]

Proposed spec for result: if URL is /cities/stats?fields=land_area,population&metrics=avg (using column headers from the sample cities.csv file)

it should return

 {"stats": [{field:"population", results: ["<average of pop>"]}, {fields: "area", result: ["<average of area>"]}]}

Thoughts?

[shawnbot]

Sorry I'm late on responding to this, @siruguri. I think it'd be great if the JSON response was all objects rather than arrays, so you could address each stat directly via, say stats.population.avg:

{
  "stats": {
    "land_area": {
      "avg": 49762.5
    },
    "population": {
      "avg": 684310
    }
  }
}

[siruguri]

I take it that you would like to be able to do stats on subsets of the index, so it'd be nice to retain the search call syntax and somehow extend it to include stats.

I didn't want to code hastily before thinking through the design a bit. There seem to be two approaches - have all possible new endpoints by default behave like search, with additional parameters specific to that endpoint, OR have only one endpoint per index as you have it currently but have a namespaced set of arguments in the Rails format that essentially act like you had an advanced search interface with dropdowns/checkboxes etc.

The former makes for simpler parameter structure, but seems to have various problems with extensibility - for one, it means that each endpoint will have its own reserved keywords, progressively constraining index column names; and two, composing multiple actions will be hard.

So in this case, the examples would look like (for all cities in Texas):

1. /cities/stats?state=TX&fields=land_area,population&metric=avg
2. /cities?state=TX&postprocess_command[name]=stats&postprocess_command[args][fields][]=land_area&postprocess_command[args][fields][]=population&postprocess_command[args][metric]=avg

A non-routes based option is to have a POST body with a query language of some sort in the body.

Thoughts?

[ultrasaurus]

how about:

/cities/stats?state=TX&fields=land_area,population&options[metric]=avg

then we could move to having other options (like paging and sorting) use the same kind of syntax and we wouldn't pollute the namespace

[siruguri]

That works ... I'll try to get to this in the next week or so.

On Tue, Aug 18, 2015 at 7:51 PM, Sarah Allen notifications@github.com
wrote:

how about:

/cities/stats?state=TX&fields=land_area,population&options[metric]=avg

then we could move to having other options (like paging and sorting) use
the same kind of syntax and we wouldn't pollute the namespace

—
Reply to this email directly or view it on GitHub
#63 (comment).

[ultrasaurus]

awesome!

[siruguri]

fyi, just started on this - figured out how to allow for the optional /stats/ parameter in the routes definitions... gonna look into integrating the aggs parameter that @shawnbot mentioned earlier - looking like Stretchy will support it.

[siruguri]

@ultrasaurus - and more generally I assume you mean:

/cities/stats?state=TX&fields=land_area,population&options[metric]=avg,max

so you get back all given aggregations for all given fields.

Also, fyi - I will for now silently ignore unknown aggregation names if ElasticSearch doesn't already do that.

[ultrasaurus]

@siruguri so, what are the implications for this? will elasticsearch calculate these dynamically and cache them? or is this something that needs to happen at index time for every field (which probably doesn't make sense?

Thinking more about how we use the API, we may want something more like

/cities?state=TX&options[fields]=land_area,population&options[stats]=population[max,min,avg]

that would allow the display of a chart with the right axis labels and a line that shows how cities in Texas compare to the national average. Not sure it's the best syntax, just trying to convey the direction... the idea is that the result might look like:

total: 40
stats: {
    population: {
        max: 756666
        min: 25
        avg: 3453
   }
}
results: [
    { ....

cc @yozlet @shawnbot