Allow documents to be divided into chapters and sections

[atrigent]

The old thefinalclub website had a concept of "works" that were divided into "chapters" that were subdivided into "sections". This allowed large works (books, plays, etc) to be used on the website, because only a certain section would need to be shown to the user at any given time. Showing an entire work on a single page would require a lot of data transfer and would be a strain on the user's browser.

Annotation studio only has a concept of "documents". When creating a document, annotation studio allows the user to specify the code for chapter navigation, but does not store chapters in a structured way and requires the user to write HTML code for the navigation. An entire document will be displayed on a single page.

We need a structured way to break works down into chapters and sections. This will probably require messing with the existing models and creating some new ones.

[AndrewMagliozzi]

If by "will probably require messing with the existing models" you meant
*definitely, then I am totally on board with this ticket.

On Fri, May 16, 2014 at 3:45 PM, Ari Entlich notifications@github.comwrote:

The old thefinalclub website had a concept of "works" that were divided
into "chapters" that were subdivided into "sections". This allowed large
works (books, plays, etc) to be used on the website, because only a certain
section would need to be shown to the user at any given time. Showing an
entire work on a single page would require a lot of data transfer and would
be a strain on the user's browser.

Annotation studio only has a concept of "documents". When creating a
document, annotation studio allows the user to specify the code for chapter
navigation, but does not store chapters in a structured way and requires
the user to write HTML code for the navigation. An entire document will be
displayed on a single page.

We need a structured way to break works down into chapters and sections.
This will probably require messing with the existing models and creating
some new ones.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26
.

[atrigent]

Here's some analysis of how the old site is structured. There are three tables involved:

CREATE TABLE `works` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `title` text NOT NULL,
  `author` text NOT NULL,
  `summary` text NOT NULL,
  `year` int(11) NOT NULL,
  `page_views` bigint(20) unsigned NOT NULL,
  `wordpress_url` text NOT NULL,
  `intro_essay` mediumtext NOT NULL,
  `created_on` datetime NOT NULL,
  PRIMARY KEY (`id`),
  FULLTEXT KEY `author` (`author`),
  FULLTEXT KEY `title` (`title`)
) ENGINE=MyISAM AUTO_INCREMENT=391 DEFAULT CHARSET=latin1

CREATE TABLE `sections` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `work_id` int(10) unsigned NOT NULL,
  `order` int(10) unsigned NOT NULL,
  `name` text NOT NULL,
  PRIMARY KEY (`id`),
  KEY `work_id` (`work_id`)
) ENGINE=MyISAM AUTO_INCREMENT=19140 DEFAULT CHARSET=latin1

CREATE TABLE `content` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `section_id` int(10) unsigned NOT NULL,
  `content` mediumtext NOT NULL,
  PRIMARY KEY (`id`),
  KEY `section_id` (`section_id`)
) ENGINE=MyISAM AUTO_INCREMENT=18400 DEFAULT CHARSET=latin1

Each row in works encapsulates a whole work, and includes metadata such as author, publication year, etc.

Each row in sections denotes a subdivision of a work. The work_id column denotes the section's parent work. Most, but not all, works have sections with meaningful order values, denoting the order of sections within a work. For some works, all section order values are 0. One interesting thing is that chapters are also stored in this table.

Each row in content contains the text of a section. The section_id column denotes which section this is the content for. The way to distinguish between chapters and chapter subdivisions in the sections table seems to be whether that section has any content.

[atrigent]

The tree structure of chapters and subchapters seems to be derived from the names of the sections. For example, in the following sections from Hamlet (work id 5):

id	work_id	order	name
13	5	0	Act I
14	5	0	Act I, Scene I: Elsinore. A platform before the castle.
15	5	0	Act I, Scene II: A room of state in the castle.
16	5	0	Act I, Scene III: A room in Polonius' house.
17	5	0	Act I, Scene IV: The Platform.
18	5	0	Act I, Scene V: Another part of the platform.
19	5	0	Act II
20	5	0	Act II, Scene I: A room in Polonius' house
199	5	0	Act II, Scene II: A room in the castle.
202	5	0	Act III
204	5	0	Act III, Scene I: A room in the castle.
213	5	0	Act III, Scene II: A hall in the castle.
217	5	0	Act III, Scene III: A room in the castle.
224	5	0	Act III, Scene IV: The Queen's closet.
226	5	0	Act IV
229	5	0	Act IV , Scene I: A room in the castle.
233	5	0	Act IV , Scene II: Another room in the castle,
236	5	0	Act IV , Scene III: Another room in the castle.
240	5	0	Act IV , Scene IV: A plain in Denmark.
245	5	0	Act IV , Scene V: Elsinore. A room in the castle.
250	5	0	Act IV , Scene VI: Another room in the castle.
254	5	0	Act IV , Scene VII: Another room in the castle.
257	5	0	Act V
260	5	0	Act V , Scene I: A churchyard.
263	5	0	Act V , Scene II: A hall in the castle.

Act I, Scene I: Elsinore. A platform before the castle. is under Act I because it is prefixed with Act I and then a comma. The website seems to only visually indent the first sub-level of the tree.

[stephskardal]

https://github.com/stefankroes/ancestry - gem i've used a bit and consider as a possibility here

[stephskardal]

I don't feel I fully communicated what I was thinking yesterday in terms of a potential proposed change to the documents structure, so I want to communicate that better to get things ironed out.

What I propose is that section represents all the sections for a document. The structure would look something like this:

section (id: 1, ancestry: nil)
-- section (id: 2, ancestry: 1, sort: 1)
---- section (id: 3, ancestry: 1/2, sort: 1)
---- section (id: 4, ancestry: 1/2, sort: 2)
-- section (id: 5, ancestry: 1, sort: 2)
---- section (id: 6, ancestry: 1/5, sort: 1)
---- section (id: 7, ancestry: 1/5, sort: 2)

Then, instead of a documents model, introduce a Metadata model (subject to different name), which has many of the fields document has, plus a section_id field, as it would belong to a section. In my example above of a single document, there would only be one metadata entry that section_id is equal to 1 because it only belongs to that top item.

The benefit of my approach, IMO, is a two fold:
a) Only one model (section) fully represents the hierarchy of content
b) We can leverage an activeadmin nestable plugin to provide the admin interface for setting the hierarchy for documents (it wouldn't work with Document + Section model)

Another benefit of this approach is then, on the single document view page, we could call that top level section children to get those immediate children (ie section > section).

Thoughts?

[stephskardal]

I suppose we could also keep the existing documents data model as is and simply add a section_id to it so that it would be the acting "Metadata" model, and then we would create a root section instance for each document instance.

[atrigent]

I definitely agree that it makes the most sense for the root node of the ancestry tree to represent the entire work. I just did a quick google about having multiple models in an ancestry tree, and came up with this: stefankroes/ancestry#155 question on github. The suggestion given was to use single-table inheritance. Since all nodes would be in the same table, their ids would all come from the same "id-space", and ancestry would work as expected.

Here's a proposed class hierarchy. Let me know if I'm overthinking this. Names are subject to change, of course.

DocumentNode - id, title, ancestry
    Work - metadata about an entire work (author, etc)
    Section - sort number
        ContentSection - text content

So the root of the tree would be a Work, sections which include other sections, such as a chapter, would be a Section, and a section which contains content would be a ContentSection. I'm not sure if there's a way to enforce certain invariants, such as that Work has no parents, ContentSection has no children, etc. Perhaps this could be done with validations?

[stephskardal]

I don't see how that github question / answer / reference to STI helps solve what we are trying to do. The reference to STI is implying we are saving a type to specify what type the ancestry is, but I don't see how has_ancestry accommodates that type based on what I know of has_ancestry. Your proposed data model has 4 different models, which seems overly complex to me. I understand the separation between Section & ContentSection in that you are proposing that the single responsibility of the Section table be to store the ancestry, while all the other data for the content is stored elsewhere, but IMO, DocumentNode & Work don't need to be separate models.

I would go back to the following data model, to keep it most consistent with what we have now:
Models (2): Document, Section.

• Document contains all the meta data for a document.
• Section contains the ancestry, title, and content for each section.
• Upon creation of Document, a root Section instance is created. All children sections / hierarchy are tied to that Section.

I think there are really multiple routes to take here and there is no "best" answer, ie pros & cons to various approaches. I prefer the more simple approach as I've described because of how I know this will integrate with an admin interface and be manageable.

[atrigent]

My understanding is that the type of a row would specify the type of that row.

I made DocumentNode and Work separate so that only the root node would contain the metadata for the overall work.

[stephskardal]

Ok, sorry I mispoke about STI. Here's a quick blog post I referenced: http://maulanaruby.wordpress.com/2007/02/17/sti-vs-polymorphic-association/. Even if STI is used for has_ancestry, I'm still unclear as to how to use has_ancestry with it.

In your model, how is STI applied? I'm still not seeing it.

[stephskardal]

The approach / question I think I'm trying to get at here, is how important is it to utilize an existing Admin framework (activeadmin). The more complex, STI / polymorphism included, the more custom admin interface will need to be built out b/c I do not expect it to support a complex data model. In my mind, I see a huge value in leveraging an existing admin interface and erring on the side of simplicity.

[atrigent]

Ok, perhaps I'm making some assumptions here. Here's my understanding of how this would work. Let me know if I'm off-base.

Essentially, the methods that ancestry adds to a model just do various parsing and querying based on the ancestry paths. From these, it figures out an id or a list of ids that the user is interested in. Them I'm assuming that it would ask activerecord to fetch these ids from the database, and I'm assuming that activerecord would then instantiate the correct model based on the type field. ancestry would not have to have any idea what the type field means or which actual models are used for any of the objects it is fetching.

I don't think the admin interface would look much different with single-table inheritance. It doesn't look like the admin interface handles ancestry in any special way at the moment - it just treats the ancestry field as a text string. What would the admin interface not be able to handle that it handles now?

[stephskardal]

Here's a nestable plugin for activeadmin: https://github.com/nebirhos/activeadmin-sortable-tree.

Those are all correct assumptions, except that I don't believe has_ancestry provides the ability for you to specify which type of model it's pulling the instance from, meaning has_ancestry always looks in the same model for the id. This tells me that we cannot have a Section instance reference any other type of model as the root node.

[atrigent]

I was thinking that you would put the has_ancestry in the DocumentNode and then it would always do DocumentNode.find(num), for example, and activerecord would then instantiate the correct model.

Where I'm coming from with this more complex proposal is that I tend to like for the storage of the data to reflect its structure. This allows me to be more sure of the integrity of the data. The downside is, of course, the complexity, and also that changing the structure can be difficult if that becomes necessary.