Mr. Doc Next

[iwatakeshi]

@FGRibreau

I don't plan to start working on this any time soon but I am definitely interested in making a blueprint for the next Mr. Doc. This will definitely require some help because I'm pretty much limited in the following field. To begin, I'll just put my thoughts on the table and see where we can go from here.

My thoughts:

• Create a parser that is language agnostic and pluggable (we'll focus on Javascript and CoffeeScript but others can extend the parser's capabilities for other languages).
• As mentioned in Theme output in docs folder #93, I think Mr. Doc depends too much on dox.
  • dox isn't very active
  • dox's documentation sucks (oh the irony)
  • Personally, I don't understand how dox works or even how parsing works in general but it's been in my interest in creating one. So far I somewhat understand how lexing/scanning works but I haven't had time to look into parsing so if we create one for Mr. Doc, it will be a plus for me.
• Create our own standard which will be a subset of JSDoc and Closure compiler
• Add additional template compilers besides Jade. This will bring more web developers to enhance the themes.

The "Next" just means "the next version of Mr. Doc" which could be 4.0.0 or 5.0.0 depending on what happens after 3.0.0.

[FGRibreau]

Indeed, if we create an intermediate language (just like LLVM does) we will be able to plug any language parser to it and still use the same frontend :)

[iwatakeshi]

👍 Yeah, LLVM might be something to look into. I'll be going over their tutorial.
Would you have any thoughts of your own on how we could implement this? :)

[gr2m]

Thanks for all your efforts! I think automated documentation from source is one of these things that don't have a good solution yet. We use doxx all over, but are not happy with, so I'm glad you address the problems. I'm happy to help reach out to find more contributors / help creating the new parser. I'm pretty sure many people have the same struggles, and moving away from dox sounds like a good idea at this point

[iwatakeshi]

@gr2m Thanks for the feedback! I would really appreciate any help! : ) As mentioned above, this is beyond my realm but would love to learn. It would be great to have a few contributors to put some thought into this so that we can come up with a blueprint/spec that outlines how to build a language agnostic documenter that isn't completely complex and extendable (such as in the languages to support and the themes). Let me know if there is anything I can do for you ;)

[gr2m]

I'd put the parsers into their own npm modules. I'd probably start mr-doc with a parser for JavaScript only and make it work, then think about a plugabble API as a second step.

From my experience, I wouldn't try to come up with a full blueprint/spec and make it fully language agnostic, instead I'd think of what a first useful mr-doc version would look like which aligns with that goal in the long term, then build the first version and build upon it

[iwatakeshi]

@gr2m Good point. We'll have to create a repo for it...maybe name it core and have the tokenizer and parser in the same repo but under separate folders?

JS will definitely be the focus. I don't plan to build a plugin for other languages but will leave it to others who want to extend the core. We'll have to make sure that we don't mix things in the core that is JS specific.

[iwatakeshi]

@gr2m Any thoughts?

[iwatakeshi]

@gr2m In the meantime, I just wrote a small gist that shows a basic usage of the scanner.
https://gist.github.com/iwatakeshi/d1973038c1417443749f

[FGRibreau]

@iwatakeshi If we go this way, we should use for http://esprima.org/ and then convert the AST into our own intermediate language (IL) to support other languages as well.

I know writing a parser/lexer is a fun thing to do but it's really hard to get it right that's why project like esprima exists :)

[gr2m]

I'm super sorry @iwatakeshi, I'm still catching up with a trillion things right now after vacation, but I did not forget about you. I need to give this some more thoughts, I'll get back to you next week, pinky promise

[iwatakeshi]

@FGRibreau I've looked into esprima but it's Javascript specific correct? I wouldn't mind using esprima or acorn as a guide line for building an AST though. I prefer the way babel outputs the AST (see http://astexplorer.net/ and change the parser to babylon) because it includes the comments. :)

I do agree that we could just use esprima but again I think that is Javascript specific and I would like to extend beyond Javascript by using plugins. Its true that its a hard project, but if we do it as a team, keep things simple, and take our time then I'm sure we can get it right without worrying about difficulty. ;)

@gr2m No worries, this project isn't in a big rush. I've just released 3.0.0 so it will be a long ways until we release the next big major update. ;)

@FGRibreau @gr2m Check out the gist and try it with babel-node and let me know what you think.