TWPRO-6918 - Documentation system

We need to have a system to document our code and our processes. This system should be easy to use and to maintain and to keep up to date, along with the code that it documents.

Technical requirements

(NOTE: this list of requirements usually goes into Jira Task description)

• We, as developers, should be able to write documentation in markdown and have it automatically published and nicely formatted.
• We, as reviewers, should be able to review the proposed documentation and suggest changes as we do for the code.
• Documentation should be written in a familiar format, so that we can easily write it and review it. Also markdown is good since it's a plain text format and it's easy to version control. Final point on this is that we already have documentation in this format in our wiki and we can reuse it.
• Documentation should be searchable and easy to navigate. GH wiki format doesn't work well because is not full text searchable and it has a flat document structure with no folders, nor categories, nor additional metadata.
• Browsing and build documentation should be easy and fast. We don't want to have to wait for a build to see the result of our changes.
• Documentation system should support charts and diagrams, and they should be written in a textual (and versiionable) format. MermaidJS is a good candidate for this.

Proposal

In this PR I'd like to introduce Retype as documentation system for our codebase. It's a proprietary system that works with plain markdown files that supports a superset of what is supported by Github Markdown but also has a lot of additional features like charts, diagrams, search, navigation, etc.

Layout of built documentation is very readable and it's easy to navigate and its embedded JS search engine is very fast and accurate.

It does support document categorization via folders and/or document metadata (embedded as YAML in the markdown file) and it's possible to add custom metadata to the document to add additional information.

So here's a quick todo list for this PR:

• Add a new folder docs to the root of the project
• Add a new configuration file retype.yml to the root of the project
• Add instruction on how to write and build documentation in the README.md file
• Add a Dockerfile to build documentation in a container (see Hosting doc)
• Add oauth authentication flow to the built doc (see auth)
• Add a new docs job to the CI pipeline (GH or drone?) to build documentation and publish it somewhere on our cluster
• Update pull request template to include a checklist for documentation and a link to the documentation system
• Define how to fill RFC documents (Create an RFC template and exclude it from documentation)
• Import and categorize existing documentation from the wiki
• Create one folder per team as base of documentation for that team (currently: core, cash, warehouse)

Here's an high level overview of the proposed solution:

Here's a state diagram for the task lifecycle:

Here's the flowchart to describe the process:

Existing documentation categorization

In order to have a starting point for the documentation we can use the existing categorization of the wiki, we should migrate it into the repository and improve it by grouping relevant documentation together. Also it could become a good starting point to have a clear and linear documentation for onboarding new developers.

Looking at current documents present on wiki I propose the following folders to group them:

• Onboarding: This should be the starting point for every new developer, it should contain all the information needed to get started with the project and to understand how it works. It should be a linear and easy to follow documentation.
• Architecture: This should contain all the information about the architecture of the project, how it works, how it's structured, how to add new features, how to add new services, etc.
• Snippet: Code snippets, pages that describes how to do something, like this one.

The first two categories should be the most important ones, the third one is a nice to have and it will act as kind of catch all for the rest of the documentation that's currently on our wiki.

Documentation authentication

Since we don't want our documentation to be public we need to add some kind of authentication to it. We can use a basic auth like other projects do, but I think that a better solution would be to use OAuth2 and integrate it with our existing authentication system.

For this reason we choose to implement authentication using this amazing oauth2-proxy project. It works out of the box as reverse proxy and/or static file server. It can be configured using environment variables and it supports multiple providers (Github, Google, etc).

We choose to build our doc as a static site and protect it with Google authentication bound to our company domain. This is implemented in /doc/Dockerfile and it's configured using environment variables at runtime.

:::code source="../Dockerfile" :::

The final image will require google authentication to access the documentation and then after a successful login it will serve our static doc.