Contributing to Documentation

NorduGrid ARC documentation is mainly written in reStructuredText and build with Sphinx to HTML pages, LaTeX (for printable PDF versions) and ePub.

Recent source tree (master branch) build is available instantly via Coderefinery GitLab Pages and deployed to the nordugrid.org on nightly basis.

Everyone is welcome to contribute. For trivial fixes just click on the “Edit in GitLab” button and commmit the changes.

You need to have an account in the Coderefinery GitLab and be a member of the project to be allowed to edit. Contact us if you want to contribute, but are not a member yet.

Commiting and reviewing changes

Contribution is possible by direct push to the repo, no need for merge requests if you are fixing typos or other content that does not requires review!

Note

In the renderend HTML version there is a link at the bottom of each page labeled Edit page source on GitLab. Using this link you can edit the page source in your web-browser without the need to checkout source tree

In case you want to add something new that requires the review process:

  • create the new branch (in the same repository, no need to fork)

  • open a merge request to master branch

  • in-line and general discussion of the particular contribution is inside merge request.

Documentation structure

As reflected in the main index page, the documentation logically divided into the following groups:

Admins documentation is the most developed part for ARC. We aimed to have dedicated documents for each particular topic and then bound them in the following way:

  • The introductory /admins/try_arc that contains a zero-configuration case hands-on instruction to help new users and admins getting started with ARC CE.

  • Main ARC Computing Element Installation and Configuration Guide that contains general installation and configuration flow for production Computing Element deployment. The guide itself should be brief enough and holds only main examples. All detailed instruction for each particular subsystem configuration should be written in the dedicated document that linked to this guide.

  • ARC Configuration Reference Document automatically rendered from text version in the code source tree. Targeted for web-search. The ultimate configuration options description we have.

  • All other dedicated documents goes to Operating ARC CE Subsystems section in case someone wants to access them directly instead of following installation guides links.

Source tree directory structure

Sources of documents in rST format are placed into the source directory that contains the following structure:

  • _static - logo and CSS files [1]

  • _templates - Sphinx HTML theme layout tuning [1]

  • _extensions - Custom sphinx extensions developed for ARC [1]

  • common - directory used to store common for all ARC customers documentation, including admins, users and developes. This documents are hidden in the main TOC tree, instead they are linked from other various documents when needed.

    • repos - this directory holds repository configuration instructions for release, nightlies, etc. Main index page have repos pointers as well as Try ARC and Install Guide.

    • changelog - documents that describe changes between ARC versions.

  • admins - directory used to store Documentation for Infrastructure Admins. The main documents that represent the entry points to ARC Computing Element installation and configuration resides directly in this directory.

    • details - this subdirectory designed to hold detailed configuration instructions for ARC CE subsystems, like RunTime Environments in ARC that describe all aspects of RTEs in-depth. This documents intentionally moved deeper in the main TOC tree to prevent first levels flooding. If you can find a chapter in main install guide that cat point to the in-depth document in question - this is a right place to store it.

    • commands - place for automatically generated command line option reference for ARC tools. At the time of writing the Python tools represented there.

    • archery - for the documents that not related to Computing Element another subdirectory should be created. At the time of writing only ARCHERY documentation of that kind is available.

  • developers - directory used to store Documentation for Developers. As long as we don’t have many of this kind of documents everything that contains implementation details (how stuff coded, which variable used, etc) should go there.

  • tech - directory used to store Technical Documents Describing ARC Components.

  • users - directory used to store Documentation for Infrastructure Users. If document is user-oriented - place it here.

  • sdk - place for Doxygen SDK documentation integration to Sphinx build [1]

  • testing - directory used to store the documents for ARC 7 Testing Area.

  • wip - “work in progress” area hidden in the TOC tree. You can use it for incomplete documents that should not be publicly advertised yet, but still will be built and available via /wip/ URL in produced HTML files.

Index files

Each directory in the source tree contains index.rst file that is used to link other documents in the same TOC tree. Upper-level index.rst file contains references to index.rst files in the subdirectories, that in turns contains pointers to the other documents.

Note

When you add a new document, add a reference to the index.rst in the same directory

Just follow the index.rst chain (look for toctree keyword) starting from the source directory to get familiar with TOC linking structure.

Storing images

When you need to add images to your document you should upload the image file itself and refer to it from the .rst.

ARC6 documentation structure implies that image files are stored inside the images subdirectory in the document location. Then referencing is done by relative path, e.g:

.. figure:: images/shared_sessiondir_yes.svg

Building the docs

The top directory in the source tree contains build.sh script that:

  • checkout the ARC source code tree and configures it with packaging-like paths

  • builds Doxygen SDK documentation

  • converts arc.conf.reference to rST

  • copies documentation parts from ARC source code tree (for developers section)

  • prepare automatically generated documentation for CLI commands

  • builds HTML

  • builds PDF (with LaTeX)

  • builds ePub

On commit the GitLab CI configured to automatically invoke the build.sh to produce rendered documentation. Documentation archive is avaiable as CI job artifacts and for master branch is deployed to GitLab Pages.

To build the docs on your local machine you should have at least Sphinx installed. Additionally you should be able to configure ARC source tree and ARC Python command dependencies for auto-generated parts.

Complete and up-to-date list of dependencies defined for CI build and can be found in .gitlab-ci.yml.

PDF builds with LaTeX is the most heavy part from the both time and needed additional packages perspective. For HTML rendering local debugging it is recommended to use the html script argument the skips PDF and ePUB:

[user@localhost doc]$ ./build.sh html

Point your browser to file:///path/to/doc/build/html/index.html to view the HTML rendering locally.

Writing Documentation in reStructuredText

reStructuredText (reST) is the default plaintext markup language used by Sphinx. There is possible to render other markups but for consistency and better cross-referencing NorduGrid ARC6 documentation written solely in reStructuredText.

General Syntax

reStructuredText markup specification is well documented in the several sources and was designed to be a simple and readable in plain-text. Common text editors (including vim and emacs) recognize reST markup and provide syntax highlighting out of the box.

Start with reStructuredText Primer on Sphinx docs.

Complete reST Markup Syntax can be found on Docutils starting with Quick reStructuredText document.

Sphinx also uses interpreted text roles to insert semantic markup (cross-referencing, etc) into docuemnts. To get familiar read this document.

Just study the markup following the documentation and readinig the already written documents just here.

Code snippets

There are different options for representing the code, starting with the simple literal blocks identified with ::.

It is advised to use code-block:: directive to enable syntax highlighted rendering:

.. code-block:: ini

[gridftpd/jobs]
allowaccess = staticdn dnfromfile

In this example, the ini tag represents the syntax highlighting lexer identfier for Pygments. Look for available lexers to serve your needs. The most common for our docs are: console, bash, ini and cfg.

References

Referencing another parts of documentation is necessary to achive usability. The typical referencing cases are:

Using custom label

Create label just above any of the paragraph headers with. The following markup creates my_label label.

 .. _my_label:

My Heading
==========

Refering to a lable is possible from any other doument with:

read the :ref:`my_label`

In this case paragraph heading will be used for hyperlink text. If you want some custom text for hyperlink text use the following syntax:

read this :ref:`text <my_label>`

Referring arc.conf.reference

Autogenerater reST rendering of arc.conf.reference already contains labels for all configuration options and blocks that can be used.

The label name has the following structure:

  • reference_<block name>[_<sub_block>...]_<option name> represents configration option inside block

  • reference_<block name>[_<sub_block>...] represents block itself

For example:

In the ``arc.conf`` there is a dedicated :ref:`[lrms] <reference_lrms>` block that defines the type of your LRMS.
*Job session directory* is configured with :ref:`reference_arex_sessiondir` configurration option.

Referring bugz

Custom Sphinx plugin for ARC documentation introduces new reST roles that can be used to mention bugs in NorduGrid Bugzilla or GitLab Issues:

More details can be found in :bugz:`3802` and :issue:`57`.

Referencing docs

Referencing the the whole document is similar to using labels, but instead of label name the document name (filename without extension) is used with :doc: keyword.

Example 1: Refer to the try_arc.rst in the same source tree directory (relative path). Use the document header as hyperlink text:

:doc:`try_arc`

Example 2: Refer to the repository.rst by absolute path (starting from sources top directory). Use the custom hyperlink text:

:doc:`NorduGrid Repositories </common/repos/repository>`

Adding notes

To highlight statement visually use notes and warnings:

.. note::
   Zero configured A-REX comes with ARC REST and EMI-ES interfaces enabled.

.. warning::
   This information is valid for releases of ARC 6 starting from 6.0.

Adding images

In the ARC6 documentation images should be included with the caption. It is accomplished with the figure keyword.

After storing image inside images subdirectory, include it in reST document as:

.. figure:: images/shared_sessiondir_yes.svg
   :align: center
   :alt: Sessiondir is shared between ARC CE and WNs

   Sessiondir is shared between ARC CE and WNs. No local scratchdir defined.

Sphinx build had been configured with image format autocoversion feature. So you can use any image format, including vector graphics.

Graphviz

If you want to illustrate some process or structure that can be shown as graph, consider using built-in Graphviz functionality.

Simple usage examples can be found in admins/details/rtes.rst document within ARC documentation. Give it a try, it is easy.

Converting from other sources

Converting the old documentation parts from LaTeX or Mediawiki markdown to reST is not a magical process that do everything automatically unfortunalely.

However you can get a good start with a pandoc tool that do the conversion.

E.g. to convert LaTeX source simply run:

[user ~]$ pandoc -f latex -t rst acix.tex > acix.rst

The ammount of efforts to edit the resulted .rst file and fix formatting issues is completely depends on the source itself (e.g. how the console ouput was formatted in the origin document). Images and references in most cases should be fixed separately after the pandoc.