NORDUGRID-MEMO-11

ARC code repository Howto (cheat-sheet)

Getting access to ARC code repository

ARC code is kept in the Subversion (a.k.a. SVN) repository. Consult the online book Version control with Subversion for details how to work with it in general.

The repository can be read or checked out anonymously via a variety of SVN client tools or through web interface.

You may subscribe to SVN update notifications.

To get write-access to the repository, write to , and explain in some details what and why you'd like to contribute.

You need a X.509 credentials (issued by an IGTF-approved authority) in order to write to ARC code repository. If You are developer from Grid community then probably You already have one. Send your X.509 certificate subject name to

Working with ARC SVN

If you are not familiar with SVN, please read appropriate documentation.

You may use any SVN client tool: either a command-line one, or any existing GUI (TortoiseSVN, RapidSVN are tested; kdesvn does not seem to like pkcs12 certificates).

The repository root is located at https://svn.nordugrid.org/repos/nordugrid/ and its web interface at http://svn.nordugrid.org/trac/nordugrid/browser/

Several modules are available:

Each module is split into has trunk>, branches and tags trees. To check out a current trunk using common command line svn tool do e.g.:

svn co https://svn.nordugrid.org/repos/nordugrid/arc1/trunk arc1-trunk


NB: Anonymous check out from http://svn.nordugrid.org/... (note http versus https) is also possible, but you will not be able to commit later unless You switch to https.


Most likely, you will need to configure your SVN client to locate your certificate. If you are using svn command line tool locate servers files in your SVN client configuration folder (e.g. ~/.subversion/servers) and add the following:

[groups]
nordugrid = svn.nordugrid.org

[nordugrid]
ssl-authority-files = /etc/grid-security/certificates/1f0e8352.0
ssl-client-cert-file = /home/john/.globus/usercert.p12

Here 1f0e8352.0 is your issuer CA certificate, and usercert.p12 is your Grid certificate in pkcs12 format (see e.g. "Grid Certificate Mini How-to" on how to create it). Adjust those values according to your setup. If You are using different SVN client please refer to documentation of your tool.

Commit Policy

It is good practice not to go and change code in other peoples domain without consulting them first.

Trunk is considered the development mainline and should be kept reasonably stable at all times. If you want to perform large changes or experimental development, this work should happend in a separate branch. Braches are created branch inside branches tree. To have branch for your work created (or get write access to branches), please contact When the work is considered suitable for distribution branch can be merged into trunk. For more information about trunks, branches, and merging between them, see the Redbook (link at top of the page). Alternatively if your changes are small and/or do not interfere with someone's else development domain You may choose to develop in private copy of trunk kept on your computer and then commit directly to trunk. First approach is especailly recommended for new developers who are not familiar with ARC code and need their changes to be reviewed by others. Or for massive code re-arrangement.

In case You think your changes are touching someone's work or You are simply not sure feel free to ask. If you know the responsible person you can contact him/her directly. Otherwise please use the mailing list to discuss changes.

Adding files to SVN

Adding - and renaming - files to the NorduGrid SVN can be a daring task, mostly because the repository also contains build structure, and smallest mistakes can cause nightly build crashing. Now is probably a good time to get acquainted with basic SVN and Autotools (Autoconf, Automake and Libtool). Most of the following describes good practice and standard SVN and Autotools procedures. There is simple example below. Repository also contains information related to packaging produced software in a way specific to supported operating system. You must take that into account while adding new files and components to ARC. There is be more information about that below.

  1. Trunk and branches contain sets of sub-directories (modules) representing different parts of ARC - currently most important top level modules are hed, libs, clients and services. Those are then split into more specific ones - like clients/data for utilities dealing with file transfers. This cheat-sheet assumes new files are added under already existing modules. If a new module has to be added, it must be discussed with all the developers.
  2. Identify yourself with one of the modules. The list of such can be found, e.g., in the toplevel Makefile.am file in the trunk root or in Makefile.am of deeper nested sub-directories : check the list SUBDIRS. Let's assume your code will be located inside a directory <dir>
  3. Add your files to <dir> in a usual SVN way, e.g.:

    svn add myfile

  4. Edit the file <dir>/Makefile.am so that it takes into account your files. You will always need Makefile.am, even if it only contains a dummy list of files in the directory (EXTRA_DIST list). Makefile.am will be transformed into Makefile by Autotools, by substituting @...@ variables with proper values. Examples of such variables are:
    • @GLOBUS_INCLUDES@, @DEFAULT_GLOBUS_LOCATION@,@GLOBUS_FLAVOR_NAME@
    • @MYSQL_LIBS@, @MYSQL_INCLUDES@
    • @GSOAP_LIBS@, @GSOAP_INCLUDES@, @GSOAP_LOCATION@
    • @VERSION@
    • ...
  5. If you'd rather need a subdirectory <ndir> to accommodate your files, then add it, and add the files into it.
  6. Create the file <dir>/<ndir>/Makefile.am, which in the simplest case is just your normal Makefile, or, if you are committing interpretable scripts, the file should contain the line listing the contents of <ndir>: $ cat Makefile.am
    mystuff_DATA = README *.pl
    EXTRA_DIST = $(mystuff_DATA)
  7. Edit the file <dir>/Makefile.am to take into account existence of the new subdirectory <ndir>. Make sure that <ndir> is listed in the SUBDIRS list: SUBDIRS = dira dirb dirc <ndir>
  8. Edit the file configure.in the root directory:
    • find the block starting with AC_OUTPUT(
    • find the lines <dir>/...
    • add below those the new line: <dir>/<ndir>/Makefile
  9. If new files are installed the package management systems (RPM, DEB) need to know about them explicitly. Normally your files will belong to an existing sub-package. If you do not think so, then discuss it with the developers. The instructions to add new files are different depending on whether you are working with arc0 or arc1 branches.
    arc0 currently has the following sub-packages:
    • compat
    • compat-dbg
    In arc1 the current list of subpackages besides the main package are:
    • arex
    • aris
    • cache-service
    • ca-utils
    • chelonia
    • client
    • client-tools
    • dbg
    • devel
    • doc
    • doxygen
    • egiis
    • gridftpd
    • gridmap-utils
    • hed
    • hopi
    • information-index
    • isis
    • janitor
    • java
    • ldap-monitor
    • plugins-needed
    • plugins-globus
    • python
    • server
    • ws-monitor
    If you are adding files to say the client package you would have to edit the file debian/client.filelist.in and nordugrid-arc.spec.in Note that these files go through Autotools substitutions and thus there must be basically no hard-coded paths.
    Note that the dbg package is a special package created by the DEB and RPM packaging and should be ignored. It contains debug symbols for ARC packages.
    For RPM packages simply change the toplevel nordugrid-arc.spec.in and modify the appropriate %files section.
    For DEB packages, modify the appropriate file: debian/nordugrid-arc-{subpackage}.install Some files have the suffix .in, then change that instead of the prior, which will then be generated at configure time. That approach grants the opportunity to have the DEB packages adapted to the configuration settings that may specify or omit features that partially influences the file list.

Documentation

Man pages

Did you add a new executable? If so please write a man page. You can use src/clients/arclib/arcsub.1.in as a template. Remember to add this file to repository as well.

Processed documents

Some documents are produced from a source. For example a LaTeX document example.tex could be turned into a PDF document. During the process of generating the document several auxiliary files are created. They should be listed in the appropriate Makefile.am. Example:

CLEANFILES = example.aux example.log

This will remove the files when doing make clean. On the other hand the final documents (eg. PDF files) should not be removed with this command. They should be listed as:

MAINTAINERCLEANFILES = example.pdf

So that only a make maintainer-clean command will remove them. Also the final produced document should be committed to SVN. In this way developers do not need to re-create the documents when doing make clean. Remember to update the PDF file in SVN whenever changes are made to the source file.

Check your modifications

It is a very good custom to ensure that the nightly code build succeeds. Therefore you should check your modifications. Copy the whole tree to a temporary directory. Then go to this new copy and do:

$ cd nordugrid
$ ./autogen.sh # (Autotools magic)
$ ./configure
$ make dist

If there were no errors you should have a nordugrid-arc-VERSION.tar.gz in the top-level directory. To test the RPM package building do:

echo "%_topdir /tmp/rpmbuild" > ~/.rpmmacros

mkdir -p /tmp/rpmbuild/{SOURCES,BUILD,RPMS,SRPMS}

cp nordugrid-arc-VERSION.tar.gz /tmp/rpmbuild/SOURCES

rpmbuild -ba nordugrid-arc.spec

This assumes that the standard RPM development is setup correctly.

On debian you do:

mkdir /tmp/build

cp nordugrid-arc-VERSION.tar.gz /tmp/build/nordugrid-arc_VERSION.orig.tar.gz

cd /tmp/build

tar zvxf nordugrid-arc_VERSION.orig.tar.gz

cd nordugrid-arc*

dpkg-buildpackage

If this commands succeeds your changes should be fine.

Go back to the checkout directory and commit the files in the usual way.

Using branches and tags for releases

Aside from using branches for development those are also used for preparing to release. Before releasing ARC software corresponding branch is created in branches. For example when developers decide that trunk is sufficiently stable and it is time for let's say 1.1 release the trunk is copied to branches/1.1 and access to that branch is granted to the release coordinator.

Binary packages generated from 1.1 branch are then used for testing and all identified issues are fixed in trunk and ported to branches/1.1 by the release coordinator. When all (or almost all) issues are fixed code from branches/1.1 is copied to tags/1.1.0 and then released to community.

Further development continues inside trunk while branches/1.1 recieves only fixes and small enhancements. When enougn fixes is collected branches/1.1 is copied into tags/1.1.1 and new release is rolled out.

NorduGrid homepage