NorduGrid technical meeting

15-16 May 2006, Copenhagen

Minutes

Participants: Anders, Jakob, Aleksandr, Michael, Mattias, Balázs, Oxana (at the minutes)

CA

NorduGrid CA key extended by 1 year; every site owner and user has 1 week to apply for a new certificate.

0.6

The last of the old serie; after 0.6 start a new architecture, components. New s/w should initially include the old components and interfaces; aiming at smooth transition. 0.6 RC is needed, 0.5.x is not extremelly well tested. Data-proximity algorithm is missing, has to be added to 0.6.+. Aleksandr has something in progress, probably will be ready in 1 week. Detailed discussion to follow on Tuesday.

New projects

KnowARC

Brussels wants to see, in order of priorities:

impact, esp. in industries
service-oriented architecture
a middleware based on standards, capable of working together with other standards-based middleware

Most tasks are computational-Grid related; plus extended Runtime environments; plus workflows. Job migration. New platforms (Mac OS, MS Windows, Solaris). Authorization ramework. P2P-like infosystem. Fault-tolerant storage (NIIF). Logging. Distribution (part of Linux distros). S/w should not be tied to interfaces, should be possible to change them.

NDGF

Common research computing infrastructure for Nordic countries based on Grid. Hosted by the NORDUnet. Based on ARC; must provide resources for Nordic researchers, primarily LCG. Must build a distributed LCG Tier1 (prime concern: storage; experiment-specific application interfaces; accounting towards LCG; T1 <-> T2 transfers). Does not have to follow standards - instead must meet the needs of existing applications (incl. proprietary stuff). Ev. MC production, distributed analysis. Long-term solutions are not a priority.

NORDUnet3

Budget enough for 2-3 PhD students; nothing left for summer schools; probably interactive applications and data management, depending on where the students will end up.

Manpower (development)

KnowARC: Lund 2, NBI 2, Uppsala 2, Oslo 2 NDGF: 6, distribution yet unknown Manpower won't materialize before September 2006. All teams will increase except of Lund.

A retreat in September (a week) is needed; participants from both KnowARC ad NDGF (NORDUNet3 umbrella?)

Worldwide projects

M/w: UNICORE, Globus, LCG/gLite, CONDOR

No standards, unclear who and how will influence them. Currently, imlementations define standards. No sense to follow any chosen implementation. On the other hand, there are clear trends and some of the main middleware projects expressed their wish AND started to work on common interfaces. See also the joint white paper by HP, IBM, Intel and Micro$oft.

Interesting Intel's analysis of GT4; reveals e.g. MDS4 problems:

Scalability - registry updates cannot handle more than few hundreds of registrants
I/O performance - use of java nio suggested
Registry content (XML doc) size seems to be limited to a few MB - bug or design flaw?

Architecture

Web service interfaces: needed in addition to legacy interfaces.

Coding:

aim at performance.
use the language that provides for the best performance of this particular component.
no specific language required, but compatibility checks are absolutelly required. Core set: Bourne shell (no Korn shell, no C shell etc), C++, Python, C, Perl.
avoid using a patchwork of languages and running into unnecessary dependencies on 3d party stuff.
Java is only good for GUIs and similar non-essential things which do not have to be fast.
Avoid dependencies on latest versions of external libraries; any external dependency must be proved to be absoltelly necessary.

Components:

Functionality of a service must be complete; a service must be a largely standalone component.

Solution should be simple enough to create quickly "private" grids; be started by a single script; have a possibility to run without heavy-weight security procedures (auto-generated certificates, VOs)

Execution component

batch job processing
database query (data mining) - zero latency
interactive jobs (?) - zero latency

Data storage component
Data transfer component

Managed data transfer
SRM (?)

Data availability component

Indexing catalogs (a la RLS)
Interfaces to such catalogs, tools

Data tagging component

Metadata catalogs

Job management component

A broker
Or a pool of jobs
May also notify users, retrieve some info, advertise something etc
Workflows

Local information

Service status; list of local services

Information indexing

Soft registration
"Hard" registration: personal bookmarks (ev. incl. indices)

CLIENT

jobs
data
monitoring
transfer

Accounting
Security

Belongs to all of the above

May 16

Anders, Aleksandr, Mattias, Balázs, Michael, Oxana

Dependencies:

System
External
Internal
Build

Should be described in the INSTALL file

System: glibc, bash, perl, libxml2, libltdl, libtool, autoconf, openssl

External dependencies: gSoap, VOMS, GPT, GLOBUS - openldap, certficates

Build: gcc, g++, lbltdl, for gSoap: byacc, bison, flex

Anders will see if gSoap can be linked statically, as the "special" NG version of it is required.

VOMS is patched, masaged and rebuilt, relocatable (def. in /usr/bin, different from gLite, can be a conflict), user "vomses" go to ~/.voms, system - to /etc/voms. Original sructure of having /etc/vomses/ is kept (also exists /etc/vomses *file*). Will be enabled by default for build.

Build README is in works.

FTP server reshuffled (again)

ftp.nordugrid.org
                  /repos
                        /...
                  /software
                        /arc
                            /nightly
                            /releases
                        /globus
                        /...
                        /VOMS

SRM proxy

Twofold task: T0 -> T1 writing; SRM proxy will help, Aleksandr works on it. ATLAS data access: read from 53 SEs at 9 sites, may take large amount of time to query each server, besides, it is unfeasibe to install SRM interfaces (dCache explicitly, it needs a clean system to be installed) at each of those. Aleksandr can think of writing an SRM interface for a GridFTP server, which will be lighter than dCache.

Testing

Log of tests (date, test, result, comment). Oxana to make a page, Anders to make a pre-0.6 tag (0.5.48, announced as a Release Candidate) this week, latest on Friday.