NorduGrid technical meeting

22-24 January 2004, Tallinn

Minutes

Present: Andi Hektor, Mattias Ellert, Mario Kadastik, Daniel Kalici, Martin Folkman, Niels Elgaard, Balázs Kónya, Jakob Nielsen, Martti Raidal, Aleksandr Konstantinov, Anders Wäänänen, Oxana Smirnova, Arto Teräs, Juha Lento, Michael Gindonis, Leif Nixon

Agenda

Day 1 (2004-01-22)

We keep up with weekly meetings over VRVS (microphone and loudspeakers are needed, allow popup windows from www.vrvs.org)

Brief discussion on Reply-to to mailing lists. No result.

Newcomers are strongly encouraged to write to nordugrid-discuss and/or to join VRVS meetings with silly questions. This way is preferred to private communications with individual experts.

Nordugrid-support mailing list is now way advanced but lost the spam filter.

Next meeting

Helsinki (CSC), March 29-30 (Monday-Tuesday), preliminary. A confirmation will be distributed. Siteseeing tour is the must.

Resources

Estonia: see "Estonian Grid" . Intentions to create a Grid with 2 nodes in Tallinn and 2 in Tartu. Idea similar to the SWEGRID: purchase resources and distribute them between different research groups. CA is a bit expensive but already exists on the national level (e.g. for ID cards). Test installation is in progress. Manpower: A.H. + M.K..

Finland: CSC, see "Computers at CSC". SGI Origin will not be on any Grid; IBM cluster will eventualy. Manpower: A.T. + J.L.. Present resources are quite heavily loaded. Finnish Grid research: at CSC and HIP. Most CSC users (ca 95% ?) use proprietary software, meaning they normally can not set it ip in a distributed manner. HIP has some smallish clusters, and through the Material Sciences Grid project may get extra resources (ca 500-700 CPUs across 6 sites). M.G. volunteered to set up a new Finland GIIS with a newest Globus.

Sweden: 6x100 CPUs sites, Ingrid and Hagrid are up and running, others are coming soon. Users are coming soon (March). No plans to open up to non-SWEGRID users, not even via NDGF. Way for non-Swedish researchers to get authorized is via common projects with Swedish groups which apply via SNAC for SWEGRID resurces. 2TB of storage per cluster, look into buying extra 60 TB and tape storage. Manpower: lousely coupled cloud of people in PDC and Umeå and elsewhere, little communication. Some rumours of a Grid portal development in Umeå. Grid banking: quite some development appears to go on in Stockholm, OGSA-based, involving some existing industrial solution. See SGAS

Denmark: some development towards using Windows resources, e.g., via Linux virtual machines, see Xen. Unknown amount of clusters of uncertain size is getting equipped with NorduGrid.

NDGF deploys a 3TB storage worth 50000 DKK. Aim to deploy it both as a front-end and the Storage Element.

Globus

A.K. introduced a bit into RLS. ATLAS prefers to use RLS, for simplicity. A.K. and A.W. claim that users can try using RLS right away, on the limited amount of sites though. J.N. & M.E. will set up servers and come up with usage instructions. If there will be manpower, would be nice to implement Grid Services and compare performance. Final decision to be taken later (e.g. after GGF10) B.K. will investigate Globus' plans w.r.t. GT2 MDS and OpenLDAP. A good thing would be to move to a standard OpenLDAP instead of the Globus-modified one. Same goes for SASL. M.E. will help with SASL localisation.

Dinner

Day 2 (2004-01-23)

License

A.W., J.N. & A.K. will chose the best license. Others have no preferences except that the stuff should be free to use and to modify.

Releases

VOMS is the problem. Client now builds everywhere, while server needs MySQL4. The idea is that VOMS will eventualy become useable as an external software; this still needs patches to make it compatible with newest Globus. For the time being, A.K. and A.W. patch existing VOMS and A.W. will prepare VOMS, VOMS-server, VOMS-devel & VOMS-doc. All the combinations of what should be installed where will be documented in the installation instructions by A.W..

Tag: when the s/w builds on all the platforms (RH6.2?) and ngtest 0 succeeds on at least one, the tag is made.

All changes in the code should be explicitly sent to A.W. personally, so that he includes it into the release notes.

A.W. and A.K. will move the CVS to Copenhagen (cvs.nordugrid.org) within two weeks. Same will happen to bugzilla (bugzilla.nordugrid.org)

Release will be made as soon as VOMS will be cleaned up. Also re-organization of CVS, organizing the tree in clients, servers and common, would be very helpful. M.E. and A.K. will figure the way of doing it. The drawback is a complexity of the corresponding CVS manipulations.

Tag next week - pre-release in 2 weeks

User Interface

N.E. will coordinate the restructuring efforts: there's a student doing Java client, J.N. is doing Python one, Thomas is doing benchmarking, students in Aalborg are doing automatic job resubmission.

B.K. will incorporate [some of the] new information schema attributes.

Forum for new contributors: nordugrid-discuss

Monitoring

UTF-8 issue: although tools should be able to work with Unicode or UTF-8, but CA will not grant certificates for names written using non-ASCII characters.

Describe mapping of attributes to human-readable names - O.S..

Information System

Authorised access to Infosystem will be enabled sooner or later. B.K. will look into this.

Job status: at the moment, LRMS messages are simply parsed; B.K. will review the situation (e.g., some mapping to more generic names).

Grid Manager

Interface to various LRMS's: B.K. will coordinate the efforts and implement the changes in InfoSys.

L.N. will investigate the necessity of one-to-one mappings of Grid users to local accounts in SWEGRID (accounting?) and possibility of getting away from that using e.g. GACL.

Default configuration: GM runs as root. A.W. or A.K. will take care of explaning the options/limitations in FAQ. A separate setup document will be prepared.

Feature request #152 in GM is fixed in October, but not documented. A.K. will provide the documentation.

GridFTP server

GACL plugin works (thugh was not thoroughly tested yet), except for the recursive ngacl.

Storage Elements

New SE: works via HTTP over SSL (Globus dropped SSL v.2, this resulted in broken compatibility with Web browsers) and GSI. SOAP: admin file, out/get info, modify access rights, delete. Does not support replication. LRC appears to be the best solution for a local index.

B.K. will test the stuff. A.K. will write documentation. J.N. will set up the thing at NBI, too.

RLS: see Globus discussion

Lunch NorduGrid seminar at the Estonian Academy of Sciences

Accounting

Everybody wants to do it. B.K. will post the logger usage record so that interested developers will react and compare vs GGF etc. Somebody has to define which queries are needed and ultimately the logger database structure.

Interface to query MySQL database: M.F.

D.K., N.E. to propose a flexible usage record, ev. looking at other developments.

Full job logging: investigate (simulate) size of a full job history record (requested attributes, timestamp of status changes, used files and resources etc.), suggest an optimal location of such database, suggest the database implementation. Task for a student.

Dinner

Day3 (2004-01-24)

Job for students

Logging, bookkeeeping & accounting is an almost undeveloped field, hence not many short tasks there, while the students explicitly wish to work in this area and have only ca 3 months each.

  1. See above: Full job logging, i.e., investigate (simulate) size of a full job history record (requested attributes, timestamp of status changes, used files and resources etc.), suggest an optimal location of such database, suggest the database implementation. Task for a student.
  2. Survey of existing Grid banking tools and assessment of their applicability to NorduGrid, ev. proposal for NorduGrid own accounting system. Links: Australian Grid Bank, SGAS by PDC, Danish proposal (A.W. to find the link), GGF efforts.
  3. Implementation of dynamic/pool accounts for NorduGrid, possibly different pools for different user groups; implications for accounting: no local accounting possible; implications for PBS - no possibility of user-specific settings.
  4. Survey of existing mechanisms to securely isolate jobs, preferably cross-platform (see e.g. link posted by L.N.).

There's a student in Denmark writing a GUI for UI (Java).

O.S. to add tasks from A.K.'s talk to the list of tasks.

Applications

A.H. mentioned CMS applications; O.S. noted that all LHC experiments will provide PACMAN packages, not much of RPMs or anything else. In general, one can not expect applications s/w being installable only from RPMs, esp. when done by a non-privileged user.

Discussion fell into runtime environments issues.

Runtime environments

O.S. to think of how to present nicely runtime environments via the monitor.

Installation by users: LCG-like scheme with priveleged users running specialized "s/w install" jobs, and a dedicated runtime environment installation area, writeable by those persons, is likely to be accepted. No volunteers to develop further.

B.K. pursued the idea of runtime environment names to include namespace URLs, and forcing (non-brutally) s/w owners to provide a structured information via such URLs. Everybody agrees on namespaces, nobody likes forced information.

Minutes taken by O.S.