Dmytro Karpenko

Plots

Here we collected the plots describing the analyzed workload that are not shown in the article. Some plots are shown here, because, though depicting interesting information, they are of lesser importance for our research and do not need to be present in the article. Some plots are relevant, but we could not place them in the article due to the lack of space. Some plots are variations of the plots that are shown in the article, for example, histograms or line plots instead of survival or distibution function.

High level statistics

These pie chart illustrate general information about the workload by showing the ratios between its main parameters.

1. Succeeded/failed jobs
2. Succeeded/failed with (possible) stage of failure specifications. The specifications of stage when the failure occured is based on guesses. There is no clear registration of failure reason, so we perform the assessment of the job parameters like exit code, running time and others and try to guess if the job was in download, LRMS or upload stage when it failed. Though this method is to some extent credible and the given values are confirmed with what the ATLAS administrators usually see in their accounting system, we can not in any case claim that the numbers are 100% true.
3. Succeeded jobs with/without performed transfer
4. Transfer requests from succeeded jobs: cached inputs, not cached inputs, outputs
5. Amount of data, that is held in: cached inputs, not cached inputs, outputs
6. Number of unique input and output files
7. Size of unique input and output files

Distributions

These bar charts illustrate how different parameters are distributed among particular objects. The objects do not change their order on different plots, i.e. a cluster marked 1 on some plot is always cluster 1 on all the other plots.

1. Successful jobs across clusters
2. Successful jobs across users
3. Users across clusters
4. Walltime of successful jobs across clusters

Temporal statistics

These plots illustrate how different parameters change over time.

1. Cache hit/miss by requests over time
2. Cache hit/miss by size over time
3. Jobs arrival rate by days
4. Jobs arrival rate by hours for: the week with most jobs; the week with least jobs; the week with average amount of jobs. Average means that we picked up a week with value of jobs closest to arithmetical mean of numbers of jobs per week.
5. Jobs arrival rate by 10 minutes intervals for 10 random 4-hour intervals: 1; 2; 3; 4; 5; 6; 7; 8; 9; 10.

Proof of sample representativeness

The plots that depict analyzed parameters per each week of the collected workflow. These plots confirm, that the parameters of the workflow are stable, and thus our 6-month sample is fully representative. The plots have logarithmic scale for better readability. A few jobs that have extreme values of parameters were removed from some of the plots, also for better readability.

1. Input files per job (few jobs with 100+ files removed)
2. Output files per job (few jobs with 20+ files removed)
3. Input size per job
4. Output size per job
5. Walltimes of jobs
6. Size of input files
7. Size of output files
8. Comparative histogram of walltimes for 5 clusters with most of jobs

Other statistics

These plots illustrate various things that were difficult to put in other categories.

1. Line plot of popularity of the unique input files on log scale
2. Correlation between input files popularity and size
3. Line plot of input requests per jobs on log scale
4. Line plot of size of input per job on log scale
5. Histogram of walltimes
6. Histogram of interarrival times
7. Line plot of upload requests per jobs
8. Line plot of size of upload per jobs on log scale
9. Line plot of number of total transfer requests per jobs on log scale
10. Line plot of total transfer size per jobs on log scale
11. Correlation between total number of requests and total size of transfer per job
12. The time span of requests for 50 most popular files