Using the Fujitsu profilers in Deucalion's Arm partition
If you compile your code in Deucalion's Arm partition using the Fujitsu toolchain, you'll be able to use the two Fujitsu profilers, the Instant Performance Profiler (IPP) and the Advanced Performance Profiler (APP). These sample-based profilers point you to the hotspot functions that take the most amount of resources while running a specific case, and give you an overview of the code performance in terms of vectorization and pipelining, among other metrics.
Due to the unique architecture of the Fujitsu A64FX processor, these tools become an invaluable asset for tuning your code for this specific chip, since many scientific codes written for the x86 architecture are not well suited for the A64FX.
The Instant Performance Profiler gives you the overall trends of the behavior of your program. This is a useful starting point to understand where you can invest your time tuning your program for the A64FX. You can use this profiler without the need to recompile your code.
The Advanced Performance Profiler can be used to target specific parts of the code and to extract detailed information about these regions of the code. To do so you have to instrument your source code, which means you'll have to add the profiling library and call the profiler functions around a region of interest in the source code before compilling.
However, the APP can be used in a different manner, which is also extremely useful, that is to produce the CPU Performance Analysis Report. This visual report, when used with Fujitsu's tuning guide, provides you with a straightforward way to improve your code. In this text, we will only provide instructions to use APP to produce this visual report.
This is only a brief tutorial. However, you can find detailed instructions on how to use the Fujitsu toolchain software (compilers and profilers) in this Fujitsu Software Manuals page, hosted at the RIKEN Center for Scientific Computer website.
Compiling with the Fujitsu toolchain
To use the Fujitsu profilers, you first need to compile your code with the Fujitsu toolchain, which you can access by loading the appropriate module in Deucalion. Next follows a simple batch script to compile a sample Fortran code in the Deucalion's Arm partition:
compile.sh
#!/bin/bash
#SBATCH --job-name=comp-arm
#SBATCH --output=comp-%j.out
#SBATCH --error=comp-%j.err
#SBATCH --partition=normal-arm
#SBATCH --account=<account>
#SBATCH --time=02:00:00
#SBATCH --nodes=1
ml FJSVstclanga
# ml FFTW.FUJ/1.1.1-sve # only load this if code uses FFTW
frt example.f90 -o example-bin
# mpifrt example-mpi.f90 -o example-mpi-bin # use this instead if the code uses MPI
Where <account> should be replaced by the account reference of your project for the Arm partition CPU.h (this reference should end with a). This script can then be run with sbatch in the same directory as example.f90:
[user@ln01 build]$ ls
compile.sh example.f90
[user@ln01 build]$ sbatch compile.sh
For more information on compilation flags, please refer to the Fortran User's Guide in the Fujitsu Software Manuals page. For C and C++, please see C User's Guide and C++ User's Guide, respectively.
Instant Performance Profiler
Once your code is compiled with the Fujitsu compiler, you can then use the Instant Performance Profiler to start your investigation on the behavior of your code.
The profiler comprises two commands: fipp which measures profile data, and fipppx which processes the data collected by fipp and produces a report.
Next follows a simple batch script to run the IPP with a binary which uses MPI.
run.sh
#!/bin/bash
#SBATCH --job-name=fipp-run
#SBATCH --output=frun-%j.out
#SBATCH --error=frun-%j.err
#SBATCH --partition=normal-arm
#SBATCH --account=<account>
#SBATCH --time=02:00:00
#SBATCH --ntasks=10
ml FJSVstclanga
fipp -C -d ./profdata mpirun --bind-to core --report-bindings -np $SLURM_NTASKS ./example-mpi-bin
# NOTE: only use $SLURM_NTASKS when --ntasks is defined
fipppx -A -d ./profdata -Impi -pall -ttext -o report.txt
Please refer to the Profiler User's Guide in the Fujitsu Software Manuals page for more details on the flags used.
These commands produce a plain text report, an example of which is presented below.
---------------------------------------------------------------------------------------------
Fujitsu Instant Performance Profiler Version 4.2.6
Measured time : Fri Oct 16 16:06:14 2026
CPU frequency : Process 0 - 1 2000 (MHz)
Type of program : MPI
Average at sampling interval : 100.2 (ms)
Measured region : All
Virtual coordinate : (2, 0, 0)
---------------------------------------------------------------------------------------------
_____________________________________________________________________________________________
Time statistics
Elapsed(s) User(s) System(s)
---------------------------------------------
8.0536 15.0199 0.9709 Application
---------------------------------------------
8.0536 7.5149 0.4807 Process 1
8.0529 7.5051 0.4902 Process 0
_____________________________________________________________________________________________
Performance monitor event:statistics
*************************************************************************************
Application - performance monitors
*************************************************************************************
Execution Floating-point Mem throughput Mem throughput
time(s) GFLOPS peak ratio(%) (GB/s) peak ratio(%)
--------------------------------------------------------------------------------
8.0536 3.9213 3.4039 0.7786 0.0760 Application
--------------------------------------------------------------------------------
8.0536 1.9607 3.4039 0.3845 0.0375 Process 1
8.0529 1.9608 3.4042 0.3942 0.0385 Process 0
Effective Floating-point SIMD inst. SVE operation
instruction operation rate(%) rate(%)
--------------------------------------------------------------------------------
39607256592 31580796391 10.9024 72.9528 Application
--------------------------------------------------------------------------------
19852876980 15790393692 10.8754 72.9528 Process 1
19754379612 15790402699 10.9296 72.9528 Process 0
IPC GIPS
--------------------------------------------------------------------------------
1.2442 4.9180 Application
--------------------------------------------------------------------------------
1.2471 2.4651 Process 1
1.2413 2.4531 Process 0
_____________________________________________________________________________________________
Procedures profile (Total thread cost basis)
*************************************************************************************
Application - procedures
Application and Process outputs the total value of the cost of each thread.
Procedure outputs the total value of the procedure cost of each thread.
*************************************************************************************
Cost % Operation (s) Barrier % Start End
-------------------------------------------------------------------------------------
158 100.0000 15.8315 -- -- -- -- Application
-------------------------------------------------------------------------------------
43 27.2152 4.3084 -- -- 652 779 m_emf_interpolate.get_emf_3d_s2_
29 18.3544 2.9056 -- -- 348 392 m_random_class.genrand_gaussian_
26 16.4557 2.6051 -- -- 3115 3220 m_species_current.dep_current_3d_s2_
9 5.6962 0.9018 -- -- -- -- __jwe_cls_typebound
6 3.7975 0.6013 -- -- 393 413 m_species_current.wl_s2_
6 3.7975 0.6013 -- -- 502 518 m_species_current.spline_s2_
5 3.1646 0.5010 -- -- 350 626 m_species_push.dudt_boris_
4 2.5316 0.4009 -- -- 1326 1353 m_species_push.ntrim_
4 2.5316 0.4008 -- -- -- -- syscall
4 2.5316 0.4008 -- -- -- -- __GI___mmap64
MPI % Communication (s) Start End
---------------------------------------------------------------
0 0.0000 0.0000 -- -- Application
---------------------------------------------------------------
*************************************************************************************
Process 1 - procedures
*************************************************************************************
Cost % Operation (s) Barrier % Start End
-------------------------------------------------------------------------------------
79 100.0000 7.9122 -- -- -- -- Process 1
-------------------------------------------------------------------------------------
23 29.1139 2.3035 -- -- 652 779 m_emf_interpolate.get_emf_3d_s2_
16 20.2532 1.6025 -- -- 348 392 m_random_class.genrand_gaussian_
14 17.7215 1.4022 -- -- 3115 3220 m_species_current.dep_current_3d_s2_
4 5.0633 0.4006 -- -- -- -- __jwe_cls_typebound
2 2.5316 0.2003 -- -- 393 413 m_species_current.wl_s2_
2 2.5316 0.2003 -- -- 502 518 m_species_current.spline_s2_
2 2.5316 0.2003 -- -- 894 1600 m_species_current.split_3d_
2 2.5316 0.2003 -- -- 350 626 m_species_push.dudt_boris_
2 2.5316 0.2003 -- -- 1516 1599 m_species_push.advance_deposit_3d_
2 2.5316 0.2003 -- -- -- -- syscall
MPI % Communication (s) Start End
---------------------------------------------------------------
0 0.0000 0.0000 -- -- Process 1
---------------------------------------------------------------
*************************************************************************************
Process 0 - procedures
*************************************************************************************
Cost % Operation (s) Barrier % Start End
-------------------------------------------------------------------------------------
79 100.0000 7.9193 -- -- -- -- Process 0
-------------------------------------------------------------------------------------
20 25.3165 2.0049 -- -- 652 779 m_emf_interpolate.get_emf_3d_s2_
13 16.4557 1.3032 -- -- 348 392 m_random_class.genrand_gaussian_
12 15.1899 1.2029 -- -- 3115 3220 m_species_current.dep_current_3d_s2_
5 6.3291 0.5012 -- -- -- -- __jwe_cls_typebound
4 5.0633 0.4010 -- -- 393 413 m_species_current.wl_s2_
4 5.0633 0.4010 -- -- 502 518 m_species_current.spline_s2_
3 3.7975 0.3007 -- -- 350 626 m_species_push.dudt_boris_
3 3.7975 0.3007 -- -- 1326 1353 m_species_push.ntrim_
2 2.5316 0.2005 -- -- 84 124 m_random_mt.mt_int32_
2 2.5316 0.2005 -- -- 199 215 m_emf_interpolate.spline_s2_
MPI % Communication (s) Start End
---------------------------------------------------------------
0 0.0000 0.0000 -- -- Process 0
---------------------------------------------------------------
_____________________________________________________________________________________________
Loops profile (Total thread cost basis)
*************************************************************************************
Application - loops
Application and Process outputs the total value of the cost of each thread.
Procedure outputs the total value of the loop cost of each thread.
*************************************************************************************
Cost % Operation (s) Barrier % Nest Kind Exec Start End
--------------------------------------------------------------------------------------------------------------
158 100.0000 15.8315 -- -- -- -- -- -- -- Application
--------------------------------------------------------------------------------------------------------------
14 8.8608 1.4027 -- -- 4 DO SERIAL 3200 3213 m_species_current.dep_current_3d_s2_
10 6.3291 1.0020 -- -- 4 DO SERIAL 721 725 m_emf_interpolate.get_emf_3d_s2_
8 5.0633 0.8017 -- -- 2 DO SERIAL 711 735 m_emf_interpolate.get_emf_3d_s2_
8 5.0633 0.8015 -- -- 1 DO SERIAL 679 776 m_emf_interpolate.get_emf_3d_s2_
8 5.0633 0.8015 -- -- 4 DO SERIAL 756 760 m_emf_interpolate.get_emf_3d_s2_
7 4.4304 0.7013 -- -- 1 DO SERIAL 361 373 m_random_class.genrand_gaussian_
6 3.7975 0.6013 -- -- 3 DO SERIAL 3190 3194 m_species_current.dep_current_3d_s2_
6 3.7975 0.6012 -- -- 2 DO SERIAL 746 770 m_emf_interpolate.get_emf_3d_s2_
4 2.5316 0.4007 -- -- 1 DO SERIAL 3156 3218 m_species_current.dep_current_3d_s2_
3 1.8987 0.3006 -- -- 2 DO SERIAL 575 580 m_species_udist.set_momentum_
MPI % Communication (s) Nest Kind Exec Start End
----------------------------------------------------------------------------------------
0 0.0000 0.0000 -- -- -- -- -- Application
----------------------------------------------------------------------------------------
*************************************************************************************
Process 1 - loops
*************************************************************************************
Cost % Operation (s) Barrier % Nest Kind Exec Start End
--------------------------------------------------------------------------------------------------------------
79 100.0000 7.9122 -- -- -- -- -- -- -- Process 1
--------------------------------------------------------------------------------------------------------------
8 10.1266 0.8012 -- -- 4 DO SERIAL 3200 3213 m_species_current.dep_current_3d_s2_
5 6.3291 0.5008 -- -- 1 DO SERIAL 679 776 m_emf_interpolate.get_emf_3d_s2_
5 6.3291 0.5008 -- -- 4 DO SERIAL 756 760 m_emf_interpolate.get_emf_3d_s2_
5 6.3291 0.5008 -- -- 4 DO SERIAL 721 725 m_emf_interpolate.get_emf_3d_s2_
4 5.0633 0.4006 -- -- 1 DO SERIAL 361 373 m_random_class.genrand_gaussian_
3 3.7975 0.3005 -- -- 2 DO SERIAL 746 770 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3005 -- -- 2 DO SERIAL 711 735 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3005 -- -- 1 DO SERIAL 3156 3218 m_species_current.dep_current_3d_s2_
2 2.5316 0.2003 -- -- 3 DO SERIAL 716 730 m_emf_interpolate.get_emf_3d_s2_
2 2.5316 0.2003 -- -- 3 DO SERIAL 3190 3194 m_species_current.dep_current_3d_s2_
MPI % Communication (s) Nest Kind Exec Start End
----------------------------------------------------------------------------------------
0 0.0000 0.0000 -- -- -- -- -- Process 1
----------------------------------------------------------------------------------------
*************************************************************************************
Process 0 - loops
*************************************************************************************
Cost % Operation (s) Barrier % Nest Kind Exec Start End
--------------------------------------------------------------------------------------------------------------
79 100.0000 7.9193 -- -- -- -- -- -- -- Process 0
--------------------------------------------------------------------------------------------------------------
6 7.5949 0.6015 -- -- 4 DO SERIAL 3200 3213 m_species_current.dep_current_3d_s2_
5 6.3291 0.5012 -- -- 2 DO SERIAL 711 735 m_emf_interpolate.get_emf_3d_s2_
5 6.3291 0.5012 -- -- 4 DO SERIAL 721 725 m_emf_interpolate.get_emf_3d_s2_
4 5.0633 0.4010 -- -- 3 DO SERIAL 3190 3194 m_species_current.dep_current_3d_s2_
3 3.7975 0.3007 -- -- 1 DO SERIAL 679 776 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3007 -- -- 2 DO SERIAL 746 770 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3007 -- -- 4 DO SERIAL 756 760 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3007 -- -- 1 DO SERIAL 361 373 m_random_class.genrand_gaussian_
2 2.5316 0.2005 -- -- 2 DO SERIAL 546 551 m_species_push.dudt_boris_
2 2.5316 0.2005 -- -- 2 DO SERIAL 575 580 m_species_udist.set_momentum_
MPI % Communication (s) Nest Kind Exec Start End
----------------------------------------------------------------------------------------
0 0.0000 0.0000 -- -- -- -- -- Process 0
----------------------------------------------------------------------------------------
_____________________________________________________________________________________________
Lines profile (Total thread cost basis)
*************************************************************************************
Application - lines
Application and Process outputs the total value of the cost of each thread.
Procedure outputs the total value of the line cost of each thread.
*************************************************************************************
Cost % Operation (s) Barrier % Line
----------------------------------------------------------------------------
158 100.0000 15.8315 -- -- -- Application
----------------------------------------------------------------------------
12 7.5949 1.2024 -- -- 377 m_random_class.genrand_gaussian_
6 3.7975 0.6012 -- -- 732 m_emf_interpolate.get_emf_3d_s2_
5 3.1646 0.5011 -- -- 3201 m_species_current.dep_current_3d_s2_
4 2.5316 0.4010 -- -- 410 m_species_current.wl_s2_
4 2.5316 0.4009 -- -- 723 m_emf_interpolate.get_emf_3d_s2_
4 2.5316 0.4009 -- -- 757 m_emf_interpolate.get_emf_3d_s2_
4 2.5316 0.4008 -- -- 392 m_random_class.genrand_gaussian_
4 2.5316 0.4007 -- -- 724 m_emf_interpolate.get_emf_3d_s2_
3 1.8987 0.3007 -- -- 746 m_emf_interpolate.get_emf_3d_s2_
3 1.8987 0.3006 -- -- 361 m_random_class.genrand_gaussian_
MPI % Communication (s) Line
------------------------------------------------------
0 0.0000 0.0000 -- Application
------------------------------------------------------
*************************************************************************************
Process 1 - lines
*************************************************************************************
Cost % Operation (s) Barrier % Line
----------------------------------------------------------------------------
79 100.0000 7.9122 -- -- -- Process 1
----------------------------------------------------------------------------
6 7.5949 0.6009 -- -- 377 m_random_class.genrand_gaussian_
3 3.7975 0.3005 -- -- 724 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3005 -- -- 732 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3005 -- -- 737 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3005 -- -- 759 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3005 -- -- 372 m_random_class.genrand_gaussian_
2 2.5316 0.2003 -- -- 348 m_random_class.genrand_gaussian_
2 2.5316 0.2003 -- -- 392 m_random_class.genrand_gaussian_
2 2.5316 0.2003 -- -- 3164 m_species_current.dep_current_3d_s2_
2 2.5316 0.2003 -- -- 3203 m_species_current.dep_current_3d_s2_
MPI % Communication (s) Line
------------------------------------------------------
0 0.0000 0.0000 -- Process 1
------------------------------------------------------
*************************************************************************************
Process 0 - lines
*************************************************************************************
Cost % Operation (s) Barrier % Line
----------------------------------------------------------------------------
79 100.0000 7.9193 -- -- -- Process 0
----------------------------------------------------------------------------
6 7.5949 0.6015 -- -- 377 m_random_class.genrand_gaussian_
4 5.0633 0.4010 -- -- 3201 m_species_current.dep_current_3d_s2_
4 5.0633 0.4010 -- -- 410 m_species_current.wl_s2_
3 3.7975 0.3007 -- -- 723 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3007 -- -- 732 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3007 -- -- 746 m_emf_interpolate.get_emf_3d_s2_
3 3.7975 0.3007 -- -- 757 m_emf_interpolate.get_emf_3d_s2_
2 2.5316 0.2005 -- -- 711 m_emf_interpolate.get_emf_3d_s2_
2 2.5316 0.2005 -- -- 361 m_random_class.genrand_gaussian_
2 2.5316 0.2005 -- -- 392 m_random_class.genrand_gaussian_
MPI % Communication (s) Line
------------------------------------------------------
0 0.0000 0.0000 -- Process 0
------------------------------------------------------
_____________________________________________________________________________________________
CPU Performance Analysis Report
Introduction
From Chapter 4 of the Profiler User's Guide in the Fujitsu Software Manuals page:
The CPU Performance Analysis Report aggregates a lot of CPU performance analysis information measured through multiple executions and visualizes it in an easy-to-understand way, using tables and associated graphs. The Advanced Performance Profiler is used to measure CPU performance analysis information.
With the APP we can thus produce the CPU Performance Analysis Report. This report is generated using a macro-enabled Excel workbook provided by Fujitsu, which you can download from Deucalion to your local machine by running the command:
scp deucalion:/share/data/profile/cpu_pa_report.xlsm .
The first step to produce this report is to collect and process the profile information with the APP. However, depending on the amount of information we wish to collect, we can get a simpler or a more detailed report. APP uses the A64FX performance-monitoring unit to measure the different profile metrics and, since there is a limited number of performance counter registers, the profiler cannot measure all the values necessary to produce a detailed report in one pass. With this in mind, the macro-enabled Excel workbook can thus produce 4 different types of reports depending on the amount of information we provide:
- the Single Report (1 pass)
- the Brief Report (5 passes)
- the Standard Report (11 passes)
- or the Detailed Report (17 passes)
(see Chapter 4 of the Profiler User's Guide in the Fujitsu Software Manuals page for more details on these different types of reports). This means that if we wish to produce a report other than the Single Report, we need to run our case more than once, recording different metrics in each of these runs, while maintaining the case setup constant throughout these repetitions.
Regarding the use of APP, this profiler comprises two commands, similar to IPP: fapp which measures profile data, and fapppx which processes the data collected by fapp and produces a report.
Example run
To understand how to produce the CPU Performance Analysis Report, we can start by looking at the batch script used to collect the necessary metrics to produce the Single Report, presented below.
#!/bin/bash
#SBATCH --job-name=cpa_run
#SBATCH --output=out-%j.out
#SBATCH --error=out-%j.err
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=10
#SBATCH --partition=normal-arm
#SBATCH --account=<account>
ml FJSVstclanga
fapp -C -d ./profdata -Hevent=pa1 mpirun --bind-to core --report-bindings -np $SLURM_NTASKS ./example-mpi-bin
# NOTE: only use $SLURM_NTASKS when --ntasks is defined
fapppx -A -d ./profdata -Icpupa,nompi -pall -tcsv -o pa1.csv
Where the flag -Hevent is used to select the group of metrics recorded in the run, which is this case is just pa1. These groups are named from pa1 up to pa17. Only one group can be selected in each run. To produce the Single Report, the CSV file produced by fapppx needs to be downloaded to your local machine, which you can do so by running the following command:
scp deucalion:/path/to/your/run/pa1.csv .
cpu_pa_report.xlsm. Once the two files are together, you can open the Excel file and go through the different dialog boxes until the report is generated.
To produce the other types of reports, you can use the last script multiple times, changing the Hevent and the name of the CSV file the required number of times to ensure that the Excel workbook can produce the desired type of report. The name of the CSV file needs to be the same as the Hevent parameter. Next follows a table with the summary of the different parameters and files needed to produce the different types of report.
| Report type | Number of runs | Hevent parameters |
CSV files |
|---|---|---|---|
| Single Report | 1 | pa1 |
pa1.csv |
| Brief Report | 5 | from pa1 up to pa5 |
pa1.csv,pa2.csv..pa5.csv |
| Standard Report | 11 | from pa1 up to pa11 |
pa1.csv,pa2.csv..pa11.csv |
| Detailed Report | 17 | from pa1 up to pa17 |
pa1.csv,pa2.csv..pa17.csv |
For example, to produce the Detailed Report, the directory in your local machine should contain the following files:
cpu_pa_report.xlsm
pa1.csv
pa2.csv
pa3.csv
pa4.csv
pa5.csv
pa6.csv
pa7.csv
pa8.csv
pa9.csv
pa10.csv
pa11.csv
pa12.csv
pa13.csv
pa14.csv
pa15.csv
pa16.csv
pa17.csv
Using JUBE to automate multiple runs
In order to automate the production of reports other than the Single Report, we can use JUBE to orchestrate the submission of multiple jobs. You can learn more about JUBE in its documentation page.
We can setup a template case directory, which JUBE uses to create the different case directories for each of the runs. Considering that, in this example, the exmaple-mpi-bin uses the input.in file for input parameters, we can setup the directory example_run_tmpl containing the following files:
example-mpi-bin
input.in
run_tmpl.sh
run_tmpl.sh is
#!/bin/bash
#SBATCH --job-name=cpa_run
#SBATCH --output=out-%j.out
#SBATCH --error=out-%j.err
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=10
#SBATCH --partition=normal-arm
#SBATCH --account=<account>
ml FJSVstclanga
fapp -C -d ./profdata -Hevent=#EVENT# mpirun --bind-to core --report-bindings -np $SLURM_NTASKS ./example-mpi-bin
# NOTE: only use $SLURM_NTASKS when --ntasks is defined
fapppx -A -d ./profdata -Icpupa,nompi -pall -tcsv -o #EVENT#.csv
#EVENT#. This template directory can then be used with the following JUBE input file:
jube_run.xml
<?xml version="1.0" encoding="UTF-8"?>
<jube>
<benchmark name="cpa_fapp" outpath="cpa_fapp">
<comment>CPU Performance Analysis Report with fapp</comment>
<!-- Configuration -->
<parameterset name="params">
<parameter name="event" type="text" mode="python">",".join(["pa"+str(i+1) for i in range(17)])</parameter>
</parameterset>
<!-- Load jobfiles -->
<fileset name="files">
<copy>example_run_tmpl_fapp/*</copy>
</fileset>
<!-- Substitute -->
<substituteset name="substitute">
<!-- Substitute files -->
<iofile in="run_tmpl.sh" out="run.sh"/>
<!-- Substitute commands -->
<sub source="#EVENT#" dest="$event"/>
</substituteset>
<!-- Operation -->
<step name="job">
<use>params</use>
<use>files</use>
<use>substitute</use>
<do>sbatch run.sh</do>
</step>
</benchmark>
</jube>
<parameter name="event" type="text" mode="python">",".join(["pa"+str(i+1) for i in range(17)])</parameter>
pa, from pa1 up to pa17 in this case, are defined. As such, with this input file we will collect all the 17 CSV files needed to produce a Detailed Report. If you only wish to collect the first 11 pa values for the Standard Report, for example, you can change range(17) to range(11).
With the file jube_run.xml and the example_run_tmpl together in the same directory, we can then run JUBE by doing
[user@ln01 JUBE_run]$ ml JUBE && jube run jube_run.xml
scp deucalion:"/path/to/JUBE_run/cpa_fapp/000000/0*/work/*.csv" .
/path/to/JUBE_run/cpa_fapp/000001, so be careful to change the previous command to download the desired run. Be sure to inspect the directory structure created by JUBE to understand the previous command.