Carbogo

Overview

Carbogo is a web-based monitoring platform that lets you track the energy consumption and CO₂ emissions generated by your jobs on the GenOuest cluster. The goal is to enable an accounting of computational resource usage and environmental impact. Carbogo also aims to reduce the platform's CO₂ emissions by giving users feedbacks on the efficiency of their computations.

What you can monitor

Through Carbogo, you can see reports spaning from April 10, 2026 to today for:

• Number of your jobs that were successful
• CPU time used for your successful jobs
• Energy consumption for your successful calculations and your storage consumption (your home and scratch storage + project storage if you are the owner of the project)
• CO₂ emissions associated with your successful jobs and your storage consumption
• Number of jobs submitted that have failed and the CPU time they used
• Failed jobs consumption and emissions to understand the impact of your unsuccessful computations
• CPU efficiency to understand how much you really use the ressources you asked for
• Project-level storage energy usage if you're part of one or multiple project(s)

Using the calendar, you can select the time span you want or access predefined time spans quickly by using preset buttons.

How energy consumption and CO₂ emissions are estimated

Data sources

The GenOuest platform uses PDU (Power Delivery Units) outlets which allows to track the energy consumption of each machines. The energy consumption of each type of equipment (CPUs, GPUs and storage) is extracted and is used for calculation. The energy usage of the climatisation is also tracked and used in our calculation.

The data about user jobs, user storage and projects storage are gathered from the cluster.

These information are used to calculate the proportion of your energy consumption for each type of hardware component.

Energy consumption calculation

• For each user the CPUs and GPUs consumption is calculated as: user consumption = total resource consumption * (user's resource hours / cluster total allocated resource hours).

• The same calculations are repeated for jobs that failed, providing separate metrics for unsuccessful computations.

• For each user the storage consumption is calculated as : user consumption = total resource consumption * (user's used storage / cluster total allocated storage). If the user is owner of one or multiple project(s), the user owned project(s) energy consumption is added to the user storage energy consumption.

• Projects' energy consumption is based on their disk storage usage: project energy = total storage energy * (project disk space / cluster total allocated disk space).

• The user total consumption is calculated as : total cluster energy consumption * ((user CPUs energy consumption + user GPUs energy consumption + user storage energy consumption) / (cluster CPUs energy consumption, cluster GPUs energy consumption + cluster storage energy consumption )).

CO₂ estimation

To estimate the CO₂ emissions generated by the jobs and the storage of a user, we get the mean CO₂/kWh for the day, and we apply this rate to the electrical consumption of the user for the day. This means we can provide an accurate estimate of CO₂ emissions for each user, as these estimates take into account day-to-day fluctuations in emissions from the power grid. When the user select a time range on the Carbogo interface, we sum the precalculated emissions of each individual day of this user inside the selected time range.

To get CO₂/kWh data, we use the éCO2mix platform. This platform is operated by the France's Transmission System Operator (RTE), which is the authoritative body responsible for monitoring consumption and assessing emissions within the national system.

Key assumptions and limitations

The energy calculation model relies on these important assumptions:

1. Equal cost for CPU time: We assume that between our machines each CPU hour have the same cost, so that the electrical consumption of each machine in the cluster is equal for the same calculation time.
2. Project storage attribution: Currently, the power consumption and the CO₂ emissions of the project storage are only added on the energy and emission counts of the project owner. The members of the project see the energy consumption of the project under the project part but their personal energy and emission values are not incremented with the values of the project.
3. Machines manufacturing and transportation : We also don't take into account the CO₂ emissions generated by the manufacturing and the transportation of the equipments. But in reality, these factors account for the majority of CO₂ emissions of the equipments.

The first assumption have a very limited impact on the final results and can't be circumvented.

The second assumption reflect a decision regarding who is decided to be responsible for the emission. Currently, the project owner is considered to be the person who owns the project and is therefore "responsible" for the emissions. But in reality, the project is often shared among several members, each of whom has a specific role in the use of storage space. For users who wish to do so, they can calculate their own emissions using electricity consumption data that are given under the project part and their own estimation of their share of the project storage usage.

The third limitation is a big approximation, because the manufacturing and the transportation account for the most part of the CO₂ emissions of a machine. So this information should be kept in mind when the results are used as you only see the emissions of your electrical consumption.

How to reduce my environmental impact using the cluster ?

Context

A big part of the cluster’s environmental impact stems from the manufacture and transport of components and machinery. So the best way to reduce the CO₂ emission of our machines is to not create the machines. This means that we should aim at buying the lowest number of machines possible.

Carbogo exists because users play a very important role in reducing equipment purchases and electricity consumption.

What can I do ?

For a user, the simplest way to reduce the cluster’s carbon footprint is to request only the resources they actually need. In fact, CPUs are currently used only 15% of the time they are allocated to a user. When the ressources are allocated to users but not used, it sends a false signal to admins, giving them the impression that resources are being heavily used. This forces administrators to purchase additional hardware to meet the demand for resources. This new hardware is manufactured, shipped, and runs 24 hours a day, creating the majority of the CO₂ emissions of the cluster.

How do I dimension my jobs effectively ?

Unfortunately, there is no one-size-fit-all parameters for the jobs. The parameters of the jobs will vary depending on the software used : some require a lot of RAM, others require multiple CPUs to run faster through parallelization, and others require only a single CPU because they cannot parallelize their calculations.

However, finding the right combination of parameters isn't difficult once you have performed a little bit of testing with your software. The most effective way to dimension a job is as follows:

• Launch a first job and wait it to finish.

• If it's a large job that uses a lot of data and/or takes a lot of time, it's a good idea to first run a test job on a small subset of your dataset. This has three advantages:

  • It allows you to verify that the job doesn't crash due to a human error in the script.
  • It allows you to verify that the job doesn't crash after multiple hours of calculation because it needs more RAM (OOM error).
  • It also gives you an idea of the parameters to use without (hopefully) waiting for too long.

• Once the job is finished, you can use the command seff [job_id] (e.g. seff 1884158). This command will print out summary statistics and efficiency information about the job. There is an exemple :

  Job ID: 1884158 Cluster: GenOuest User/Group: foo/bar State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 16 CPU Utilized: 01:06:03 CPU Efficiency: 38.76% of 02:50:24 core-walltime Job Wall-clock time: 00:10:39 Memory Utilized: 21.93 GB Memory Efficiency: 17.14% of 128.00 GB

• We are interested in three lines :

  • CPU Efficiency : This shows how much CPU time you used compared to CPU time you asked for (the higher the better)
  • Memory Utilized : This shows the maximum amount of RAM your job used
  • Memory Efficiency : This shows how much RAM you used compared to the RAM you asked for (the higher the better)

• In this exemple, the job is not well calibrated:

  • We can see that only 17.14% of the RAM requested for have been used (only 21.93 GB used of the 128 GB requested). This means 106.07 GB of RAM have been locked for the user without a use.
  • Same for the CPU, only a little bit more than a third of the CPU have been used, meaning two-third of the CPU time have been allocated to the user without a use. (But it's well above the CPU efficiency of our cluster which is 15% currently, so in reality, this score is not so bad in perspective).
  • Because we see that the job has run quickly (in 10 minutes and 39 secondes) and that not a lot of RAM has been used, we can make the hypothesis that this is not a problem of bottleneck from data reading or data manipulation. So it's just that the software doesn't need that many resources and isn't using them all.
  • In that case, we can simply reduce the amount of RAM and CPU allocated to the job for our future uses (if we use a similar data size). For example, we could give 30 Gb of RAM and 8 CPUs (to allow for a little leeway and prevent the job from failing if the data uses slightly more RAM than expected).
  • Please note: If you used a test job using a small subset of your dataset, be sure to adjust the job's RAM accordingly. For example, if the program loads all the data into RAM, you'll need to increase the RAM when using your real data.

Some tips :

• Understanding how the software works can speed up testing. For example, there’s no need to test the number of CPUs for software whose documentation states that it uses only one core because it cannot parallelize its calculations.

• In some cases, it can be interesting to split your job into multiple sub-jobs. Let's imagine we have a script with two steps (a fairly common scenario in bioinformatics):

  • A first step of data preparation, that manipulates the data (for exemple to convert the data into a new format). This step needs a lot of RAM and time but not a lot of CPU.
  • A second step of calculation, that uses the newly generated format. This step requires a lot of CPU.

  If you launch your script inside one job, the CPU will be idle during the data preparation step (that takes a lot of time). So you will waste a lot of CPU time. If you split the script into two jobs, one that uses a lot of RAM but not a lot of CPU to perform the data manipulation and then a second job that uses a lot of CPU to perform the calculation, your CPU efficiency will be way higher.

  Creating two jobs can also help you reduce your wait time before the job launch. To determine when to launch your job, SLURM takes into account the quantity of resources you ask for. The smaller your job is, the quicker it will be launched. So, if you have two jobs that require less resources, your two jobs will have a higher chance of being launched quickly than a big job requiring a lot of resources. (Other information are also taken into account).