Wiki software last updated 28-Jul-2014 - Comments to Ruben Squartini <>

SD Shift

  1. SD shifts were recommended by the Service Task committee
  2. As expressed in recommendations, SD shifts aim to check the quality of the data

In SD shifts

  • Not requested to go through all alarms, to check every LS/PMT in alarms : The Observatory staff is taking care of LS/PMT troubles

Available monitoring Web Sites

The Auger Monitoring web site at Malargue is located at [], where it is possible to access either the production version (stable) or the development one.

Replications of the web site and the associated databases are at the European mirror set in Wuppertal, [].

General documentation

(see also the logon page)

To have information on the login procedure or to request for updating your role(s), please read the documentation either on the The login and role pages or on QuickStart.pdf.

If needed feel free to contact the site administrator.

Please look ast the user guide on wiki on this auger wiki if you're not an expert of the Auger Monitoring web interface. Many SD pages are described, and the way to use display tools is explained. So refer to this wiki help when doing the checks.

General layout and SD section are explained in Gap-2010-038. It is highly recommended to read it for a properly use of the web site.

Online Data checking steps

1) Status of SD DB table filling

Since all the displayed informations correspond to data obtained via mysql request on DataBase tables, it is worthwhile to first check that the DB tables are filled.

On the web site, a colored table is displayed, showing the filling status of 5 SD tables. You should not care about the “CDAS” one, always red. Nobody seems to care about its filling among CDAS experts.

MonitCalib, LongTermMonitCalib, T2LSinfo and T2LSStatus should be filled. If the corresponding cells are colored, the situation is not normal. If the color is orange or red, there is clearly a problem.

If no new data since 12 hours, send a mail to : t.b.d. FIXME

If no new data since 12 hours,

  1. either there is no filling of the data base table on moni (previous case)
  2. either the internet link between Malargue/Wuppertal is broken (test it by doing a ssh for example, if you have to mean to check send a mail to : t.b.d. )
  3. either it can be that the replication has a trouble. send a mail to :

If tables are updated correctly, you can check the online data.

2) Efficiency of the array

Black Tanks

A black tank is a tank (i.e. a station) that is expected to send T2 to CDAS but has not sent any during the last 24 hours, meaning in particular that it does not appear in the database T2LSStatus table.

Array map and list

on moni site : Black Tank

on mirror site: Black Tank

The displayed interactive map allows to locate and identify any black tank, drawn as black points, with a link to the corresponding SD/LS page.

If the number of black tanks seem to be abnormal (increase of total numbers over last days)

Evolution of Black tank numbers with time - Efficiency of the array

Only on mirror site

The number of Black tanks per day, as well as the array efficiency (LS in array - Black tanks) can be checked on:

If the number of Black Tanks has abnormally increased, send a mail to : FIXME

T2 rates

To be more quantitative, check the trigger rates over the full array using

Select “MonitCalib” in the first combo, then the list of available monitoring variables will appear in a new combo (it may take time). Select “T2rate” in this one. By default, the average value over the time period (set in the footer) is displayed.

3) Status of LS and PMTs

Different checks can be done. They concern the SD stations (batterie status, disk, software version), and the PMTs.

Check the number of Masked PMTs.

The evolution of the number of masked PMT with time is available

Use the combo to display MaskedPMT vs time, and choose an appropriate time period (use the footer). In case of abnormal increase, the reason has to be identified. Looking at the alarm history could help. In case of abnormal number, contact FIXME

It is worthwhile to check the number of stations having two masked PMTs : that means that only one PMT is used in data taking, and that the station can become a Blank Tank if the working PMT has a failure (menu SD/BlackTank/Greytanks) []

Check the number of raining PMTs.

The variation is smooth, following a light seasonal effect.

Other checks

Beside this, other tools are available, if needed, to check the status of each PMT, as well as the status of Batteries, etc.

select the variables or items you want to look at (:!: help is: here )

The Alarms of the Auger Monitoring system are described in the alarm list.

Automatic report edition

From the web site, you can produce 2 reports (pdf files). They contains most of the information mentioned above.

1 - SD Monitoring report

2 - PMT Commisionning report

Checks using information from Offline Data files

The following checks concern information retrieved by analysing data files and/or production files. So they are not available online. In these cases, there is a delay (due to data transfer to Lyon, where files are analysed) between the time your are looking at the monitoring web site and the last input in the database.

Check T3 errors

:!: available only on the mirror site (paomon) since the DB table in which related information are stored exist only on the paomonDB.

:!: Moreover, the table is not filled online, but offline, using AugerSd data files stored on CC IN2P3

The aim is to display the rate of T3 errors (in particular, the T3 Lost) and the T2 Inconsistencies.

see : SD Triggers for a full description of the displays

Check number of events

It is a “by-product” of the previous checks. The number of events is stord per half a day. Check that the rate is constant. If it is not, is it probably due to a problem identified with the previous checks. If no problem has been identified, the event rate decrease has to be reported.

Data Files

The number of files per day as well as the size can be check using the tools available:

The data is filled looking at two types of files:

  • Sd data sd/yyyy/mm/sd_yyyy_mm_dd*.root
  • Production files v0r2/Auger/ad_yyyy_mm_dd*.root and v0r2/Auger/adsd_v2r0_yyyy_mm_dd*.root

With the menu and the selectors, it is possible to check if files are there for each day, if some are missing, if the size is normal… see Data / Prod Files


  • PMS : (Parts Management System database)
* [[monitoring:shifter:bigbrother|BigBrother]] : do not use it. Use the [[monitoring:user|Auger Monitoring software]]
* [[monitoring:shifter:littlebrother|LittleBrother]]: do not use it. Use the [[monitoring:user|Auger Monitoring software]]

Log In