- Calibsun | Fachkompetenz
Data Quality Check & performance monitoring
Why sensor quality check matters ?
On a PV plant, data integrity directly conditions every downstream process: solar and power forecasting, performance monitoring and asset management, and solar resource assessment. A pyranometer reading affected by soiling, frost, or misalignment does not produce an obvious error signal (sensor data continues to arrive, timestamps are valid, values appear plausible). Without a structured quality check process, corrupted measurements propagate silently into machine learning algorithms and monitoring systems, degrading data quality and energy production estimates.
CalibSun’s Sensors Expertise is designed to address this challenge across the full lifecycle of a solar PV plant: from historical data validation at onboarding, to automated real-time flagging on every incoming measurement. This data management approach ensures reliable data quality for both performance analysis and power forecasting, regardless of site size, weather conditions, or hardware configuration.
Key elements
- Pyranometer misalignment, soiling, and decalibration are the primary sources of data quality issues on operational PV plants - none produce obvious discontinuities in the time series. This technical approach ensures accurate sensor readings and protects data integrity throughout operational periods.
- CalibSun führt eine zweistufige Qualitätsprüfung durch: eine einmalige Qualitätskontrolle der historischen Daten bei der Erfassung für maschinelles Lernen sowie ein kontinuierlicher, automatisierter Markierungsprozess für jede eingehende Messung.
- Meteorological cross-validation — temperature, humidity, wind — is a critical layer for detecting invalid data and sensor anomalies, particularly under difficult winter weather conditions.
- On utility-scale sites, one pyranometer per 5 to 10 MW, distributed regularly across the solar power plant footprint, is required to resolve spatial irradiance data gradients.
- Data quality directly impacts performance ratio calculations, satellite calibration, and the reliability of machine learning forecasting models trained on historical data.
What is Sensor Quality Check ?
On an operational PV plant, the pyranometer is the primary reference for incoming solar irradiance, GHI on a horizontal plane, GTI on a tilted plane matching panel orientation. It feeds performance monitoring systems, provides ground-truth irradiance data for satellite calibration, and serves as the main data acquisition input for photovoltaic power forecasting models. This active role makes sensor readings and quality inspection essential to ensure reliable data flows through every downstream application and technology platform.
Pyranometers are precise instruments operating in harsh outdoor conditions. Three main failure modes affect sensor performance and data quality in practice:
- Soiling: dust, pollen, or pollution accumulate on the sensor dome, progressively attenuating measured solar irradiance without producing obvious discontinuities.
- Decalibration: sensor sensitivity drifts over time, introducing systematic bias difficult to detect without a reference instrument or cross-validation.
- Misalignment: following incorrect installation or physical damage, the sensor no longer sits in its nominal plane, directly distorting GHI or GTI measurement readings.
None of these failure modes are self-evident or produce missing data rates. Without data quality control, erroneous sensor data is treated as valid by forecasting models, satellite calibration pipelines, and monitoring systems alike – with direct impact on performance estimates and energy output calculations.
Beyond irradiance sensors, meteorological sensors (measuring ambient temperature, wind speed, and relative humidity) also require data validation and quality assurance. These measurement sensors and environmental conditions data are used to cross-validate irradiance data, identify sensor anomalies, and refine photovoltaic production models. By performing rigorous testing, evaluation, and visual inspection of sensor accuracy and environmental conditions, operators ensure that sensor performance remains trustworthy across operational periods.
- Quality check steps
Data Quality Control Criteria: How Invalid Data Is Detected
Physical Consistency: Extremely Rare Limits (ERL)
The first layer of data quality verification applies physical consistency checks to the irradiance data time series. CalibSun uses the Extremely Rare Limits methodology (Long & Dutton, 2002), which defines bounds based on astronomical constraints.
Two categories of physically invalid data are flagged at this stage:
- GHI values exceeding extraterrestrial irradiance: the solar resource received above the atmosphere defines an absolute physical upper bound. Any measured value exceeding this threshold is physically impossible and is flagged unconditionally.
- Negative irradiance values during daylight hours: while small negative readings can occur at night due to sensor noise, negative values during solar hours indicate a hardware fault or data processing anomaly.
These ERL checks form the ground layer of any rigorous data quality process, filtering out values that no statistical methods or contextual analysis could rehabilitate.
Meteorological Cross-Validation
The second layer of quality assurance exploits the physical relationships between solar irradiance and concurrent meteorological variables. A measurement inconsistent with observed weather conditions is a strong indicator of a sensor issue.
The most operationally significant case involves winter conditions. When ambient temperature drops below 0°C and relative humidity exceeds 90%, non-heated, non-ventilated pyranometers are exposed to frost and freezing fog. The impact is systematic:
- The sensor dome frosts overnight, rendering irradiance data invalid from the first morning hours.
- As temperature rises above 0–2°C, the sensor gradually defrosts, but water droplets on the dome can temporarily act as a lens, producing irradiance readings that exceed the actual incoming solar resource.
- Until full defrosting is confirmed, all measurements from non-heated sensors must be flagged as invalid data.
Heated and ventilated pyranometers are specifically designed to prevent this failure mode. In the field, this distinction has a measurable impact on data quality: during a winter campaign conducted by CalibSun at a PV site in Ardennes, France, the heated and ventilated reference pyranometer showed only 6.6% of data points excluded, while non-heated sensors required 44.7% of data points to be flagged over the same period, due solely to frost, freezing fog, and condensation effects.
This field result illustrates why heated and ventilated pyranometers are the correct choice for the reference sensor in any solar energy project located in a climate with cold and humid winters.
Inter-Sensor Spatial Consistency
On sites equipped with multiple pyranometers, data quality can be further assessed through spatial cross-validation. When two sensors at different locations on the same solar power plant show a persistent divergence that is inconsistent with expected spatial irradiance gradients, the dataset flags this as a potential sensor anomaly requiring investigation.
This approach also enables detection of localised shading events that affect one part of the plant but not another, a situation that a single-sensor monitoring setup would either miss or misinterpret as a global irradiance drop.
Orientation and Horizontality Validation
Misalignment of a pyranometer, whether due to incorrect installation or mechanical drift, directly distorts GHI and GTI measurements. CalibSun validates sensor orientation by comparing measured irradiance data against the theoretical clear-sky irradiance profile on cloud-free days. Under clear-sky conditions, a correctly oriented and horizontal pyranometer must track the expected clear-sky envelope with high precision in both amplitude and temporal shape.
Any systematic deviation, a consistent offset, an asymmetry between morning and afternoon readings, or a peak occurring earlier or later than the theoretical maximum, indicates an orientation issue that requires field correction. This validation step is implemented as part of every sensor audit and at the start of all services : during satellite calibration campaigns and when implementing forecasting algorithms. If the quality check detects an issue on one of the on-site sensors, our expert team can provide direct recommendations to the operators to help them resolve it and improve forecast accuracy.
Historical and Real-Time Quality Check
Historical Data Quality Control
When CalibSun onboards a new solar PV plant, the first step is a systematic data quality control pass over the full available historical data archive. This process typically covers one to several years of time series from SCADA systems, pyranometers, and meteorological sensors.
The objective is to produce a clean, validated dataset that can serve as the training base for the machine learning forecasting model. Every flagged or missing data point is documented. Missing values are not filled arbitrarily – gaps in the historical data are either left explicitly marked or reconstructed using physically consistent methods where the gap duration and surrounding context allow it.
The quality of this initial data processing step has a direct and lasting impact on forecast accuracy: a model trained on corrupted or biased historical data will encode systematic errors that persist throughout its operational lifetime. Accurate data collection and rigorous data validation at this stage are therefore not optional, they are the foundation of performance and reliability for every subsequent forecast.
Automated Real-Time Data Quality Control
Once a solar power plant is operational within CalibSun’s forecasting pipeline, sensor data is ingested continuously via API at the highest available resolution, optimally 1 minute, operationally 5 to 15 minutes depending on SCADA configuration.
Every incoming measurement passes through the automated quality check pipeline before being used in the forecasting process. This pipeline applies, in sequence:
- ERL physical consistency checks, filtering physically invalid data unconditionally.
- Meteorological cross-validation, best practices for flagging readings inconsistent with concurrent temperature, humidity, and wind parameters.
- Inter-sensor consistency checks, comparing simultaneous readings across distributed pyranometers where available.
- Clear-sky envelope verification, detecting values that exceed the theoretical maximum Sonneneinstrahlung for the current solar geometry and atmospheric conditions.
When a measurement is flagged, the system does not simply discard it. The flagging logic records the reason, the time of occurrence, and the associated meteorological context. This structured approach to data management enables retrospective analysis of sensor behaviour over time, supports maintenance scheduling, and feeds back into the calibration of quality thresholds.
Alerts can be triggered when a sensor produces an abnormal rate of flagged values over a defined time window, providing an early alert signal before a hardware failure fully compromises the dataset.
Sensor Quality Check and Its Impact on Forecasting Performance
Data quality issues at the sensor level have cascading effects on every layer of the forecasting process.
Rigorous sensor quality check is therefore not a standalone engineering task. It is structurally integrated into forecasting accuracy, satellite calibration quality, performance monitoring reliability, and energy trading risk exposure.
Satellite calibration
CalibSun uses local ground-truth irradiance data to calibrate satellite-derived irradiance products. If the reference pyranometer carries a systematic bias, the satellite calibration absorbs that bias, degrading the spatial representativeness of the corrected irradiance field across the entire plant footprint.
Model training
Machine learning models trained on historical data containing undetected data quality issues will encode those errors as signal. The resulting model will generalise poorly to clean input conditions, producing forecasts with degraded accuracy and poor performance metrics.
Real-time recalibration
CalibSun’s NEXT algorithm recalibrates continuously using the most recent local sensor data. If incoming measurements are corrupted and pass undetected through the quality check, the recalibration step will adjust the model in the wrong direction, temporarily increasing forecast error rather than reducing it.
Performance ratio and asset management
Performance monitoring relies on accurate irradiance data to compute the performance ratio and detect underperforming strings or inverters. Biased irradiance inputs produce biased performance ratio values, masking real degradation or generating false alerts, both of which carry direct asset management consequences.
- Fragen
Frequently asked questions about Sensor Quality Check
Wie viele Pyranometer werden in einer Solarkraftanlage im Großmaßstab benötigt?
For small PV plants, a single quality pyranometer is generally sufficient. For utility-scale sites, cloud cover moves across the plant footprint at varying speeds, creating spatial irradiance gradients that a single sensor cannot resolve. CalibSun recommends one pyranometer per 5 to 10 MW, distributed regularly across the site, to enable spatially resolved irradiance data analysis and robust data quality cross-validation. This standard design follows recommendation from Task 16 from IEA PVPS (Best Practices Handbook for the Collection and Use of Solar Resource Data for Solar Energy Applications: Fourth Edition / Manajit Sengupta, Aron Habte, Stefan Wilbert, Christian Gueymard, Jan Remund, Elke Lorenz, Wilfried van Sark, and Adam R. Jensen – 10.69766/ENEH5295 978-3-907281-66-6 ) ensures product quality and accurate measurement across the supply chain.
What happens when sensor data is flagged or missing?
When sensor data is flagged as invalid or temporarily unavailable, the forecasting system continues to operate using satellite and NWP inputs alone. Forecast accuracy is reduced during these gaps, but the service does not interrupt. As valid measurements resume, the model recalibrates automatically. Persistent missing data or high flagging rates trigger an alert for field investigation – enabling you to identify and diagnose potential sensor issues before they propagate into the forecasting dataset.
How does sensor data quality affect the performance ratio?
The performance ratio is computed as actual energy output divided by expected energy output based on measured irradiance data. If the pyranometer carries a systematic bias — whether from soiling, decalibration, or misalignment — the expected energy output reference is incorrect. This produces a biased performance ratio that either masks real degradation or generates false alerts — both of which have direct asset management and O&M impact.
Can data quality issues be detected automatically?
Yes. CalibSun’s automated quality check pipeline applies physical consistency checks, meteorological cross-validation, inter-sensor spatial consistency analysis, and clear-sky envelope verification to every incoming measurement in real time. When a sensor produces an abnormal rate of flagged values over a defined time window, an alert is triggered for field investigation before the issue propagates into the forecasting dataset.
