This new version introduces a fundamentally redesigned architecture and enhanced data processing capabilities in near real-time, marking a significant step towards real-time meteorological observations from aviation surveillance data. A timeline for this transition is provided at the end of this announcement.
Major Architectural Upgrade
EMADDC 3.0 represents a shift from a single monolithic system to a distributed, modular architecture. This new setup enables parallel processing, improved scalability and robustness, and the ability to handle (near) real-time data streams. Real-time data ingestion is supported through Apache Kafka (AWS MSK), while a dedicated database ensures efficient storage and retrieval.
To ensure system robustness, observations delayed by more than 10 seconds are automatically redirected through an alternative "delayed observations" processing pipeline and processed within 5 to 15 minutes. This approach allows delayed data to be integrated without disrupting real-time operations. The delayed observations are processed subsequently and stored in a file corresponding to their respective time window. Multiple files may be generated for the same time frame, distinguished by a suffix in the filename.
Data Access and Formats
With EMADDC 3.0, EMADDC will transition fully to file-based distribution via the KNMI Data Platform (KDP). FTP services will be discontinued once R2.2 is retired.
Key data delivery updates include:
- 5-minute datasets produced and published on KDP only. FTP file delivery is not supported by EMADDC 3.0
- Typical availability within 6.5 minutes (including processing and publication delays) relative to the start of the 5-minute file window. This reduces the delay from R2.2 by at least 5 minutes
- File formats limited to CSV and NetCDF (BUFR is no longer supported for performance and cost reasons)
- Updated KDP download scripts will support filtering by file format and new functionality to convert to BUFR upon request
- A Python conversion tool will be provided to convert CSV or NetCDF to the legacy BUFR format locally once a CSV or NetCDF file is downloaded
New Datasets
Four new datasets will be introduced on the KNMI DataPlatform (KDP):
- EMADDC General and Met Office Global
- All sources with deduplication
- Requires data licenses with UK Met Office and EMADDC
- EMADDC Met Office Global
- FR24-derived observations only
- Requires UK Met Office license
- EMADDC General
- Similar to the first dataset but excluding FR24
- Supplemented with ADS-B Support data where possible
- Available to EMADDC NDA signees
- Public/Open Dataset
- To be defined and released at a later stage
EMADDC aims to make these first 3 datasets available starting in April 2026 for testing by users and requests feedback. Quality of the provided data is equal to the data provided by R2.2. EMADDC’s current datasets on KDP will become deprecated and users are urged to transition to the new datasets in due time.
A potential streaming data option is currently under review.
Real-Time Data Expansion
EMADDC 3.0 supports real-time data ingestion and (near) real-time processing. The following sources of surveillance data are currently processed in near real-time on EMADDC 3.0:
- ADSB-SUPP
- FR24
- MUAC
- NaviAir
- CANI
- ENAIRE
- NavPortugal
- SloveniaControl
EMADDC aims to transition all existing data providers from batch/file delivery to real-time streaming. This initiative commenced in 2025, and EMADDC is actively requesting users to contact their local ATC provider (ANSP) and collaborate with EMADDC to receive real-time surveillance data through NewPENS. Current data providers who are unable to transition may be phased out, potentially resulting in reduced coverage, a situation EMADDC seeks to avoid.
Data Quality and Processing Improvements
EMADDC 3.0 maintains consistency and quality of its products with R2.2 by using the same correction algorithms, now applied to real-time data. Corrections are currently generated using R2.2 but will be fully integrated into EMADDC 3.0 before the system is considered operational. Collocation processing of observations with NWP forecast data has been modernized using Python-based workflows and global NWP data from ECMWF IFS, removing the need for regional data used by R2.2. Several code updates had to be made to work with real-time data, such as the deduplication algorithm, removal of outlier checks, and several quality control checks which were no longer valid or possible in a real-time processing system. These changes result in an increased number of observations (specifically temperature) whilst maintaining or even improving quality. Future development will focus on real-time correction methods that can also incorporate delayed observations, planned beyond EMADDC 3.0.
Coverage
The current observation coverage of EMADDC for users that have access to the full dataset (e.g., Met Office and EMADDC data licenses signed) is illustrated in the accompanying figure, showing dense data availability across Europe, North America, and parts of Asia, with expanding coverage in other regions in the future.
Timeline
- April 2026: New 5-minute datasets available on KDP from EMADDC 3.0 for testing
- July 2026: EMADDC 3.0 considered operational
- Autumn 2026: R2.2 will be officially terminated, data no longer available on FTP
EMADDC 3.0 marks a major milestone in delivering timely, scalable, and high-quality aircraft-based meteorological observations, aeronautical stakeholders, operational forecasting, NWP and research communities.
For any questions regarding this transition, please contact EMADDC at emaddc@knmi.nl