10/24/2006 Data Management Issues Related to Drought Monitoring at Environment Canada Robert Morris Data Analysis and Archive Division Meteorological Service of Canada Toronto, Canada Presentation at the North America Drought Workshop Mexico City, October 19-20, 2006 10/24/2006 Outline • Introduction • Observational Networks and Data Characteristics • Quality Control • Forward Looking – Data Management Framework • Summary 10/24/2006 Introduction • The Canadian climatological archives contain temperature and precipitation observations starting in 1840 1900 A.D. 159 stations 2000 A.D. 1313 stations 10/24/2006 Trends and Challenges • Automatic observing technology introduces changes often faster than data management keeps up resulting in – dissimilar data sets available for analysis – loss of long-term continuity (e.g. for comparison with long-term normals) • Rationalization of networks (e.g. reductions and operational control reverting to other agencies such as observing sites at airports) • Accessing and incorporating data from other agencies networks (e.g. provinces, conservation authorities, electrical utilities) • Increased requirement for accessing data in real-time • Use of remotely sensed data (e.g. weather RADAR) and numerical weather prediction model output to augment data sparse areas and assist the quality control of in-situ data 10/24/2006 Environment Canada Daily T&P Data • Synoptic data (e.g. data transmitted in WMO standard synoptic formats on the GTS) – Human observations – mainly airports – Automatic stations • Aviation • Surface weather • Reference Climate Stations (RCS) • Cooperative / volunteer climate network – Processed from paper forms – Observer enters data on an Internet screen allowing for more timely access if the observer is so inclined Climate Volunteers Contractors Autostations Surface weather Aviation Autostations Manned National Archive System (NAS) DLY02 DLY44 DLY04 via 2304 paper form keypunch via SM via 2322 paper form keypunch PRIM04 QC RQC via SM via SM via COOLTAP …web or IVR Environment Canada Daily Climate Data Reporting and Processing ACCESS Available within hours or days Available within weeks or months SMQC A datamart with access via web services is updated twice daily with options for each station based on prioity DLY04, then DLY02, then DLY44 10/24/2006 Currently for September 2006 From Synoptic Reports 449 stations From volunteer observers (COOLTAP) 187 stations Number stations with at least 28 observations 10/24/2006 Quality Control of Climate Data • Based on “hard-wired” checks – Physical and probable limits – Physical consistency amongst elements –Trends – Spatial consistency • Legacy system lacks flexibility to add new elements or data streams or to incorporate new related data into the system to assist checking e.g. RADAR, satellite, NWP model output • The Data Management Framework (DMF) will provide a moderized approach for the future Distribution Station Maintenance Interface Data Access Policy/Maintenance Interface Middleware: Common Data Access Routines (Real-time, General Purpose) Replicate In Replicate out Sites Gateways Servers Modules for non Real-time QA/QC (e.g. spatial, manual) Data Acquisition Decode and Transformations Automatic (Real-time) QA/QC Alerting Web Services Data Access Policy Station metadata Station metadata Short-Term Storage Long-Term Storage Original concept of the DMF component-based architecture* User Applications (e.g. Value Added) Product Generators Sites G a t e w a y s Servers Data Management Framework QC Flag Summary Determination QC Category 3 Physical Limits (Range) QC Category 5 Temporal Consistency QC Category 4 Inter-Variable Consistency QC Category 6 Spatial Consistency QC Functions Category 1 Flag Category 3 Summary Flag Category 4 Summary Flag Category 5 Summary Flag QC Summary Flag Outputs QC Flag Basic Test Default tests Test Test … Default tests Test Test … QC Flag QC Flag Default tests Test Test … QC Flag QC Flag Default tests Test Test … QC Flag QC Flag QC Category 1 Data Element Presence Category 6 Summary Flag Interact with other data elements and QC flags QC Category n QC Flag QC Flag QC Category 2 Data Element Integrity QC Flag Default tests Test Category 2 Flag QC Category Data Element QC Flag QC Flag QC Flag QC Flag QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message Data/Metadata Out QC in the DMF [...]...Summary • Climate data for drought monitoring in Environment Canada is steadily improving, especially in the area of availability of near-real time data • Improvements in data management are planned in the areas of – Near-real time quality control – Incorporating new data elements and other agencies’ data – Use of related data (e.g RADAR) to assist quality control and data completeness – Metadata, access... Incorporating new data elements and other agencies’ data – Use of related data (e.g RADAR) to assist quality control and data completeness – Metadata, access systems (e.g web services), geomatics systems 10/24/2006 Gracias por su atención Thank you 10/24/2006 . 10/24/2006 Data Management Issues Related to Drought Monitoring at Environment Canada Robert Morris Data Analysis and Archive Division Meteorological. – Spatial consistency • Legacy system lacks flexibility to add new elements or data streams or to incorporate new related data into the system to assist