Ronald H. Brown IMET Data Quality Control Report: 1999 and 2000

Jeremy J. Rolph and Shawn R. Smith Research Vessel Surface Meteorological Data Center Center for Ocean Atmospheric Prediction Studies Florida State University July 2003 Report 03-01 Version 1.0
1.0 Introduction

This report summarizes the quality of surface meteorological data collected by the research vessel Ronald H. Brown (identifier: WTEC) IMET system during 10 cruises beginning 22 February 1999 and ending 04 June 2000. The data were provided to the Florida State University-Research Vessel Surface Meteorology Data Center (RVSMDC) in electronic format by Jonathan Shannahoff and converted to standard RVSMDC netCDF format. The remaining data for 2000 are not available at the Data Center. Original data received were in ten-second intervals. One minute centered averages were constructed by the Data Center. The available data were preprocessed using an automated screening program, which adds quality control flags to the data, highlighting potential problems. Finally, the Data Quality Evaluator (DQE) reviewed the data and current flags, whereby flags were added, removed, or modified according to the judgment of the DQE and other RVSMDC personnel. Details of the quality control procedures can be found in Smith et al. (1996). The data quality control report summarizes the flags for the Ronald H. Brown IMET surface meteorological data, including those added by both the preprocessor and the DQE.


2.0 Statistical Information

The Ronald H. Brown IMET data include observations averaged every minute. Values for the following variables were provided:

Time(TIME)
Latitude(LAT)
Longitude(LON)
Platform Heading(PL_HD)
Platform Heading 2(PL_HD2)
Platform Course Over Ground(PL_CRS)
Platform Speed Over Ground(PL_SPD)
IMET Platform Relative Wind Direction (14.12 m)(PL_WDIR)
IMET Platform Relative Wind Speed (14.12 m)(PL_WSPD)
Earth Relative Wind Direction (14.12 m)(DIR)
Earth Relative Wind Speed (14.12 m)(SPD)
IMET Platform Relative Wind Direction 2 (14.12 m or 25.5m)*(PL_WDIR2)
IMET Platform Relative Wind Speed 2 (14.12 m or 25.5m)*(PL_WSPD2)
Earth Relative Wind Direction 2 (14.12 m or 25.5 m)*(DIR2)
Earth Relative Wind Speed 2 (14.12 m or 25.5 m)*(SPD2)
IMET Platform Relative Wind Direction 3 (25.5 m)(PL_WDIR3)
IMET Platform Relative Wind Speed 3 (25.5 m)(PL_WSPD3)
Earth Relative Wind Direction 3 (25.5 m)(DIR3)
Earth Relative Wind Speed 3 (25.5 m)(SPD3)
Atmospheric Pressure(P)
Air Temperature(T)
Sea Temperature(TS)
Relative Humidity(RH)
Atmospheric Radiation (Shortwave)**(RAD)

*-From Cruise 99-F through the cruise of 2000, data from the anemometer at 25.5 m was stored in the second set of wind variables. PL_WDIR3, PL_WSPD 3, DIR 3, and SPD 3 do not exist for these files.

**-Shortwave atmospheric radiation was only measured on the 99-B cruise and a high percentage of these data were flagged(Table 2).

A number of parameters on the cruises were found to be of poor quality but are included in the statistical results, Table 1 & 2, although some of these values will not be publicly released with the data (see details below, Section 3). The true winds from cruise 99-C and 99-D were removed because most of the heading data, required to calculate the true winds, were missing during the cruises. Lack of data from the second anemometer, from 28 March around 7Z and until the end of the cruise, resulted in the removal of these data for cruise 99-D. 99-E had substantial problems with the second anemometer, erroneous heading data, and the course data were suspect. The result was 3 of the variables used to calculate the true winds having major problems. Consequently, derivation of the true winds using these necessary, yet suspect, variables results in highly questionable true wind values which were flagged but left in the 99-E data. QC on 99-A was not completed due to problems found during the conversion of the data to netCDF format. Note also that flow distortion was found to be a major problem on seven of the ten Ronald H. Brown cruises, resulting in extensive flagging (discussion follows, section 4.2).

Details of the cruises are listed in Table 1 and include cruise dates, number of records, number of values, number of flags, and total percentage of data flagged. A total of 4,545,620 values were evaluated with 392,824 flags added by the preprocessor and the DQE resulting in a total of 8.55 percent of the values being flagged.


Table 1: Statistical Cruise Information
Cruise Identifier*Cruise DatesNumber of RecordsNumber of ValuesNumber of FlagsPercent Flagged
99-A02/11/99 ­ 02/20/99ExcludedExcludedExcludedExcluded
99-B02/22/99 - 03/01/999,702194,0409,1414.71
99-C03/09/99 - 03/23/9919,228423,01680,51819.03
99-D03/26/99 - 04/02/9910,306226,73262,01727.38
99-E04/08/99 - 04/18/9914,298314,55643,31913.77
99-F07/28/99 - 08/19/9932,684588,31213,2342.25
99-G08/24/99 - 09/21/9938,892700,05631,0234.43
99-H09/25/99 ­ 10/23/9937,570676,26033,4734.95
99-I11/02/99 ­ 11/06/994,80286,4361,2291.42
99-J11/10/99 ­ 12/16/9953,216957,88897,23410.15
200005/21/00 ­ 06/04/0021,018378,32421,5285.70
*assigned by RVSMDC to ease identification


3.0 Quality Control Information

The quality of the IMET data from the research vessel Ronald H. Brown ranged from good to very poor, depending on the cruise and variable. It was determined that some of the variables from individual cruises need to be removed from the data set altogether, although these data are available upon request. Some of the platform relative winds and headings are suspect, yet remain in the data set and should be used with caution. Table 2 details the distribution of flags among the remaining variables. Discussions of the flagged and removed variables follow.


Table 2: Number of Flags and Percentage Flagged for Each Variable
VariableBEFGHIJKSTotal Number of FlagsPercentage of Variables Flagged
TIME          0.00*
LAT  25   61371,1991,3220.50
LON  21   67391,8001,9270.73
PL_HD      8,16323,246831,41711.96
PL_HD2          0.00*
PL_CRS      46537635650.21
PL_SPD      231501504310.16
PL_WDIR       355173720.14
PL_WSPD    1  209300.01
DIR 22,670    1,03832,657556,37021.46
SPD 23,447 75  98135,184659,96322.72
PL_WDIR2      18,6012171218,8307.17
PL_WSPD221   1 14,4253,1041717,5686.69
DIR2 17,191    15,28037,3271069,80826.57
SPD2 4,550 152  12,27359,434 76,40929.08
PL_WDIR3       263290.06
PL_WSPD35,491   1    5,49212.53
DIR3 20,483     282 20,76547.37
SPD3 18,836 100 1 149219,08843.55
P   734 1 25418010.33
T4  98 1 666 7690.32
TS31  33  1171,3771901,7480.72
RH743  258   1 1,0020.41
RAD4,755        4,75549.01
Total Number of Flags11,045107,177461,4503371,702194,2333,532389, 91
Percentage of All Values Flagged0.242.360.00*0.030.00*0.00*1.584.260.078.55
*Percentages < 0.01


3.1 Deleted Data

The DQE determined that a large amount of the 1999 and 2000 Ronald H. Brown IMET data were unusable due to extensive missing, highly suspect, or erroneous data. As a result, these data were removed from the final quality controlled data set.

True winds were removed from cruises (99-C, 99-D, and 99-E), due to very poor quality. The reason for the true winds in 99-C and 99-D being removed is stair stepping of the winds due to the large amount of missing heading data and some suspicious course data in the set. The course and heading are key variables used in calculating the true winds. If they are in error, the results of the calculations will also be incorrect. For the 99-E cruise, the heading data was suspect, if not missing, as was the course data and the second anemometer's platform relative winds were mostly missing. An attempt was made by the RVSMDC to repair the winds with no success. The suspect data (course, heading and platform relative winds) were left in the set although the true winds were not.

On the 99-E cruise the platform relative wind speed from the second anemometer is identical to the platform-relative wind direction for the second anemometer on 8 April. Due to the inability to determine which variable was correct, both variables were K-flagged. After this time, these data, PL_WSPD2 and PL_WDIR2, are missing. During the 99-D cruise on April 1, the platform speed is equal to the true wind speed of anemometer two because the platform relative wind speed from anemometer two is missing. Therefore the platform speed was used in calculating the actual wind speed. The platform relative winds recorded for anemometer 2 need to be removed from both of the sets because of lack of data and quality of the data that was recorded.


3.2 Missing Data

During the 99-B cruise, all data went missing from about 4:38 ­ 17:10Z 24 February. Throughout Cruise 99-C, 14 March for example, true winds were recorded [DIR (1 & 2) and SPD (1 &2)] during the time both of the platform relative winds and the heading went missing. These values recorded obviously mimicked the movement of the ship, stair-stepping (section 4.1) right along with platform speed, course and heading (when recorded). All values recorded for DIR and SPD were flagged as unusable with the J-flag. This also occurred during Cruise 99-E from 8 August to the end of the cruise, and 99-J with the second anemometer frequently failing to record any values, subsequently being flagged in the same fashion, and for the first anemometer for a short time on 29 August 1999. 99-G has platform wind data drop out from around 6:00Z 12 September until 21:32Z 13 September. On 31 March the platform relative winds were not recorded and were flagged appropriately during cruise 99- D. Cruise 99-J failed to record any data from 12 December to the next day.

The pressure is missing in cruise 99-C from 9-10 March.

Missing heading data results in true wind errors, since the heading is required to determine the true winds, on cruises 99-C, 99-D, and 99-G cruises. 99-G cruise has some of the course data missing on 12 September for a day, also affecting the validity of the true winds.

Values for some, if not all, variables were missing from the following cruises: 99-B, 99-C, 99-D, 99-E, 99-G, 99-H, 99-I, 99-J and finally the 2000 cruise. The variables begin to go missing on the cruises, leaving large gaps in the data, but they sporadically record only a few minutes of data values during these gaps. These data values were highly suspect and received the K-flag.

The missing data, or gaps in the time series, for all variables in the 99-I cruise were not labeled ­9999.f. Therefore, for the period of time from just after 00:00Z 6 November 1999 and until 18:07Z, where the data was missing, the visual editor creating the time series plots simply drew a straight line connecting the two end points of recorded data. This yielded values that were not actually measured. This occurred again in all of the variables for the cruise of 2000 from 1520 ­ 1618Z 27 May.


4.0 Variable Flagging


4.1 Stair stepping

Stair stepping of the meteorological variables was an inherent problem found in this data set. The problem was experienced by most variables. Stair stepping is a response in the data to the research vessel's change in platform speed or direction, which often stair-step. Meteorological data readings, in the absence of flow distortion (section 4.2), should not reflect ship-motion and therefore such values received the cautionary K-flag. The true winds of this set commonly stair-step along with the motion of the boat. The pressure data was one parameter that did not reflect any stair stepping.


4.2 Flow Distortion

Flow distortion was suspected to be the cause of some serious problems revealed by all three of the wind vanes on the Ronald H. Brown. Flow distortion is caused by the wind flow around the superstructure and cargo of the ship and the relative position of the wind vanes. Being that the ship's deck cargo likely changed from cruise to cruise, depending on the type of research being done and the type of instruments carried on board, the severity of the flow distortion also varied from cruise to cruise. On the majority of the cruises, suspect wind data (K-flagged by the DQE) caused by flow distortion could be identified by large differences in magnitude and direction of the platform relative winds and their resultant true winds between the two anemometers. However, on cruises where distortion was severe, identification of correct data was not possible. Flow distortion that was identifiable occurred on the following cruises: 99-B, 99-C, 99-E, 99-F, 99-G, 99-H, and the cruise of 2000*.

*Note: On the cruises, only the clearly identifiable distorted winds were K-flagged. Some values were left unflagged with large differences in magnitude and direction since the correct anemometer was unable to be determined.


4.3 Winds

As noted above, the winds were questionable throughout the data set. Some of the wind data was quite noisy. The DQE recommends a smoother be used for wind data that experience an anomalously high degree of variability. Earth Relative Wind Direction (DIR) and Earth Relative Wind Speed (SPD) calculated for each anemometer (1, 2, & 3) have periods in which the variables used to calculate them are missing or highly suspect; therefore, errors are found in the true winds.

In the 99-J cruise, the second set of true winds are left in the data set although they are highly suspect with 40-47% of the true speed data being flagged with the E, J, and K flags. These data should be used with extreme caution. This cruise's wind data are also suspect when the ship is moving slow or turning. This is apparent on 4 and 5 December for example. True wind errors resulting from the ship's change of course are also apparent in the cruise of 2000.

On the 99-E cruise, there are periods where the platform-relative winds are the same time series as the true winds with a step in magnitudes and/or directions. This usually occurs when some of the navigation data are erroneous. The platform relative wind direction 2 and platform relative wind speed 2 exhibit the same trace and magnitude on 8 April. The same is true at times for the ship-relative speed and true wind speed, they are the same or one is a constant multiple of the other. There are also times when adding a constant will make the two variables equal for an entire day. The cruise 99-E has a period like this on 8 April in which the platform-relative wind speed is the same graph and twice the magnitude as the earth-relative wind speed, in obvious error.


4.4 Ventilation

An insufficiently ventilated thermometer can experience steep temperature increases in a relatively short amount of time when the platform relative wind speeds are low, or when wind flow over the instrument is blocked. A ventilation problem was apparent for both temperature and relative humidity on several of the cruises (99-G, 99-H, 99-J, and 2000) and was appropriately flagged with the K-flag. The relative humidity is an inverse function of the temperature and dew point (RH = Td/T*100%) and one would expect problems to occur in both T and RH. We can assume that the dew point is relatively constant at any given location over the ocean, and therefore, if the temperature rises and the dew point is fairly constant, the relative humidity will decrease. The Ron Brown uses independent T and RH sensors resulting in ventilation problems not always being apparent in both the T and RH data. The RH sensor does vary with the heating issues of the ship, just at a different rate than the temperature due to their differing locations and sensitivity.

Another problem with ventilation occurs when exhaust from the ship is able to affect the sensor of the instrument. This may have happened in the 99-C cruise on 18 March when there is a Temperature- maximum when the platform-relative wind direction is from around 180 degrees, but there was not enough meteorological support, i.e.; reliable wind data, to flag this confidently. Another reason this could not be flagged with confidence is because it was an isolated occurrence in the data set.


4.5 Navigation Data

A smoother is recommended on Platform Heading (PL_HD), and Platform Course (PL_CRS) for the parts of each cruise where the data gets anomalously noisy. The 99-C cruise's heading (PL_HD) should not be used. Subsequently, the earth-relative winds calculated using these navigation data will be of highly suspect quality. Similar concerns exist for the 99-D cruise.

Spikes in the latitude and longitude were also common for the Ron Brown. This sometimes influenced the platform speed, (PL_SPD), reaching upwards of 2500 m/s, a highly unrealistic speed for any vessel. These values were flagged with the spike flag (S).


4.6 Port Data

It is possible, while the ship is in port, for some of the meteorological variables to experience data values quite different than would be experienced in the open ocean. If the data values taken in port were relatively out of the trend of the data experienced at sea (i.e. sea temperature recording several degrees Celsius above the open waters) the values were K-flagged. Use of a port flag is under consideration by RVSMDC to mark data recorded while the ship is in port.


4.7 B Flag

Platform speed over ground (PL_SPD) received B-flags from the preprocessor for recorded negative values. At very low platform speeds, it is possible to have negative ship speeds, relative to the water caused by winds, waves, and currents. The values of PL_SPD here are relative to earth and not the water and are flagged B due to negative recorded values.

There were also 4,755 B-flags assessed to atmospheric radiation on cruise 99-B by the preprocessor representing radiation values less than 0 Watts per meter squared. These physically unrealistic negative radiation values are likely the result of the instrument not being tuned to low radiation values.


4.8 G Flag

There were G-flags assigned by the preprocessor to Pressure (P) values that were greater than four standard deviations from the climatological mean (da Silva et al. 1994). The flagged values are only slightly lower than 4 standard deviations and may represent a realistic extreme low pressure for the region.

The preprocessor also G-flagged the true wind speed (SPD), SPD2, and SPD3 which are feasible values during strong thunderstorms or even with the passage of a deep low pressure system be it tropical or extra-tropical.

Temperature (T) and relative humidity (RH) were G-flagged as well. These values could be associated with a cold, arctic air mass or a warm, tropical air mass and were not too far removed from the four standard deviation limit.

The same is also true for the relative humidity's (RH) G-flags.


4.9 Data Spikes

Isolated spikes occurred in most of the variables throughout the data. Spikes are a relatively common occurrence with automated data, caused by various factors (e.g. electrical interference, ship movement, etc.). These individual points were assigned the S-flag.

There were also acceleration spikes caused by slow instrument response to changes in ship motion. These were K-flagged when the DQE felt that these spikes had negative influence on the other variables. These mostly occur in the platform relative wind direction and speed, course and heading of the ship.

There were also groups of spikes in the temperature data that were prolonged for several minutes to several hours. These drops are likely due to passing clouds and showers dropping the temperature for some time and were not flagged by the DQE as true meteorological events have the same environmental response. These periods of lower temperatures were also associated with relative humidity increases, slight wind shifts and higher wind speeds.


5.0 Final Comments:


5.1 Winds and Overall Quality

The vast majority of the flagging was due to the flow distortion experienced in the winds, and the lack of heading, course, and platform-relative wind data causing the winds to be poorly computed. Even though all of the meteorological experienced some stair stepping, over all, they proved to be of fair quality. Note also that the flow distortion problem does not necessarily mean the winds were calculated incorrectly, or that the sensor was malfunctioning. The DQE suspects it is a realistic response to variations in deck cargo or permanent ship structures affecting the winds.


5.2 Insufficient Data

In parts of each of the cruises, the DQE would like to note that some data may have been left unflagged because of insufficient meteorological backing due to the lack of quality in the true winds. In these cases, there was not enough evidence to say whether the certain questionable data should be flagged. It is very possible that some data left unflagged on these cruises are questionable and should be used with caution.




References:

da Silva, A.M., C.C. Young and S. Levitus, 1994: Atlas of Surface Marine Data 1994, Volume 1: Algorithms and Procedures. NOAA Atlas Series.

Smith, S.R., C. Harvey, and D.M. Legler, 1996: Handbook of Quality Control Procedures and Methods for Surface Meteorology Data. WOCE Report No. 141/96, Report WOCEMET 96-1, Center for Ocean-Atmospheric Prediction Studies Florida State University, Tallahassee FL 32306-2840

Smith, S.R., M.A. Bourassa, and R.J. Sharp, 1999: Establishing More Truth in True Winds. J. Atmos. Oceanic Technol., 16, 939-952.