OPCW

NOTE BY THE DIRECTOR-GENERAL

REVISED STANDARD OPERATING PROCEDURE

FOR EVALUATION OF THE RESULTS OF OPCW PROFICIENCY TESTS

1. The Director-General wishes to inform Member States that, in accordance with the recommendations presented by the evaluators of the Third Official Proficiency Test (see Note by the Director-General, Evaluation of the Third Official Proficiency Test, S/22/97, dated 25 November 1997, and Evaluation of Results of the Third Official Proficiency Test, version 2, S/23/97, dated 21 November 1997), the Secretariat has revised the "Standard Operating Procedure (SOP) for Evaluation of Results of OPCW Proficiency Tests" (Annex 1 to PC-XIII/B/WP.5, dated 16 February 1996) noted by Working Group B of the Preparatory Commission in subparagraph 5.3(a) of PC-XIII/B/6, dated 21 March 1996. The revised version of this SOP is presented in the annex to this Note by the Director-General.

2. This revised SOP incorporates both the interpretations of the evaluation criteria used so far and suggestions related to a set of more precise requirements for evaluating the analytical results of proficiency tests in the future. These more precise requirements were first introduced at the meeting on 16 September 1997 between the Secretariat and the participants in the Third Official Proficiency Test. In addition, the comments on these suggestions made by the evaluators of the Third Official Proficiency Test were welcomed in the above-mentioned Note by the Director-General (S/22/97). Only a few laboratories commented on the suggestions. Bearing in mind that the next official proficiency test will be held soon, the Secretariat, supported by the TNO Prins Maurits Laboratory in the Netherlands and the Finnish Institute for the Verification of Chemical Weapons Convention (VERIFIN) in Finland, proceeded with the revision. The input of these two institutions - the evaluators of the First, Third and Fourth Official Proficiency Tests - is gratefully acknowledged.

3 The revised "Standard Operating Procedure (SOP) for Evaluation of Results of OPCW Proficiency Tests" replaces the original version and shall be applied starting from the next official proficiency test i.e. from the Fourth Official Proficiency Test.

4. To assist the reader, in this revised version of the SOP, all new text is indicated in bold type, and all text that has been deleted is crossed through.

Annex

STANDARD OPERATING PROCEDURE (SOP)

FOR EVALUATION OF RESULTS OF

OPCW PROFICIENCY TESTS

Version 2, 31 March 1998

1. Scope

1.1 This revised "Standard Operating Procedure (SOP) for Evaluation of Results of OPCW Proficiency Tests" replaces the previous version presented in annex 1 to PC-XIII/B/WP.5 that was noted by Working Group B of the Preparatory Commission in PC-XIII/B/6, subparagraph 5.3(a).

1.2 The aim of OPCW proficiency tests is to establish and maintain a recognised and transparent methodology for the continued assessment of the technical competence of the participating laboratories. For this reason, every effort shall be made to ensure that the evaluation of results will be uniform and fair for all participating laboratories and that the test interpretation will be consistent and unambiguous. The Secretariat has the responsibility to evaluate the analytical results. If the Secretariat is not in a position to perform evaluation of the analytical results, an accredited (or seeking accreditation) laboratory with demonstrated experience in the analysis of chemicals related to the Convention will be selected to support this process. To facilitate open and transparent testing, the laboratory evaluating the analytical results shall not participate in the same proficiency test. However, such a laboratory must analyse the samples.

1.3 This SOP addresses the following:

(a) the evaluation of the analytical results; and

(b) the overall evaluation of the proficiency testing.

1.4 The evaluating laboratory shall evaluate the analytical results obtained by the test participants as set out in paragraph 5 below and the Appendix to this SOP, shall document the evaluation results, and shall report the results to the Secretariat. These activities shall be carried out under the direction of the Secretariat and in accordance with:

(a) an appropriate quality assurance/quality control (QA/QC) system;

(b) "ISO Guide 43" and "WELAC Criteria for Proficiency Testing in Accreditation" relevant to evaluation of results for proficiency tests;

(c) "Criteria for Acceptable Performance of Laboratories in Proficiency Testing" (PC-XI/B/WP.6, Annex 1) (C-I/DEC.62) and "Conditions in Relation to Proficiency Tests Following the First Test" (C-I/DEC.66); and

(d) an appropriate confidentiality policy required for accreditation.

2. Contact point

The Secretariat shall appoint a contact person responsible for coordination of the test.

3. Information provided by the Secretariat

3.1 The Secretariat shall inform in time agreed upon between the Secretariat and the laboratory evaluating the analytical results of the following:

(a) the test plan;

(b) the availability of the test samples to the evaluating laboratory;

(c) the number of participating laboratories;

(d) the estimated time within which the analytical results shall be sent for evaluation and the chosen means of transportation; and

(e) sample composition (test sample preparation and analysis report).

3.2 The delivery of information and reports shall be done in a way which ensures confidentiality.

4. Reports of the participating laboratories

4.1 The reports from the participating laboratories as well as the report of the sample preparation shall be kept confidential. The Secretariat shall ensure that laboratory identifications have been removed and replaced with laboratory codes in the participants' reports before sending the report copies to the evaluating laboratory. The participating laboratories shall be requested to use the accepted test forms. Measurements carried out on the blank samples shall also be documented on the test forms. When filling out the test forms participants shall be requested:

(a) to use their laboratory code number and not their names;

(b) not to provide more or less data than requested (unless asked for during the evaluation phase);

(c) to number all pages and to indicate the total page number in the beginning of the report; and

(d) to provide the report as loose sheets; and

(e) to avoid any type of identification (such as use of other languages than English).

4.2 When including the supporting analytical data (spectra, chromatograms, etc.) with the test report the participating laboratories shall be requested:

(a) to delete the names of the laboratory and the analysts;

(b) to clearly indicate the analytical method, sample code, subsample code and identified chemical(s); and

(c) to provide copy of sufficient quality for use as supporting analytical data.

4.3 QA/QC data shall be submitted upon request of the Secretariat only. In case additional information not found in the participant report is required for the evaluation, the evaluating laboratory shall contact the Secretariat which shall forward the matter to the laboratory in question and shall forward the answer with participant identification data removed to the evaluator. The laboratories are not obliged to include QA/QC data in their report. This data shall be submitted upon specific request of the Secretariat only.

5. Evaluation of analytical results

5.1 The experts responsible for evaluation of analytical results shall verify the correctness of the reported analytical data on the basis of:

(a) the test sample preparation report;

(b) the reported analysis information from participating laboratories;

(c) their experience and knowledge of available state-of-the-art analytical techniques and methods; and

(d) additional experimental work when required.

5.2 The main emphasis is to assess whether the reported chemicals have been correctly identified and whether the presented data supports the identifications. Each identification shall be backed up by the measurements carried out on the blank samples. Analytical results obtained using different techniques shall also be confirmed and consistent. There must be an unbroken chain of evidence linking each test sample to each chemical identified by a defined method of analysis.

5.3 No information other than that found in the participating laboratory's report shall be considered. In accordance with the strict interpretation of the adopted criteria and SOPs, corrigenda or additional information sent to the Secretariat after the test time can not be accepted for evaluation, and no clarification can be sought when it comes to the content of the report. Furthermore, there is only one report – the one provided to the Secretariat within the test time frame – and only one list of chemicals shall be reported. The report shall not be divided into sections entitled "Official report" and "Appendix".

5.4 Furthermore comments shall be given on the possible reasons for false positives and false negatives (e.g. cross-contamination or misinterpretation of analytical data) and the possible reasons for false negatives (e.g. poor separation, unsuitable detection system, lack of reference data). The available equipment and analytical procedures used including the QA/QC procedures shall be assessed to see whether they could explain the mistake.

5.5 The evaluation of sample preparation and the analytical results obtained with the commonly used analytical techniques (i.e. GC, MS, IR and NMR) is described in the Appendix to this SOP. Results obtained by using analytical techniques shall be assessed by an expert in the relevant field in accordance with the guidelines presented in Appendix to this SOP.

6. Evaluation report of analytical results

The evaluation of the results of the participants shall start as soon as the evaluating laboratory obtains the set of complete participant reports from the Secretariat. The evaluation report of analytical results shall be submitted within four weeks to the Secretariat. The Secretariat shall compile and produce the preliminary proficiency test report. The Secretariat shall submit this preliminary report to the participating laboratories for comment. The laboratory evaluating the results shall check the comments received from participating laboratories and shall make the necessary corrections to the evaluation report. Then the final analytical evaluation report shall be submitted to the Secretariat for further actions.

7. Assessment of the proficiency test

7.1 The Secretariat shall make an assessment of the applicability of the test prior to reaching its final conclusions on the performance of individual laboratories participating in the proficiency test. The applicability of the test shall be judged on the basis of the sample preparation and analysis report, the analytical evaluation report, and the test plan, including the test scenario, the timetable, and other relevant matters sent to participants before the test start. The test samples shall be accepted as equivalent for all participating laboratories with respect to the chemicals in question when:

(a) the concentration range of spiking chemicals corresponds to the one agreed for the proficiency tests;

(b) the samples can be regarded as sufficiently homogeneous; and

(c) the reported degradation can be found acceptable within the test scenario.

7.2 The chemicals shall be categorised in accordance with the test scenario as spiking chemicals, as degradation products, as non-irrelevant chemicals, and as false positive chemicals as follows:

(a) S = spiking chemicals used for scoring;

(b) s = spiking chemical not to be used for scoring (= degradation product, impurity or equivalent of a spiking chemical);

(c) F = false positive; and

(d) I = irrelevant chemical.

7.3 The performance of participating laboratories shall be assessed on the basis of "Criteria for Acceptable Performance of Laboratories in Proficiency Testing" (PC-XI/B/WP.6, Annex 1) (C-I/DEC.62) and "Conditions in Relation to Proficiency Tests Following the First Test" (C-I/DEC.66). The interpretation of the criteria is presented in the Appendix to this SOP. The Secretariat shall calculate a score on the basis of final results for each participating laboratory fulfilling the required performance criteria.

8. Proficiency test report

8.1 The Secretariat shall compile a full proficiency test report including:

(a) evaluation of participating laboratories' performance and scoring;

(b) sample preparation and analysis information; and

(c) information on evaluation of analytical results.

8.2 The Secretariat shall provide a preliminary version of the proficiency test report for participating laboratories. Participating laboratories are given a minimum of one week to inform the Secretariat whether they accept the proficiency test evaluation results and to provide their comments on the preliminary report. Comments shall be assessed by relevant experts and incorporated in the final report as appropriate.

9. Follow-up actions

The Secretariat shall inform the appropriate laboratory of the errors (false positives and negatives; false negatives arising from not finding a spiking chemical or not providing sufficient supporting data for a chemical that was found) that may have occurred. The Secretariat shall request information on remedial actions taken by the laboratory. The participating laboratory in question shall submit a full report to the Secretariat stating the cause of the problem and remedial actions taken before their participation in another test. Depending on the problem, the evaluating laboratory may be asked to provide an expert opinion on whether or not the remedial actions taken by the participating laboratory are found to be effective and to submit this report to the Secretariat for further action.

10. Additional information

1 Nomenclature of Inter-laboratory Analytical Studies (IUPAC Recommendations 1994).

2 ISO Guide 43, Development and Operation of Laboratory Proficiency testing, ISO/IEC Guide 43-1984 (E).

3 WELAC Criteria for Proficiency Testing in Accreditation, WELAC Guidance Document No. WGD 4, Final proposal June 1993.

4 AOAC Official Methods of Analysis, Appendix: Guidelines for Collaborative Study Procedure to Validate Characteristics of a Method of Analysis.

5 Thompson M., Wood R., J. AOAC Int. 76 (1993) 926. "International Harmonised Protocol for Proficiency Testing of (Chemical) Analytical Laboratories".

Appendix

1. Interpretation of the criteria presented in C-I/DEC.62

The interpretation of the criteria presented in C-I/DEC.62 is clarified in the table below.

Table. Interpretation of the criteria presented in C-I/DEC.62

Table cont. Interpretation of the criteria presented in C-I/DEC.62

2. Evaluation of sample preparation

2.1 The effectiveness of the sample preparation methods used to recover the spiking chemicals shall be assessed. Especially, the differences between the recommended procedures (ROPs) and the methods used by a participant should be evaluated with a view to assessing whether possible false negatives can be explained. Possible reported artefacts resulting from sample preparation shall be assessed. The evidence linking each test sample, its sample preparation phases and corresponding subsamples that were analysed shall be assessed. The links between (sub)samples and the analysis results should be precise, referring to flow-charts, sample preparation forms and legends of figures. Information provided by the analysis of the corresponding blank samples shall be taken into account as well. Each measurement performed by each analytical technique shall be backed up by the evidence of the analysis of the blank samples, if appropriate.

2.2 On the basis of the evaluation the expert shall comment on the possible reasons for false results.

3. Chromatographic techniques and capillary zone electrophoresis

3.1 Results of chromatographic techniques such as gas chromatography (GC) and liquid chromatography (LC) techniques, and capillary zone electrophoresis (CZE) shall be checked that they support the correct identification. The evaluating laboratory shall check that each method reported is named; described in sufficient detail as described in the test forms and that representative chromatograms or electropherograms are found to support results obtained for each sample. The data required for chromatographic techniques and capillary electrophoresis is as follows:

(a) blank chromatogram/electropherogram;

(b) sample chromatogram/electropherogram; and

(c) chromatogram/electropherogram of an authentic compound; and retention times, or retention indexes.

3.2 Gas chromatographic data shall be evaluated to assess whether the available equipment and procedures used allow separation and detection of the spiking chemicals. It shall be clear which samples (and which subsample solutions) have been analysed by which method. A representative chromatogram with the datafile header still attached shall be found to support findings from each subsample solution of which the detection was made and corresponding blank subsample solution. The source of reference data shall be presented precisely. When evaluating the gas chromatographic data the following features shall be checked:

(a) the retention times of GC-peaks must fall within a window of ± 20 seconds of that from an authentic compound;

(b) the retention indices of GC-peaks must fall within a window of ± 10 units as of that from authentic compound;

(c) the signal-to-noise ratio of a chromatographic peak in GC-based techniques must be greater than 5; and

(d) if more than one retention parameter is reported, all of them should fulfil the evaluation criteria.

4. Evaluation of mass spectrometric (MS) analyses

4.1 Mass spectrometric data is evaluated to assess whether the spectra correspond to the proposed chemicals. The evaluating expert shall check whether each mass spectrometric method reported to be used by a participant is described in sufficient detail; whether it is clearly linked using appropriate names to a corresponding gas chromatographic or liquid chromatographic method; whether a representative spectrum is found to support identification of each chemical found from each sample; and whether a reference spectrum and/or spectral interpretation is found to support each identified chemical and the source of reference data is presented precisely. Total ion chromatogram (TIC) shall be provided both to the sample and its corresponding blank at corresponding range. Total ion chromatograms and mass spectra shall be provided with the datafile header still attached. Especially in case of coeluting background material in GC/MS it shall be demonstrated that blank samples do not contain spiking chemicals. Interpretations of mass spectra shall be provided including fragmentation pathways and ion structures. The data required for EI are as follows:

(a) blank TIC;

(b) sample TIC;

(c) EI spectrum from sample; and

(d) EI spectrum from reference (authentic/library) or spectral interpretation.

4.2 The interpretation of the MS spectral data of chemicals whose reference spectra are not available shall be supported by spectral information derived from closely related chemicals, with a specific indication of methods used.

4.3 When evaluating the mass spectral data, the following features shall be checked:

(a) is the full mass spectral region presented for the proposed chemical consistent with identified chemicals?

(b) is the retention data in the applied GC or LC conditions consistent with the identified chemical?

(c) are all ions belonging to the proposed structure present?

(d) are the isotopic ion ratios appropriate?

(e) are any impurity peaks present?

(f) does the spectrum correspond with the ionisation method used (e.g. in CI; M+1, adduct ions) or analysis mode (e.g. MS/MS)?

(g) the quality of reference data used (which database was used, is the spectrum correct, what kind of reference material was used to produce an authentic reference spectrum?) Is the information sufficient?

(h) in case of spectral interpretation, is the information sufficient for identification? Are the mass fragments found credibly explained by spectral interpretation? and

(i) does the comparison of the TIC of the blank and the sample demonstrate that the blank samples do not contain spiking chemicals?

4.4 The data required for CI, when used as a supportive technique for EI, are as follows:

(a) blank TIC;

(b) sample TIC; and

(c) CI spectrum from a sample with pseudomolecular ion (M+1) indication, together with CI spectrum from reference (authentic/library), or a spectrum interpretation.

4.5 In relation to the CI mass spectrum, the following specific requirement shall also be met:

(a) the relative intensity of the pseudomolecular ion in the CI mass spectra should be at least 10%.

5. Evaluation of infrared spectrometric (IR) analyses

5.1 The infrared spectrometric data shall be evaluated to assess (i) whether the quality of spectra and the information obtained is sufficient for identification, (ii) how the identification was made, and (iii) whether the data can only be considered to support identification made by other techniques. The expert shall check whether each IR method reported to have been used by a participant is described in sufficient detail and, if applicable, whether it is clearly linked using appropriate names to a corresponding gas chromatographic method; whether a representative spectrum is be found to support identification of each chemical found from each sample; and whether a reference spectrum and/or spectral interpretation is found to support each identified chemical. The IR spectrum shall be compared with a reference spectrum arising from an authentic chemical or library, and shall be compared with spectra recorded under spectroscopically comparable conditions. The source of the reference data shall be presented precisely. In the case of GC/FTIR Gram-Schmidt chromatograms shall be provided for both the sample and its corresponding blank at corresponding ranges. Gram-Schmidt chromatograms and spectra shall be provided with the datafile header still attached. Peak lists of the most intensive IR bands shall be added to the spectra. Especially in case of coeluting background material in GC/FTIR, it shall be demonstrated that the blank samples do not contain spiking chemicals. The data required for IR under similar conditions are as follows:

(a) blank chromatogram (GC/FTIR);

(b) sample chromatogram (GC/FTIR);

(c) IR spectrum from sample; and

(d) IR spectrum from reference (authentic/library).

5.2 When evaluating the IR spectra, the following features shall be assessed:

(a) is the spectral range (e.g. 4000–700 cm–1) presented adequate for the proposed structure?

(b) in the case of GC/FTIR, is the retention data in the applied GC-conditions consistent with the identified chemical ?

(c) are all significant bands present?

(d) are any impurity bands present?

(e) the quality of reference data used (which database was used, is the spectrum correct, what kind of reference material was used to produce an authentic reference spectrum?) Is the information sufficient? and

(f) in case of spectral interpretation is the information sufficient for identification? Are the bands found credibly explained by spectral interpretation?

(f) does the comparison of the Gram-Schmidt chromatogram of the blank and the sample demonstrate that the blank samples do not contain spiking chemicals?

6. Evaluation of nuclear magnetic resonance (NMR) spectrometric analyses

6.1 The NMR spectrometric data shall be evaluated to assess whether the quality of spectra and the information provided are sufficient for identification, how the identification was made, or whether the data can only be considered to support identification made by other techniques. The expert shall check whether each NMR method reported to have been used by a participant is described in sufficient detail; whether a representative spectrum is found to support identification of each chemical found from each sample; and whether a reference spectrum and/or spectral interpretation is found to support each identified chemical, and the source of the reference data is presented precisely. The NMR spectra shall be presented with the datafile header still attached. Resonances in the NMR spectra shall be marked with compound numbers. The data required for NMR under similar conditions are as follows:

(a) blank spectrum;

(b) sample spectrum; and

(c) spectrum of compound (authentic/library) or spectral interpretation.

6.2 The interpretation of the NMR spectral data of chemicals whose reference spectra are not available shall be supported by spectral information derived from closely related chemicals, with a specific indication of methods used.

6.3 When the NMR spectral data is being evaluated, the following features shall be assessed:

(a) are all significant resonances revealed or do resonances of other chemicals in the same sample overlap?

(b) are the effects of solvent, concentration, pH, and the chemical shift reference taken into account?

(c) are comparable resonances found in the sample spectrum and reference spectrum? Are the resonances found credibly explained by spectral interpretation (chemical shifts, coupling constants)?

(d) the quality of reference data used (is the spectrum correct, has it been acquired under the same conditions, what kind of reference material was used to produce an authentic reference spectrum, does the published data support spectral interpretation?). Is the information provided sufficient for interpretation? and

(e) does the comparison of the spectra of the blank and the sample demonstrate that the blank samples do not contain spiking chemicals?

- - - o - - -