Ensuring the use of the best available information in the management of fisheries is a central tenet of the MLMA. One step in achieving this is external peer review of certain scientific information used in management. This chapter describes the requirements of the MLMA regarding best available scientific information and external peer review. As part of the information gathering phase of the Master Plan amendment process, OST developed a report on best practices regarding peer review under the MLMA (PDF)(opens in new tab). This chapter as well as the additional details provided in Appendix Q draw from that report.
Section 7050(b)(6)(opens in new tab) of the MLMA states that management should be based on “the best available scientific information and other relevant information.” This includes the following:
The Department should apply the criteria developed by the National Research Council (NRC) in determining the best available scientific information (NRC 2004):
- Relevance: Scientific information should be representative of the fish stock, habitat, and socioeconomic context of the fishery being managed, although the data need not be site specific or species specific. In some cases, analogous information from a different region or the biological characteristics of a related species or species with similar life-history strategies will be informative and relevant, and may constitute the best information available.
- Inclusiveness: Scientific advice should be sought widely and should involve scientists from all relevant disciplines. The goal should be to capture the full range of scientific thought and scientific opinion on the topic at hand. Critiques and alternative points of view should be acknowledged and addressed openly. Anecdotal (experiential, narrative, or local) information should be acknowledged and evaluated during the process of assembling scientific information. When no other information is available, anecdotal information may constitute the best information available. In addition, anecdotal information may be used to help validate other sources of information and identify topics for research.
- Objectivity: Data collection and analysis should be unbiased and obtained from credible sources. Scientific processes should be free of undue nonscientific influences and considerations.
- Openness: The public should have information about each phase of the process from data collection to data analysis to decision-making. Decision makers should provide a clear rationale for the choice of the information they use or exclude when making management decisions. The process of collecting data and selecting research for use in support of management decision-making should be open, broad-based, and carefully documented. All scientific findings and the analysis underlying management decisions should be readily accessible to the public. The limitations of research used in support of decision-making should be identified and explained fully. Stock assessments and economic and social impact assessments should clearly describe the strengths and weaknesses of the data used in analyses.
- Timeliness: There are two primary aspects to timeliness. First, timeliness refers to the acquisition of data in such a manner that sufficient time exists to analyze it adequately before it is used to make management decisions. Second, timeliness refers to whether the data are applicable to the current situation. Uncertainties that arise from an incomplete study should be acknowledged, but interim results may be better than no new results at all. Management decisions should not be delayed indefinitely on the promise of future data collection or analysis.
Recovering the RV Beagle after a dive in Big Creek State Marine Reserve. (CDFW photo by M. Prall)
Peer review
In §7062(opens in new tab), the MLMA requires that the Department “establish a program for external peer review of the scientific basis of marine living resources management documents.” Peer review is the most accepted and reliable process for assessing the quality of scientific information. Its use as a quality control measure enhances the confidence of the community (including scientists, managers, and stakeholders) in the findings presented in scientific reports and, consequently, in decisions based on that scientific information.
The MLMA identifies some types of documents that can be submitted for external peer review, including marine resource and fishery research plans (§7062(a)(opens in new tab)), interim fishery research protocols (§7074(c)(opens in new tab) if justified), and FMPs or plan amendments (§7075(a)(opens in new tab)). The MLMA does not address data sets, analyses, and other documents developed by the Department or other entities, which may be cited within a management document (e.g., ESRs). However, scientific information developed by the Department is subject to the Department’s Scientific Integrity Policy (CDFW 2017a (PDF)(opens in new tab)), which allows for internal review unless the documents will have “a substantial management impact or large expenditure of funds”.
The MLMA does not provide guidance on other documents that should be submitted for peer review, but limits peer review to the scientific basis of management documents. In general, the Department and Commission should consider submitting all scientific analyses central to the development of FMPs and management measures, as well as the scientific portions of FMPs, for peer review. The process for this review is described below.
Exemption of documents from external peer review
The MLMA authorizes the Commission, with the advice of the Department, to adopt criteria for exempting certain documents from external peer review (§7074(d)(opens in new tab) and §7075(c)(opens in new tab)). In making this determination, the Commission should be guided by the following criteria:
- The product does not contain scientific or technical information upon which decisions are based.
- The product has already been subject to a prior adequate peer review within a reasonable time period.
- A peer review process would significantly interfere with the need for promptness in decision-making or secrecy of information.
- The information is routine data, generated using properly applied, scientifically-accepted methods.
- Information involving a health or safety issue where dissemination is time-sensitive.
- The information consists of accounting, budget, actuarial and financial information.
Scope of external peer review
At a general level, the MLMA characterizes the scope of external peer review as “the scientific basis of marine living resources management documents” (§7062(a)(opens in new tab)). Section 7062(c)(opens in new tab) calls for the external review panel to determine whether “a scientific portion of the document is based on sound scientific knowledge, methods, and practices.” Given the breadth of issues in FMPs and related documents, properly establishing the scope of an external peer review so that it focuses upon the scientific elements of the documents is crucial to implementing these provisions of the MLMA. Due to the significant workload associated with conducting an independent peer review, including the logistics and coordination among reviewers, it is expected that it will not be possible to accomplish most reviews with volunteers and therefore contractors will likely be engaged. This will require dedicated funding and capacity to manage.
Regardless of whether contractors or volunteers are employed, to conduct an external peer review, the Department and coordinating entity managing the external peer review process should develop a detailed scope for scientific review of the target documents before selection of the panel of reviewers. The Department should notify the public of the scope upon its formulation. In many cases, it will be useful to delineate between the scientific basis of the management document undergoing review and the management recommendations contained therein, which typically would not be subject to peer review. Table 2 provides guidance on types of reviews, example applications, and the benefits and limitations of the use of review types.
Levels of peer review
The intensity of peer review may vary for different documents. For example, routine updates based upon previously reviewed methods may be reviewed internally while novel or complex methods, data, and analysis will require more formal review by an external panel of experts. Table 2 identifies four levels of external scientific peer review and considerations associated with each. See Appendix Q for additional details on best practices regarding each potential work product.
Table 2: Levels of peer review and associated considerations.
| Review mode
|
Example applications
|
Potential work product
|
Benefits
|
Limitations
|
| Internal review
|
Routine actions with limited management
implications or associated controversy.
|
ESRs, fishery research protocols.
|
Agile, cost-effective.
|
Limited opportunity for alternative
perspectives.
|
| Expert written
review
|
Products of short to moderate length, and low
to moderate complexity.
Work products that are
unlikely to have highly significant management implications.
|
Draft FMP of low to moderate complexity.
|
Quick, less costly.
Multiple
independent reviews offer diverse viewpoints.
|
No group discussion or
deliberation.
Reviewers may have contrasting or
opposing views.
|
| Panel review
(remote)
|
Moderately complex methodologies, models, or
data analyses that require group discussion and participation of agency
staff.
Reviews requiring international
participants.
Work products that are likely to have
moderately to highly significant management implications.
|
Draft FMPs or methodologies of moderate to
high complexity.
|
Allows for deliberation among reviewers and
managers.
Relatively easy to accommodate public
participation.
|
Moderately costly, moderately
time-intensive.
Does not allow for in-depth group
working sessions.
|
| Panel workshop
review (in-person)
|
Complex methodologies, models, or data
analyses that require group discussion and participation of agency
staff.
Newly applied methodologies (first
application in California fisheries management).
Reviews
requiring additional analyses or model runs.
Work products that are likely to have moderately to highly
significant management implications.
|
Stock assessment, complex or highly complex
draft FMP or methodology.
Highly controversial
reviews.
|
Allows for deliberation among reviewers and
managers, real-time analysis, discovery and back and forth
deliberation.
Workshops open to the public may
increase buy-in.
|
Requires extensive pre-workshop
planning.
Costly, time-intensive.
|
| Journal peer review
|
Complex methodologies, models, or data
analyses, and novel science.
Decisions or scientific
information that would benefit from highly rigorous scientific
vetting.
Scientific information that could be built
upon or would benefit the wider academic
community.
Controversial findings or results inform
influential or costly management decisions.
|
Varied; Methodology, models, new data,
analyses.
|
Ensures product meets high standards of
scientific quality.
|
Not reviewed openly, may not allay public
concerns.
Time-intensive – may not be appropriate
for time-sensitive findings or conclusions; Manuscript must align with
journal publication timelines.
Competitive process.
|
The level of review for specific types of documents is included in the table above. However, in determining the appropriate level of review, the following criteria should be considered:
- Complexity: The nature and complexity of scientific information presented in models, analyses, and methods.
- Management risk: The significance of information and decision-making risk potential impact on sustainability for incorrect management decisions.
- Uncertainty: The level of confidence surrounding a body of scientific knowledge.
- Socioeconomics: The social and/or economic value of the fishery and economic impacts of decisions that will be informed by the scientific information; cost-benefit analysis of additional review.
- Level of previous review: A determination of the type and amount of previous peer review of the information used.
- Precedent: Whether science is regarded as “precedent setting,” particularly novel, or is the first application of a new tool or model.
- Group discussion: The benefits to be gleaned from group deliberations.
External peer review timing
The MLMA does not dictate the timing of peer review within the regulatory process, and practice has varied. In general, the Department should consider seeking peer review of scientific information that will be used to inform management decisions before regulatory options are developed and before agency or stakeholder positions have formed, to the extent that is feasible. External peer review of FMPs and similar documents might begin only upon completion of a draft document and before public review. Where feasible, it is advantageous for the Department to include an opportunity for the external peer review panel to review the Department’s responses to panel findings as well as public comments. See Figure 7 for suggested checkpoints for peer review during the management process.
Figure 7: Suggested checkpoints for scientific peer review of science in a generalized fishery management plan development process (from OST 2017).
Management and design of the external peer review process
In conducting external peer reviews of scientific information, the MLMA authorizes the Department to enter into an agreement with outside entities “that are significantly involved with research and understanding marine fisheries and are not advocacy organizations” (§7062(b)(opens in new tab)).
The contracted entity is to select and administer the external peer review panel and is responsible for the scientific integrity of the peer review process (§7062(b)(opens in new tab)). The MLMA does not define scientific integrity. However, in designing a peer review process with a contracted entity, the Department should aim for a process that has the following characteristics (Office of Management and Budget 2005):
- Incorporates the right expertise and balance.
- Identifies the key scientific issues and provides a clear charge to reviewers.
- Supports deep, focused, and high-quality discussions among members of the panel.
- Ensures that the rationale for the panel’s findings is clear and well-documented.
- Produces a highly accurate report summarizing the review findings.
The Department will also seek to ensure that external peer reviews have high process integrity, including the following characteristics:
- Are open and consistent.
- Avoid real or perceived conflicts of interest.
- Include a workable process for public comment and involvement.
- Adhere to their defined procedures.
The management and activities of external peer review panels should also be guided by the Department’s Procedural Guidelines for Ad Hoc Independent Scientific Advisory Committees (CDFW 2017b).
CDFW staff working in the lab polishing and ageing California halibut otoliths. (CDFW photo by Kristine Lesyna)
Composition of external peer review panels
Among other things, the MLMA mandates that external peer review panels be made up of “individuals with technical expertise specific to the document to be reviewed” (§7062(b)(opens in new tab)). In addition, “Peer reviewers shall not be employees or officers of the Department or the Commission and shall not have participated in the development of the document to be reviewed.” Reflecting best practices, membership of external peer review panels should have the following characteristics:
- Reflect the right types and diversity of expertise relative to the scientific information under review.
- Meet standards for expertise as demonstrated by degrees, publications, and experience.
- Have not participated in the development of the information being reviewed.
- Be free from conflicts of interest, including any financial or other interest that could impair objectivity or confer unfair competitive advantage.
The review of highly specialized information may sometimes require exceptions to these conflict of interest rules, particularly where the pool of potential reviewers is narrow. In such situations, the real or perceived conflict of interest should be promptly identified and disclosed to the public.
Dealing with disagreements among reviewers or conflicting reviews
While it is not the goal of peer review to achieve consensus among reviewers, contrasting viewpoints or recommendations about major components of the subject matter can be difficult to resolve. This may occur more frequently during written reviews where experts do not communicate with one another during the process. However, panel workshops may also produce conflicting recommendations.
Any review output should appropriately represent any dissenting or contrasting views, however it is not the role of a review coordinating body to resolve or prescribe which recommendation to consider or accept over another. This role could be deferred to the review committee chair, or, depending on the level and subject of disagreement, the Department, or the review coordinating body may choose to consult with an outside expert.
As noted here, the Department is required to provide written explanation if it disagrees with any aspect of the review findings. A written response and justification could also be appropriate when responding to conflicting reviews. The review committee chair, outside expert, or the Commission could serve as moderator to make a final determination of whether an issue was adequately addressed.
Reporting of peer review findings
Section 7062(c)(opens in new tab) of the MLMA requires that the external scientific peer review entity provide the Department with “the written report of the peer review panel that contains an evaluation of the scientific basis of the document,” including any findings of scientific deficiencies in the document and the basis for those findings. As required by the MLMA, the Department is to then accept the findings and alter the document, or if it disagrees with a finding, to include as part of the record the basis for its disagreement, including reasons for determining the document is based on sound scientific knowledge, methods, or practice. The MLMA requires that the Department submit the peer review report and its response to peer review findings with the reviewed document to the Commission and make these materials publicly accessible to strengthen the transparency of the peer review process.
While scientific review can be a resource- and time- intensive process, it can help to demonstrate that fishery management decisions are based on valid and defensible science. An open process can also demonstrate a commitment to objectivity and help to build relationships with stakeholders. Many of the recommendations contained in this chapter require standardizing and formalizing existing practices and processes, as well as dedicated funding, to ensure consistency across review implementations. For additional details regarding the peer review process including a peer review checklist, sample Terms of Reference (
TOR), and report template, see
Appendix Q.
Photo at top of page: CDFW staff collecting samples using a beach seine net. (CDFW photo by Ken Oda)