This appendix provides an overview of a software-based assessment framework, the MLMA-based assessment framework, developed by the Center for Ocean Solutions during the information gathering phase of the Master Plan amendment process (Hazen et al. 2017). As with the other appendices, it is anticipated this overview will continue to be expanded and refined as part of Master Plan implementation so it can serve as an effective resource to managers and stakeholders.
As referenced in Chapter 3, the MLMA-based assessment framework was co-developed by Department staff and scientists, and is designed to provide a systematic, practical, and flexible means to measure California fisheries management. The assessment framework can help identify future needs and direction in ESRs and can be applied at the outset of an FMP development process to help scope the effort by identifying areas where management efforts should be directed. This can allow the Department to systematically identify future management needs, prioritize limited resources, and effectively communicate decision-making rationale.
The assessment framework was created through the careful repurposing of relevant metrics from well-known, widely-applied sustainability assessment frameworks, peer-reviewed literature, and experts. It has been extensively reviewed and tested by Department staff.
Structure and organization of the Marine Life Management Act-based Assessment Framework
The assessment framework comprises six questionnaires, each containing metrics associated with the following requirements of the MLMA, respectively:
- Manage for abundance of the target stock(s).
- Minimize unacceptable bycatch.
- Maintain, restore, and enhance habitat(s).
- Conserve entire ecosystems.
- Minimize adverse effects on fishing communities.
- Ensure good management process (compliance, evaluation, and stakeholder engagement).
The main component of the questionnaires is a list of metrics. The first four questionnaires (#1-4) deal with the ecological outcomes of management efforts. These questionnaires contain metrics that assess how much scientific information is available for the fishery, the effects of the fishery on the stock(s) and associated marine resources, and the management measures currently in place to address potential and/or known effects. Specifically, the metrics within each questionnaire are organized into the following three categories, consistent with the structure of the MLMA’s goals:
- Understand: Managers understand the basic sustainability concerns for each fishery and identify scientific information relevant to affected marine resources and fishing activities.
- Assess: Managers assess the magnitude of effect the fishery has on the biophysical system and how management measures affect fishing communities.
- Manage: Managers take action to address actual and potential impacts of the fishery and management activity.
The questionnaire on minimizing adverse effects on fishing communities (#5) is based on the broad MLMA goals of recognizing the interests of fishery participants and minimizing adverse impacts to fishing communities. This questionnaire contains metrics that assess understanding of the fishery participants and their concerns and effects of regulation on fishing communities. Metrics within the final questionnaire on management processes (#6) focus on compliance, data needs, research plans, evaluation of management actions and responsiveness to those evaluations, and stakeholder engagement throughout the management process. The metrics are primarily in multiple choice format, but some require the input of narrative information.
Each questionnaire was developed with several principles in mind:
- Flexible: The questionnaire balances guidance and discretion. Metrics provide enough guidance so that the differences between various responses are clear and defined. The questionnaire also provides enough discretion to enable the assessment of a diverse array of fisheries that may be characterized by different ecological and socioeconomic issues and managed using distinct management strategies.
- Manageable: The questionnaire is a reasonable length.
- Theoretically sound: The questionnaire is based on best available science and best practices in fisheries management.
- Legally accurate: The questionnaire accurately evaluates legal compliance and requires no more or less than the MLMA.
The questionnaires that compose the assessment framework are generally linear. The respondent should answer questions in numerical order, except where the questionnaires provide explicit instructions to do otherwise. Guidance and text about navigating to various sub-questions is included in the questionnaires to demonstrate the intended flow of the self-assessment.
In addition to the metrics, each questionnaire has several additional components. User guidance is incorporated throughout the questionnaires, pointing the questionnaire-taker to specific, vetted examples, definitions, and useful tools developed elsewhere. Such guidance is expected to result in more accurate and consistent answers and direct managers to possibilities for improving management strategies and outcomes. An uncertainty scale and best available science scale are included to reduce response biases and gain further useful information for scoping and prioritizing future management actions. These components are designed to gauge the precision and rationale underlying responses to each question. The uncertainty scale appears after each question and the best available science scale follows certain questions (e.g., queries about the collection of scientific information or making decisions based on scientific information). Space for comments is provided after each question where the questionnaire-taker can identify missing information, barriers, or any other comments that allow for more discretionary and narrative responses that can inform future management decisions.
Suggested best practice for utilizing the Marine Life Management Act-based Assessment Framework
Step 1. Identify the appropriate person(s) to complete the questionnaire(s).
Several options for utilizing the assessment framework exist. The Department could self-assess their management outcomes and identify both successes and areas for potential improvement. In a complementary or collaborative manner, outside entities such as Sea Grant, OST, academic, or other institutions could use the assessment framework as a mechanism for scientific peer review. Lastly, the Department could apply the framework in collaboration with interested constituents as an approach for improved engagement and dialogue.
Step 2. Conduct assessment.
Completing the entire assessment may require reference to management documents and/or consultation with colleagues. Two of the questionnaires—bycatch (#2) and habitat (#3)—are designed to be taken for each different fishing sector (i.e., recreational and commercial) or gear type in the fishery. The remaining questionnaires are designed to be taken only once for each fishery. However, if the reviewer feels that the geography or fishing activities of different sectors warrants multiple assessments under any of the remaining questionnaires, the reviewer has discretion to do so. For example, if a fishery has a northern and southern component, and different stock health information that is specific to each, the reviewer should take the managing target stock questionnaire (#1) separately for the two geographic components.
As noted above, each questionnaire contains metrics and several additional components. Each question is accompanied by background and guidance, designed to define key terms and provide specific examples where appropriate. Each question is followed by a comment box that may be used to provide narrative explanations, identify gaps in understanding, or specify other important information. Comments are fully incorporated into the assessment results and have the potential to add valuable information to the outputs where gaps in understanding or uncertainty about the most accurate response exist. Certain questions are also accompanied by confidence scales that track how certain the reviewer is that the response selected fully captures the fishery being assessed. This scale can be used to identify when the reviewer feels that none of the possible responses are entirely accurate, that an accurate response falls somewhere between the possible responses, or that data are too sparse to answer with full confidence. Low confidence scores should be explained in the comment box. Finally, many questions are accompanied by a request to identify the sources of information that support either the scientific understanding of the fishery or the management measures that have been implemented for the fishery. The categories of sources are defined each time they appear.
Step 3. Review results to scope and prioritize future management actions and resource allocation.
The results of this assessment framework can be used to scope and prioritize future management actions and to efficiently allocate resources. Designed to evaluate consistency with the MLMA, the framework generates a comprehensive picture of the current status of implementation. The Department can use these results to inform development of management documents (e.g., ESRs and FMPs) within the new scaled management approach and other management actions or decisions. Outputs will also be valuable for informing internal discussions, facilitating communication with constituents about management outcomes and processes, allocating limited resources to focus on areas of need, or directly supporting decision-making through clearer identification of priorities.
The following are suggested options for quantifying and visualizing results:
- Unweighted: The possible responses for each metric sum to a maximum value of 1, with each individual response allocated an equal fraction thereof. For example, for a question with four possible responses where the lowest answer represents “no information available” the values are 0, 0.33, 0.66, and 1.0. The mean value is then calculated for each set of answers, per questionnaire.
- Weighted: A weighting scheme could be applied to individual responses, questions, categories, questionnaires, or some combination of the aforementioned. For example, critical questions can be identified by reviewing the metrics and selecting those deemed most important as a policy matter. Specific multipliers (i.e., 1.5, 2) can then be applied to the results of these questions to reflect their importance. Proposed weighting schemes should be vetted by experts familiar with California fisheries and the assessment framework.
- Threshold: A threshold methodology could set pre-determined results for questions, categories, or questionnaires that are used to indicate an area of concern. Utilizing the underlying scoring methods from either the unweighted or weighted options, selecting thresholds would translate results into a system akin to “pass/fail” or “no concern/concern.”
Once an option for quantifying results is selected, results can be presented through a series of summary tables with all the assessment framework’s questions and selected responses for a hypothetical fishery. The tables can include descriptions of the questions and display the total response value, the response value per category, and the response value per question (Table F1). Tables can also include weighting and thresholds (Table F2).