Chapter 5 Finalizing the global assessment

There are several steps after getting the scores to finalizing an assessment. This guide walks your through these steps. Each of the following steps is described in detail below:

  1. Create results folder
  2. Final score review
  3. Methods document
  4. Results document
  5. Updates document
  6. Update the website
  7. Finalize and create a “release” for repositories
  8. Upload to repository

5.1 Create results folder

We summarize the results from each year’s assessment in the yearly_results/global20?? folder, located in the ohi-global repository.

After the OHI scores have been updated with the new data layers a global20?? folder should be created in the ohi-global/yearly_results folder, and populated with:

  1. an empty “data_check” and “Results” folder
  2. a copy of the README”, “Results_first_look.Rmd,” “Results/Supplement_Results.Rmd,” and “Results/functions” from the previous year
  3. you may find other files from previous years to be useful as well, but only copy them as needed (many of the files were created to meet needs specific to a particular assessment year)

The Results_first_look.Rmd creates an internal html document that provides our first look at the scores. This is mostly used internally to explore and check results. This script also does the following:

  • copies the scores.csv from the eez folder to this (yearly_results/global20??) folder and appends the current date to the filename. This allows us to version control our data so we can keep better track of it. This is especially handy during the final stages of an assessment when there are lots of versions of the data flying around email and such!
  • creates a user friendly version of the data called OHI_final_formattted_scores_date.csv. This is typically the version of data we provide to people.
  • creates most of the data in the Results/data folder and figures in the Results/figures folder.

It will be necessary to carefully walk through the previous year’s version of the Results_first_look.Rmd file and make relevant updates as needed, such as:

  • on ~ line 88 change: update_data = TRUE
  • on ~ line 84 change the dateFile to the current date, e.g., dateFile = '2018-10-03'
  • correct links: Data
  • read through to correct references to scores/regions

If there are any changes to the scores the Results_first_look.Rmd must be updated and rerun.

5.2 Final score review

Once all the data layers have been updated we conduct a final review of the scores, which involves these steps:

  1. Update and disperse the Results_first_look.html to team members to review.
  2. Meeting with team members to discuss scores, document questions/concerns in a new Github issue.
  3. To address these concerns, review the data/code and, if necessary, update the scores with corrections to the data or code. These checks are performed via R/Rmd scripts in the yearly_results/global20??/data_check folder. The results from each check is also documented in the Github issue.
  4. Update and rerun the Results_first_look.Rmd as needed
  5. If necessary, we have a second meeting to discuss the updated scores (usually this isn’t necessary).

5.3 Citation management

Here is a great reference from the fine folks at RStudio.

5.3.1 Getting Zotero and RStudio to work well together

5.3.1.1 The process: short version

  • Create a Zotero library or collection, export it as .bib (or other formats - seems flexible)
  • Call the bibliography file and a citation style (optional) in your R Markdown YAML header
  • Create your citations in the body of the R Markdown text using [@name_shorttitle_2016] format
  • Hit “knit” - done!

5.3.1.2 The process: long version

Exporting a bibliography file from Zotero

Once you’ve created your library (or collection/folder/whatever) in Zotero, you can export it in BibLaTex format (the whole library from the File menu, or the collection by right-clicking and choosing “Export Collection,” then selecting “BibLaTex” as the format:

This creates a .bib file, which contains a list of references each formatted like this (I deleted many of the fields just for the sake of space):

@article{ready_predicting_2010,
    title = {Predicting the distributions of marine organisms at the global scale},
    volume = {221},
    issn = {03043800},
    url = {http://linkinghub.elsevier.com/retrieve/pii/S030438000900711X},
    journaltitle = {Ecological Modelling},
    author = {Ready, Jonathan and Kaschner, Kristin and South, Andy B. and Eastwood, Paul D. and Rees, Tony and Rius, Josephine and Agbayani, Eli and Kullander, Sven and Froese, Rainer},
    file = {Ready_etal2010.pdf:/Users/ohara/Library/Application Support/Zotero/Profiles/th8m611h.default/zotero/storage/297AD2JJ/Ready_etal2010.pdf:application/pdf}
}

Calling the bibliography in R Markdown

In your R Markdown document, you call the bibliography file in the YAML header like so:

---
output:
 html_document:
   toc: true
   number_sections: true
bibliography: ohi.bib
csl: ecology-letters.csl

---

Notes:

  • You can easily include the bibliography file in a subfolder: bibliography: refs/ohi.bib (if you like to keep your repository folders organized)
  • the csl: allows you to include different “citation style language” formats for different journals; here is a link to Zotero’s style library.
    • I found a github repo with many .csl files too, but they didn’t seem to work - something missing or corrupted. The Zotero ones seemed to work fine.
    • I just put the .csl into the same subfolder as the .bib, so my YAML header looked like:
bibliography: refs/ohi.bib
csl: refs/ecology-letters.csl


Citing in the text of your R markdown

In the R Markdown document, the citations look like this: [@articleidentifier] where “articleidentifier” is the author/title fragment/year in the reference listed in the .bib file. So for the reference to Ready et al. 2010 from the .bib above, the citation would look like: [@ready_predicting_2010].

  • Multiple citations are separated by semicolons within the same set of square brackets: [@ready_predicting_2010; @selig_assessing_2013]
  • For in-line citations where you’ve already mentioned the author’s name and want to exclude it, e.g. Halpern et al (2012) are awesomesauce”, just put a ‘-’ sign in front of the @ sign: “Halpern et al [-@halpern_index_2012] are awesomesauce”
  • You can also put in other text inside the square brackets: [To experience more awesomeness, see also @halpern_spatial_2015]

NOTE: One challenge I ran into is figuring out the articleidentifier (aka bibtexkey):

This key doesn’t seem to be displayed anywhere in Zotero’s gui, instead you have to search through the .bib document in a text editor.

For me, a better option has been to use JabRef, which allows you to search the .bib document more easily:


Knitting it all together

The knitr package will turn those citation calls into the proper format for the particular journal (e.g. superscripts or parentheses, name or initials or just year, etc) based on the specifications in the .csl. Call a different .csl when submitting to a different journal to change the citation format.

The bibliography will be attached at the very end of the document; no need to do anything special at all. But: you probably want to include a heading title - that is not automatic. So for example, the last line of the .Rmd should be # References or the like.

The references will be in order according to the citation format, e.g. alphabetical by author, or in order of appearance in the body of the text, with an ID number or without, and formatted to the idiosyncrasies of the given journal.

If you use knit child options to attach separate .Rmd files, it still seems to work fine - only the parent gets the header, and as it knits each child .Rmd it annotates the citations and includes them in the final bibliography.

Other notes

If you have blank items in your Zotero library/collection, those will show up in the .bib file as empty records, but will crash the knitting. So don’t do it.

You can edit the .bib manually if you like (e.g. delete empty records that are crashing everything) but if you re-export the library you’ll overwrite your changes, so probably best to get your Zotero set up properly.

5.3.2 Working with Zotero

Getting started with zotero (OHIprep_v2019)

5.3.2.1 Zotero group library

We have a group OHI Zotero library. This is handy because we can save and organize our OHI references so they are available to everyone (including screenshots, pdfs, and notes). In regard to writing, Zotero offers a very intuitive method of inserting citations into Word documents and Casey Ohara describes how to use Zotero with markdown.

To use the Zotero group library for OHI, you will first need to be invited to the group. You will then need to download the standalone Zotero program and the web-browser plugin from here. Hopefully everything will sync up automatically. If it works, you should see this:

5.4 Methods document

Every year we update a document describing the methods used to calculate the assessment (example methods document: https://oceanhealthindex.org/images/htmls/Supplement.html)

The files used to create this document are located here: https://github.com/OHI-Science/ohi-global/tree/draft/documents/methods and https://github.com/OHI-Science/ohi-global/tree/draft/metadata_documentation

To maximize flexibility, we have broken our methods into component parts that can be used in multiple contexts. For example, the Rmd files that describe each data layer are incorporated into the official methods document and are also used to create a webpage describing the data layers. Previously, if we updated our methods we had to update both the methods document and the website, but with this approach we only have to make the change in one location. This helps keep our messaging more consistent.

Table: A description of the files in the ohi-global/global_supplement repository

folder/file name description updates
Supplement.Rmd combines all components to create the final methods document No changes to this script are likely necessary, but this Rmd must be knitted when any components change (see description below)
CombineLayers.R Combines all the layer Rmds located in the layers_info folder into a single Rmd file called layers_all.Rmd No changes to this script are likely necessary, but this R file must be run whenever there are changes to any of the data layer Rmds
layers_all.Rmd Created by CombineLayers.R, includes a description of all data layers Created automatically when CombineLayers.R is run
OHI.bib A .bib file of the OHI references (more on that below) Updated yearly
BIBcorrect.R Corrects weirdness in the .bib file This probably will need to be updated yearly; this will be determined after reviewing the formatted citation list
tables folder with datatables of 10 goal and 8 subgoals No changes likely necessary
index_scores.Rmd description of OHI model for calculating index scores No changes likely necessary
goal_model.Rmd description of OHI model for calculating goal scores No changes likely necessary
trend.Rmd description of OHI model for calculating trend scores No changes likely necessary
pressure.Rmd description of OHI model for calculating pressure scores No changes likely necessary
resilience. Rmd description of OHI model for calculating resilience scores No changes likely necessary
goal_descriptions folder individual Rmd files describing each goal/subgoal Minimal changes to some files may be necessary if the model or data used in a goal changes
layers_info folder individual Rmd files with descriptions of each data layer, filenames must match layer names in metadata_documentation datatables Changes to some layers will probably be necessary, but most layers will require no changes
figures folder figures displayed in the Supplement.html No changes likely necessary
methods-in-ecology-and-evolution.csl instructions for displaying references no need to change unless a different way of displaying references is desired

5.4.1 Steps to updating methods document

The methods document is created by knitting ohi-global/documents/methods/Supplement.Rmd. However, many preparations need to be made prior to knitting.

5.4.1.1 Step 1: Update metadata_documentation

The data tables in ohi-global/metadata_documentation are used to create the methods document, and consequently must be fully up-to-date and accurate! This includes: layers_eez_base.csv, layers_eez_data_sources.csv, layers_eez_methods.csv, and layers_eez_targets.csv.

Currently, the layers_eez_gapfill.csv is not used to create the methods document, but it is still a good idea to update this as well.


layers_eez_base.csv

This file includes the directory and file name for each of the updated layers. Updates to this file should be completed when you are updating each layer to global (see Chapter 4, Updating Scores).


layers_eez_data_sources.csv

In the OHI group Zotero library (invite required), you will create a new reference for each of the data sources that have been updated for the current assessment year. NOTE: For the OHI global assessment, you should do all Zotero work in the “OHI Global Refs” folder. This can be done easily by downloading Zotero to your computer and installing the extension in your browser to save the data source directly to the library, where the information can then be edited. (See section 5.3: Citation Management for more information).

Create citations in Zotero or update the existing citations, including title, author, abstract, date, URL, and accessed date. “Date” should indicate the year listed in the data citation, OR the most recent year in the dataset if no citation exists. “Accessed” should align with the date indicated in the data prep .Rmd file, OR the timestamp on the saved raw data files if a download date is not obvious. If there are new data sources, fill out the information and add them to “OHI Global Refs” folder. If the datasource has an additional year of data, find that datasource in “OHI Global Refs” and update the date and any other relevant information. Example of completed citation information in Zotero:

One you have completed filling in citation information, right click the item in Zotero and choose “Export Item….” Export as a BibLaTeX file and save as a .bib to your computer. It is VERY important that when you right click the item and export it, you are working from the “OHI Global Refs” folder.

Open this file in a text editing app and copy the tag at the beginning:

Paste this tag into layers_eez_data_sources.csv in the ds_reference column. If it is an update to an existing data source, copy over it and update the ds_years column for the time range of the data (also found in the Rmd, or within the data source itself). Check to make sure the other information is still accurate, and add a new column “ds_updated_20??” to indicate which references have been updated. Fill in “y” if you are updating an existing data source.

If it is a new data source, paste the tag into a new line and fill in the other information (name, description, years, and native resolution).

For all references that were not updated, fill in the ds_updated_20?? with “n” or “NA.”


layers_eez_dataprep

dataprep_url: File path to the knitted html file . Note: finalized data prep files should all be knitted BEFORE attempting to update this document, as this can take some time!


layers_eez_reference

reference: Alias tag created BibLaTeX file as described above. This tag should match exactly with the tag in layers_eez_data_sources.csv. Always double check this!


layers_eez_updates

updates: Short description of updates to the data; usually is additional year of data and/or changes to gapfilling methods or other calculations.


layers_eez_gapfill.csv

Add file path location of gapfilling report for each layer.


layers_eez_targets.csv

This really only needs to be updated when layers are removed/added.

5.4.1.2 Step 2: Update OHI.bib

Make sure all references are included in the OHI Zotero library (see Section 5.3: Citation Management for more). Then export the folder “OHI Global Refs” as a BibLaTeX file and replace the OHI.bib file in ohi-global/documents/methods/ with the updated file.

5.4.1.3 Step 3: Update layers_info

The metadata_documentation/ohi_model/layers_info folder includes an Rmd file for every data layer used to calculate the global OHI assessment. The files describe the general methods and data used to generate the layer. Ideally, these Rmd files do not include information that changes each year (e.g., final year of data used in analysis, links to data preparation files on Github, etc), so they can be used each year with minimal changes. Components that change every year should be included in tables, such as those in ohi-global/metadata_documentation.

However, if there have been changes to the model or data used to generate a layer it will be necessary to update the corresponding Rmd file.

After the Rmds have been updated it is necessary to run the CombineLayers.R file, which merges all the layer information into a single document, layers_all.Rmd.

5.4.1.4 Step 4: Review other files

Review whether any of the other files require editing. There may be changes to the text to make it clearer, but otherwise, the following files are unlikely to change:

There will usually be no changes to the following files: goal_model.Rmd, index_scores.Rmd, pressure.Rmd, resilience.Rmd, or trend.Rmd, unless changes are made to the OHI models.

The Rmds describing the goal models in the goal_descriptions and goal_descriptions_short folders will only change if there have been changes to the goal model or data.

Changes to the goal/subgoal descriptions in the “tables” folder are unlikely.

5.4.1.5 Step 5: Knit Supplement.Rmd

Review the Supplement.Rmd document to update the assessment year (i.e., title at top, assessmentYear variable in this code chunk, update figure 3.1). Knit the file and review the html output. Make any necessary changes and repeat…seemingly endlessly until everything is correct.

5.5 Results document

After producing the final scores, we create a document that describes the results of the assessment: https://oceanhealthindex.org/images/htmls/Supplement_Results.html. This file can be accessed from our website.

The file is created using this Rmd: ohi-global/yearly_results/global20??/Results/Supplement_Results.Rmd.

It is critical to carefully step through this script and modify as needed; for example, the date of the radicalFile object must match the date appended to the csv file of scores created by Results_first_look.rmd (e.g., radicalFile = ‘2018-10-10,’ ~line 92).

And, of course, freely modify this script to reflect the data/figures you believe will be most useful to people. This could involve editing text, removing particular figures, adding figures, etc.

NOTE: If there are any changes to the scores, it is important to run the Results_first_look.Rmd file prior to running this script. Many of the figures in the Supplement_Results document are actually generated by the Results_first_look.Rmd script.

5.6 Update ESRI Living Atlas of the World OHI map

Every year we update our OHI ARCGIS online map to reflect the current assessment year’s scores. You can find the map here: ARCGIS Living Atlas of the World by searching for “Ocean Health Index.”

Through UCSB, you should have an ARCGIS online account. You will need to be added to the group “Global Ocean Health Index” by either Gage Clawson or Melanie Frazier (or whoever is running OHI at this point). https://ucsb.maps.arcgis.com/home/group.html?id=e2857950e3314d77a1fd14959f3cce8e#overview

First, you will run the script in the ohi-global repo, located in ohi-global/yearly_results/globalxxxx/ESRI_data/prepping_esri_data.Rmd. Link to 2021 prep.

Once you have run that script, you will need to upload the created data into the ARCGIS OHI global map.

To obtain the zip file which you will upload, export the .dbf, .shp, .shx, and .prj files from the ohi-global/yearly_results/globalxxxx/ESRI_data/scores_xxxx folder on Mazu, and rename to “ohi_scores_2020_final.zip.”

NOTE: It is important that this file is named “ohi_scores_2020_final.zip,” even if the assessment year is not 2020. This is because we originally uploaded the data and created an ARCGIS map from it when it was named “ohi_scores_2020_final.zip.” I’m sure we could figure out how to change the name of the source file on ARCGIS online if we wanted to, or recreate the map with a non-year specific name, but at this time its not worth the effort… just roll with 2020 in the name.

Now log into https://ucsb.maps.arcgis.com/. You should’ve been added to the “Global Ocean Health Index” group by now. Click on Groups —> Global Ocean Health Index —> and update the files accordingly as instructed below.

Upload the ohi_scores_2020_final.zip file to “OHI Scores Source” by clicking “Update Data” —> “Overwrite Entire Layer.” Now the data is updated on ARCGIS Online! All you have left to do is edit the text on “OHI Scores Source” and “Historical Global Ocean Health Index Scores (2012-2021)” to match the current assessment year, and all of the new data should be reflected in the ARCGIS Living Atlas of the World map for OHI.

5.7 Github Releases

One of the things we like to add to our final scores blog post is a link to a github release. This makes the ohi-global, ohiprep_vxxxx, and ohicore repositories easily downloadable as zip files (and we can add those zip files to knb, see next section). The link to our current releases is here: https://github.com/OHI-Science/ohi-global/releases

  1. Make a release for ohicore repository (if there were any changes to ohicore)
  2. Make a release for the ohi-global repository and link to ohicore and ohiprep_vxxxx repositories.

To make a release:

  • go to the repositories release page (https://github.com/OHI-Science/ohi-global/releases)
  • click “Draft a new release”
  • Copy/paste the information from previous releases, updating the year, version of the release, and all links. I.e. if we were making a release for v2022, we update to year to v2022, and since it it the first release, the name will be “v2022.1.”
  • Target the “published” branch of ohi-global
  • Create a new tag for vxxxx.x (i.e. if v2022 first release, it will be v2022.1)
  • Publish the release. This will create the zip files for ohi-global and show up on the releases page now.

Note: If you need to make a new release for ohicore, go to the ohicore repository and follow the same steps as above.

5.8 Upload to a repository

One of the final steps of an assessment is submitting the final data (and related files) to a data repository. This makes the files publicly available and provides a DOI (digital object identifier, https://www.doi.org/) used to officially cite the data. We have been using the Knowledge Network for Biocomplexity, or KNB (https://knb.ecoinformatics.org/) as our data repository. An example of our data on KNB is here: https://knb.ecoinformatics.org/view/doi:10.5063/F12Z13R6

Signing into KNB requires an ORCID account. An ORCID account is obtained here: https://orcid.org/register. You will then make your way to https://knb.ecoinformatics.org/. Select the SIGN IN tab, and then “Sign in with ORCID”:




Select the “Upload data,” and start adding files and information:

The following is the information we used for the 2018 assessment, but please modify and improve as desired!

5.8.0.1 Overview

Title: Global Ocean Health Index 2018 assessment

Abstract: Scores from the 2018 global Ocean Health Index (OHI) assessment and accompanying data and models. The global Ocean Health Index assesses ocean health for 220 coastal countries and territories and has been conducted yearly since 2012. The Index describes how well we are sustainably managing 10 goals which represent the full suite of benefits that people want and need from the ocean. These goals include: artisanal fishing opportunity, biodiversity, carbon storage, clean waters, coastal livelihoods and economies, coastal protection, food provision, natural products, sense of place, and tourism and recreation. Each goal is given a score ranging from 0 to 100, and the full suite of goal scores are then averaged to obtain an overall index score for each region. Please see http://ohi-science.org/ for additional resources and information.

Keywords: OHI, index, ocean

5.8.0.2 People

OHI team (including the OHI Fellows) Contacts: If in doubt, put Benjamin Halpern

5.8.0.3 Dates

First data is 2012. The later date is the year of the assessment.

5.8.0.4 Locations

Short geographic description: Global country and territorial EEZ regions (N=220)

Northwest coordinates: 90 -180 Southeast coordinates: -90 180

5.8.0.5 Taxa

NA

5.8.0.6 Methods

Step 1: See Methods.html file

5.8.0.7 Add Files

Please use your judgment, but we included the following files in the global 2018 assessment:

  • scores.csv: scores file that is created secondarily to include region names and expanded goal names.
  • Methods.html: methods used to calculate the global OHI scores (this is created in the global_supplement folder in the ohi-global repository).
  • Global_OHI_Results: Figures and tables of global results. This is currently an informal document, but I would like to improve this in future years.
  • Zip files of the 3 repositories used to calculate the global OHI assessment: ohiprep, ohi-global, and ohicore. These files are downloaded from the “release” version of repository (this assumes you have already created the release). You navigate to the release by first going to the repository of interest and selecting “releases”:




Then download the .zip option:


Once all the information has been entered and the files uploaded to the KNB site you can obtain a link for your data. Navigate to “My data sets” and select the one you are working on from the list and copy the web address. Create an issue and provide the link so anyone who wants to review the information can do so. You can easily edit everything at this point! NOTE: At this point the data does not have an official DOI.

Once everyone is satisfied with the content, you can officially “Publish with DOI”:


The data now officially has a DOI, and you can mark this off your list of things to do!