dev_how_to_maintain_my_tests_and_examples.RmdThe tests and examples in this package are based on the outputs of
the various functions in the package. For example, the
fm_fit_model function needs the output of the
fm_load_data function. One could, for each example or test,
run the fm_load_data function upstream to retrieve the
outputs and run fm_fit_model. The problem is that these
functions take a long time to run and so we prefer to store the output
of each function for later use.
There are two benefits to this solution:
Two models can be distinguished, a model with little data, which we will call “small”, and a model with a lot of data, which we will call “big”. These two models oblige us to have different strategies depending on the model.
We will detail :
We have chosen to store the results of the functions in the
inst/examples folder. This allows the outputs to be reused
in the examples. The content of this inst/ folder is
accessible at the root directory of FishMap, once the
package is installed or after running the following code inside the
FishMap project (it simulates the installation of FishMap) :
pkgload::load_all()On ce FishMap is installed, you can access the files with the following code:
fm_data_inputs <- readr::read_rds(system.file("examples", "part1_output_small.rds", package = "FishMap"))These outputs are also reused in tests in the same way.
The strategy for how data is stored is not the same depending on the model. Nevertheless, they are based on the same principle. Two environment variables are defined, each with a specific purpose.
The environment variable “FISHMAP_TEST_RESOLUTION” can take two
values “small” or “big”. By default, the value is “small”. You will need
to explicitly set the value to “big” in a .Renviron file.
This variable allows unit tests to be run under the right conditions for
the chosen model.
The “FISHMAP_UPDATE_TEST_OUTPUTS” environment variable is used to update the outputs to be tested. If its value is “TRUE” then the outputs will be replaced and everything else will remain the same.
We will therefore detail how this works for both models.
By default, unit tests are run with this model. They read the results
of the necessary functions and execute only the function to be tested.
Here is a simplified example of a test implementation for the
fm_fit_model function (you can find the full example in
tests/testthat/test-fm_fit_model.R):
# Testing for small model
test_resolution <- Sys.getenv("FISHMAP_TEST_RESOLUTION", unset = "small")
if(test_resolution == "small"){
expected_outputs <- readr::read_rds(
system.file(
"examples",
paste0("part2_output_", test_resolution , ".rds"),
package = "FishMap")
)
#' @description Testing that the result of `fm_fit_model` is stable
expect_equal(
object = expected_result,
expected = expected_outputs,
tolerance = 10e-6
)
}Explanation of what the code does:
Here is another example using the output of the
fm_load_data function:
if(test_resolution == "small"){
fm_data_inputs <- readr::read_rds(system.file("examples", "part1_output_small.rds", package = "FishMap"))
}else if (test_resolution == "big") {
fm_data_inputs <- readr::read_rds(system.file("examples", "part1_output_big.rds", package = "FishMap"))
}
# run part2
withr::with_seed(1234,{
fm_model_results <- fm_fit_model(fm_data_inputs = fm_data_inputs,
SE = 1,
data_source = 1,
data_obs = 2,
samp_process = 0,
b_constraint = 2,
cov_samp_process = 0,
biomass_temporal = 1,
sampling_temporal = 0,
lf_link = 0,
ref_data = "com",
EM = "est_b",
month_ref = 1)
})Code explanation:
fm_load_data functionThe unit tests are identical between the small and large model. They will compare the results of the current model with the recorded results (you will find a section below that explains how this works)
The difference between the two model is in the storage of the data.
In the case of the “big” model, they are stored in a sub-folder
data which is located in tests/testthat as
explained at the beginning.
As a dev, you will need to update the outputs if the functions or dependent packages change.
When you launch your tests and if you decide to update the outputs. The comparison of the results, by definition, will always be ok because the output read corresponds to the output you just created.
To do this, in each function, the unit tests allow you to update the
outputs thanks to the modification of an environment variable named
FISHMAP_UPDATE_TEST_OUTPUTS. In the code of unit tests, you
will see code as follows :
# Update expected outputs here
if (Sys.getenv("FISHMAP_UPDATE_TEST_OUTPUTS") == "TRUE") {
# save output depending if we are in flat or in test
output_inst_dir <- here::here("inst", "examples")
if (test_resolution == "small") {
readr::write_rds(x = fm_model_results,
file = file.path(output_inst_dir, paste0("part2_output_", test_resolution , ".rds")))
}else if (test_resolution == "big") {
readr::write_rds(x = fm_data_inputs,
file = file.path("data", paste0("part1_output_", test_resolution , ".rds")))
}
}
}
# check output is saved as rds
if (Sys.getenv("FISHMAP_UPDATE_TEST_OUTPUTS") == "TRUE") {
output_inst_dir <- here::here("inst", "examples")
#' @description Test to check if we can save output
expect_true(file.exists(file.path(output_inst_dir, paste0("part2_output_", test_resolution , ".rds"))))
}Code explanation:
To update all example & tests files in your package following a change in functions code, or package version, here is the procedure to follow:
.Renviron (you can access it with
usethis::edit_r_environ()) to add the following line:FISHMAP_UPDATE_TEST_OUTPUTS=TRUE.Renviron
variable has been correctly set with the command
Sys.getenv("FISHMAP_UPDATE_TEST_OUTPUTS"). It should return
TRUE now.You can run the unit tests from the command line using the devtools package with the following command:
devtools::test()inst/examples
have been updated.FISHMAP_UPDATE_TEST_OUTPUTS=FALSE.Renviron variable has been
correctly reset with the command
Sys.getenv("FISHMAP_UPDATE_TEST_OUTPUTS"). It should return
FALSE now.To run the “big” model in the CI, you will need to modify the
R-CMD-check.yaml workflow in
.github/workflows.
Proceed with the following steps:
FISHMAP_TEST_RESOLUTION to
“big”.[test] Running big model to tests changes.Because you are running a big model, this will take a long time.
FISHMAP_TEST_RESOLUTION back
to “small” and push on your branch.Why: We don’t want the big model to always run on the main branch.
While developing the package, we noticed significant differences between the INLA and TMB package versions.
To overcome the problem of the INLA and TMB package versions, we
invite you to keep the DESCRIPTION file up to date. This
file allows you to know which package version is used by the developers.
It should evolve over time and versions are not meant to be static.
If unit tests don’t pass because of differences in results, make sure
you have the correct package versions for INLA and TMB. You can find the
information on minimal version required in the DESCRIPTION
file with the lines Config/FishMap/INLA/version and
Config/FishMap/TMB/version. You can find your current
package version with the commands packageVersion("TMB") and
packageVersion("INLA").
Here is the code to update if needed:
desc_file <- here::here("DESCRIPTION")
version_inla <- as.character(utils::packageVersion("INLA"))
version_tmb <- as.character(utils::packageVersion("TMB"))
the_desc <- desc::desc(file = desc_file)
the_desc$set(`Config/FishMap/INLA/version` = version_inla)
the_desc$set(`Config/FishMap/TMB/version` = version_tmb)
the_desc$write(file = desc_file)