# Making Modules

# Introduction

This vignette will provide an overview of the formods framework for creating reproducable modules that interact with each other. Each module has its own namespace that is mantained by using a module short name as a prefix for functions. For example the figure generation module uses `FG`. If you want to create a module, please submit an issue at the formods github repository with the following information:

- The short name you intend to use
- A brief description of the module
- A list of modules it depends on

## Current modules

The following modules are currently available:

- ASM - App State Management
- DW - Data Wrangling
- FG - Figure Generation
- FM - Formods
- NCA - Non-Compartmental Analysis
- UD - Upload Data

Other short names in use:

- MC - This is a reserved word

Currently in development:

- MB - Model Builder
- CTS - Clinical Trial Simuilations

# formods framework

To get started you need to create some template files. The example below assumes you are creating this module for a package called `mypackage` and that you are running this command in a git repository. Say that this module is used to produce `widgets`, the short name is `MM` which stands for `My Module`:

    use_formods(SN          = "MM",
                Module_Name = "My Module",
                element     = "widgets",
                package     = "mypackage")

This command will create the following files:

- `MM_module_components.R` - An app that can be used for testing the module and highlighting the different UI elements that are used within the module (found in `inst/templates`).
- `MM_Server.R`: A bare bones file containing the expected functions and their minimal inputs. (found in `R`).
- `MM.yaml`: This module configuration file contains the the minimal elements expected, but you can add your own fields to suit your modules needs (found in `inst/templates`).
- `MM_funcs.R`: This script contains example code for the different elements of the module that can be used in the function examples and also to quickly test the different module functions while you develop them (found in `inst/test_apps`).
- `MM_preload.yaml`: This yaml file contains a skeleton of a preload configuration file. This will need to be populated to work with `MM_test_mksession()` (found in `MM_funcs.R`) in order to create a testing environment for your module (found in `inst/prelaod`).

# Expected functions

The module template will create a standard set of functions for you. The `MM` below will be replaced with whatever short name you choose above when you create the templates. These functions can be customized for your specific module. Some are optional and can be deleted. For example the `MM_fetch_ds` function is only needed if your module creates datasets and provides them for other modules to use (like the `DW` module exports data views to be used by other modules). The modules are designed to create elements. For example the `DW` module creates data view elements, the `FG` module is used to create figure elements, etc.

- `MM_Server` Shiny server function.
- `MM_init_state` Creates an empty the formods state for the module.
- `MM_fetch_state` Each module has a function to fetch the state. Within this function there should be no interactivity. Any access of the elements to the Shiny input should be isolated. Based on differences between the input elements (current state of the app) and the stored app state can be used to trigger different things.
- `MM_fetch_code` This takes as it’s first argument the module state. When called with only this argument it should return a character object containing all of the code needed to generate the elements of this module represented in the app. You can assume that any modules this one depends on will be defined previously. For example the `FG` module will return only that code associated with generating figures. It will be appended to the code from the `UD` and `DW` modules that define loading the dataset and creation of the data views. For modules where no code is generated (e.g. `ASM`) just return `NULL`.
- `MM_append_report` If a module generates reportable outputs, this function will be used to append those outputs to the overall reports generated by formods.
- `MM_fetch_ds` If a module provides data sets to be used in other modules you need to create this function. For an example see the `DW_fetch_ds()` in the `DW` module This function should return at least the following:
  - `hasds` Boolean variable indicating if the modules currently has any exportable datasets.
  - `isgood` General return status of the funciton. Set to FALSE if any errors were encoutered.
  - `msgs` A character vector of any messages to pass back to the user.
  - `ds` A module can provide multiple datasets. This is a list with the following elements for each dataset:
    - `label` Text label for the dataset
    - `MOD_TYPE` Short name for the type of module.
    - `id` module ID.
    - `DS` Dataframe containing the actual dataset.
    - `DSMETA` Metadata describing DS.
    - `code` Complete code to build dataset.
    - `checksum` Module checksum.
    - `DSchecksum` Dataset checksum.
- `MM_fetch_mdl` If a module provides ODE models to be used in other modules you need to create this function. To see an example of this check out the `MB_fetch_mdl()` in the `MB` module This function should return at least the following:
  - `hasmdl` Boolean variable indicating if the modules currently has any exportable models.
  - `isgood` General return status of the funciton. Set to FALSE if any errors were encountered.
  - `msgs` A character vector of any messages to pass back to the user.
  - `ts_details` List of timescale details
  - `mdl` A module can provide multiple models. This is a list with the following elements for each model:
    - `label` Text label for the model.
    - `MOD_TYPE` Short name for the type of module.
    - `id` module ID.
    - `rx_obj` The rxode2 object that holds the model.
    - `rx_obj_name` The rxode2 object name in generated code.
    - `ts_obj` The timescale object that holds timescales list.
    - `rx_obj_name` The timescale object name in generated code.
    - `fcn_def` Text to define the model
    - `MDLMETA` Verbose metadata describing model.
    - `code` Complete code to build the model.
    - `checksum` Checksum of the module the model came from.
    - `MDLchecksum` Checksum of the model.
- `MM_preload` The ASM module provides the ability to load a previous analysis from yaml files. This function is called with the contents of those yaml files to process and load them.
- `MM_mk_preload` This will create a preload list of the current state of the module.
- `MM_test_mksession` When testing outside of Shiny it is useful to have a pre-populated session, input, etc. objects with actual data. Each module should have a `test_mksession` function that populates these objects with useful data. If your module depends on a different module. The function should return the following:
  - `isgood` Boolean indicating the exit status of the function.
  - `msgs` Any messages generated when creating the test environment.
  - `session` The value Shiny session variable (in app) or a list (outside of app) after initialization.
  - `input` The value of the shiny input at the end of the session initialization.
  - `state` App state.
  - `react_tate`` The`react_state` components.
- `MM_new_element` Creates a new module element.
- `MM_fetch_current_element` Extracts the current element from the state object.
- `MM_set_current_element` Sets the current element to the provided value.
- `MM_del_element` Deletes the current active element.

# Expected UI components

- `ui_mm_compact` This is a UI output that contains a compact view of your module that can be called from the main ui functions for the App. It is composed of the individual UI elements that are shown in the `MM_module_components.R` file. This allows the user a quick way to utilize a model (using the `ui_mm_compact`), and the ability to customize the module UI by manually arranging the pieces found in `MM_module_components.R`.

# Module interaction

Say you are using the UD module to feed data into the DW module and the user goes back to the upload form and uploads a different data set. This will need to trigger a reset of the Data Wrangling module as well as tell your larger app that something has changed.

## Module state and reacting to changes

Changes in module states are detected with the `react_state` object. For a given module of type `"MM"` with a module id of `"ID"` you would detect changes by reacting to `react_state[["ID"]]` and looking for changes in the checksum element below:

`react_state[["ID"]][["MM"]][["checksum"]]`

- `checksum` A checksum that can be used to detect changes in this module. For example in the UD module this will change if the uploaded file or the sheet selected from a currently uploaded file changes.

# Helper functions in formods

- `FM_le()` - Creates log entries (`le`) that are displayed in the console and recorded in the log file for the current session.
- `FM_tc()` - This can be used to evaluate code, trap errors, and process results.
- `has_changed()` - Depreciated see `has_updated()`
- `has_updated()` - Used to compare objects to see if they have changed.
- `icon_link()` - Used to create the ui for a link using an icon.
- `is_installed()` - Used to test if a package is available.
- `is_shiny()` - Used to test if a session object is simply a list or a legitimate Shiny session object.
- `set_hold()` - Used to set a hold on one or more UI elements. This prevents internal updating of that UI element based on the current value in the App.
- `fetch_hold()` - This will retrieve the hold status of a UI element.
- `remove_hold()` - This will remove any holds set on a UI element.
- `FM_build_comment()` - This creates comments from strings so they will form sections when viewed in RStudio.
- `FM_add_ui_tooltip()` - Attaches a tooltip to a UI element.
- `FM_init_state()` - Called at the top of your module state initialization function to create a skeleton of a module state that you can then build upon.
- `FM_set_notification()` - Within you code you can create notifications and attach them to a module state.
- `FM_notify()` - Used in `observeEvent()` to show notifications that have not yet been displayed.
- `FM_set_mod_state()` - Used to save any changes to the module state.
- `FM_fetch_mod_state()` - Used to get the module state.
- `FM_set_ui_msg()` - Attach verbose messages or errors that need to be pushed back to the user in the app.
- `FM_pretty_sort()` - Used as a general sorting function that will try to make the sorted results prettier.
- `FM_pause_screen()` - Pauses the screen when doing something on the server side that takes a while.
- `FM_resume_screen()` - Resumes activity (unpauses the screen) when you’re done with the pause.
- `FM_fetch_data_format()` - Creates formatting information for display for a given data frame.

The examples below require a Shiny session variable and a formods state object. Here we create some examples and other objects needed to demonstrate the functions below.

    library(formods)
    # This creates the state and session objects
    sess_res = UD_test_mksession(session=list())
    #> → ASM: including file
    #> → ASM:   source: file.path(system.file(package="onbrand"), "templates", "report.docx")
    #> → ASM:   dest:   file.path("config","report.docx")
    #> → ASM: including file
    #> → ASM:   source: file.path(system.file(package="onbrand"), "templates", "report.pptx")
    #> → ASM:   dest:   file.path("config","report.pptx")
    #> → ASM: including file
    #> → ASM:   source: file.path(system.file(package="onbrand"), "templates", "report.yaml")
    #> → ASM:   dest:   file.path("config","report.yaml")
    #> → ASM: State initialized
    #> → ASM: setting word placeholders:
    #> → ASM:   -> setting docx ph: HEADERLEFT = left header
    #> → ASM:   -> setting docx ph: HEADERRIGHT = right header
    #> → ASM:   -> setting docx ph: FOOTERLEFT = left footer
    #> → ASM: module isgood: TRUE
    #> → UD: including file
    #> → UD:   source: file.path(system.file(package="onbrand"), "templates", "report.docx")
    #> → UD:   dest:   file.path("config","report.docx")
    #> → UD: including file
    #> → UD:   source: file.path(system.file(package="onbrand"), "templates", "report.pptx")
    #> → UD:   dest:   file.path("config","report.pptx")
    #> → UD: including file
    #> → UD:   source: file.path(system.file(package="onbrand"), "templates", "report.yaml")
    #> → UD:   dest:   file.path("config","report.yaml")
    #> → UD: State initialized
    #> → UD: module checksum updated:897d952fecbc804999396a96f9df4b20
    #> → UD: module isgood: TRUE
    state    = sess_res$state
    session  = sess_res$session

    # Here we load an example dataset into the df object.
    data_file_local =  system.file(package="formods", "test_data", "TEST_DATA.xlsx")
    sheet           = "DATA"

    df = readxl::read_excel(path=data_file_local, sheet=sheet)

## Setting holds on UI elements

The mechanics of the fetch state functions mean that each time a fetch state is called, all of the UI elements in the App are pulled and placed in the app state. This generally works well with some exceptions. The main exception is when you want to have a UI element that changes another UI element. Say for example you have a selection box with a UI id of `my_selection`. You want that selection to alter a text input with an id of `my_text`. However if you just poll the ui elements you may update `my_text` based on changes to `my_selection` then have those overwritten by the current value of `my_text`. To prevent this, you need to do two things:

- When processing `my_selection` you need to set a hold on `my_text` (done with `set_hold()`).
- When processing `my_text` you need to do that only if there is no hold set. This is checked with `fetch_hold()`

Lastly you need to remove the hold. This is done after the UI has refreshed with the new text value populated in to `my_text` (with the appropriate reactions set). This is done with an observeEvent that is triggered after everything else (with a priority of -100 below):

    remove_hold_listen  <- reactive({
      list(input$my_selection)
    })
    observeEvent(remove_hold_listen(), {
      # Once the UI has been regenerated we
      # remove any holds for this module
      state = MM_fetch_state(id              = id,
                             input           = input,
                             session         = session,
                             FM_yaml_file    = FM_yaml_file,
                             MOD_yaml_file   = MOD_yaml_file,
                             react_state     = react_state)

      FM_le(state, "removing holds")
      # Removing all holds
      for(hname in names(state[["MM"]][["ui_hold"]])){
        remove_hold(state, session, hname)
      }
    }, priority = -100)

The `remove_hold_listen` object should contain all of the inputs that create holds.

## Dataframe formatting information

If you want to tables and pulldown menues based on the types of data in each column you can use the `FM_fetch_data_format()` function.

    hfmt = FM_fetch_data_format(df, state)

    # Descriptive headers
    head(as.vector(unlist( hfmt[["col_heads"]])))
    #> [1] "<span style='color:#3C8DBC'><b>ID</b><br/><font size='-3'>num</font></span>"
    #> [2] "<span style='color:#3C8DBC'><b>TIME_DY</b><br/><font size='-3'>num</font></span>"
    #> [3] "<span style='color:#3C8DBC'><b>TIME_HR</b><br/><font size='-3'>num</font></span>"
    #> [4] "<span style='color:#3C8DBC'><b>NTIME_DY</b><br/><font size='-3'>num</font></span>"
    #> [5] "<span style='color:#3C8DBC'><b>NTIME_HR</b><br/><font size='-3'>num</font></span>"
    #> [6] "<span style='color:#3C8DBC'><b>TIME</b><br/><font size='-3'>num</font></span>"

    # Subtext
    head(as.vector(unlist( hfmt[["col_subtext"]])))
    #> [1] "num: 1,...,360"  "num: 0,...,84"   "num: 0,...,2016" "num: 0,...,42"
    #> [5] "num: 0,...,1008" "num: 0,...,2016"

The custom headers can be used with the `{rhandsontable}` package.

    hot = rhandsontable::rhandsontable(
      head(df),
      width      = "100%",
      height     = "100%",
      colHeaders = as.vector(unlist(hfmt[["col_heads"]])),
      rowHeaders = NULL
      )

To add subtext to a selection widget in Shiny you need to use the `{shinyWidgets}` package.

    sel_subtext = as.vector(unlist( hfmt[["col_subtext"]]))
    library(shinyWidgets)
    shinyWidgets::pickerInput(
        inputId    = "select_example",
        choices    = names(df),
        label      = "Select with subtext",
        choicesOpt = list(subtext = sel_subtext))

To alter the formats shown here you need to edit the `formods.yaml` configuration file and look at the `FM`\(\rightarrow\)`data_meta` section.

## Notifications

Notifications are created using the `{shinybusy}` package and are produced with two different functions: `FM_set_notification()` and `FM_notify()`. This is done in a centralized fashion where notifications are added to the state object as user information is processed. This will set a notification called `Example Notification`. Along with that a timestamp is set:

       state = FM_set_notification(state, "Something happened", "Example Notification")

That timestamp is used to track and prevent the notification from being shown multiple times. Next you need to setup the reactions to display the notifications. Here you can create a reactive expression of the inputs that will lead to a notification:

        toNotify <- reactive({
          list(input$input1,
               input$input2)
        })

Next you use `observeEvent()` with that reactive expression to trigger notifications. You need to use the fetch state function for that module to get the state object with the notifications. Then `FM_notify()` will be called an any unprocessed notifications will be displayed:

        observeEvent(toNotify(), {
          state = MM_fetch_state(id              = id,
                                 input           = input,
                                 session         = session,
                                 FM_yaml_file    = FM_yaml_file,
                                 MOD_yaml_file   = MOD_yaml_file,
                                 react_state     = react_state)

          # Triggering optional notifications
          notify_res =
          FM_notify(state    = state,
                    session  = session)
        })

## Adding tooltips

Tooltips are created internally using the suggested `{prompter}` package. To add a tool tip to a ui element you would use the `FM_add_ui_tooltip()` function. For example to add the tool tip, `You need to type harder!` to a text input you would do the following:

    uiele = shiny::textInput(
              inputId = "some_text",
              label   = "You need to type harder!")
    uiele = FM_add_ui_tooltip(state, uiele,
          tooltip  = "This is a tooltip",
          position = "left")

## Pausing the screen

To pause the screen the `{shinybusy}` package is also used. This is controlled with two functions: `FM_pause_screen()` is used to pause the screen and/or update the pause message, and `FM_resume_screen()` is used end the pause and resume interaction with the user.

    FM_pause_screen(state, session)
    FM_resume_screen(state, session)

# formods state objects

When you create a formods state object it can have the following fields:

- `yaml`- Contents of the formods configuration file.
- `MC` - Contents of the module configuration file.
- `MM` - MM here is the short name of the current module. `MOD_TYPE` below), this is where you would store any app information. (see below).
- `MOD_TYPE` - Short name of the module.
- `id` - ID of the module.
- `FM_yaml_file` - formods configuration file.
- `MOD_yaml_file` - Module configuration file.
- `notifications` - Contains notifications set by the user through `FM_set_notification()`.

## App information in MM

This field `state$MM` is relatively free form but there are some reserved elements. These reserved keyword are:

- `button_counters` - Counter that tracks button clicks
- `ui_hold` - List of hold elements that is populated with `set_hold()`
- `isgood` - Boolean variable indicating the state of the module.
- `ui_msg` - Messaages returned to the UI with captured errors populated with `FM_set_ui_msg()`

Other than those fields you can store whatever else you need for your module.

# Checklist

The following is a suggested checkist to go over when making a module:

- Try the following functions with and without content
  - `MM_mk_preload`
  - `MM_fetch_code`

## Configuration file

# YAML configuration files

    # https://www.glyphicons.com/sets/basic/
    #General formods (FM) configuration across modules
    FM:
      include:
        # This is where you can put files to include files in the working directory
        # of the app. For the files listed below you shouldn't change the 'dest'
        # portion but you can change the source to use custom report templates.
        # If relative paths are used they will be relative to
        # the user directory (either the temp formods directory running in shiny
        # or the top level of the zip file structure when saving the app state).
        files:
        - file:
            source: 'file.path(system.file(package="onbrand"), "templates", "report.docx")'
            dest:   'file.path("config","report.docx")'
        - file:
            source: 'file.path(system.file(package="onbrand"), "templates", "report.pptx")'
            dest:   'file.path("config","report.pptx")'
        - file:
            source: 'file.path(system.file(package="onbrand"), "templates", "report.yaml")'
            dest:   'file.path("config","report.yaml")'
      # Some features (e.g. copy to clipboard) don't work when deployed
      deployed: FALSE
      #General code options for the modules
      code:
        theme:           "vibrant_ink"
        showLineNumbers: TRUE
        # File name of the R script to contain generation code
        gen_file: run_analysis.R
        # This is the preamble used in script generation. It goes on the
        # top. Feel free to add to it if you need to. Note that packages should be
        # listed in the packages section at the same level.
        gen_preamble: |-
          # formods automated output ------------------------------------------------
          # https://formods.ubiquity.tools/
          rm(list=ls())
        # Each module should have a packages section that lists the packages
        # needed for code generated for that module.
        packages: ["onbrand", "writexl"]
      notifications:
        config:
          # You can put any arguments here that would be arguments for
          # config_notify(). See ?shinybusy::config_notify() for more information
          success:
            useFontAwesome: FALSE
            useIcon:        FALSE
            background:     "#5bb85b"
          failure:
            useFontAwesome: FALSE
            useIcon:        FALSE
            background:     "#d9534f"
          info:
            useFontAwesome: FALSE
            useIcon:        FALSE
            background:     "#5bc0de"
          warning:
            useFontAwesome: FALSE
            useIcon:        FALSE
            background:     "#f0ac4d"
      reporting:
        # enabled here controls reporting for the app. Individual modules can be
        # controlled in their respective configuration files
        enabled: TRUE
        # The content_init section is used to initialize reports. You shouldn't
        # change the xlsx rpt but the docx and pptx rpt can be altered to
        # pre-process reports if you need to such as adding default.
        content_init:
          xlsx: |-
               rpt = list(summary = NULL,
                          sheets  = list())
          docx: |-
               rpt  = onbrand::read_template(
                 template = file.path("config", "report.docx"),
                 mapping  = file.path("config", "report.yaml"))
          pptx: |-
               rpt  = onbrand::read_template(
                 template = file.path("config", "report.pptx"),
                 mapping  = file.path("config", "report.yaml"))
        # Word template can contain placeholders. This is where you can put the
        # default values for placeholders. There are some nuances associated with
        # creating placeholders and you should see the documentation about them in
        # the onbrand package to better understand that:
        #
        #   https://onbrand.ubiquity.tools/articles/Creating_Templated_Office_Workflows.html#placeholder-text
        #
        # Each element below contains the placeholder used and should contain a
        # location and a value. For example the default document contains a header
        # placeholder in the upper right that looks like:
        #
        #  ===HEADERRIGHT===
        # The placeholder below is HEADERRIGHT, the location is header and by
        # default we will replace it with nothing. Placeholder text can have no
        # spaces.
        #
        # These defaults can be overwritten in the save section. If your template
        # has no placeholders you can comment out the entire phs section.
        #
        phs:
          - name:      "HEADERLEFT"
            location:  "header"
            value:     ""
            tooltip:   "left header text"
          - name:      "HEADERRIGHT"
            location:  "header"
            value:     ""
            tooltip:   "right header text"
          - name:      "FOOTERLEFT"
            location:  "footer"
            value:     ""
            tooltip:   "left footer text"
        phs_formatting:
          width:       "100%"
          tt_position: "left"
          tt_size:     "medium"
      ui:
        # See ?actionBttn for styles
        button_style: "fill"
        # Max size for picker inputs
        select_size:  10
        color_green:  "#00BB8A"
        color_red:    "#FF475E"
        color_blue:   "#0088FF"
        color_purple: "#bd2cf4"
      data_meta:
        # This controls the overall format of headers and the select subtext for
        # data frames with the following placeholders surround by ===:
        # COLOR  - font color
        # NAME   - column name
        # LABEL  - type label
        # RANGE  - this depends on the nature of the data in the column:
        #        - If there are between 1 and 3 values then those values are shown.
        #        - If there are more than 3 values then the min and max are show.
        data_header:  "<span style='color:===COLOR==='><b>===NAME===</b><br/><font size='-3'>===LABEL===</font></span>"
        subtext:      "===LABEL===: ===RANGE==="
        # Separator when showing more than three in a column. For example if you
        # had a dataset with 1,2,3,4,5,6 and many_sep was ",...," then it would
        # appear as "1,...,6"
        many_sep: ",...,"
        # This controls the differences for different data types. Take the output
        # of typeof(df$colname) and put an entry for that output here.
        data_types:
          character:
            color:    "#DD4B39"
            label:    "text"
          double:
            color:    "#3C8DBC"
            label:    "num"
          integer:
            color:    "#3C8DBC"
            label:    "num"
          other:
            color:    "black"
            label:    "other"
      workflows:
        example:
          group:      "Examples"
          desc:       "Example Workflow"
          # Set to true if the workflow requires a dataset
          require_ds: TRUE
          # this can contain an absolute path as a string or R evaluable code
          preload:    "file.path('.', 'example.yaml')"
      labels:
        # JMH remove this once the dataset stuff has been moved over
        # default_ds:   " Original data set"
        ui_label:      "put labels here"
      user_files:
        use_tmpdir:     TRUE
      logging:
        enabled:        TRUE
        timestamp:      TRUE
        timestamp_fmt: "%Y-%m-%d %H:%M:%S"
        log_file:      "formods_log.txt"
        console:       TRUE