# globaltrends 0.2.1

## Data Structure

* Change from sqlite/parquet files to `data.table` for performance reasons
  * All `globaltrends` data is stored in a single, non-compressed RDS file
  * The data is stored in a list objectiv with the following `data.table` objects:
    * `batch_keywords`
    * `batch_time`
    * `data_control`
    * `data_object`
    * `data_score`
    * `data_doi`
    * `data_locations`
    * `data_region`
    * `data_related`
    * `keyword_synonyms`
  * All `globaltrends` data is loaded into memory as a list of `data.table` objects
  * **This makes the storage structure incompatiable with the structure used up to `0.2.0`**
    * It is possible to integrate the data manually. To this end, extract the data from the parquet files, transform it to `data.table` objects and enter it into the respective elements in the `gt.env` environment.
* Drop `vacuum_data()` function, since the data architecture is not built on a database file any more

## Efficiency

* `compute_score()` now persists in-memory data to disk after every 1,000 computed locations, matching the periodic persistence already used by the `download_xxx()` functions

## Testing

* All tests now run fully isolated, in their own temporary directory, with automatic cleanup (`withr`) - no shared state or leftover files between tests
* Live Google Trends API calls in the test suite are now opt-in (set `GLOBALTRENDS_LIVE_TESTS=1`), so automated test runs are offline and deterministic by default
* Added test coverage for previously untested internals: API retry/backoff logic, quota handling, input validators, database persistence, and `initialize_python()`
* Enabled parallel test execution (`Config/testthat/parallel: true`)

# globaltrends 0.2.0

* General revision and improvement of code base
* General revision and improvement of tests
* General revision and improvement of documentation
* Inclusion of API-usage counter for downloads through Google Trends API (10,000 daily downloads)

## Change in database handling

* Switch from sqlite to parquet
  * Database tables are stored as individual parquet files
  * `start_db()` loads the parquet files to memory
  * All database actions are conducted in memory and later written to parquet
  * The download functions write in-memory data to parquet after every 1,000 downloads

## Technical issues and bug fixes

* Updated dependency to testthat 3.0.0
* Improved error handling for Google Trends API calls:
  `.get_trend()` now catches three classes of errors from the Research API backend (HTTP via reticulate) instead of crashing with a Python traceback
  * Quota exceeded (HTTP 429): writes in-memory data to parquet and then aborts with a clear message asking the user to wait for the quota to reset.
  * Invalid argument (HTTP 400): skips the offending term/location/date combination silently and continues.
  * Timeout (WinError 10060): skips the affected location and continues; the missing data can be filled by re-running download_object().

# globaltrends 0.1.0

## Change in downloading approach

* Run downloads from Google Trends through the official Google Trends API
  * Access available from Google: [https://support.google.com/trends/contact/trends_api](https://support.google.com/trends/contact/trends_api)
  * Downloads are made with the [Google API Python Client](https://github.com/googleapis/google-api-python-client)
  * `globaltrends`uses `reticulate` to run the respective Python script
* Users can choose to download through the official API or the `gtrendsr`package
  * If users call `initialized_python()`, the downloads will be made through
    the official Google Trends API
  * If users run downloads without calling `initialized_python()`, the downloads
    will be made through the `gtrendsr`package
* Control the waiting time between queries through the environmental variable
  `gt.env$query_wait` (default value = `0.1s`)

## Package simplifications

* Remove time-series adjustments from `compute_score` for package simplification
* Remove `get_abnorm_hist` function for package simplification
* Remove plotting functions and export classes for package simplification
* Accelerate `compute_score`
* Accelerate handling of synonyms in separate function `aggregate_synonyms`

## Technical issues and bug fixes

* Increase dependencies to `dplyr 1.1.1` and replace the `multiple` argument by
  `relationship`
* Fixed issues in sample data

# globaltrends 0.0.14

* Remove dependency on `glue` package, calls of `glue::glue` are replaced with `paste` and `paste0`
* Remove dependency on `ineq` package, calls of `ineq::ineq` are replaced with internal functions
* Use functions from `DBI 1.1.0`: `dbCreateTable` and `dbAppendTable`
* Remove the function `plot_map` from the package:
  * The function proofed too inflexible, only world map was included
  * We point users to the functions from the packages `maps` and `ggmap`
  * Remove dependency on the `maps` package
* Remove reference to non-existent table `data_global` in function `remove_data`

# globaltrends 0.0.13

* Set minimum dependency of `stats` and `utils` to version `3.5.0` in line with minimum `R` dependency
* Store globaltrends logo in the *vignettes* folder
* Add explanation of Google's data preparation methodology to vignette
* Change wait intervals for status responses != 200 and add message about automatic retry of download
  * Status == 500: wait 1 second
  * All other responses: wait 60 seconds
* Add function `vacuum_data` to free unused memory after `remove_data`
* Remove all usage of `.data` to comply with `tidyselect 1.2.0`, this applies to calls of:
  * `dplyr::rename`
  * `dplyr::select`
  * `purrr::map`
  * `purrr::walk`
  * `tidyr::nest`
  * `tidyr::pivot_longer`
  * `tidyr::pivot_wider`
  * `tidyr::unnest`
* As a consequence of the changes in `tidyselect 1.2.0`, several objects are defined as global variables (see `globals.r` for details) 
* Replace `size` with `linewidth` to comply with `ggplot2 3.4.0`
* Add the argument `multiple = "all"` and `multiple = "error"` to comply with `dplyr 1.1.0`
* Facilitate handling of location = "NA - Namibia", the function `add_locations` does automatically drop the location
* Remove dependency on `WDI` package, references to `WDI::WDI_data` are replaced by data objects `countries` and `countries_wdi`

# globaltrends 0.0.12

* Stop direct exports from functions to .GlobalEnv
* Exports of objects (e.g., SQL connections, keyword tables) are redirected to package environment `gt.env`
* Add handle for Namibia ISO code

# globaltrends 0.0.11

* Add function plot_map()
* Add automatic stringr::str_squish for keywords and synonyms
* Add documentation on usage of search topics

# globaltrends 0.0.10

* Change indication and handling of synonyms (synonyms are now ignored in `export_score` and `export_voi`)

# globaltrends 0.0.9

* Fasten computation of search scores for data with synonyms
* Fix bugs for control keywords that have a mean of 0 for their search volume time series

# globaltrends 0.0.8

* Adapt `export_xxx` for `vector` and `list` inputs
* Optimize RAM usage of `export_xxx`
* Check whether "db/globaltrends_db.sqlite" file exists in working directory
* Inclusion of "workaround" plot functions that set class `xxx` automatically:
  * `plot_xxx_box`
  * `plot_xxx_bar`
  * `plot_xxx_ts`

# globaltrends 0.0.7

* Change waiting times for errors
  * Status Code != 200/Limit exceeded -> 60 seconds wait
  * Status Code == 500 -> 1 second wait

# globaltrends 0.0.6

* To distinguish them from the actual database tables, names of the example data
  object were adapted. Documentation is still available for the respective database
  table.
  * `batch_keywords` -> `example_keywords`
  * `batch_time` -> `example_time`
  * `data_control` -> `example_control`
  * `data_doi` -> `example_doi`
  * `data_object` -> `example_object`
  * `data_score` -> `example_score`
* Waiting period between downloads reduced from 20-30 seconds to 5-10 seconds

# globaltrends 0.0.5

* Added classes for output from `export_xxx`
  * `export_score` -> class("exp_score")
  * `export_voi` -> class("exp_voi")
  * `export_doi` -> class("exp_doi")
* `export_xxx` + `get_abnorm_hist` supersedes `export_xxx_change`
  * method for `export_score` -> class("exp_score")
  * method for `export_voi` -> class("exp_voi")
  * method for `export_doi` -> class("exp_doi")
* `plot_bar` supersedes `plot_score`
  * method for `export_score` -> class("exp_score")
  * method for `export_score` -> `get_abnorm_hist` -> class("abnorm_score")
* `plot_ts` supersedes `plot_voi_ts` and `plot_doi_ts`
  * method for `export_score` -> class("exp_score")
  * method for `export_score` -> `get_abnorm_hist` -> class("abnorm_score")
  * method for `export_voi` -> class("exp_voi")
  * method for `export_voi` -> `get_abnorm_hist` -> class("abnorm_voi")
  * method for `export_doi` -> class("exp_doi")
  * method for `export_doi` -> `get_abnorm_hist` -> class("abnorm_doi")
* `plot_box` supersedes `plot_voi_box` and `plot_doi_box`
  * method for `export_score` -> class("exp_score")
  * method for `export_score` -> `get_abnorm_hist` -> class("abnorm_score")
  * method for `export_voi` -> class("exp_voi")
  * method for `export_voi` -> `get_abnorm_hist` -> class("abnorm_voi")
  * method for `export_doi` -> class("exp_doi")
  * method for `export_doi` -> `get_abnorm_hist` -> class("abnorm_doi")
