Query Google Trends with retry and backoff

Source: R/gtrends_with_backoff.R

Thin wrapper around gtrendsR::gtrends() that retries failed requests using jittered exponential backoff. Google Trends regularly returns transient errors (non-200 responses, HTTP 429 rate limits, 5xx server errors) and the connection itself can fail intermittently; retrying with growing, randomized waits is far more robust than failing on the first hiccup.

gtrends_with_backoff(
  keyword = NA,
  geo = "ch",
  time = "today+5-y",
  gprop = "web",
  category = "0",
  hl = "en-US",
  low_search_volume = FALSE,
  cookie_url = "http://trends.google.com/Cookies/NID",
  tz = 0,
  onlyInterest = FALSE,
  retry = 6,
  empty_retry = 4,
  wait = 4,
  max_wait = 30,
  quiet = FALSE,
  attempt = 1
)

Arguments

keyword

A character vector with the actual Google Trends query keywords. Multiple keywords are possible using gtrends(c("NHL", "NBA", "MLB", "MLS")).

geo

A character vector denoting geographic regions for the query, default to “all” for global queries. Multiple regions are possible using gtrends("NHL", c("CA", "US")).

time

A string specifying the time span of the query. Possible values are:

"now 1-H"

Last hour

"now 4-H"

Last four hours

"now 1-d"

Last day

"now 7-d"

Last seven days

"today 1-m"

Past 30 days

"today 3-m"

Past 90 days

"today 12-m"

Past 12 months

"today+5-y"

Last five years (default)

"all"

Since the beginning of Google Trends (2004)

"Y-m-d Y-m-d"

Time span between two dates (ex.: "2010-01-01 2010-04-03")

gprop

A character string defining the Google product for which the trend query if preformed. Valid options are:

  • "web" (default)

  • "news"

  • "images"

  • "froogle"

  • "youtube"

category

A character denoting the category, defaults to “0”.

hl

A string specifying the ISO language code (ex.: “en-US” or “fr”). Default is “en-US”. Note that this is only influencing the data returned by related topics.

low_search_volume

Logical. Should include low search volume regions?

A string specifying the URL from which to obtain cookies. Default should work in general; should only be changed by advanced users.

tz

A number specifying the minutes the returned dates should be offset to UTC. Note the parameter 'time' above is specified in UTC. E.g. choosing "time=2018-01-01T01 2018-01-01T03" and "tz=-120" will yield data between 2018-01-01T03 and 2018-01-01T05, i.e. data specified to be in UTC+2.

onlyInterest

If you only want the interest over time set it to TRUE.

retry

Maximum number of attempts before giving up on a hard error (429, 5xx, network, ...).

empty_retry

Maximum attempts for a "no data returned" response before accepting the window as genuinely empty (zero volume). Kept small: a truly empty window never recovers, so retrying it the full retry ladder just wastes time; a few quick attempts still let a transient rate limit (which can also surface as "no data") recover. This is the lever that keeps a rate-limited backfill from grinding for hours.

wait

Base wait in seconds. The wait before attempt n grows exponentially (wait * 2^(n - 1)), is capped at max_wait, and has random jitter added to avoid synchronized retries hammering the server.

max_wait

Upper bound (seconds) for a single backoff wait.

quiet

If TRUE, suppress progress messages.

attempt

Internal counter, do not set manually.