FRED-E is a system that allows you to manage, download, and browse FRED series data and catalogs programmatically as Pandas dataframes. While the FRED site provides a fairly robust API to access historical time series account for edits / changes, there lacks a programmatic means to conduct analyses of the full catalog of series. FRED-E downloads and saves the full corpus (definitions), and provides the means to maintain a watchlist of series which the framework can programatically auto-update on a daily / periodic cadence.

The FRED-E framework wraps the FRED API w/ resilience and retry logic to protect against intermittent outages (default is 5 retries separated by 5 seconds in the event of a failure); However, FRED-E cannot shield the system from extended outages of FRED, changes to rate limiting policy, or incorrect credentials / API Keys etc.

Setup Environment

Fred-E follows a standard environment setup. Create a virtual environment, install the dependencies in setup.py. Make sure you activate your virtual environment whenever you utilize the library.

If you are developing / changing FRED-E, set up virtual environment:

cd /path/to/fred-e/root
python -m venv venv
source venv/bin/activate # note: to deactivate, run: venv/bin/deactivate
python setup.py install

If you are just using FRED-E as a library, pip install from github or locally

# Download direct from github to install as a package (will prompt for your credentials) 
pip install git+https://github.com/davantixyz/fred-e

# If you've already cloned the repo, create a package from the local repo (this is probably preferred) 
git clone https://github.com/liangh/fred-e # will clone this to a local folder
pip install git+file:///path/to/project/root/fred-e

API Keys / Credentials

FRED API Key

If you are just consuming the data times series in BigQuery, then you won't need to create a FRED API key. If you will be download time series (i.e. updating the series on the watchlist - more on this below), then an API integration into FRED will be required. Details on how to create an API key on FRED can be found here.

Google BigQuery Credentials

The project workspace hosting the bigquery instance populated by the Fred-E framework will have to give you access to the dataset. The workpace administrator will have to create a service account credential for your google account. After assigning access, you will be able to download a credential JSON file w/ the private key. This is set as a system property loaded into the FRED-E runtime (more on this in the next section)

Note, for a local relational database, a connection URI will be required - again, this is set as a system property passed into the FRED-E runtime.

Runtime + File Properties

The FRED-E runtime accepts a number of properties that will specify everything from connection details to credentials to runtime settings. A full list of available properties is below. There are two ways to set properties into the FRED-E runtime: (a) via a properties file that is loaded automatically into the FRED-E runtime, or (b) by setting the properties directly into the runtime programmatically. Examples of both are below.

Properties File

On startup, FRED-E will load the file set in the system property FRED_E_SETTINGS. Note that this is the OS system property, not the FRED-E runtime properties below. The module / function which loads this file is in: fred_e.core.settings_env (function: read_env()). Within this file, runtime properties are set like the below

FRED_API_KEY = 'my_fred_api_key'
API_SLEEP_SEC = 1.0
SERIES_RELEASE = 'latest'

JSON Config File

If the FRED_E_SETTINGS file is a JSON file, Fred-E will be able to parse the file automatically. Additionally - and this is true even for runtime properties - if the property has the extension .json or .properties, Fred-E will auto-load the file configuration properties and set the loaded dictionary / properties in the original property name.

Runtime Properties

FRED-E properties can also be set programmatically during runtime (this may be helpful when working in an interactive session (i.e. jupyter notebook, for example). These properties take effect immediately upon setting. Additionally (discussed in JSON config file section) - the property value is set to a .json suffix / extension, Fred-E will proactively attempt to load the file and set the loaded properties into the target property.

# Imports: all properties / session singleton reference
from fred_e.core.settings_env import *
from fred_e.core import session

# Get session settings - we'll add properties to this object 
settings = session.get_settings()

# Add properties like a dict to the session settings
settings[PROP_FRED_API_KEY] = 'my_fred_api_key'
settings[PROP_API_SLEEP_SEC] = 1.0
settings[PROP_SERIES_RELEASE] = 'latest'

# Will auto-load / read / parse file and assign to PROP_GOOGLE_CREDENTIALS
settings[PROP_GOOGLE_CREDENTIALS] = '/Users/me/security/my-gcs-security-credentials.json'

Available Properties (Full list)

All available FRED-E properties are listed as constants in the module fred_e.core.settings_env.

FRED Properties

- FRED_API_KEY: FRED API key, from the FRED API / FRED website
- API_SLEEP_SEC: number of seconds to sleep between requests (can be fractional i.e. 0.5s). FRED rate limits requests to their API
- SERIES_RELEASE: *all, earliest, latest* - when downloading series from the FRED API, what mode do we run.  latest = latest revision, earliest = earliest version, all = all revisions (this includes retroactive changes to data points)

General Properties

- DATA_ROOT: data root where files are stored for project
- DRY_RUN: if true, then will prevent writes / changes (experimental)

Persistence and Google BigQuery Properties

- PERSISTENCE_IMPL: the full-qualified concrete persistence implementation class.  For relational databases:  fred_e.persistence.impl.PersistenceRelational.  For google bigquery: fred_e.persistence.impl.PersistenceGoogleBigQuery
- PERSISTENCE_CONN_URI: in the case of a local relational database, this is the URI connection string (i.e. may contain credentials etc):  e.g. sqlite:///Users/myuser/data/fred-e/fred-e.db
- GOOGLE_APPLICATION_CREDENTIALS: path to the Google service account credentials file.  This is a json file that the dataset owner will be able to assign and that you can download in the google cloud dashboard.  (see section: Google BigQuery Credentials).  Note that if this property is set to a value w/ an extension of .json, Fred-E will auto-expand this set.
- GOOGLE_PROJECT_ID: Project where dataset is located
- GOOGLE_DATASET_ID: Name of dataset

Retrieve Series Data Catalog

You've done all of the setup in the above sections. For users / consumers of the data, the below sections are of the most interest.

Get all categories

Retrieve all records in categories table (all categories in FRED, recursive)

from fred_e import catalog
catalog.get_categories_all()

Get all series meta (i.e. series catalog)

The following function call will retrieve all series meta-data (i.e. series catalog / corpus) into a local pandas dataset for analysis. The corpus info contains the series id / name as well as other meta-info such as frequency / popularity ranking, etc.

# Most functionality will be in the catalog module
from fred_e import catalog
catalog.get_series_meta_all()

Get all series names of a particular frequency

Filters series corpus for a particular frequency (daily/monthly/weekly/annually, etc)

from fred_e import catalog
# Convenience funcs by periodicity / frequency type
catalog.get_series_meta_daily_all()
catalog.get_series_meta_weekly_all()
catalog.get_series_meta_monthly_all()
catalog.get_series_meta_quarterly_all()
catalog.get_series_meta_annual_all()

Get series meta for a selected list of series by id

Retrieve meta information for a selected group of series

from fred_e import catalog

# Note in below we can pass in the columns to retrieve
# or leave blank to retrieve all applicable columns on the table
series_ids = ['ABC', 'XYZ', ... ]
catalog.get_series_meta_by_series_id(series_ids)
catalog.get_series_meta_by_series_id(series_ids, columns=['id', 'title'])

Get all series above a particular popularity rating

from fred_e import catalog
catalog.get_series_meta_min_popularity(popularity=50)  # everything above popularity score of 50

Get series data (time series)

Retrieve all available data (dates) for a given series, given series id. Note that, depending on the SERIES_RELEASE property that is set, we may get multiple values for a given date given revisions to data. This takes multiple series ids. To be efficient, should pass in the names of all of the series you intend to utilize.

from fred_e import catalog
catalog.get_series_data(['series_a', 'series_b']). 

Get series data on watchlist (time series)

Retrieve data for all series that are on the watchlist.

from fred_e import watchlist
watchlist.get_all_watchlist_series_data()

Watchlist Maintenance

The watchlist in FRED-E are all of the series that the FRED-E system will auto-update / keep up to date. Only series that are on the watchlist will be retrieved into FRED-E.

Get all series in watchlist

Get all series that are currently in the watchlist

from fred_e import watchlist
watchlist.get_all_watchlist_series()

Add series to the watchlist

To add new series to the watchlist

from fred_e import watchlist
watchlist.add_to_watchlist(['series_a']) # add to watchlist
watchlist.is_in_watchlist(['series_a']) # checks if series are already in watchlist

Update series in the watchlist

FRED-E will examine all series in the watchlist, filter to a list of update candidates (this is a list of the series on the watchlist that will require an update (depends on periodicity) - note: dailies will always be updated.

from fred_e import updates
updates.update_watchlist_candidate_series() # if series not passed, will update all available candidate series

Updating Series Corpus

FRED-E recursively retrieves all available series categories (starting within the top-level categories). FRED-E will then proceed to retrieve all the available series meta-data within each category. These results are saved to the underlying data storage (i.e. BigQuery or relational database). Note: due to the rate-limiting imposed by the FRED backend, this is a long operation. Additionally, the series corpus will not update the actual data series. Only series added to the watchlist (in the sections above) are allowed.

from fred_e import catalog

# This will build the full category dataset (full category tree) (table: categories)
catalog.build_save_series_meta()

# This will build the series metadata corpus (table: series_meta)
catalog.build_save_series_meta()