Add process which produces a JSON cache of all public feeds #67
Description
Sites like environmentaldata.org suffer from painfully slow load times as they try to load all of ESDR's public feeds. It might be nice to have a JSON cache of all public feeds which gets updated regularly (every 1-5 minutes?) and contains some essential, but minimal info about each feed.
Perhaps the following:
- id
- name
- latitude
- longitude
- lastUpload / maxTimeSecs
- exposure?
A query like this is a decent start:
select productId,
id,
name,
latitude,
longitude,
UNIX_TIMESTAMP(lastUpload) as lastUploadSecs,
maxTimeSecs,
deviceId
from Feeds
where isPublic = 1
order by productId, deviceId, id desc;Ideas to consider:
- Store the JSON under ESDR's public directory, maybe in some subdirectory denoting it as a cache.
- Multiple versions, with differing amounts of info.
- Abbreviated field names in the interest of file size OR using some more compact format such as an array of arrays.
- Group by product ID?
- Sort by productId asc, deviceId asc, feedId desc...so that the most recent feed for a device comes first?
- Also generate separate JSON files per product?
Possible JSON format:
{
"version" : 1,
"fields" : ["id", "name", "latitude", "longitude", "lastUploadSecs", "maxTimeSecs", "deviceId"],
"feedsByProductId" : {
"1" : [
[26087, "West Mifflin ACHD", 40.363144, -79.864837, 1576762626, 1576686600, 26017],
[59665, "Pittsburgh ACHD", 40.4656, -79.9611, 1648222891, 1648218600, 56652]
],
"8" : [
[4268, "CREATE Lab Test", 40.44382296127876, -79.94647309184074, 1635191877, 1635189738.36, 4260],
[4277, "Red Room", 40.34107763959465, -80.069620013237, 1484140498, 1484084287, 4265]
],
"9" : [
[4245, "Wifi Speck 6", 40.443738, -79.946481, 1422565308, 1422565433, 4230],
[4313, "Outdoors", 40.50156314996969, -80.06125688552856, 1432395167, 1431359910, 4291]
]
}
}