This repository contains the source code for an app that synchronizes data with specified content types from sources to storages on behalf of users.
See also the API documentation or the Neotoma project in general.
SSL certificates and the following environment variables are managed by Park Ranger:
SYNC_SERVER_SESSION_SECRET: Secret, non-obvious string used to prevent session tampering (e.g.oc]7kwM)R*UX3&but generate your own)
SYNC_SERVER_DIR: Local path to app base directory (defaults to/var/www/sync-server)SYNC_SERVER_HOST: Host address for the app (defaults to127.0.0.1)SYNC_SERVER_HTTP_PORT: Port through which to run the app with HTTP (defaults9001)SYNC_SERVER_HTTPS_PORT: Port through which to run the app with HTTPS (defaults to9002)SYNC_SERVER_LOGGER_FILE_PATH: File system path where to store log events (e.g./tmp/sync-server.log)SYNC_SERVER_LOGGER_MAILER_LEVEL: Numeric value between 0 and 5 to designate level of errors to email to SYNC_SERVER_MAILER_LOGGER_EMAILSYNC_SERVER_MAILER_LOGGER_EMAIL: Email to which logger errors should be sentSYNC_SERVER_MAILER_RECIPIENT_EMAIL: Email address to which to restrict all email delivery for testing purposesSYNC_SERVER_MAILER_SENDER_EMAIL: Email address used by app to send email (e.g.support@example.com; required to send email)SYNC_SERVER_MONGODB_DATABASE: Name of a MongoDB database (defaults tosync_server)SYNC_SERVER_MONGODB_HOST: Host address for a MongoDB service (defaults to127.0.0.1)SYNC_SERVER_MONGODB_PORT: Port through which to access a MongoDB service (defaults to27017)SYNC_SERVER_NAME: Name used by the app to identity itself with users (defaults toNeotoma)SYNC_SERVER_SENDGRID_API_KEY: API key for SendGrid account for delivering email (required to send email)SYNC_SERVER_WEB_HOST: Host address for the web client intended to communicate with the app exclusively via cross-origin HTTP requests; used to set HTTP access control (CORS) (e.g.http://example.com:9019)
Tests will use Park Ranger to establish environment variables available for the "test" environment after loading those for whatever environment initially indicated upon execution.
Be sure to set any of the above variables to a different value within .env-test if you don't want the tests to use the variables available for the indicated environment.
For example, set a different SYNC_SERVER_MONGODB_DATABASE to prevent your development database from getting reset every time you run tests, and SYNC_SERVER_SENDGRID_API_KEY to null to prevent email delivery.
Once the environment is ready per above, and Node.js with NPM is installed, simply run npm install to install dependencies in the node_modules directory and npm start to fire the server up.
Deployment scripts are available through Hoist. The following are also supported:
npm run dev: Runs the app and automatically reloads it when code changes are made during developmentnpm run test: Runs all tests locallynpm run data-seed: Runs the data seed script
This script loads any data objects available as JSON files in the data directory into the Mongo database after deleting existing data in the database that belong to those objects' collections.
For example, if you create the file data/users.json and place the following JSON conformant to the JSONAPI specification, the script will clear existing users and add this new one as an administrator:
{
"data": [{
"type": "users",
"attributes": {
"admin": true,
"email": "ghopper@example.com",
"name": "Grace Hopper"
}
}]
}
This script is intended to be run only in development environments to help seed data without concern for the possible side effects of deleting existing data.
It's also especially helpful when used to seed content type, source and storage records, which are fundamental to the application.
For example, the following could be placed into data/storages.json to populate storage records for Dropbox and Google Drive:
{
"data": [{
"type": "storages",
"attributes": {
"apiVersion": "2",
"name": "Dropbox",
"host": "content.dropboxapi.com",
"passportStrategy": "passport-dropbox-oauth2",
"clientId": "[YOUR DROPBOX DEVELOPER APP'S CLIENT ID]",
"clientSecret": "[YOUR DROPBOX DEVELOPER APP'S CLIENT SECRET]",
"itemPutUrlTemplate": "https://${host}/2/files/upload",
"logoGlyphPath": "/images/logos/dropbox-glyph.svg",
"itemStorageEnabled": true,
"slug": "dropbox"
}
}, {
"type": "storages",
"attributes": {
"name": "Google Drive",
"logoGlyphPath": "/images/logos/google-drive-glyph.svg"
}
}]
}
The following could be placed into data/sources.json to populate source records for Foursquare and Facebook:
{
"type": "sources",
"attributes": {
"name": "Foursquare",
"itemStorageEnabled": true,
"logoGlyphPath": "/images/logos/foursquare-glyph.svg",
"host": "api.foursquare.com",
"apiVersion": "20180202",
"itemsLimit": 100,
"clientId": "[YOUR FOURSQUARE DEVELOPER APP'S CLIENT ID]",
"clientSecret": "[YOUR FOURSQUARE DEVELOPER APP'S CLIENT SECRET]",
"passportStrategy": "passport-foursquare",
"itemsGetUrlTemplate": "https://${host}/v2/users/self/${contentTypePluralLowercaseName}?v=${apiVersion}&oauth_token=${accessToken}&limit=${limit}&offset=${offset}",
"itemDataObjectsFromPagePathTemplate": "response.${contentTypePluralLowercaseName}.items",
"totalItemsAvailableFromPagePathTemplate": "response.${contentTypePluralLowercaseName}.count",
"slug": "foursquare"
},
"relationships": {
"contentTypes": {
"data": [{
"type": "contentTypes",
"attributes": {
"name": "Check-in"
}
}, {
"type": "contentTypes",
"attributes": {
"name": "Friend"
}
}, {
"type": "contentTypes",
"attributes": {
"name": "Tip"
}
}]
}
}
}, {
"type": "sources",
"attributes": {
"apiVersion": "v2.8",
"authScope": ["user_posts","email"],
"name": "Facebook",
"itemStorageEnabled": true,
"logoGlyphPath": "/images/logos/facebook-glyph.svg",
"host": "graph.facebook.com",
"clientId": "[YOUR FACEBOOK DEVELOPER APP'S CLIENT ID]",
"clientSecret": "[YOUR FACEBOOK DEVELOPER APP'S CLIENT SECRET]",
"passportStrategy": "passport-facebook",
"itemsGetUrlTemplate": "https://${host}/${apiVersion}/me/${contentTypePluralLowercaseName}?access_token=${accessToken}",
"totalItemsAvailableFromPagePathTemplate": "summary.total_count",
"slug": "facebook"
},
"relationships": {
"contentTypes": {
"data": [{
"type": "contentTypes",
"attributes": {
"name": "Friend"
}
}, {
"type": "contentTypes",
"attributes": {
"name": "Photo"
}
}, {
"type": "contentTypes",
"attributes": {
"name": "Post"
}
}]
}
}
}
And finally, the following could be placed into data/contentTypes.json to populate the content type records associated with those sources:
{
"data": [{
"type": "contentTypes",
"attributes": {
"name": "Check-in",
"dataTemplate": {
"place-state": "venue.location.state",
"place-postal": "venue.location.postalCode",
"place-name": "venue.name",
"place-longitude": "venue.location.lng",
"place-latitude": "venue.location.lat",
"place-country-code": "venue.location.cc",
"place-country": "venue.location.country",
"place-city": "venue.location.city",
"place-category": "venue.categories[0].pluralName",
"place-address": "venue.location.address",
"likes-count": "likes.count",
"foursquare-venue-id": "venue.id"
}
}
}, {
"type": "contentTypes",
"attributes": {
"name": "Friend"
}
}, {
"type": "contentTypes",
"attributes": {
"name": "Photo"
}
}, {
"type": "contentTypes",
"attributes": {
"name": "Post"
}
}, {
"type": "contentTypes",
"attributes": {
"name": "Tip"
}
}]
}
The dataTemplate attribute for content types is used to indicate how you'd like the data from a source to get formatted before copied to storage.
For example, using the dataTemplate value above, the following raw data pulled from the Foursquare API for a check-in item:
{
"comments": {
"count": 0
},
"createdAt": 1517560416,
"id": "5a742260ea1e440aa76d01a4",
"isMayor": true,
"like": false,
"likes": {
"count": 0,
"groups": []
},
"photos": {
"count": 0,
"items": []
},
"posts": {
"count": 0,
"textCount": 0
},
"source": {
"name": "Swarm for iOS",
"url": "https://www.swarmapp.com"
},
"timeZoneOffset": 60,
"type": "checkin",
"venue": {
"beenHere": {
"lastCheckinExpiredAt": 0
},
"categories": [
{
"icon": {
"prefix": "https://ss3.4sqi.net/img/categories_v2/building/office_coworkingspace_",
"suffix": ".png"
},
"id": "4bf58dd8d48988d174941735",
"name": "Coworking Space",
"pluralName": "Coworking Spaces",
"primary": true,
"shortName": "Coworking Space"
}
],
"contact": {
"facebook": "195202760509656",
"facebookName": "MOB",
"facebookUsername": "MOB.BCN",
"formattedPhone": "936 67 41 65",
"phone": "936674165",
"twitter": "mob_bcn"
},
"id": "4ed4fe31f5b975def54c94dd",
"location": {
"address": "C. Bailèn, 11",
"cc": "ES",
"city": "Barcelona",
"country": "Spain",
"crossStreet": "C. d'Ausiàs Marc",
"formattedAddress": [
"C. Bailèn, 11 (C. d'Ausiàs Marc)",
"08010 Barcelona Catalonia"
],
"labeledLatLngs": [
{
"label": "display",
"lat": 41.39174253552806,
"lng": 2.177135786885419
}
],
"lat": 41.39174253552806,
"lng": 2.177135786885419,
"postalCode": "08010",
"state": "Catalonia"
},
"name": "MOB - Makers of Barcelona",
"stats": {
"checkinsCount": 3025,
"tipCount": 14,
"usersCount": 698
},
"url": "http://www.mob-barcelona.com",
"venueRatingBlacklisted": true,
"verified": false
}
}
...would get formatting into the following data for storage:
{
"id": "foursquare-5a742260ea1e440aa76d01a4",
"type": "checkins",
"attributes": {
"place-state": "Catalonia",
"place-postal": "08010",
"place-name": "MOB - Makers of Barcelona",
"place-longitude": 2.177135786885419,
"place-latitude": 41.39174253552806,
"place-country-code": "ES",
"place-country": "Spain",
"place-city": "Barcelona",
"place-category": "Coworking Space",
"place-address": "C. Bailèn, 11",
"likes-count": 0,
"foursquare-venue-id": "4ed4fe31f5b975def54c94dd",
"created-at": "2018-02-02T15:14:08+01:00"
}
}