In my last post, I discussed SNAP data. In this post, we will cover the process of harvesting USDA SNAP CSV data. We will store data in MongoDB and augment it to support fast geo queries. These geo queries will be used to locate nearby SNAP stores from a particular location.
We will exploit two Cloud-hosted services to reduce the amount of effort it takes to create an environment for working with the data. For our purposes, the free plan of both services will be adequate.
Cloud9 is a Cloud-hosted IDE. We will use the shell environment it provides to perform the tasks necessary to download SNAP data and import it into a mongo database.
MongoLab provides a Cloud-hosted MongoDB service. We will use it to import and serve SNAP data to support our queries.
Create a new workspace
Name your new workspace
Call it “snapdb” and select the “Custom” option (which gives you a barebones workspace with just a README.md file). Cloud9 will take a moment while it provisions your new cloud-hosted development workspace.
Open your new workspace
Click the “Start Editing” button. Your workspace will open in a new window (or tab) and initialize.
Prepare your workspace
Right-click on README.md and choose “Delete” from the context menu to get rid of it.
Right-click on the snapdata workspace folder and create a new file. Call it “notes.md”. Double-click to open the file. We’ll use the file to save notes about our mongo database that we’ll create in the next step.
Create a new database
Go to MongoLab and sign up for an account. Once you’re signed in, create a new database.
Create a new database - configuration options
Choose a free sandbox database on Amazon; name it “snapdb”.
Open the database page
Add a database user
To access the database from a client, we need to create a database user. The database user credentials will be used when connecting to snapdb. Click the link that says “Click here”.
Set database user credentials
Choose “apiuser” for the username and “snapdb” for the password.
Save the database information
You’re done with database creation. Save the connection information and database credentials in your Cloud9 notes.
We now have everything in place to get SNAP data and loaded into a mongo database. In our next post, we’ll walk through the steps of downloading, importing, transforming, and querying SNAP data interactively from the Cloud9 shell.