Tips for Grid Creators

If {you are convinced about the need for Creating a Grid + need help Getting Started} ;

Then {you have come to the right place!};

This post can give you some useful tips and examples.

Grid submission can be done on the #new-submission channel; requires you to join the Grid Creators server.

1. New Submission

Submission of a grid is as easy as posting a simple chat message, however the hard part is doing all the thinking upfront to make your submission awesome.

A New Submission must include the following: Grid Name, Grid Description, Who is this for? , Why is this important, Sources, Team

a) Picking a Grid Name :

Short and sweet is better than long and wordy.
E.g. a Grid Name : Disease_Treatment is better than treating_diseases_grid or grid-for-disease-treatment

b) Grid Description :

What will this grid contain, stick to the what. There are other sections to describe the who and why.

E.g. Our grid will contain a list of available treatments for each known disease and illness, with extensive information about each disease and treatment option, including medical terminology, potential complications, symptoms, cost, reliability, and availability.

c) Who is it for? :

Call out the different roles clearly.
E.g.
This …
1. People afflicted with life-threatening disease. (Presents every option available in an instant)
2. The poor. (Alternatives or affordable prices for previously unobtainable treatment)
3. The general populace. (Quickly discover the cheapest and most appropriate medicine for their disease)

is better than…this…

This grid is for people afflicted with life-threatening diseases, poor people and general use by anyone.

d) Why is this important? :

Describe the impact it would make. Does your Grid Save Lives, or Make someone’s life incredibly better.

E.g.
Many people either make uninformed decisions about their treatment, or do not get treatment simply because they are unaware of the available treatment options that may be affordable for them. We seek to change this by making information available on every treatment, allowing people to compare options and make more informed treatment and health care decision, ultimately improving quality of life and possibly even saving the lives of people who would otherwise not receive care for a life-threatening disease.

e) Source:

Specify the source categories, adding 1 or 2 examples URLs is sufficient. Do not need to add all the URLs, the place for that would be the Grid Data Model.

Note: All sources must be public and downloadable under

E.g.
The medical information constituting our grid will be from publicly accessible databases and archives established by reputable health organizations, such as the National Institute of Health and the National Library of Medicine.

f) Team:

The power of 3. The ideal team size we have seen work well, is 3 people.

3 people is all you need to create a movement

2. Data Model

a) Defining a Data Model, is the first key step once your Grid is approved by the Community for build out.

Data model for a Grid consists of : the name of each Column in the grid, Column Descriptions for each column, Resources (a tab with source urls for each data Column) and Sample Rows of Data manually collected from each of these sources.

Here is an example of a good grid data model:

Refugee.grid : This is a good example of how to show resources for each column.

Healing.grid : This is a good example of how to show column descriptions and sample data

b) Resource tab: Ensure your Grid has a Resource tab where you list out the references for each column, i.e. the URL from where you are crawling that element.

Robots.txt verification: You must verify that this URL is crawlable.
Data Licence/Terms of Service verification: You must verify that the data can be crawled for public consumption and experimental purposes.

NOTE: If you wish to build a commercial app or monetize your app using any kind of inapp purchases, proper data licensing must be obtained for your data elements from the website or data source.

c) submit the Data Model for review

Data Models are initially created as Google Sheets and the url is submitted to the #model-review channel.

Once your model is approved, a Channel with your grid name will be created and shared with you and your team. A community leader will be assigned to your grid to help you through the Grid Creation process. You can now use this channel to collaborate, ask questions from a leader and complete your Grid.

Now you should be ready with your data!

3. Publishing your Data

Publishing data to BigParser is just a matter of a few clicks!

Simply follow the steps below and you will be all set to start sharing your Grid with others and make the life saving impact with your grid.

a) Add your Grid to BigParser

My Data is on my Hard Drive : You can simply Upload your Data to BigParser

My Data is in Google Drive or Dropbox : Install the Google Drive or Dropbox app from the BigParser App Store (requires login to BigParser). Once installed you can simply search for your Google sheet or file in the drive and sync it to BigParser. If you want the updates to flow in automatically, keep you can switch on Autosync under Drive settings.

My Data is on some URL on the Web : Simply Paste the URL of the File in the Search box at the top in BigParser. This will pull the file from the web to your My Data tab.

b) Make your Grid Public

Once your data is added to BigParser, your can simply click on your File under My Data to Launch your Grid.

To make this grid public:
Click on the Share button, on the top right corner
Select the option Public, under Permissions
Click on the Share button, at the bottom of your window

You can read more here on how to do advanced settings, to hide specific columns when sharing, or include other settings to prevent downloads or copying.

Your Grid is now Public and Connecting the World’s Data

4. Other Tips for Data Wranglers!

Various Pythons Libraries are available for your convenience, you can get them from your Community Leader or via the BigParser API page.

Wrangler.py : A simple python program for crawling data from any website, and write to a local file on your computer.
Auth.py : A simply python program to push data to a Google Sheet, incrementally adding additional rows.
Custom.py : A shell python program that you can use to build your own custom web crawler or wrangler in Python

Have any useful code that you would like to share?

Node, Java or any other language, you can simply start by sharing your GitHub public repo url with #grid-creators on the BigParser Developers server.

Please remember our community is on a mission to Connect the World’s Data, so share your good work, so others can use it to connect more data.

Keep an eye out for our related posts coming soon:

BigParser at HackTJ
BigParser API & Grids Trophy
BigParser API & Grids GrandPrize