The problem: you want to split up a few million U.S. address records into equally-sized chunks that retain spatial hierarchy. You want to do this without anything other than a street address (geocoding is expensive!). Maybe you want to do this as part of a map/reduce process (we certainly do), maybe you want to do some sampling, who knows?
Anyone who's ever used U.S. ZIP codes as a way to subdivide datasets can tell you: 60608 (pop 79,607) is a totally different beast than 05851 (pop 525). They're not census tracts; it's not really appropriate to compare them statistically or thematically.
Our solution - largely the work of platform wizard and Rust enthusiast Eric Kidd - is to bake census data into a tool that does the splitting for you at a level that allows for easy comparison. More specifically:
It provides a deterministic mapping from zip codes to "geochunks" that you can count on remaining stable.
Check out the Jupyter notebook that explains the algorithm in detail, but it works like so:
Install rust first if you don't have it:
curl https://sh.rustup.rs -sSf | sh
. . . then geochunk, using the rust package manager:
cargo install geochunk
Use 1: Indexing
Build a table that assigns every U.S. zipcode to a geochunk that contains 250,000 people:
geochunk export zip2010 250000 > chunks_of_250k_people.csv
Use 2: List processing
Alternately, let's try a pipeline example that uses
geochunk csv: say you want to parallel-process every address in the state of Colorado, and you need equal-size but contiguous slices to do it.
- Get and unpack sample data from the openaddresses project:
wget -c https://s3.amazonaws.com/data.openaddresses.io/runs/283082/us/co/statewide.zip && unzip statewide.zip
- Pipe the full file through geochunk, into slices of about 250,000 people each:
cat us/co/statewide.csv | geochunk csv zip2010 250000 POSTCODE > statewide_chunks_150k.csv
. . . and now you have 2 million addresses, chopped into ~8 equally-sized slices with rough contiguity:
Geochunk works on this scale in 1.38s (Have you heard us evangelizing about Rust yet?), leaving you plenty of time for the real processing.
This tool is serious dogfood for us; it's baked into our ETL system, and we use it to try making a tiny dent in the Modifiable Areal Unit Problem. We hope you'll find it useful too.