How Shovels combines data streams to improve their product

Harvest Season at Shovels

Company
Alex Brown

Alex Brown

Technical Support & Writer

Fall is here. For those of us with actual seasons (cough cough), we only need to look outside 🍁. For Californians, just as everyone else that relies on agriculture, a good harvest is absolutely essential.

Here at Shovels, we have done a ton of work sharpening our plows, oiling our machinery, and sowing the seeds of a data harvest which will keep us fed through the winter. If you’re curious about some of that prep work, read how we partnered with Prolific to supercharge our LLM-training.

At the risk of over-extending this farming metaphor, the key to a good harvest is crop diversity. And at Shovels, we do not rely on a single data source to deliver our insights. Permit data from local jurisdictions will always be our bread and butter, but that alone is not enough.

Hybridization

Just as cross-breeding species of plants can make them hardier for a given climate, or make them more insect- or blight-resistant, hybridized data is combined in such a way that it might not exist anywhere else. With AI-cleaning, clever columns, and SQL wizardry, we can join previously disjointed data into a new, multi-dimensional data models. Unique data is hard to come by, but the rewards far outweigh the effort.

What does that mean? It means that building permit data can be cross-referenced against property and tax assessor data. That contractor history and performance can be upgraded into distinct profiles with contact info and company websites. It means that everything in the Shovels dataset can be paired with residential demographics, which is a (the?) golden goose for behavior models and trend analysis.

Hybridization improves a product by keeping the best pieces of it’s individual components. Use cases vary, and needs may evolve. Imagine if you were stuck with a single SQL join for an entire season, like the farmers with their new crop hybrids? Luckily, our platform provides a national market-sized sandbox to play in. Evaluate and recombine the context-rich data to your heart’s content to see what new insights you can glean.

Enrichment

Even with a perfect harvest, where quality is unmatched and efficiency is optimized, more can always be added to improve the final product. Try baking something without eggs, dairy, or sugar, and let us know how it goes (and tastes — sorry vegans!)

Data and dough are similar in this regard, in that they benefit enormously from enrichment.

At Shovels, a key process for hybridizing our data is simple enrichment: take the messy and half-filled data we’re given by the local jurisdictions, and fill in the gaps ourselves. Someone really needs to pass a grant for additional federal funding to increase permit metadata requirements, it’d make our lives so much easier.

Needless to say, simple enrichment is anything but. Simple means focused, in scope. It’s easy to get carried away, especially with enrichment vendors promising everything from phone numbers to timestamped Alexa commands. We make sure that our enrichment is ethical (ie, publicly-available) and relevant (filling a gap). To that end, we’re nearly ready to reap the unique data we’ve sown. When it’s here, you’ll be the first to know.

Future Harvests

Speaking of, we wanted to share a little sneak-peak at some things coming up soon.

All of this new data means new API endpoints. We’re releasing V2.0 of our API, plus a brand new web app, Shovels Online, this week(!!) on October 24th. If you’ve been using the beta thus far, thank you for your patience — this new version is so much better.

(If you happen to be in the Bay Area, we’re having a launch party that very evening: reach out to us directly and we’ll add you to the guest list.)

The entire Shovels team is hard at work preparing for the release of both products, and we can’t wait to share it with you. There will be plenty of fanfare, including an in-depth blog post outlining the key improvements of API v2.0, plus some other marketing material for the new Shovels Online.

Dinner Is Served, Dig In!

Whether you use a spoon or a shovel, we’ve got all the intel you can eat (or analyze).

If you need verified and activity-based lead generation, we’ve got you covered. If you want to do some enrichment of your own by directly integrating your product with our data, our API will serve you well. If you need years of localized or nationwide property and construction data to feed your models, just tell us your preferred delivery method and we’ll pipe it over.

Get in touch with us directly to be the first to access the new Shovels Online and fine-tune a custom plan that matches your needs and budget.

For those of you that have been on this journey with us for a while, we can’t thank you enough (last call for the launch party!). For those that recently joined (welcome!), you got here just in time 🚀

Happy Digging!