Startup hacks and engineering miracles from your exhausted friends at Faraday

ActiveRecord ↔︎ Sequel raw sql cheat sheet

Seamus Abshere on

UPDATED with fixes from @jeremyevans

Going back and forth between ActiveRecord and Sequel can be very confusing... especially because the Sequel documentation assumes that you use "datasets" instead of raw sql.

What ActiveRecord Sequel
One value ActiveRecord::Base.connection.select_value(sql) DB[sql].get
One row ActiveRecord::Base.connection.select_row(sql) DB[sql].first
One column across many rows ActiveRecord::Base.connection.select_values(sql) DB[sql].select_map
Many rows as hashes ActiveRecord::Base.connection.select_all(sql) DB[sql].all
Quote value ActiveRecord::Base.connection.quote(value) DB.literal(value)
Quote identifier (e.g. table name, column name) ActiveRecord::Base.connection.quote_ident(ident) DB.quote_identifier(ident)

Please email us if you think of any other ones!

Customer Success, cookies, and diminishing returns: don't announce features all at once

Thomas Bryenton on

Chocolate chip cookies

Right this moment, how much would you pay for a cookie at a coffeeshop? Let's say the amount is $3.

picture_of_cookies

(I could really go for one, so my personal estimate might be higher. I mean, check out how good that looks.)

How much would you be open to paying the coffeeshop for a subsequent cookie? A third cookie? A fifth?

Probably it would be steadily less than $3. Why? Later cookies have lower value.

Well . . . for now at least. If you come back to the coffeeshop tomorrow, you could be ready for more!

Customer Success

A related principle applies in Customer Success: people love hearing when you ship a feature they wanted. But tell them about a third feature or a fifth? Their eyes will glaze right over.

(Speaking of glaze, if cookies aren't your thing, how about these glazed scones?)

scone_picture

Sure, sometimes an engineering team works in bursts. Progress can come all at once. Then, the temptation for Customer Success is to "catch up" by reporting things to clients exactly when they happen.

This doesn't mean that as a Customer Success team you can't be strategic about who to announce features to, and when.

So:

  • Notice each client's rhythm of hunger for new features
  • Keep a stash of "feature snacks" to give clients
  • Treat yourself to a literal or figurative cookie (or scone)

How to aggregate JSONB in PostgreSQL 9.5+

Seamus Abshere on

This is part of our series on PostgreSQL and things that are obvious once you see them. It's a 2017 update to our 2016 article How to merge JSON fields in Postgres Nice!

Update We used to call this jsonb_collect, but then we realized it was very similar to json_object_agg(name, value)... except that there is no version of that function that just takes an expression. So, copying jsonb_agg(expression), we give you...

How do you aggregate (aka merge aka combine aka collect) JSONB fields in Postgres 9.5+? You define a new aggregate jsonb_object_agg(expression):

CREATE AGGREGATE jsonb_object_agg(jsonb) (  
  SFUNC = 'jsonb_concat',
  STYPE = jsonb,
  INITCOND = '{}'
);

Here's how you use it:

# select * from greetings;
         data
-----------------------
 {"es":"Saludos"}
 {"en":"Hello"}
 {"ru":"Здравствуйте"}
(3 rows)

# select jsonb_object_agg(data) from greetings;
                        jsonb_object_agg
-------------------------------------------------------------
 { "es" : "Saludos", "en" : "Hello", "ru" : "Здравствуйте" }
(1 row)

If you're curious about the aggregate we just added, note that jsonb_concat(jsonb, jsonb) is the function backing || that was introduced in Postgres 9.5.

It's just that simple!

Be not afraid of ZCTAs

Bill Morris on

This post is part of our practical cartography series.

Most American geographers will note that - as much as we'd like it to be otherwise - ZIP Codes are not polygons. Rather, they're constantly-changing lines used by the USPS to coordinate delivery in an efficient network. Many of us polygon-happy mappers use ZIP Code Tabulation Areas (ZCTAs) instead; these are provided by the US Census as a reasonable open data alternative to ZIPs. They're particularly nice for thematic mapping (though their shortcomings have also been well-documented):

map

But why use ZCTAs if they can never be reconciled with their ground-truth ZIP cousins?

Because the difference is small.

Faraday has address and location records for every household in the country, and it was straightforward to check for disagreement between the ZIP Code of each physical address and the ZCTA polygon that contains it.

Here are the results, broken down by state

The national error rate of ZCTAs is 1.4%. That might be too high for some use cases, but perfectly acceptable for others. There's some regional variation, too: you're usually safe to use ZCTAs in Hawaii and Maine, but might want to exercise caution in Oregon and Utah.

Happy mapping!

How to finally use headless Chrome to power your automated tests

Derek Kastner on

Google Chrome version 59 will ship with the headless
option
. This means you can test your web applications using chrome without needing xvfb. One problem: the latest chromedriver (version 2.29) doesn't support versions of Chrome higher than 58.

The solution is to build the latest chromedriver that supports the latest chrome/chromium. Google does not make nightly builds of chromedriver public, you have to download the chromium source and build
chromedriver yourself.

How all the pieces work together

Cucumber uses the capybara gem to send commands to selenium-webdriver. Selenium-webdriver in turn starts up your local copy of chromedriver, which then starts up chrome and controls the browser through a special debug port.

To get the latest chromium, I used a tool on GitHub that downloads the latest snapshot compiled for Linux.

To get the latest chromedriver, I followed the
build instructions.

Configuring the tests

When configuring capybara, you need to tell selenium-webdriver the path to your custom chromium binary and send the --headless flag, along with other flags
you'll likely need in a CI build node environment.

For running in docker:

Capybara.register_driver :headless_chromium do |app|  
  caps = Selenium::WebDriver::Remote::Capabilities.chrome(                       
    "chromeOptions" => {                                                         
      'binary' => "/chromium-latest-linux/466395/chrome-linux/chrome",           
      'args' => %w{headless no-sandbox disable-gpu}                              
    }                                                                            
  )                                                                              
  driver = Capybara::Selenium::Driver.new(                                       
    app,                                                                         
    browser: :chrome,                                                            
    desired_capabilities: caps                                                   
  )                                                                              
end                                                                              

Capybara.default_driver = :headless_chromium  

Now, capybara will drive a headless chromium!

Two magic words for greater independence and communication

Thomas Bryenton on

What if there were two words you could add to any email to get your team to weigh in quickly?

There are: DEFAULT DO

How default-do works

  1. Write up a final version of what you'll be doing (your "default do")
  2. Tell teammates you're about to do this thing
  3. Do the thing

Just default do it

I use default-do every day to keep Faraday moving:

  • Emails
  • Code
  • Mockups
  • Decisions
  • Policies
  • Blog posts!

It doesn't matter what it is: if I'm confident it's the right thing to do, I'll tell my team I'm about to do it, pause, then just do it.

Don't wear it out. If you don't really want feedback, don't ask, and if you need it, your default isn't "do," it's "don't."

Use headless chromium with capybara and selenium webdriver - today!

Seamus Abshere on

UPDATE: we have a new version of this post out that resolves some of the gotchas below.

Here's a Rubyist's magic incantation to run headless chromium with selenium-webdriver and capybara: (it will be similar in other languages)

require 'selenium-webdriver'

Capybara.register_driver :headless_chromium do |app|  
  caps = Selenium::WebDriver::Remote::Capabilities.chrome(
    "chromeOptions" => {
      'binary' => '/home/myuser/chrome-linux-440004/chrome',
      'args' => ['headless', 'disable-gpu']
    }
  )
  driver = Capybara::Selenium::Driver.new(
    app,
    browser: :chrome,
    desired_capabilities: caps
  )
end  

Hints

  1. You need a chromium (chrome) binary that reports version 57 (version 59 is too new). For example, snapshot 440004 - just download and unzip.
  2. You need a recent chromedriver binary with support for headless chrome. For example, snapshot 460342 - just download and unzip.
  3. If you get the error unrecognized chrome version, then see (1) above - you probably have a too-recent chromium.

Thanks to @dkastner!

Antipattern: Using Ruby's Hash#[]

Seamus Abshere on

This is part of our antipatterns series. Ouch! Updated for 2017!

Ask yourself why you're using Hash#[]. It is is a great way to introduce silent bugs into your app.

Use Hash#fetch if you expect the value to exist

That way you get sensible error messages.

#> params = {}

#> params.fetch('really').fetch('important')
KeyError: key not found: "really"  

Use Hash#dig if you don't care

Because you don't get idiotic, non-semantic NoMethodError: undefined method '[]' for nil:NilClass errors.

#> params.dig('really', 'important')
=> nil

Avoid Hash#[] because... just... why?

#> params['really']['important']
NoMethodError: undefined method `[]' for nil:NilClass  

Special case: ENV

The Twelve-Factor App has us all using environment variables. But most of us default to ENV#[] to look stuff up... even if it's critical. Bad idea! Use fetch!

#> ENV['REALLY_IMPORTANT'] == 'thing'
=> false # well i hope you didn't need that

#> ENV.fetch('REALLY_IMPORTANT') == 'thing'
KeyError: key not found: "REALLY_IMPORTANT"  

Plancha: how to flatten multi-sheet excel workbooks

Bill Morris on

This is part of our series on data science because it belongs in your toolchain.

If you work with data long enough - actually scratch that; if you work with data for more than a week - you'll run into the dreaded multi sheet (or tab) excel workbook. Sometimes the sheets are unrelated, but other times they should really all be stacked together in the same table, ideally in a more-interoperable format than .xlsx:

in

Enter plancha. Named for the trusty tortilla press, we built this simple CLI tool to flatten multi-sheet excel files, resolve header mismatches, and return a pipeline-friendly csv, like this:

out

Install

This is a node.js tool, so use npm:

npm install plancha -g

Usage

Just feed it an input .xlsx file:

plancha -i myfile.xlsx


Happy data-pressing!

scrubcsv: now with null value removal

Seamus Abshere on

This is part of our series on data science because it belongs in your toolchain. Happy Null Removal!

The latest version of scrubcsv has built-in null value removal:

$ cat a.csv
name,breed,age  
jerry,beagle,n/a  
tater,null,1

$ scrubcsv -n 'null|n/a' a.csv
name,breed,age  
jerry,beagle,  
tater,,1  

See how null and n/a went away?

Get the latest version with

$ cargo install scrubcsv -f