Startup hacks and engineering miracles from your exhausted friends at Faraday

3 reasons Citus is the best (non-Heroku) Postgres host available today

Seamus Abshere on

Citus Cloud is the hosted Postgres that our app uses. We tried EnterpriseDB Postgres Plus Cloud and couldn't even get going; we used Amazon RDS for a year but migrated away about 6 months ago. Heroku Postgres wasn't bare-metal enough for us.

1. Log directly to Papertrail

Screenshot of logging setup It's not a world-shattering invention, but features like sending your Postgres logs to a syslog host is very useful for debugging.

Citus takes features requests and actually implements them (this was one of mine).

RDS has a weird home-grown in-browser log viewer which is useless without things like rds-pgbadger.

2. citus extension preinstalled

You know that feeling where you're so happy you can use SQL, but you worry just a bit 🙏🏽 about your horizontal scaling story? Citus is the company behind the citus extension to Postgres:

Citus parallelizes incoming queries by breaking it into multiple fragment queries which run on the worker shards in parallel [...] The workers are simply running extended PostgreSQL servers and they apply PostgreSQL’s standard planning and execution logic to run these fragment SQL queries. Therefore, any optimization that helps PostgreSQL also helps Citus.

(from Querying Distributed Tables)

Speaking of extensions, EnterpriseDB didn't even come with PostGIS the last time we checked; to get it you needed a Windows-based (!!) "StackBuilder" tool. Support was based in India and used an ancient Salesforce interface. You get the idea.

3. Everything is negotiable

Citus gave me a great price to move over from RDS.

You're not juggling Amazon on-demand and reserved instances, or forgetting your snapshot settings and paying $2000/mo for obsolete EBS snapshot storage (did that!), or being a tiny fish in the massive AWS pond.

You're not spending a month of negotiations on the phone with EnterpriseDB "execs," paying a huge up-front fee, and then seeing the project fail utterly when your top devops engineer couldn't install a basic Postgres extension.

This is a company with people like Craig Kerstiens (helped build Heroku Postgres) on the other end of the phone, live support chat, and a real value prop.

Saved by the compiler: Parallelizing a loop with Rust and rayon

Eric Kidd on

This post is part of our rust series.

The Rust compiler just saved me from a nasty threading bug. I was working on cage (our open source development tool for Docker apps with lots of microservices), and I decided to parallelize the routine that transformed docker-compose.yml files. This was mostly an excuse to check out the awesome rayon library, but it turned into a great example of what real-world Rust development is like.

The original routine looked something like this:

/// Process our pods, flattening and transforming them using our
/// plugins, and output them to the specified directory.
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {  
    // Output each pod.
    for pod in &self.pods {
        // Don't export pods which aren't enabled.
        if !pod.enabled_in(&self.current_target) {

        // Figure out where to put our pod.
        // ...

        // Combine overrides, make it standalone, tweak as needed, and
        // output.
        let mut file = try!(pod.merged_file(&self.current_target));
        let ctx = plugins::Context::new(self, pod);
        try!(self.plugins().transform(op, &ctx, &mut file));

To convert this to a parallel loop, I started by changing:

for pod in &self.pods {  


self.pods.par_iter().map(|pod| -> Result<()> {  

Here, Result<()> means "this closure might return an error, or it might return an empty tuple, basically void." (For more information on Rust error handling, check out the Rust book.) But it couldn't be that easy, could it?

Nope. The Rust compiler showed me the following error message. It's long, but I'll translate:

error[E0277]: the trait bound `plugins::PluginTransform + 'static: std::marker::Sync` is not satisfied  
   --> src/
397 |         self.pods.par_iter().map(|pod| -> Result<()> {  
    |                              ^^^ trait `plugins::PluginTransform + 'static: std::marker::Sync` not satisfied
    = note: `plugins::PluginTransform + 'static` cannot be shared between threads safely
    = note: required because it appears within the type `Box<plugins::PluginTransform + 'static>`
    = note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `alloc::raw_vec::RawVec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `std::vec::Vec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `plugins::Manager`
    = note: required because it appears within the type `std::option::Option<plugins::Manager>`
    = note: required because it appears within the type `project::Project`
    = note: required because it appears within the type `&project::Project`
    = note: required because it appears within the type `&&project::Project`
    = note: required because it appears within the type `[closure@src/ 424:10 op:&plugins::Operation, self:&&project::Project, export_dir:&&std::path::Path]`

The key bit to take away here is that "Sync is not satisfied" for the trait plugins::PluginTransform. In Rust, Sync is a special trait that tells the compiler that it's safe to share an object between two threads. That whole long list of "notes" afterwards looks intimidating, but it's really just telling me that my PluginTransform objects live inside my plugins::Manager type, which in turn lives inside my Project. The Rust compiler is verbose, but it tries to be helpful.

A bit of investigation reveals that PluginTransform inherits from the Plugin trait:

pub trait PluginTransform: Plugin {  

So we can fix this problem by changing the declaration of Plugin from:

pub trait Plugin {  


pub trait Plugin: Sync {  

This says, "All types implementing Plugin must also allow me to access them from multiple threads." With that fixed, I re-run cargo test and get a similar message:

error[E0277]: the trait bound `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` is not satisfied  
   --> src/plugins/transform/
230 | impl plugins::Plugin for Plugin {  
    |      ^^^^^^^^^^^^^^^ trait `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` not satisfied

This time, I change:

trait GenerateToken: Debug {  


trait GenerateToken: Debug + Sync {  

This says, "All types which implement GenerateToken must also implement Debug (so I can print them), as well as Sync (so I can share them). We're making progress!

Re-running cargo test, however, reveals the actual bug, and it would have been a nightmare to debug:

error[E0277]: the trait bound `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` is not satisfied  
   --> src/plugins/transform/
124 | impl GenerateToken for MockVault {  
    |      ^^^^^^^^^^^^^ trait `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` not satisfied

Uh-oh. In our test harness, we have a type MockVault, which contains an Rc<RefCell<_>>:

type MockVaultCalls = Rc<RefCell<Vec<(String, Vec<String>, VaultDuration)>>>;

/// A fake interface to vault for testing purposes.
struct MockVault {  
    /// The tokens we were asked to generate.  We store these in a RefCell
    /// so that we can have "interior" mutability, because we don't want
    /// `generate_token` to be `&mut self` in the general case.
    calls: MockVaultCalls,

We use the MockVault to simulate a connection to Hashicorp's Vault, a secure central storage for passwords and other secrets, which issues time-limited credentials. And when we test our Vault code, we use the calls member here to record all the requests that we would have made to Vault.

The type Rc<RefCell<_>> is a hack. The GenerateToken API assumes that our token-generator is a read-only object. This is good, because we want to access it from multiple threads! But in the test code, we need to create some "interior" mutable state. Basically, we ask Rust to replace compile-time borrow checks with run-time borrow checks. (For more details, see the Rust book.) But Rc<RefCell<_>> is a lightweight mechanism designed for single-threaded code.

The fix is to replace Rc<RefCell<_>> with Arc<RwLock<_>>, which is fully thread-safe:

type MockVaultCalls = Arc<RwLock<Vec<(String, Vec<String>, VaultDuration)>>>;  

Once we make this change, Rust reminds us to change all the code that accesses calls, too:

error: no method named `borrow_mut` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope  
   --> src/plugins/transform/
132 |         self.calls.borrow_mut().push((display_name.to_owned(), policies, ttl));  
    |                    ^^^^^^^^^^
error: no method named `borrow` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope  
   --> src/plugins/transform/
368 |     let calls = calls.borrow();  
    |                       ^^^^^^

Phew. If we hadn't caught that Rc<RefCell<_>>, our test suites might have randomly segfaulted a couple of times a week, and it would have taken us weeks to track it down. I don't even think our current test suites could trigger this bug, but I bet a future version would have been able to, leaving a nasty surprise for us someday. The Rust compiler dug down through layers of data structures and found the one bit that wasn't thread safe.

Wrapping it up

Rust also complains about that fact that we're calling continue from inside a closure:

error[E0267]: `continue` inside of a closure  
   --> src/
402 |                 continue;  
    |                 ^^^^^^^^ cannot break inside of a closure

We can fix this by moving this code:

// Don't export pods which aren't enabled.
if !pod.enabled_in(&self.current_target) {  

...into a filter call on our parallel iterator:

    // Don't export pods which aren't enabled.
    .filter(|pod| pod.enabled_in(&self.current_target))
    // Process each pod in parallel.
    .map(|pod| -> Result<()> {

The final problem is that we now have a parallel computation that returns a bunch of Result<()> objects, and we need to boil them down to a single Result<()> like we had before. If we have a bunch of error messages, I'm happy to just pick one. No need to overwhelm the user.

I spoke to Josh Stone, and he helped me come up with the following:

    .map(|pod| -> Result<()> {
        // ...
    // If more than one parallel branch fails, just return one error.
    .reduce_with(|result1, result2| result1.and(result2))

This says, "Given any two results named result1 and result2, take result1 if it's an error, otherwise take result2." It's basically a short-circuit && operator, but for Result values instead of booleans. I've filed an issue suggesting that rayon should provide those last two lines as a built-in function.

And yes, that's a genuine parallel map-reduce in Rust!

So here's our final loop, using work-stealing parallelism:

fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {  
    // Output each pod.  This isn't especially slow (except maybe the
    // Vault plugin), but parallelizing things is easy.

        // Don't export pods which aren't enabled.
        .filter(|pod| pod.enabled_in(&self.current_target))

        // Process each pod in parallel.
        .map(|pod| -> Result<()> {
            // Figure out where to put our pod.
            // ...

            // Combine overrides, make it standalone, tweak as needed, and
            // output.
            let mut file = try!(pod.merged_file(&self.current_target));
            let ctx = plugins::Context::new(self, pod);
            try!(self.plugins().transform(op, &ctx, &mut file));

        // If more than one parallel branch fails, just return one error.
        .reduce_with(|result1, result2| result1.and(result2))

This was pretty painless! I have a lot more things to parallelize, of course. But this is what day-to-day Rust development is like: I have to do a bit of extra work to satisfy the compiler (which mostly becomes a reflex). But in turn, the compiler ferrets out all kinds of subtle concurrency errors and generally watches my back. It's an interesting tradeoff, and overall I like it.

Announcing Cage: Develop and deploy complex Docker applications

Eric Kidd on


This post is part of our devops series.

At Faraday, we rely heavily on microservices to analyze data, generate maps and make predictions. All our microservices run inside Docker containers, which makes it easy to run our code either locally or in the cloud.

But if you've ever worked on a large project with lots of services, you're aware that local development can be difficult:

  • You need some way to run a complex set of microservices locally on your laptop.
  • You need to be able access the source code for multiple projects easily, edit it locally, and see the changes immediately.
  • You need to remember how to run tests for services written in multiple languages by multiple teams.

Originally, we used docker-compose to work on our services locally. It offered some great features, but it didn't quite do enough:

  • We very quickly wound up needing multiple docker-compose.yml files. For example, you're encouraged to use them to define "task" containers. They're also helpful if you need to selectively restart individual portions of your application or if you use multiple load balancers.
  • docker-compose.yml files often contain a fair bit of duplication. There are ways to reduce this using extends: and env_file:, but it still requires manual maintenance.
  • docker-compose provides limited support for working with multiple source repositories.

But what if there were a tool that made complex microservice projects as simple and easy as a Rails web application? We decided to build some tools and see how simple we could make it.

Introducing Cage

Cage is an open source tool that wraps around docker-compose, and it tries to make local development as easy as possible.

We can get started by using cage to generate a new project:

cage new myproj  
cd myproj  

Next, we can start up our database server and create a new database. This part should be familiar to docker-compose users:

cage up db  
cage run rake db:create  
cage run rake db:migrate  

Once the database is set up, we can start the rest of the application:

cage up  

This should make a web application available at http://localhost:3000/.

Diving deeper

If we open up pods/frontend.yml, we'll see a standard docker-compose.yml file:

version: "2"  
    image: "faraday/rails_hello"
    build: ""
    - "3000:3000"
      io.fdy.cage.srcdir: "/usr/src/app" "bash"
      io.fdy.cage.test: "bundle exec rake"

We see that frontend.yml defines a single web service using the faraday/rails_hello image, with source code available from (There are also some labels that we'll explain later.)

Let's get this source code and make a change! First, we need to "mount" the source code into our service, and restart the app:

cage source mount rails_hello  
cage up  

This will clone a copy of the rails_hello source code in src/rails_hello, and mount it into our web service in the directory specified by io.fdy.cage.srcdir above. So we can just go ahead and create an HTML file at src/rails_hello/public/index.html:

  <head><title>Sample page</title></head>
  <body><h1>Sample page</h1></body>

If we go back to http://localhost:3000/ and reload, we should see our new page!

Testing and shell access

One challenge on large microservice projects is remembering how to test other people's code! We specified io.fdy.cage.test above, which specifies how to run tests for our web service. We can invoke this as:

cage test web  

If we have other services written in other languages, we could also test them using cage test $SERVICE_NAME.

Similarly, if we want to get command-line access to our web service, we can run:

cage shell web  

How we built cage

cage is a single binary with no dependencies. It's written in Rust and the Linux version links against musl-libc, so you should be able to install it on any modern Linux distribution using cp.

cage relies heavily on the compose_yml library, which provides a typesafe API for working with the complex data structures in a docker-compose.yml file.

Internally, cage is structured a bit like a multi-pass compiler. In this case, the intermediate language would be docker-compose.yml files, and various transformation plugins each transform the files in some way.

Rust has been a great language for this project:

  • Rust allows us to build fast, standalone binaries.
  • Refactoring Rust code is a joy, because the compiler can catch so much.
  • The cargo build tool and the the crates ecosystem is great.
  • Rust's type system allows us keep careful track of exactly what's in a docker-compose.yml file, which fields are optional, and which fields require shell variable interpolation. (It's far more complex than it looks.) Without a strong type system, it would be very easy to overlook an important case when writing a transformation plugin.

Getting started

Cage is still extremely new, and few people outside of Faraday have ever used it! So we encourage you to contact us and to ask us questions.

We're interested in hearing about what works, what doesn't, and what's too confusing. We're also interested in ideas for new features to simplify your development workflow.

Meet Dealbot, the open-source sales cadence automation system for Pipedrive

Andy Rossmeissl on

Dealbot illustration

We've just released Dealbot, our lightweight sales automation system for Pipedrive, as open source. Over the coming weeks we'll introduce Dealbot's features along with its cadence ecosystem in a series of blog posts.

Here's the secret behind the most successful sales development teams out there:

They're using cadences.

A cadence is a named, prescribed schedule of activities like phone calls and emails that you can use to engage and qualify your leads.

7x7 cadence

SalesLoft's classic 7x7 cadence

Using the cadence approach to outreach is one of those cases where a good tool makes all the difference.

Dealbot applies cadences to your deals in Pipedrive

Paid services like SalesLoft and Outreach are the gold standard for cadence management. But these tools might be overboard for your team, and can be expensive.

Faraday's Dealbot provides everything you need to get started using cadences with your Pipedrive CRM in an open-source package that can be hosted at Heroku for free.

Here's how to get started quickly

  1. Create a free Pipedrive account if you don't already have one.

  2. Click the magic button:

  3. Fill out the required config fields and click "Deploy for Free." Don't close the tab! You'll get an email telling you what to do next.

Stay tuned

Over the next few weeks we'll dive into Dealbot features, use cases, cadences, and technology. In the meantime, check out the Dealbot site for more, or sign up below for future Dealbot blog posts.

How to create an RDS instance with Terraform

Seamus Abshere on

This post is part of our PostgreSQL series.

Terraform's RDS support makes it easy to create a database instance. Here's a cheatsheet:

resource "aws_db_instance" "mydb1" {  
  allocated_storage        = 256 # gigabytes
  backup_retention_period  = 7   # in days
  db_subnet_group_name     = "${var.rds_public_subnet_group}"
  engine                   = "postgres"
  engine_version           = "9.5.4"
  identifier               = "mydb1"
  instance_class           = "db.r3.large"
  multi_az                 = false
  name                     = "mydb1"
  parameter_group_name     = "mydbparamgroup1" # if you have tuned it
  password                 = "${trimspace(file("${path.module}/secrets/mydb1-password.txt"))}"
  port                     = 5432
  publicly_accessible      = true
  storage_encrypted        = true # you should always do this
  storage_type             = "gp2"
  username                 = "mydb1"
  vpc_security_group_ids   = ["${}"]

Here's the security group you need:

resource "aws_security_group" "mydb1" {  
  name = "mydb1"

  description = "RDS postgres servers (terraform-managed)"
  vpc_id = "${var.rds_vpc_id}"

  # Only postgres in
  ingress {
    from_port = 5432
    to_port = 5432
    protocol = "tcp"
    cidr_blocks = [""]

  # Allow all outbound traffic.
  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = [""]

You can get these values from the EC2 console (don't forget them!):

variable "rds_vpc_id" {  
  default = "vpc-XXXXXXXX"
  description = "Our default RDS virtual private cloud (rds_vpc)."

variable "rds_public_subnets" {  
  default = "subnet-YYYYYYYY,subnet-YYYYYYYY,subnet-YYYYYYYY,subnet-YYYYYYYY"
  description = "The public subnets of our RDS VPC rds-vpc."

variable "rds_public_subnet_group" {  
  default = "default-vpc-XXXXXXXX"
  description = "Apparently the group name, according to the RDS launch wizard."

How to crunch lots of geodata in parallel

Bill Morris on

This post is part of our data science and practical cartography series.

GNU parallel + ogr2ogr = happy data scientists

These power tools in combination make it very easy to process lots of geodata at once, in as many parallel operations as your local machine or server can support.

Reprojecting in bulk

Here's an example, assuming you have a folder full of shapefiles you want to reproject into Geographic coordinates. Make a directory for the output, then pipe every shapefile through ogr2ogr in parallel:

mkdir wgs84  
ls *.shp | parallel ogr2ogr -t_srs 'EPSG:4326' wgs84/{} {}  

Running a sequence of commands on many files

In order to build whole data workflows, you can wrap your sequence of commands in a bash function. Here's an example, where we:

  1. Download each state landmarks file from the census FTP
  2. Extract each file
  3. Create a new file for each consisting of only airport landmarks, projected to WGS84
# grab this handy list of all state FIPS codes
wget -c

# define the function
get_airports() {  
  # grab the data from the census server
  wget -c$
  unzip tl_2016_$
  # extract just airports (code K2451) and reproject to WGS84
  ogr2ogr -t_srs "EPSG:4326" -where "MTFCC = 'K2451'" tl_2016_$1_airports.shp tl_2016_$1_pointlm.shp
  echo "done with state $1"
export -f get_airports

# kick off the parallel processing!
cat state_fips_codes.txt | parallel get_airports {}

This crunches through 52 states and territories in 21.8 seconds on a small ec2 server, limited only by network speed.


Install the tools

  • GNU parallel
    • OSX: brew install parallel
    • Ubuntu: apt-get install parallel
  • ogr2ogr
    • OSX: brew install gdal --HEAD
    • Ubuntu: sudo apt-get install gdal-bin

Bonus toolkit: From Derek Watkins, here are a few dozen examples of the awesome geoprocessing you can you with GDAL/OGR.

Happy mapping!

How to do histograms in PostgreSQL

Bill Morris on

This post is part of our data science and PostgreSQL series.

UPDATED FOR 2017 Now with easy subquery support and a more sensible argument order!

We adapted this excellent piece by Dmitri Fontaine and turned it into a function.

It can be invoked like this:

SELECT * FROM histogram($table_name_or_subquery, $column_name);  

. . . to give sweet results like this, in a check of the distribution of 2016 political contributions in Vermont:

fec=# SELECT * FROM histogram('(SELECT * FROM small_donors_vt LIMIT 50000)', 'transaction_amt');

 bucket |   range   | freq |       bar       
      1 | [0,9]     | 2744 | ******
      2 | [10,19]   | 5630 | *************
      3 | [20,29]   | 6383 | ***************
      4 | [30,39]   | 1290 | ***
      5 | [40,49]   |  369 | *
      6 | [50,59]   | 3541 | ********
      7 | [60,69]   |  174 | 
      8 | [70,79]   |  313 | *
      9 | [80,89]   |  171 | 
     10 | [90,99]   |   65 | 
     11 | [100,109] | 2363 | ******
     12 | [110,119] |   51 | 
     13 | [120,129] |  115 | 
     14 | [130,139] |   32 | 
     15 | [140,146] |   11 | 
     16 | [150,159] |  187 | 
     17 | [160,169] |   24 | 
     18 | [170,177] |   33 | 
     19 | [180,189] |   19 | 
     20 | [191,199] |   24 | 
     21 | [200,200] |  795 | **

Use it yourself by adding this to your postgres setup:

CREATE OR REPLACE FUNCTION histogram(table_name_or_subquery text, column_name text)  
RETURNS TABLE(bucket int, "range" numrange, freq bigint, bar text)  
AS $func$  
  source AS (
    SELECT * FROM %s
  min_max AS (
    SELECT min(%s) AS min, max(%s) AS max FROM source
  histogram AS (
      width_bucket(%s, min_max.min, min_max.max, 20) AS bucket,
      numrange(min(%s)::numeric, max(%s)::numeric, ''[]'') AS "range",
      count(%s) AS freq
    FROM source, min_max
    GROUP BY bucket
    ORDER BY bucket
    repeat(''*'', (freq::float / (max(freq) over() + 1) * 15)::int) AS bar
  FROM histogram',
$func$ LANGUAGE plpgsql;


  1. You don't need to use a subquery - you can also just provide a table name as the first argument.
  2. Subqueries are useful for sampling. For example, (SELECT * FROM bigtable LIMIT 500 ORDER BY RANDOM(). Don't forget parentheses!
  3. You can call this as SELECT histogram() or SELECT * FROM histogram(). The latter form is much more legible!

Happy querying!

How to preview PostGIS maps on your command line

Bill Morris on

This is part of our practical cartography and PostgreSQL series. Put a map on it!

Sometimes it's a pain to open up QGIS and load a PostGIS-enabled DB. Sometimes I don't feel like writing a custom tileserver and hooking it up to Leaflet or Mapbox GL just so I can see if my map looks right.

Sometimes I use the psql command line and a nifty tool by Morgan Herlocker called "geotype" to view my map data.

npm install -g geotype  

. . . which enables fast and simple maps like this:


Yep. That's New York, alright.


. . . and that sure looks like the population distribution of the District of Columbia.

These maps are nothing to show to customers, but they make QA/QC a lot easier. Here's the syntax, piping psql output directly into geotype:

psql $DB_URL -t -c "SELECT ST_AsGeoJSON(ST_Collect(the_geom)) FROM mytable" | geotype  

(The -t and ST_Collect() coerce the output into the type of data that geotype can read)

Happy mapping!

Good, strict default warnings for Rust code with Clippy

Eric Kidd on

Do you maintain a project written in Rust? Would you like to enable really aggressive warnings to make code review easier? Here's what it looks like:

We can set this up using a combination of Rust's built-in warnings and the excellent Clippy package. For best results with Clippy, it's easiest to install the nightly release of Rust:

curl -sSf | sh  
rustup update nightly  

You can then update your Cargo.toml file as follows:

unstable = ["clippy"]

clippy = { version = "0.0.*", optional = true }  

Near the top of your project's src/ file, add this enormous list of warnings:

// Enable clippy if our Cargo.toml file asked us to do so.
#![cfg_attr(feature="clippy", feature(plugin))]
#![cfg_attr(feature="clippy", plugin(clippy))]

// Enable as many useful Rust and Clippy warnings as we can stand.  We'd
// also enable `trivial_casts`, but we're waiting for
#![cfg_attr(feature="clippy", warn(cast_possible_truncation))]
#![cfg_attr(feature="clippy", warn(cast_possible_wrap))]
#![cfg_attr(feature="clippy", warn(cast_precision_loss))]
#![cfg_attr(feature="clippy", warn(cast_sign_loss))]
#![cfg_attr(feature="clippy", warn(missing_docs_in_private_items))]
#![cfg_attr(feature="clippy", warn(mut_mut))]
// Disallow `println!`. Use `debug!` for debug output
// (which is provided by the `log` crate).
#![cfg_attr(feature="clippy", warn(print_stdout))]
// This allows us to use `unwrap` on `Option` values (because doing makes
// working with Regex matches much nicer) and when compiling in test mode
// (because using it in tests is idiomatic).
#![cfg_attr(all(not(test), feature="clippy"), warn(result_unwrap_used))]
#![cfg_attr(feature="clippy", warn(unseparated_literal_suffix))]
#![cfg_attr(feature="clippy", warn(wrong_pub_self_convention))]

For documentation about individual warnings, run rustc -W help and look at the Clippy lint list. This will show you all available warnings, and explain what each one does.

If you also want to turn warnings into compilation errors, you can add:

// Fail hard on warnings.  This will be automatically disabled when we're
// used as a dependency by other crates, thanks to Cargo magic.

To see your new warnings, run:

rustup run nightly cargo build --features unstable  

This will compile your code using the nightly build of Rust, activating the unstable feature that we defined in Cargo.toml, which will in turn activate the clippy feature. Surprisingly, even with all these warnings activated, I see very few false positives in practice.

Because Clippy uses nightly Rust, you may occasionally get compilation errors. If this happens, get the latest nightly build and the latest version of Clippy:

rustup update nightly  
cargo update  

If this still doesn't work, just leave off --features unstable and try again tomorrow.

How to convert a fixed-width file into CSV

Seamus Abshere on

This is part of our data science series. How predictive!

(The more valuable and massive a data set is, the less likely it's in a format you can just parse. Has anybody else noticed that?)

Here's how to convert a fixed-width file to CSV with the standard GNU unix tool gawk:


Thanks to stackoverflow: (reproducing verbatim)

gawk '$1=$1' OFS=, FIELDWIDTHS='4 2 5 1 1' infile > outfile.csv  

Where FIELDWIDTHS is a list of field widths and OFS is the output file separator.

Real life

In real life, fixed width files contain commas and double quotes.

# put this in a file called fixed2csv.awk
  for (i=1;i<=NF;i++) {
    printf "\"%s\"%s", $i, (i<NF?OFS:ORS)

Then run it on your data:

gawk -f fixed2csv.awk OFS=, FIELDWIDTHS='4 2 5 1 1' infile > outfile.csv  

Thanks to Ed Morton on Stackoverflow for inspiration!