AI, startup hacks, and engineering miracles from your friends at Faraday

Buy or build: things we built

Seamus Abshere on

Here are Faraday's contributions to open source that we use every day in production. No experiments here; this is the stuff that we looked for on the shelf, found the options wanting, and built ourselves.

A new standard for secrets: Secretfile

secret_garden (Ruby), vault-env (JS), and credentials-to-env (Rust) all implement a standard we call Secretfile(s):

# /app/Secretfile
DATABASE_URL secrets/database/$VAULT_ENV:url
REDIS_URL secrets/redis/$VAULT_ENV:url

Then you use it like this SecretGarden.fetch('DATABASE_URL').
Clients implementing this standard are meant to first check the environment for DATABASE_URL, then failing that look up the secret in Hashicorp Vault (interpolating $VAULT_ENV into production, staging, etc. first). It's very useful for development where your DATABASE_URL is just postgres://seamus@127.0.0.1:5432/myapp - you can save this in a local .env file and only mess with Vault in production/staging.

Lightning fast CSV processing: catcsv and scrubcsv

catcsv is a very fast CSV concatenation tool that gracefully handles headers and compression. It also supports Google's Snappy compression. We store everything on S3 and GCS szip'ed using burntsushi's szip.

$ cat a.csv
city,state
burlington,vt

$ cat b.csv
city,state
madison,wi

$ szip a.csv

$ ls
a.csv.sz
b.csv

$ catcsv a.csv.sz b.csv
city,state
burlington,vt
madison,wi

Of course, before you cat files, sometimes you need to clean them up with scrubcsv:

$ scrubcsv giant.csv > scrubbed.csv
3000001 rows (1 bad) in 51.58 seconds, 72.23 MiB/sec

Lightning-fast fixed-width to CSV: fixed2csv

fixed2csv converts fixed-width files to CSV very fast. You start with this:

first     last      middle
John      Smith     Q
Sally     Jones

You should be able to run:

$ fixed2csv -v 10 10 6 < input.txt
first,last,middle
John,Smith,Q
Sally,Jones,

World's fastest geocoder: node_smartystreets

node_smartystreets is the world's fastest geocoder client. We shell out to its binary rather than using it as a library. It will do 10k records/second against the smartystreets geocoding API. If you don't have an Unlimited plan, use it with extreme caution.

Better caching: lock_and_cache

lock_and_cache (Ruby) and lock_and_cache_js (JS) go beyond normal caching libraries: they lock the calculation while it's being performed. Most caching libraries don't do locking, meaning that >1 process can be calculating a cached value at the same time. Since you presumably cache things because they cost CPU, database reads, or money, doesn't it make sense to lock while caching?

def expensive_thing
  @expensive_thing ||= LockAndCache.lock_and_cache("expensive_thing/#{id}", expires: 30) do
    # do expensive calculation
  end
end

It uses Redis for distributed caching and locking, so this is not only cross-process but also cross-machine.

Better state machine: status_workflow

status_workflow handles state transitions with distributed locking using Redis. Most state machine libraries either don't do locking or use Postgres advisory locks.

class Document < ActiveRecord::Base
  include StatusWorkflow
  status_workflow(
    archive_requested: [:archiving],
    archiving: [:archived],
  )
end

Then you can do

document.enter_archive_requested!

It's safe to use in a horizontally sharded environment because it uses distributed locking - the second process that tries to do this will get a InvalidTransition error even if it's the same microsecond.

Rust build tools: rust-musl-builder and heroku-buildpack-rust

rust-musl-builder is how we build Rust apps on top of Alpine. It also drives heroku-buildpack-rust, the preeminent way of running Rust on Heroku.

Minimal postgres for node: simple-postgres

simple-postgres (JS) is just the essentials to talk to Postgres from Node. We particularly love its use of template literals for apparently magical escaping:

let account = await db.row`
  SELECT *
  FROM accounts
  WHERE id = ${id}
`

Yes, that's safe!

Minimal HTTP server: srvr

srvr (JS) is a small HTTP server that speaks for itself:

  • everything express does
  • better
  • less code
  • no dependencies
  • websockets

Proper Docker API support for Rust: boondock

boondock is a rewrite of rust-docker to be more correct.

Coordinate docker-compose: cage

cage boots multiple docker-compose.ymls, each as a pod. It's sortof like a local k8s. You configure it with a bunch of docker-compose files:

pods/
├── admin.yml (a pod containing adminweb and horse)
├── common.env (common env vars)
├── donkey.yml (a pod containing donkey)
├── placeholders.yml (development-only pod with redis, db, etc.)
[...]

Local development looks like this:

$ cage pull
==== Fetching secrets from vault into config/secrets.yml
==== Logging into ECR
Fetching temporary AWS 'administrator' credentials from vault
Pulling citus        ... done
Pulling citusworker1 ... done
Pulling citusworker2 ... done
Pulling queue        ... done
Pulling redis        ... done
Pulling s3           ... done
Pulling smtp         ... done
Pulling vault        ... done
Pulling horse        ... done
Pulling adminweb     ... done
[...]
$ cage up
Starting fdy_citusworker2_1 ... done
Starting fdy_smtp_1         ... done
Starting fdy_citus_1        ... done
Starting fdy_vault_1        ... done
Starting fdy_citusworker1_1 ... done
Starting fdy_queue_1        ... done
Starting fdy_s3_1           ... done
Starting fdy_redis_1        ... done
Starting fdy_horse_1 ... done
Starting fdy_adminweb_1 ... done
[...]
$ cage stop
Stopping fdy_citusworker2_1 ... done
Stopping fdy_vault_1        ... done
Stopping fdy_citus_1        ... done
Stopping fdy_s3_1           ... done
[...]

Fixed up rust crates: rust-amqp

rust-amqp@tokio (Rust) is our rewrite of the internals of the rust-amqp crate in proper tokio. It is much more reliable and needs to be merged upstream.

(beta release) 3rd gen batch processing on k8s: falconeri

falconeri is a distributed batch job runner for kubernetes (k8s). It is compatible with Pachyderm pipeline definitions, but is simpler and handles autoscaling, etc. properly.

(alpha release) Seamless transfer between Postgres/Citus and BigQuery: dbcrossbar

dbcrossbar handles all the details of transferring tables and data to and from Postgres and Google BigQuery. Additionally, it knows about citus, the leading Postgres horizontal sharding solution - so it can do highly efficient transfers between Citus clusters and BigQuery.

Conclusion

That's it. I only mentioned tools that we use every day.

A big takeaway from the Symitar Fintech Panel on Practical AI Use Cases

Rob Trail on

I recently participated in the Fintech Panel on Practical AI Use Cases at the 2018 Symitar Educational Conference (SEC). It was great to see that there's a lot of excitement about AI and how it's being used by credit unions and other consumer finance organizations.

On top of that, Jack Henry did an amazing job of organizing the panel, providing valuable insights to attendees in varying stages of AI adoption by bringing together participants from companies operating in very different areas of the field.

  • Infosys consults on large-scale AI projects, helping credit unions and banks develop complex, in-house capabilities to be applied across numerous business functions.
  • Clinc specializes in advanced conversational AI, helping banks develop, train, and deploy superior conversational AI solutions.
  • Faraday, yours truly, specializes in AI-powered customer lifecycle optimization, helping credit unions, banks, and fintech platforms better understand their consumers, personalize experiences, and improve member interactions from acquisition to retention.

The big takeaway

We covered a lot, but the biggest concern in the room was how credit unions with little to no experience with AI can operationalize the right solutions quickly.

There are seemingly endless ways AI is being used by innovative consumer finance organizations. Every day, we're hearing about new, creative ways organizations are applying data science and machine learning to extract more value from their data. Frankly, it can become overwhelming for credit unions that are just getting started.

You're not going to transform every process with AI overnight (or even over the next year). Start with one or two solutions that are relatively easy to implement and will yield significant results in the short term.

If you're considering using AI to draw deeper member insights, personalize experiences, and optimize outreach, Faraday can get you up and running in 6-8 weeks. Here are a few ways to get started:


cs02_mhvfcu_promo


How Peacock Alley began their journey to an AI-driven growth strategy

Perry McDermott on

Over the last 45 years, Peacock Alley has established itself as a leader in high-end, luxury linens and bedding by placing their valued customers first, from product design to enjoyable, evolving shopping experiences.

Peacock Alley recently signed on with Faraday to leverage artificial intelligence and nationwide consumer data to continue to deepen their understanding of their current customers and intelligently target and acquire new customers.

Ryne Higgins, Head of eCommerce and Digital Marketing, spearheaded the efforts to incorporate AI into Peacock Alley's marketing strategy. We had a chance to speak with Ryne about Peacock Alley, their decision to implement an AI solution, and what their onboarding experience has been like so far.

We'd like to thank Ryne for sharing his thoughts and experience, and hope you enjoy the interview!


Peacock Alley AI Interview

Can you provide a brief history on Peacock Alley?

Ryne:

"Our founder, Mary Ella Gabler founded Peacock Alley in 1973 (yes, we are 45 years old and working with an AI / machine learning software!) on the "Little Black Dress" theory to bedding and bath linens; own the best basics and then thoughtfully work everything else in.

About 9 years ago, she handed the reins over to her sons – Jason and Josh - and they are continuing to run the business today with her watchful eye still on product development and brand.

The company has been through a lot of iterations throughout the years but our core business channels are perhaps not that much different than you'd expect – we have a growing wholesale, retail, and eCommerce presence still today."

Who is a typical Peacock Alley customer? Is your customer base changing?

Ryne:

"The typical Peacock Alley customer is a difficult question and one of the reasons why we have paired with Faraday.

Across our channels, you may get a lot of answers. For example; our wholesale channel sells to a number of specialty stores, online dropship channels, and direct-to-designer. Our retail stores, while open to the public, mostly focus on the professional, accredited designer who is looking for the touch and feel experience in Dallas, Austin, Atlanta, or Nashville.

For my purview of the business (eCommerce), we are mostly focused on striking the cord with the do-it-yourself designers (think of what you see on Pinterest and beyond), whether they need coverlets and bedspreads, bath towels, or luxury fitted sheets.

I wouldn't say that our customer base is necessarily changing – we're still committed to the people that have made us successful over the past 45 years – however, as a brand, we are always interested in understanding what makes our customers tick and finding ways to attract new customers."

What does a typical Peacock Alley customer lifecycle look like? For example, are a lot of your customers repeat buyers? How do you communicate with past customers?

Ryne:

"Like a lot of 45 year old brands, our changes in the online space have been more revolution vs. evolution. In the past year; we've redesigned / replatformed our website, paired with companies like Faraday, and have really been trying to reshape our marketing mix between traditional and digital channels. For digital channels, we are seeing a high rate of new customers as we've built an experience that they are interested in interacting with.

We see very strong customer loyalty and to be frank with you, our product and brand has really stood on its own in bringing customers back historically. Our messaging and retention strategies have really been based more on the personal relationships we've built with our customers over the years.

As a brand with high-touch customer service and interaction, we are always looking for ways to supplement the old-fashioned "how are you doing" with digital communication that adds value to our customer's experience with the brand."

How has your marketing strategy evolved over the last 5 years? Where have you found success? Where have you struggled?

Ryne:

"Without getting into too many details, I will say that it is definitely in a place of evolution. Our business was built on the wholesale channel and sales tools, swatchfolios, etc have always been a staple of our marketing efforts.

That is still true today but we are also trying to find ways to improve those assets while also mixing in digital marketing (be it SEO, paid search, paid social, etc). As with most traditional media efforts, tracking can be difficult compared to the relative ease of digital. Faraday's toolset will allow us to continue to close that gap."

To what extent has data guided your marketing strategy throughout that period?

Ryne:

"Data and the ever-evolving digital landscape has been paramount in our evolution of our marketing strategy. When Mary Ella started the business, you could have never imagined directly tying a dollar spent to a dollar earned in marketing.

As these tools and technologies continue to evolve, for us it is about finding ways to marry the digital tools out there today with the traditional efforts that have really built and grown this business over the years."

Can you describe your current tech stack from your e-commerce system to customer data management? Has this changed significantly over the last few years?

Ryne:

"Sure. The fun thing about digital is it is pretty easy to find out what tools and technology people are using so it's pretty much public information.

Last year, we made a transition from Magento to Shopify, we signed on with a tool that allows us to visualize the true color of a product without having to shoot our products in a dozen different colors, and have signed on data and machine learning partners like Faraday.

We believe that to scale the marketing efforts of a luxury business through digital – we had to have a partner that could help us solve for the black box of channels like Facebook and Instagram.

It is pretty unbelievable that, with as great of a platform some of these channels are, they can only build look-a-like type prospecting audiences with 2.1M+ people in them, making it very difficult to understand why something is performing or not.

By leveraging a partner like Faraday, we have the ability to pull back the curtain and communicate with members of their account management, leadership, and theoretically their engineering team if ever needed.

For us to make significant adjustments in our marketing budgets, we need the ability to find the right people and send the right message at the right time. It is a fairly cliché statement but partners like Faraday should help us close that gap."

Prior to learning about Faraday, had you considered leveraging AI in any aspect of your business? If so, how? If not, why?

Ryne:

"I mean, sort of? I come from a background that is extremely digital in nature and I'm constantly trying to follow the trends that will help us build our brand. I will say – before talking to Faraday – I didn't think that AI was accessible to brands like ours quite yet (we have a small, scrappy team).

I was extremely impressed with what a relatively small team in Vermont can produce, and I liked their ability to "flex" their messaging to someone who considers themselves digital savvy but would have never considered scaling an in-house data science team in the short-term."

Why did you choose to leverage Faraday's AI solution?

Ryne:

"I've said a lot in the rest of the interview that I think answers this question but I would also say that the ability to take our use cases and personalize their product to what we needed to accomplish was key.

The Faraday team (shout out Robbie) is the type that listens, and I genuinely got the feeling that, in a world of technology partners who say a lot, they actually had the ability to walk the walk."

Can you describe your onboarding experience with Faraday?

Ryne:

"I've been through countless onboarding meetings and they are often met with a disconnect between sales and account management. It seemed like sales and account management was in sync. They had communicated our efforts, asked for refreshers as needed, and without the need for constant follow up have been meeting the quick deadlines that we needed them to.

As a business manager, I'm constantly met with "how" or "why" type questions and the team has been very responsive and thorough. It really sets the stage for our entire engagement."

We're excited to be working with you! Do you have any other comments or suggestions you'd like to add for companies looking to get started with AI?

Ryne:

"This might sound weird from a customer of the company, but don't just sign up for AI because you'd like to say you work with an AI firm. Take the time through the process and see if you can link Faraday's AI solution to your business outcomes.

Consider your ability to execute with the platform. We've been waiting for the solutions that we think Faraday can solve for and it seemed like a great fit for our business."

Thanks a bunch, Ryne!

Curious about how your team can use AI? A quick demo and consultation is a great way to find out! Click below to get started.


Faraday AI Platform Demo



Is your marketing team AI-ready? 3 considerations for consumer marketers

Perry McDermott on

No doubt you're hearing about the power of AI to transform marketing. But if you're like many marketers, the prospect of integrating artificial intelligence (AI) into your strategy may be daunting. It's a rapidly evolving field with seemingly endless applications, and you may have had relatively little experience with AI to date.

However, if you're committed to a data-driven marketing strategy, it's worth getting acquainted with AI and machine learning (ML); doing so will allow you to further monetize your existing data, and put the immense power of big data to work for your business.

According to Capgemini Consulting, 3 out of 4 organizations using AI and ML increase their sales and enhance customer satisfaction by more than 10%. Marketers already use AI to optimize various initiatives and automate repetitive, time-consuming tasks. Some applications include:

  • Audience expansion (AKA lookalike targeting)
  • Personalization at scale
  • Programmatic advertising
  • Automated content creation

If you're curious about what AI can do for your marketing team, but still trying to decide whether you're ready to implement the technology, here are three crucial points to consider:



3_considerations_ai_ready-3

1) Goals and strategy: Which outcomes do you want to optimize?

Establishing clear goals will help you make decisive choices about which applications of AI are best for your team, which data sources to tap into, and how to implement the technology with the resources at your disposal.

Are you more focused on optimizing customer acquisition, turning one-time buyers into repeat customers, or perhaps something much more specific?

If your goal is to optimize ad spend across your digital campaigns, you might consider leveraging applications like propensity modeling and programmatic advertising. If your goal is to drive upsells or cross-sells amongst your existing customer base, propensity modeling and personalized offers will be more helpful.

It's important to clearly define your target outcomes before making decisions about how to apply AI within the context of your particular marketing strategy. Then, research and strategic planning will help you decide which solutions are worth prioritizing in pursuit of your established goals, and which will involve more trouble and cost than they're worth.

2) Data: Is your data AI-ready?

Data is the fuel that powers AI. Quantity and quality will ultimately determine the effectiveness of your AI applications. This means it's critically important to ask whether you have enough historical data to train your AI — typically, at least 1,000 "past examples" — and whether your data is clean and rich.

Does your data have sufficient breadth and depth? More specifically, how many attributes and events are being considered per record, and are those additional data points predictive of the outcomes you want to optimize?

Thanks to big data vendors, consumer marketers don't need to rely purely on first-party data to train their AI models. Enriching existing lead and customer data with third-party consumer data can improve the predictive accuracy of your AI applications by an order of magnitude.

Here are a few ways to ensure your data is AI-ready:

  • Identify predictive data sources: If the goal is to increase lead conversion rates, your ESP and e-commerce platform will likely contain the most predictive data. Lifecycle mapping is a useful exercise to help you discover which data sources are most predictive of a target outcome.

  • Do a data audit: Clean your existing data by eliminating inconsistent, incomplete, or duplicate records. You can do this in-house or hire a consultant. Alternatively, certain AI platforms and third-party services can help automate the process.

  • Add depth and breadth: Enrich your first-party lead and customer records with third-party demographic, behavioral, and property-based data. While licensing third-party data can be costly and time-consuming, certain AI platforms and third-party services include third-party data and automate the data enrichment process. The Faraday Identity Graph is a prime example.

wp01_clo_promo-2

3) Implementation: Which approach makes the most sense for your team?

The resources needed to operationalize AI depend on how you choose to implement your solution. Business Insider's AI in Marketing report lists three common approaches: building in-house capabilities, using a third-party service, or using a platform. Each approach has its pros and cons, so it's important to carefully consider whether a solution's benefits outweigh implementation time and costs.

In-house capabilities

This is the most customizable, yet most resource-intensive approach to operationalizing AI, and should only be considered if your entire company is committed to an AI-first mindset. At minimum, building in-house capabilities from scratch requires:

  • Rich training data
  • Data storage capabilities
  • ML algorithms
  • Data science automation systems
  • Deployment features and integrations
  • Experienced talent (data scientists, ML experts, and engineers)

Third-party services

For companies that require numerous AI applications across different business functions, but don't necessarily need to feed insights and predictions between applications, third-party services like IBM's Watson can be a good alternative.

These services help facilitate AI implementation by including data science automation systems and built-in ML algorithms that in-house data science teams can use to build AI applications for marketing, accounting, SCM, or any other function worthy of optimization and automation. At minimum implementing AI with a third-party service requires:

  • Rich training data
  • Deployment features and integrations
  • Experienced talent (data scientists, ML experts, and engineers)

AI platforms

Platforms are designed to streamline AI implementation for relatively specific sets of use cases. ML capabilities and features are built in to platforms and supported by data science teams, enabling marketers to easily interact with, and act on AI-powered insights and predictions through the platform's user interface.

While specific requirements will vary depending on the platform, at minimum, operationalizing AI with a platform requires:

  • Basic training data

So, is your marketing team ready to implement AI?

If you have enough customer data, there's probably a solution that will fit your needs, but choosing the right implementation approach for your specific objectives can be challenging. If you're having trouble, consider the following questions:

  • Is your company committed to an AI-first strategy? If so, where do your marketing objectives fit into the strategy?
  • Does your company already have an in-house data science team? If so, do they have the bandwidth to focus on your marketing objectives? If not, do you have the resources to recruit and hire the right talent?
  • How important is customization? Will your AI-powered marketing insights and predictions help optimize AI efforts in another business function?
  • What's your timeline? In-house capabilities generally take at least 18 months to build from scratch, third-party services can take between 12-18 months depending on customization requirements, and platforms will take 6-12 weeks depending on predictive modeling requirements.

How to build your own consumer marketing AI stack

If customization is a priority and you have the resources to build in-house capabilities or use a third-party service, we've created a guide to building your own consumer marketing AI stack.

Alternatively, if it seems like your team should be using a platform, the guide will give you a good understanding of the data, systems, and features included in Faraday's AI platform.

faradiy_promo


What is customer lifecycle optimization and how does it work?

Perry McDermott on

customer_lifecycle_optimization

Revenue and the customer lifecycle

For nearly every B2C company, revenue is linked to the customer lifecycle. Customer acquisition, engagement, and retention initiatives directly or indirectly impact revenue growth and sustainability by either reducing customer acquisition costs, increasing customer lifetime value (LTV), or ideally, both.

Effective marketing strategies revolve around the customer lifecycle; understanding key lifecycle stages, identifying events that are likely to trigger transitions between stages, and finding ways to optimize interactions across the lifecycle.

Naturally, the more you know about your customers and how to influence them to transition into a desired stage, the more efficiently you can grow and sustain revenue with the resources at your disposal.

Simply put, that's what customer lifecycle optimization is all about: leveraging rich customer data and predictive analytics techniques to generate insights and make predictions that measurably improve outcomes at each stage of the customer lifecycle.

Customer lifecycle optimization (a 4-step practice)

Customer lifecycle optimization (CLO) is a practice. As with other practices, CLO involves a series of prescribed steps to be done effectively. While specific lifecycle stage names, data sources, and analytics techniques will vary depending on your industry, company, and objectives, the canonical CLO process comprises four steps: lifecycle mapping; data discovery; predictive groundwork; and implementation and action.

1) Lifecycle mapping

The first step in any CLO initiative is always identifying and defining key lifecycle stages and transitions between stages. Mapping these stages and transitions to a uniform customer lifecycle is crucial to uncovering rich, predictive data.


The uniform B2C customer lifecycle:

B2C_customer_lifecycle

While terminology will change from business to business, we found that this formulation is rich enough to capture important boundaries, yet simple enough to avoid stages with ambiguous transitions.

Think about your customer lifecycle. How are stages defined? Which attributes qualify individuals to be placed in those stages? What events trigger transitions between stages?

Feel free to use the following table to help organize your findings:

Lifecycle_mapping_table

Note: The Customer lifecycle optimization whitepaper dives deeper into transitions and litmus tests. It's free to download.

2) Data discovery

At each stage, prospects, leads, and customers will complete certain events that will individually or collectively trigger a transition in or out of that stage. Individuals will also have different attributes, which help determine whether or not they belong in any given stage.

Go through each cell of your customer lifecycle map and think about which events or attributes in your data could be used to trigger a transition or pass the litmus test. Then, think about where you can find that data. Is it in your ESP, CRM, or a custom data warehouse?

3) Predictive groundwork

Once you've formalized your lifecycle and its representation in data, you can start recognizing patterns and eventually predict outcomes. Imagine loading up your converted leads (customers in "retention and expansion") alongside your stale leads that never converted. What differences can you find?

This comparison analysis is especially effective when you've added depth and breadth to your existing data.

As you experiment with this type of pattern recognition, you'll quickly realize that it's the kind of thing that computers do very well. That's where machine learning comes in handy.

4) Guided outreach

Whether you want to increase conversion rates from lead generation campaigns or reduce churn from existing customers, properly targeted outreach is essential to engaging and motivating the right leads and customers to take a desired action.

Outbound communication is the strongest and most versatile intervention at an organization's disposal to compel progress and therefore expand revenue. This includes individually targeted digital advertising, a form of direct outreach.

Consider the following examples of outreach initiatives:

Customer_lifecycle_outreach_intervention

Having identified a stage transition you'd like to motivate with an outreach intervention, the question becomes, "Who do I reach?" Regardless of the desired transition, the general technique is called audience expansion, also known as "lookalike" audiences.

To apply the audience expansion technique, we must always identify the audience and a set of candidates with similar characteristics and attributes.

With these groups defined, the next step is to apply your predictive groundwork. This could involve using patterns you identified in your data to look for similar opportunities among your candidates, or in more advanced cases, using artificial intelligence to build a predictive model trained to discriminate between likely and unlikely transitioners.

Finally, you will be left with a well-defined group of candidates likely to transition into a desired lifecycle stage when reached with relevant content.

Putting it all together

To summarize the CLO practice, it's useful to recall the original motivation: leveraging rich data and statistical predictions to optimize revenue-building initiatives throughout the customer lifecycle. This means motivating transitions from one stage to the next.

At this point, you should have good understanding of what CLO is and the four canonical steps involved. For a deeper dive into specific data requirements and predictive analysis techniques, download the whitepaper below.

Customer lifecycle optimization whitepaper