AI, startup hacks, and engineering miracles from your friends at Faraday

Finding public S3 objects

Seamus Abshere on

There's lot of concern over public objects in S3 buckets. Now Amazon gives you a way to lock down entire buckets - but what if you legitimately have a mix of public and private objects?

Ruby script to find public S3 objects

First get 2 gems:

source 'https://rubygems.org' do
  gem 'aws-sdk'
  gem 'thread'
end

Then the script itself:

# find_public_s3_objects.rb
require 'aws-sdk-s3'
require 'thread/pool'
BUCKET = ARGV[0] or raise("expected bucket")
s3 = Aws::S3::Resource.new(region: 'us-east-1')
count = 0
pool = Thread.pool 8
mutex = Mutex.new
s3.bucket(BUCKET).objects.each do |object|
  pool.process do
    grants = object.acl.grants
    mutex.synchronize do
      count += 1
      if count % 100 == 0
        $stderr.write "#{count}..."
      end
    end
    if grants.map { |x| x.grantee.uri }.any? { |x| x =~ /AllUsers/ }
      mutex.synchronize do
        puts object.key
      end
    end
  end
end
pool.shutdown

Then you run it like this:

bundle exec ruby find_public_s3_objects.rb my-bucket-name

It's much faster than the bash-based example on StackOverflow.

Good, strict default warnings for Rust code with Clippy

Eric Kidd on

Do you maintain a project written in Rust? Would you like to enable really aggressive warnings to make code review easier? Here's what it looks like:

We can set this up using a combination of Rust's built-in warnings and the excellent Clippy package. For best results with Clippy, it's easiest to install the nightly release of Rust:

curl https://sh.rustup.rs -sSf | sh
rustup update nightly

You can then update your Cargo.toml file as follows:

[features]
unstable = ["clippy"]

[dependencies]
clippy = { version = "0.0.*", optional = true }

Near the top of your project's src/lib.rs file, add this enormous list of warnings:

// Enable clippy if our Cargo.toml file asked us to do so.
#![cfg_attr(feature="clippy", feature(plugin))]
#![cfg_attr(feature="clippy", plugin(clippy))]

// Enable as many useful Rust and Clippy warnings as we can stand.  We'd
// also enable `trivial_casts`, but we're waiting for
// https://github.com/rust-lang/rust/issues/23416.
#![warn(missing_copy_implementations,
        missing_debug_implementations,
        missing_docs,
        trivial_numeric_casts,
        unsafe_code,
        unused_extern_crates,
        unused_import_braces,
        unused_qualifications)]
#![cfg_attr(feature="clippy", warn(cast_possible_truncation))]
#![cfg_attr(feature="clippy", warn(cast_possible_wrap))]
#![cfg_attr(feature="clippy", warn(cast_precision_loss))]
#![cfg_attr(feature="clippy", warn(cast_sign_loss))]
#![cfg_attr(feature="clippy", warn(missing_docs_in_private_items))]
#![cfg_attr(feature="clippy", warn(mut_mut))]
// Disallow `println!`. Use `debug!` for debug output
// (which is provided by the `log` crate).
#![cfg_attr(feature="clippy", warn(print_stdout))]
// This allows us to use `unwrap` on `Option` values (because doing makes
// working with Regex matches much nicer) and when compiling in test mode
// (because using it in tests is idiomatic).
#![cfg_attr(all(not(test), feature="clippy"), warn(result_unwrap_used))]
#![cfg_attr(feature="clippy", warn(unseparated_literal_suffix))]
#![cfg_attr(feature="clippy", warn(wrong_pub_self_convention))]

For documentation about individual warnings, run rustc -W help and look at the Clippy lint list. This will show you all available warnings, and explain what each one does.

If you also want to turn warnings into compilation errors, you can add:

// Fail hard on warnings.  This will be automatically disabled when we're
// used as a dependency by other crates, thanks to Cargo magic.
#![deny(warnings)]

To see your new warnings, run:

rustup run nightly cargo build --features unstable

This will compile your code using the nightly build of Rust, activating the unstable feature that we defined in Cargo.toml, which will in turn activate the clippy feature. Surprisingly, even with all these warnings activated, I see very few false positives in practice.

Because Clippy uses nightly Rust, you may occasionally get compilation errors. If this happens, get the latest nightly build and the latest version of Clippy:

rustup update nightly
cargo update

If this still doesn't work, just leave off --features unstable and try again tomorrow.