AI, startup hacks, and engineering miracles from your friends at Faraday

Deleting /var/lib/docker with devicemapper

If you're running docker with devicemapper as the storage driver, there are a few steps to recover after deleting /var/lib/docker:

$ sudo service docker stop
$ sudo rm -rf /var/lib/docker
$ sudo lvremove docker
$ docker-storage-setup
$ sudo service docker start

This will help you recover from errors like

Error starting daemon: error initializing graphdriver: devicemapper: Non existing device docker-docker–pool


Error starting daemon: error initializing graphdriver: Unable to take ownership of thin-pool (dockerVG-docker--pool) that already has used data blocks

Thanks to hints from redhat bugzilla and docker forums.

Antipattern: ECS + yum update

This is part of our antipatterns series. Ouch!

With the recent bugs in ecs-agent 1.8.0, you may be trying to roll back to amzn-ami-2015.09.e or earlier to get a last-known-good ecs agent.

If you have yum update in your userdata, however, it updates ecs-init and that, in turn, will auto-upgrade you to 1.8.0—rolling back to an older image won't help!

But... you don't want to get rid of yum update it from your userdata because of fun CVEs in glibc and openssh.

Solution: yum update --exclude=ecs-init

Best of both worlds: you get the latest security patches and you can roll back to whatever agent you want!

Confirmed to work with Julien of AWS Support. Thanks Julien!