AI, startup hacks, and engineering miracles from your friends at Faraday

Deleting /var/lib/docker with devicemapper

Seamus Abshere on

If you're running docker with devicemapper as the storage driver, there are a few steps to recover after deleting /var/lib/docker:

$ sudo service docker stop
$ sudo rm -rf /var/lib/docker
$ sudo lvremove docker
$ docker-storage-setup
$ sudo service docker start

This will help you recover from errors like

Error starting daemon: error initializing graphdriver: devicemapper: Non existing device docker-docker–pool  


Error starting daemon: error initializing graphdriver: Unable to take ownership of thin-pool (dockerVG-docker--pool) that already has used data blocks  

Thanks to hints from redhat bugzilla and docker forums.

Antipattern: ECS + yum update

Seamus Abshere on

This is part of our antipatterns series. Ouch!

With the recent bugs in ecs-agent 1.8.0, you may be trying to roll back to amzn-ami-2015.09.e or earlier to get a last-known-good ecs agent.

If you have yum update in your userdata, however, it updates ecs-init and that, in turn, will auto-upgrade you to 1.8.0—rolling back to an older image won't help!

But... you don't want to get rid of yum update it from your userdata because of fun CVEs in glibc and openssh.

Solution: yum update --exclude=ecs-init

Best of both worlds: you get the latest security patches and you can roll back to whatever agent you want!

Confirmed to work with Julien of AWS Support. Thanks Julien!