UPDATED: quickly download lots of Python packages

Dev, DevOps
This trick downloads Python packages up to 9x faster than normal:egrep -o '^([A-Za-z].*==[^ ]+)' requirements.txt | xargs -n1 -P9 pip downloadAfter things are downloaded, actually build and install the packages:pip install -r requirements.txtEDIT: original code gave `egrep: Invalid range end` -- fixed. Also added "-n1" to xargs so it'll download in parallel, vs sequentially.
Read More

tip: easily run Postgres administration commands

Dev, DevOps
TIP: on Linux it helps to be the Postgres user to do administration stuff with the database.  Either do some configuration twiddling, or run commands as the postgres user:sudo su -c 'dropdb mydb' postgresIn Ansible, this is: - name: database -- zap database      command: dropdb mydb       sudo: yes      sudo_user: postgres
Read More

talk: Better Browser Tests with Selenium

Dev, DevOps
Nicer Browser Tests with Selenium presentationslides as Google docme presenting at the Python Meetup on 7/23:Youtube videos of Python3 and Nicer Browser TestsThanks to Esther for putting everything together, Hulu for sponsoring, Philip for helping, and Carl Mullins for the video!
Read More

TIP: tweak number of Gunicorn workers

Dev, DevOps
Count number of Gunicorn workers$ pgrep gunicorn.*work | wc -l5Add another worker by signaling the Gunicorn master$ sudo pkill -TTIN  gunicorn.*mastVerify another worker has appeared$ pgrep gunicorn.*work | wc -l6Decrease the number of workers, verify its effect$ sudo pkill -TTOU  gunicorn.*mast$ pgrep gunicorn.*work | wc -l5
Read More

OSX Docker Tip: set datetime

Dev, DevOps
OSX doesn't directly support the awesomeness that is Docker containers. It uses an intermediate Linux virtual machine, and that runs containers for you, on your behalf.I started getting weird Amazon S3 errors. My project would work locally (in OSX), and would work inside a container (Linux), but the S3 filesystem would throw errors.It turns out Amazon doesn't like it if you request an asset two months in the past... The Mac system had its clock set correctly, but the VirtualBox Linux VM had drifted.The following hops into the VM, then sets the clock based on a global Network Time Protocol server. You should only have do this a couple times a year.boot2docker sudo ntpclient -sh pool.ntp.org
Read More

speeding up tests with Postgres

Dev, DevOps, Testing
Dear Lazyweb,I'm developing a site using Django and Postgres. How can I get my tests to run a lot faster?Answer: tell Postgres to use a tablespace in RAM. Selects, updates, and inserts will run at fast RAM speed vs slow disk speed.Caveat: this doesn't work for writes. Postgres is designed to keep data reliable at all times. Any INSERT/UPDATE/DELETE gets written to a "write-ahead log" (WAL), so that if the database crashed it'll restore the data.  Since this is on a disk no matter what the tablespace, the above trick doesn't work without defeating the WAL.Answer2: use UNLOGGED tables, TBD.Reference:Unlogged table performance in postgresql 9.1: (2011) "unlogged tables have shown an increase of output by 13~17%"; includes performance-oriented Postgres settings.WAITING FOR 9.1 – UNLOGGED TABLES (2011): "that's really fast"wal_buffers performance…
Read More

talk Thursday: Networking Tricks with the Linux Kernel

Appearance, Dev, DevOps
If you're in LA, come see me Thursday at the Unix Users meeting!Everyone knows about networking. The Linux kernel provides a wide range of networking and other services, most of which are not generally known. In fact, a lot of what "everyone knows" is wrong or is lacking in detail. In this talk I'll give a technical survey of what's available and how to use it to accomplish magic tricks. With the right knowledge,our software can do great things without much code!http://www.meetup.com/UUASC26/events/208257162/
Read More

talk: "Tricking out Linux kernel networking" tomorrow night

Appearance, DevOps
I'll be giving an excerpt of this talk tomorrow at Santa Monica Coloft -- please drop by! Summary: "We are all familiar with classical TCP sockets. The Linux kernel gives us lots of other sockets, and internal services, to accomplish cool tricks with minimal effort." The full thing will wind up to be around two hours, so this 20-30 min version will be less code and more jazz hands, which will make it fun! http://www.meetup.com/LAHackers/events/198602002/
Read More

strace to the rescue, or, program uses mysterious config defaults

Dev, DevOps, Testing
Strace is a powerful tool that lets you debug other programs, and find where config files are found.Programs have bugs.  They're written for one platform and ported to another.  In any case sometimes things don't work as they should.I manage this website using the rather nice FTP client lftp.  It can automatically "mirror" a local directory on my lappytop onto the web server. This means I don't manage the server directly, I only push files up, so if something happens I can switch to another server with very little hassle.Today I noticed I couldn't store the password to the FTP account. The documented command bookmark edit brought up an editor, but it didn't have my saved bookmark!  On my system it's misconfigured.  Bookmarks work, as I can do open jta to…
Read More