Django: speed up Sqlite 1000x!

Dev, Testing
I'm working on a project analyzing large code bases. For just messing around, I'm using Sqlite. A strange thing happened when updating ~10,000 rows: they were really slow! Sqlite was updating about 10 records a second.It turns out Sqlite correctly cares about your data, so it makes a transaction around the UPDATE statement, verifying that each and every bit of your data hits disk before returning. For me, if disaster strikes and I lose my tiny database I can recreated in a few seconds, so I decided to live life dangerously, and tell Sqlite to go ahead and try to update data, but don't wait around for it to hit disk.Here's the magic. Put this before your bulk INSERT/UPDATE statements. Note that it affects the entire session, so you don't…
Read More

tip: easily run Postgres administration commands

Dev, DevOps
TIP: on Linux it helps to be the Postgres user to do administration stuff with the database.  Either do some configuration twiddling, or run commands as the postgres user:sudo su -c 'dropdb mydb' postgresIn Ansible, this is: - name: database -- zap database      command: dropdb mydb       sudo: yes      sudo_user: postgres
Read More

talk: Better Browser Tests with Selenium

Dev, DevOps
Nicer Browser Tests with Selenium presentationslides as Google docme presenting at the Python Meetup on 7/23:Youtube videos of Python3 and Nicer Browser TestsThanks to Esther for putting everything together, Hulu for sponsoring, Philip for helping, and Carl Mullins for the video!
Read More

TIP: tweak number of Gunicorn workers

Dev, DevOps
Count number of Gunicorn workers$ pgrep gunicorn.*work | wc -l5Add another worker by signaling the Gunicorn master$ sudo pkill -TTIN  gunicorn.*mastVerify another worker has appeared$ pgrep gunicorn.*work | wc -l6Decrease the number of workers, verify its effect$ sudo pkill -TTOU  gunicorn.*mast$ pgrep gunicorn.*work | wc -l5
Read More

OSX Docker Tip: set datetime

Dev, DevOps
OSX doesn't directly support the awesomeness that is Docker containers. It uses an intermediate Linux virtual machine, and that runs containers for you, on your behalf.I started getting weird Amazon S3 errors. My project would work locally (in OSX), and would work inside a container (Linux), but the S3 filesystem would throw errors.It turns out Amazon doesn't like it if you request an asset two months in the past... The Mac system had its clock set correctly, but the VirtualBox Linux VM had drifted.The following hops into the VM, then sets the clock based on a global Network Time Protocol server. You should only have do this a couple times a year.boot2docker sudo ntpclient -sh pool.ntp.org
Read More

awful, but useful, Python

Dev
Have you ever wanted to open and decompress all log files, streaming each line to the caller?In Python, this is:import bz2, sysfrom itertools import chain# set 'lines' to all lines of all files after they've been decompressedlines = chain(*(bz2.BZ2File(path) for path in sys.argv[1:]))# as a check, print the first 100 lines, then crashlines = islice(lines, 100)print ''.join(lines)blam
Read More

speeding up tests with Postgres

Dev, DevOps, Testing
Dear Lazyweb,I'm developing a site using Django and Postgres. How can I get my tests to run a lot faster?Answer: tell Postgres to use a tablespace in RAM. Selects, updates, and inserts will run at fast RAM speed vs slow disk speed.Caveat: this doesn't work for writes. Postgres is designed to keep data reliable at all times. Any INSERT/UPDATE/DELETE gets written to a "write-ahead log" (WAL), so that if the database crashed it'll restore the data.  Since this is on a disk no matter what the tablespace, the above trick doesn't work without defeating the WAL.Answer2: use UNLOGGED tables, TBD.Reference:Unlogged table performance in postgresql 9.1: (2011) "unlogged tables have shown an increase of output by 13~17%"; includes performance-oriented Postgres settings.WAITING FOR 9.1 – UNLOGGED TABLES (2011): "that's really fast"wal_buffers performance…
Read More

talk Thursday: Networking Tricks with the Linux Kernel

Appearance, Dev, DevOps
If you're in LA, come see me Thursday at the Unix Users meeting!Everyone knows about networking. The Linux kernel provides a wide range of networking and other services, most of which are not generally known. In fact, a lot of what "everyone knows" is wrong or is lacking in detail. In this talk I'll give a technical survey of what's available and how to use it to accomplish magic tricks. With the right knowledge,our software can do great things without much code!http://www.meetup.com/UUASC26/events/208257162/
Read More

Programming vs Programmer

Dev, General
Although I have many years of professional programming experience, it's still startling to realize the distance between writing a program for "Hello World", vs working as a day-to-day programmer. The amount of stuff you have to know, and use quickly in ever-changing combinations, is substantial.A student decried that his two-month class in Ruby didn't really prepare him for being a big-P Programmer, it just taught him little-p programming.My view is that the material in that class, as well as Computer Science in general, is meaningless. What you learn is not useful, it's all about learning how to learn.The following article shows how to do professional-level deployment of your Python webapp onto a server.  It uses ten different technologies, only one of which is "Python"!  This is all after your webapp is…
Read More