prevent multiple copies of script running at same time

Dev, DevOps
Single-process servers (and cron jobs) are easier to write.  If a script fires up, then starts processing a job that another copy of the same script is working on, then bad things happen.The traditional solution to this is a bit of a hack.  Script fires up, writes its unique process id (PID) to a magic file somewhere.  When the script dies we delete the file.  This works somewhat, but fails if the script gets impolitely killed -- the "lock" file still exists but the associated script is no longer running.The following trick is useful. When the script fires up it registers itself with a global list of strings held by the OS. If the code can't register itself, it knows the program is already running, and should exit.  When a…
Read More

easy way to maintain source code decorum

Dev
It's best to maintain source code in a clean, pretty state. It's easy to read, and more importantly easy to find small code changes that might lead to dramatic effects.But, during development, people add debugging statements. Print statements, debug logging, and things like "# XXX fix this".  Letting this code hit production is not good, as it confuses the normal operation of the system.To pick bits of debugging fluff from your code, add this to your top-level Makefile.  Before merging your code with the main branch, run "make sniff" to sniff your code for unwantedly smelly bits, like leftover debugging code.The following will only check newly-changed code, and won't warn on old smelly code:sniff: git diff develop | awk '/(debug|XXX)/ { print $$1; found=1; } END { exit found }'If…
Read More

trivial Jenkins configuration

Dev, DevOps
Knowing when your software breaks is useful. If you catch subtle errors soon after the change, it's *much* easier to figure out which bit of code "optimization" broke things for the users.Another benefit is fixing "works for me" syndrome. Even if you're building software just for yourself, it's nice to have a 3rd party to verify that you didn't do silly things. Forgetting to check in South database changes is quite easy, but that simple omission will break everything.Jenkins is a cranky to set up, because it's a collection of moving parts. The best setup would be having Jenkins rebuild your tests every time GitHub detects a source code change.The following setup is not as geeky, but it's much, much easier: tell Jenkins copy local source files and test, every…
Read More

interesting database performance article

Dev, DevOps
Postgres doesn't support that much compression directly. These guys decided to take the performance hit of compressing/decompressing things at the filesystem layer... but it's actually *faster*. Turns out I/O bound stuff like the database, even if you spend more CPU on compression, there are multiplicative effects if the data is smaller.http://citusdata.com/blog/64-zfs-compression
Read More

Profiling Django

Dev, DevOps
Install django-extensionshttp://pythonhosted.org/django-extensions/runprofileserver.html1) sudo pip install django-extensions2) add 'django_extensions' to your app's INSTALLED_APPS list.Run server in profiling modepython  ./manage.py runprofileserver --prof-path=/tmp 8001Do a querytime curl -i http://localhost:8001/account/eventboard/update/Write little reporting moduleprof.py -- given laterRun report/tmp/account.eventboard.update.017442ms.1363891644.prof :         202963 function calls (197801 primitive calls) in 17.443 seconds   Ordered by: internal time, call count   List reduced from 2280 to 5 due to restriction    ncalls  tottime  percall  cumtime  percall filename:lineno(function)    26201   14.956    0.001   14.956    0.001 /usr/lib/python2.7/ssl.py:154(read)        2    0.725    0.363    0.725    0.363 /usr/lib/python2.7/ssl.py:301(do_handshake)      937    0.470    0.001   15.069    0.016 /usr/lib/python2.7/socket.py:406(readline)       10    0.318    0.032    0.318    0.032 /usr/lib/python2.7/socket.py:223(meth)        2    0.238    0.119    1.291    0.646 /usr/local/lib/python2.7/dist-packages/httplib2/__init__.py:982(connect)In this case,…
Read More

command-line JSLint

Dev
Finding syntax errors and so forth is not work for programmers, it's a job for our very fast but dumb slaves, the computers.  Here is how to harness their power for good, by showing possible bugs in our JavaScript programs.install a command-line JavaScript interpreter, Rhinosudo apt-get install rhinoinstall JavaScript checkermkdir -p ~/src/jslint/cd ~/src/jslint/wget https://raw.github.com/douglascrockford/JSLint/master/jslint.jscreate a little wrappermkdir -p ~/bin/echo '#!/bin/bash' >> ~/bin/jslintecho 'rhino -f ~/src/jslint/jslint.js $*' >> ~/bin/jslintAdd ~/bin/ to your PATHTest the checker$ echo 'alert(ding)' >>testme.js$ jslint !$jslint testme.jsjs: uncaught JavaScript runtime exception: ReferenceError: "ding" is not defined.It works! In our sample code, we call "alert(ding)" using an undefined variable.  The code should be "alert('ding')" -- a quoted string.
Read More

Ssh tunnels are great

Dev, DevOps
At work I'm connecting to multiple Munin statistics servers. On my lappytop I'm only running one server, so how do I get multiple sets of results?  Answer: create a tunnel to another server!The Munin protocol is extremely easy, send "fetch X" to get statistics on X.  In my example df=Disk File usage.  Here's how to get local information, via Munin running locally on port 4949.$ echo 'fetch df' | nc -q1 localhost 4949# munin node at freebeer_dev_sda1.value 5.29318865322941_dev.value 0.0339391645617528_dev_shm.value 0.378827479751794_var_run.value 0.00922469512382616_var_lock.value 0.Here's how to make a remote machine's Munin (on port 4949) show up on localhost (port 4950). This means we can scan multiple local ports to get information on many different machines.ssh -fNL localport/localhost/remoteport remotehostOption "-f" means drop into the background after asking for a password.  Next option "-N"…
Read More

Empirical Data behind Software Development Techniques

Dev
Our friends at Microsoft Research have published four papers testing--and measuring--development ideas. One big surprise was that a company's organization "can predict software failure-proneness with a precision and recall of 85 percent. " It's more important than using assertions or increasing test coverage!Other results: test coverage isn't a direct measure of software quality, and assertions are useful.I was glad to see some numbers behind what I've experienced, that Test Driven Development increases quality. "What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer." Hmm.Exploding Software-Engineering Myths
Read More

Fast Parallel Downloading (for apt-get) #2

Dev, DevOps
This is how to upgrade your Ubuntu system, downloading several packages in parallel. See previous article for details:cd /var/cache/apt/archives/apt-get -y --print-uris -u dselect-upgrade >debs.listegrep -o -e "http://[^']+" debs.list | xargs -l3 -P5 wget -nvapt-get -u dselect-upgrade
Read More