21Mar 2017 by johntellsall

new short talk: Distributed Processing

Dev
My new talk is about different processing tradeoffs, using good old Unix utility xargs, Python multiprocessing, gevent, and some distributed processing tips.  Also: tonight there's a seekrit announcement -- watch this space, and/or follow me at @JohnTellsAllTalk URL is http://bit.ly/jta-dist (this page, which will receive updates)Slides are in Google Docs
Read More
04Nov 2016 by johntellsall

TIP: recover file from old Git branch

Dev
In cleaning up my source code I realize I was too... aggresive in my house cleaning. I deleted a file I needed.  No biggie, just take a look at HEAD or HEAD^... Hey, that didn't work!TIP: use git rev-list to search and find the most recent commit that affected (deleted) a path. (Source: Charles Bailey on SO)$ git rev-list -n 1 HEAD -- docker/torta/torta-task.json0c400d323aff484cfb2bdc18dcdb7813de93a658Yay!  That was really easy. Now all I need to do is check out from that Git hash and I'm done:$ git checkout 0c400d323aff484cfb2bdc18dcdb7813de93a658 !$error: pathspec 'docker/torta/torta-task.json' did not match any file(s) known to git.Err?  Hmm. Oh: the 0c4 branch contains the "delete" command. If I want to get the file contents, I'll get it from the branch just before that one:$ git checkout 0c400d323aff484cfb2bdc18dcdb7813de93a658^ docker/torta/torta-task.json$ ls…
Read More
01Nov 2016 by johntellsall

Videos! For “Platonic Solids of Quality” and “Functional Programming and Django QuerySets”

Appearance, Dev
I've been remiss in posting my videos.  The last two talks went really well- Platonic Solids of QualityAs people who produce awesome stuff, there are a lot of tools we can use to work less and simplify our workflow. This talk introduces many techniques from Agile Testing and other sources so we can all do more things faster.- Functional Programming and Django QuerySetsThis talk is a quick overview of Python iterators and why to use them, then combining iterators in Functional Programming (vs procedural and object oriented programming). Lastly I show how to use these ideas in order to better understand and test Django QuerySets.I have another talk video somewhere, but it's about using Selenium for testing and thus has rather a bit of profanity in it.  No link.Thanks to…
Read More
28Oct 2016 by johntellsall

TIP: use Strace to debug issues inside Docker

Dev, Docker, Testing
Yesterday my Docker application wasn't working correctly -- the appserver is hanging.  Debugging this is a challenge: there's no crash nor stack trace to point out the issue.  Is the appserver misconfigured, so it's trying to talk to a non-existent database? Is the config okay, but the network is not set up correctly?  Can an external service not see our Docker container correctly?To debug this I used my good old "strace" command to trace exactly what is happening. It outputs log messages for all system calls the appserver does, including all the network I/O.  Alas it didn't work for me:strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work: Operation not permittedThis is odd, as the Docker container is running with root permissions, and the parent container is Debian.My buddy Loren says this is a…
Read More
04Oct 2016 by johntellsall

talk: Upping Your (Programming) Game

Appearance, Dev
(this page is http://bit.ly/jta-upping)I'm giving this talk tonight, at LA Django, hosted by our friends at Wiredrive HQ. Thanks for Marcel and Wiredrive for supporting the Los Angeles tech scene!Slides are in gdrive: Upping Your (Programming) Game.There will be cats Lil Bubzilla.pic credit: Marcel
Read More
28Sep 2016 by johntellsall

TIP: work with Amazon AWS ECS Container Service Docker containers

Dev, Docker
When running Amazon EC2 Container Service (ECS), have these two commands running in Tmux on your instances:tail -f $(ls -1t /var/log/ecs/ecs-agent.log.* | head -1)sudo tail -f /var/log/upstart/docker.logIn another Tmux tab, this command lets you look at the logs from your running Docker container:docker logs -f $(docker ps --filter name=myname,status=running --quiet)
Read More
28Aug 2016 by johntellsall

Speeding up UI Browser tests with Robot Framework

Dev, Testing
My current AWS demo project is a Cat Voting Booth.  It uses SocketIO/WebSockets, so in addition to server-side unit tests I've built a number of browser-level tests, using Robot Framework.One feature people don't realize is that Robot can actually be quite speedy. Opening and closing a browser window is very slow, on the order of 1-2 seconds. Depending on the application, this can be done once, saving 1-2 seconds for *every* test!  For my Very Important Cat Voting Booth, this is the case. Here's the magic bit:*** Settings ***Resource          resource.robotTest Setup        Reset VotesSuite Setup       Open Browser To Voting PageSuite Teardown    Close All Browsers*** Test Cases ***Valid Page    Votes Not AvailableRegister Up Vote    Vote Up    Element Text…
Read More
30May 2016 by johntellsall

Django trick: keep "runserver" from crashing on Python syntax error

Dev, DevOps, Testing
When developing Django, the "runserver" command is convenient. It runs our appserver, and reloads itself when we change our app's source code.  We can have a rapid "edit stuff then see what happened" cycle. However, runserver it has an annoying habit. If we're typing so fast that we add a Python syntax error into our code, the  command will crash. We expect that when we fix the syntax error, we can see our results, but it doesn't work. The appserver has crashed, and is no more.   The following workaround works wonders. If we add a syntax error, "runserver" will crash, and this loop will wait for a moment then re-run it. We can now type as fast as possible all the time, and Django will always show us what…
Read More
24Mar 2016 by johntellsall

UPDATED: quickly download lots of Python packages

Dev, DevOps
This trick downloads Python packages up to 9x faster than normal:egrep -o '^([A-Za-z].*==[^ ]+)' requirements.txt | xargs -n1 -P9 pip downloadAfter things are downloaded, actually build and install the packages:pip install -r requirements.txtEDIT: original code gave `egrep: Invalid range end` -- fixed. Also added "-n1" to xargs so it'll download in parallel, vs sequentially.
Read More