30Sep 2019 by johntellsall

Package Managers

Dev, DevOps, Security, Testing
I’ve wondered why Npm seems to have so many security fires compared to e.g. the Python package system. They shouldn’t be that different. Anyway, there are reasons: https://blog.acolyer.org/2019/09/30/small-world-with-high-risks/ I'm amused that new Pip is basically adopting the lockfile from Npm, for exactly the same reason: nailing down transitive dependency versions. If package A wants package B, and B doesn't specify exactly what its dependencies are, then installing package A at various times will have unpredictable results. Example As a developer you use package A, and specify its exact version. You create lots of awesome code and tests, and everything is wonderful. Then you send your code to CI... The CI system of course rebuilds everything from scratch. It installs package A, and dependent package B. Package B has been updated since the time you…
Read More
11Sep 2019 by johntellsall

Testing in DevOps

Appearance, DevOps
Here's the slides, in Googledoc. This page is http://bit.ly/jta-testing-devops And, some cats, illustrating the difference between the sweet languid life of Developers writing code with the ease of taking care of a plant, vs feeding the hungry wild beast of Production. Above: my buddy Andrey showing how cats use Multivariate Analysis to minimize error functions leading to better business outcomes. For values of "cats" equalling "Machine Learning" :)
Read More
21Nov 2018 by johntellsall

importing AWS resources into Terraform

DevOps
Terraform is a wonderful tool! It helps simplify DevOps work. It turns the thorny bramble of delicate networking, users, databases, and virtual machines into a simple and well-running machine. It allows us to chant "infrastructure as code" to the amusement of well-meaning technologists. Best of all: it lets us have consistent environments. A dev can wreak havoc, learn things, then create a Terraform patch that applies to the entire collection of systems, making everything just a little bit cleaner and better understood.Terraform, although being a moderately baked and flexible tool, has a few warts. One challenge is that it doesn't play with manually-created resources very well. If you create some users in Terraform, and some users in the AWS Console, applying Terraform later will try to delete the manual users.…
Read More
20Jul 2018 by johntellsall

running My Traceroute (aka Matt’s traceroute) (MTR) on macOS

DevOps
Mtr is a wonderful program that combines ping and traceroute. It shows you each hop along a path to another host on the internet, and how long each hop takes.  It's my #1 go-to tool to debug wifi / networking / DNS issues. And, it's pretty!Anyway it requires extra privileges, so it's a bit fiddly to run. Even worse, The Internet Is Wrong on this topic, there's lots of bad advice.Here's how to install and run mtr on a macOS machine:brew install mtrPATH=$PATH:/usr/local//Cellar/mtr/0.92/sbin sudo mtr 8.8.8.8The "8.8.8.8" is a magic IP. Easy to remember, it's a public DNS router that our friends at Google make available to the public. You can use any IP or domain name here. I use the all-8s IP, because sometimes my DNS isn't working, so…
Read More
23May 2018 by johntellsall

CI Tip: block insecure changes to your source code

Dev, DevOps, Security
As a dev or devops person, it's pretty easy to accidentally expose AWS or other secrets by putting them into "temporary" code or environment files. Once a secret appears in version control, it should no longer be trusted, go ahead and rotate the key.Here's how to write a checker, so that secrets won't make it into source code!  This is perfect for a Git commit hook. The important feature to use is Perl-style regular expressions, which allow you to say "match this exactly N number of times". Also, the "-w" flag to Ack lets us match word boundaries. Thus, if something is a word, and exactly some number of characters long, we flag it.Example:$ echo 'I love gin' | ack -w '[a-z]{3}'I love ginThis example finds words of exactly three…
Read More
08Feb 2018 by johntellsall

Terraform and iTerm2 FTW

DevOps
1) install imgcat from iTerm2 page2) brew install graphvizNow, when you're working on a complex set of Terraform dependencies, you can display them directly in your terminal!terraform graph | dot -Tpng | imgcat(Code from the excellent book Terraform: Up and Running by Yevgeniy Brikman.)
Read More
02Sep 2016 by johntellsall

reach your next-level DevOps Career

DevOps
I often get questions about how to get into development or how to progress in a dev/webdev/devops career.  Here's my most recent "From Dev to DevOps" ideas:- become very familiar with doing container-centric development.  At the moment "Docker Compose" will get you comfortable with the terminology and mechanics of containers.  It's extremely useful for local dev work. It's also very similar to Docker Cloud, and leads pretty directly to Kubernetes and Amazon EC2 Container Service (ECS).- learn one or more orchestration softwares.  Chef is the biggest, Puppet is cool, Salt Stack is the new hotness.  My favorite is Ansible: it's human-readable, flexible, fast, and obvious.  The other day I wrote a "create or update a distributed S3 bucket, and verify it has the correct permissions" in 7 lines of code!-…
Read More
17Jun 2016 by johntellsall

Quality DevOps: installing and verifying Network Time Protocol (NTP)

DevOps, Testing
I lurve Ansible. It lets me install or update software on one or 100 instances, easily. The entire system becomes a set of scripts to run and run and run again until I get things exactly the way I want them.In today's devops ecosystem, where "infrastructure is code", how do we test our infrastructure?Ansible gives us one way to do this.  When we install or update a service, run a service-specific command to make doubly sure that things are working as expected. If something's not quite right, Ansible will abort and we can figure out what went kablooey.Save the following into "ntp.yml" and run with ansible-playbook -vvi myhost ntp.ymlThanks to phillipuniverse !# ntp.yml -- install NTP time sync daemon# Adapted from https://gist.github.com/phillipuniverse/7721288#file-ntp_playbook-yml## USAGE: ansible-playbook -vvi myhost ntp.yml#---- hosts: all  become: yes  gather_facts:…
Read More
30May 2016 by johntellsall

Django trick: keep "runserver" from crashing on Python syntax error

Dev, DevOps, Testing
When developing Django, the "runserver" command is convenient. It runs our appserver, and reloads itself when we change our app's source code.  We can have a rapid "edit stuff then see what happened" cycle. However, runserver it has an annoying habit. If we're typing so fast that we add a Python syntax error into our code, the  command will crash. We expect that when we fix the syntax error, we can see our results, but it doesn't work. The appserver has crashed, and is no more.   The following workaround works wonders. If we add a syntax error, "runserver" will crash, and this loop will wait for a moment then re-run it. We can now type as fast as possible all the time, and Django will always show us what…
Read More