Package Managers

Dev, DevOps, Security, Testing
I’ve wondered why Npm seems to have so many security fires compared to e.g. the Python package system. They shouldn’t be that different. Anyway, there are reasons: https://blog.acolyer.org/2019/09/30/small-world-with-high-risks/ I'm amused that new Pip is basically adopting the lockfile from Npm, for exactly the same reason: nailing down transitive dependency versions. If package A wants package B, and B doesn't specify exactly what its dependencies are, then installing package A at various times will have unpredictable results. Example As a developer you use package A, and specify its exact version. You create lots of awesome code and tests, and everything is wonderful. Then you send your code to CI... The CI system of course rebuilds everything from scratch. It installs package A, and dependent package B. Package B has been updated since the time you…
Read More

TIP: Bash has “global search and replace”!

Dev
TIP: Bash has “global search and replace”! It works with the history mechanism. Example, the bangbang (!!) command repeats the previous command:$ !! # repeat previous commandAdding a colon (:) and then a letter or two will modify the command before running it.  A useful modifier is "p" for printing.  That is:$ !!:p # repeat previous command, but just :p-print itThis is useful because you can use up-arrow to now go to the previous command and edit it interactively.For non-interactive editing, you can do global search and replace!  Example: use the "repeat previous command" command, bangbang (!!). Then modify it (:), then say "global search" (gs).  To do this to find "one" and replace it with "two", use this command:$ !!:gs/one/twoIn my real-world case, I'd already run a command to…
Read More

CI Tip: block insecure changes to your source code

Dev, DevOps, Security
As a dev or devops person, it's pretty easy to accidentally expose AWS or other secrets by putting them into "temporary" code or environment files. Once a secret appears in version control, it should no longer be trusted, go ahead and rotate the key.Here's how to write a checker, so that secrets won't make it into source code!  This is perfect for a Git commit hook. The important feature to use is Perl-style regular expressions, which allow you to say "match this exactly N number of times". Also, the "-w" flag to Ack lets us match word boundaries. Thus, if something is a word, and exactly some number of characters long, we flag it.Example:$ echo 'I love gin' | ack -w '[a-z]{3}'I love ginThis example finds words of exactly three…
Read More

TIP: you can exclude file types when searching for stuff in Git!

Dev
Git isn't just a database, it's a comprehensive searching system XX. You can exclude file types when searching for stuff in Git!Example: the Development branch broke some time between two Git commits. I want to find the error but don't want to look at huge piles of changes. I don’t think the front end Vue files broke the build, so let's find changes excluding those files, so I can find the error.First, I list the files between the two commits, to see if it’s roughly what I want:git diff --stat c7b5d2d^..2de4986 -- ':(exclude)*.vue'Looks good. Next, I list the actual content by zapping the "--stat" argument:^--stat^Or: git diff c7b5d2d^..2de4986 -- ':(exclude)*.ts'I found the problem! Yay, Git!More info:- Git pathspec- Bash "Event Designators" (the upcaret to search and replace previous line)
Read More

NPM package tip

Dev
FYI the only way to install and save Node/Javascript libraries is like this:npm install mongoose@"~4.0.8" --saveThis installs anything 4.0.x — major and minor are respected.  The option “@4.0.8” would be only 4.0.8, which is fine, but probably too picky.  The default “^4.0" is terrible and will install anything with major version 4. Don’t ever do this.  It’s also the default. Programmer emptor.For Python, I generally do "pip install package" to install the latest version of a module, then "pip freeze mypackage >> requirements.txt" to capture the exact version number.
Read More

workshop: Zero to Webapp in Python

Dev
(This page is http://bit.ly/jta-webapp)Slides: Google SlidesWorkflowy outlineGitHub source codeWeb development is amazingly complicated. Can we get away with not learning that much?  John will do an interactive workshop on how to start with nothing and wind up with a working web app!GoalStudents will learn how to write a tiny webapp without their brains exploding. NON-GOAL: we'll simplify the stack to exclude many things required for real projects. We won't cover classes, many types, modules, the debugger, nor testing. We might get to packages, virtual environments, and 3rd party modules like Flask. The intent of this workshop is to get something working with a minimum of magic.Technologies:* Python!* HTML+CSS (a tiny amount)* Bash shell (a little)The goal is to deliver a simple working webapp, then refine it with small changes, each…
Read More

practical use of GNU Makefile for testing

Dev, Testing
Recently I reviewed a bunch of code from students. The projects are in Java. To test each one, we:compile Java source file into class filerun the class fileexamine outputOver time I'll be getting more code, thus I want a simple workflow that will automatically adapt to having more source in the same directory. I can't just list all the source files then write out a Bash script to compile + run everything, for example. As new code comes in, I want it to be included in the overall test run.How would you accomplish this?GNU Make is the bombI used my good old friend, GNU Make! This simple tool runs programs to turn files into other files. It understands dependencies, and won't do work unless it needs to. For example, it'll compile…
Read More