13Jan 2012 by johntellsall

Ssh tunnels are great

Dev, DevOps
At work I'm connecting to multiple Munin statistics servers. On my lappytop I'm only running one server, so how do I get multiple sets of results?  Answer: create a tunnel to another server!The Munin protocol is extremely easy, send "fetch X" to get statistics on X.  In my example df=Disk File usage.  Here's how to get local information, via Munin running locally on port 4949.$ echo 'fetch df' | nc -q1 localhost 4949# munin node at freebeer_dev_sda1.value 5.29318865322941_dev.value 0.0339391645617528_dev_shm.value 0.378827479751794_var_run.value 0.00922469512382616_var_lock.value 0.Here's how to make a remote machine's Munin (on port 4949) show up on localhost (port 4950). This means we can scan multiple local ports to get information on many different machines.ssh -fNL localport/localhost/remoteport remotehostOption "-f" means drop into the background after asking for a password.  Next option "-N"…
Read More
18May 2011 by johntellsall

I salute Fabrice Bellard

General
Recently the amazing hack of "boot Linux in a web page" has been circulating around. Bellard wrote a x86 emulator in JavaScript, then took the straight Linux binaries and was able to boot it. In a web page. At reasonable speeds. His distribution even includes a C compiler. This is amazing, but that's not all.Bellard wrote the emulator QEMU, which lets you run one set of software "inside" another. It's mostly used for virtual machines, so you can run multiple operating systems inside a "host" OS. The guests can run Linux or Windows or whatever. The guest virtual machines can even be of different CPU types: ARM on a x86! This can be incredibly useful. But that's not all.A large majority of video software uses the "ffmpeg" library. It's quite…
Read More
16Nov 2009 by johntellsall

Superdevelopers at Facebook

General
Facebook gets unprecedented traffic. They don't want things to go down so they... release changes constantly, a few times a day! They use peer review and their immense audience to see how well their solutions work, then iterate. The other teams *support* engineering, rather than being gatekeepers.I actually did this at a medium-sized site many years ago. Push change to live, capture logs for five minutes while people beat it, then undo the change. It worked well.
Read More
10Nov 2009 by johntellsall

ntop and RRD

DevOps
"ntop" conveniently displays web pages with charts and plots of your local network traffic. To record historical traffic, it uses Round Robin Database (rrd). The default ntop package enables the RRD plugin, but doesn't give itself permission to write out its data. Here's how to get the two to work:sudo apt-get install ntopsudo install -d /var/lib/ntopsudo install -o nobody -d /var/lib/ntop/interfacessudo install -o nobody -d /var/lib/ntop/rrdTo test, start "sudo ntop" (probably in "screen"), then open the web page http://localhost:3000To activate the RRD plugin, select Plugins / Round Robin / Describe, then click the red "no" to become a blue "yes." You'll see messages in the terminal where you started ntop.After a few moments, you can view RRD graphs. One is in Summary / Network Load.
Read More
09Oct 2009 by johntellsall

Empirical Data behind Software Development Techniques

Dev
Our friends at Microsoft Research have published four papers testing--and measuring--development ideas. One big surprise was that a company's organization "can predict software failure-proneness with a precision and recall of 85 percent. " It's more important than using assertions or increasing test coverage!Other results: test coverage isn't a direct measure of software quality, and assertions are useful.I was glad to see some numbers behind what I've experienced, that Test Driven Development increases quality. "What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer." Hmm.Exploding Software-Engineering Myths
Read More
06Oct 2009 by johntellsall

Fast Parallel Downloading (for apt-get) #2

Dev, DevOps
This is how to upgrade your Ubuntu system, downloading several packages in parallel. See previous article for details:cd /var/cache/apt/archives/apt-get -y --print-uris -u dselect-upgrade >debs.listegrep -o -e "http://[^']+" debs.list | xargs -l3 -P5 wget -nvapt-get -u dselect-upgrade
Read More
14Sep 2009 by johntellsall

Ultimate Quality Development System

Dev
Our friends at Divmod have come up with an extremely clear method of rapidly delivering high quality software. The basic idea is you write a bug report, create a code branch, then check in the branch with a "review" tag. When someone reviews and okays your code, you merge into the trunk. This method has lots of benefits, including taking the drama out of peer review, and making it very easy to safely test on multiple platforms. https://opensource.com/article/18/2/uqds (via Twisted)
Read More
19May 2009 by johntellsall

Cheap Network Measurement

Dev, DevOps
You can't control what you can't measure. I want a fast local wireless network, counting only from my workstation to the router. If I change settings or drivers, how can I tell if it's an improvement or not? How can I gauge my network speed, including driver and network settings?There are jillions of tools available*, but they mostly 1) require a specific client on the server, or 2) don't actually send traffic, they passively analyze what's there.Here's a solution: transmit a largish file to the server, where it will ignore the data. Measure the network bandwidth locally using 'cpipe' or another tool.Setup: enable SSH on the router, install "cpipe" locally.cat /boot/vmlinuz-* | cpipe -vt | ssh router 'cat >/dev/null'I get lines like this, with statistics on each of many chunks…
Read More
22Apr 2009 by johntellsall

Server names in Dnsmasq

Dev, DevOps
My domain name service (DNS) is running on a router using the wonderful Tomato firmware. When a computer boots up, it sends its hostname (i.e. "sally") to the router, which records the hostname and returns the corresponding IP ("192.168.7.10"). This is all grand. But, what if different services require a computer to have multiple names? That is, when the Puppet service fires up on my lappytop, it looks for the hostname "puppet" by default. Given the above scheme, a computer has one and only one name; DNS doesn't know that "sally" is also known as "puppet". Here's how to fix that. The router is running Tomato, which uses Dnsmasq as the DNS service. It automatically figures out the normal hostname/IP mapping, for instance "sally" in the above example. We can…
Read More