The Cog

That Little Bit of Knowledge That Makes Everything Work

Ubuntu 16.04 Runs Automatic Updates Whether You Say So or Not in the Installer

After selecting “No automatic updates” in the Ubuntu 16.04 server installer, I expected that the system would not attempt to upgrade anything without my permission, just like every version before. Turns out that is not the case. I found out the hard way when my production database shut down to upgrade in the middle of the day.

You can turn it off totally by editing the following lines in /etc/apt/apt.conf.d/10periodic:

APT::Periodic::Download-Upgradeable-Packages “0”;
APT::Periodic::Download-Upgradeable-Packages-Debdelta “0”;
APT::Periodic::AutocleanInterval “0”;

And editing this line in /etc/apt/apt.conf.d/20auto-upgrades:

APT::Periodic::Unattended-Upgrade “0”;

Install zfs-auto-snapshot on Ubuntu 16.04 LTS/Debian

zfs-auto-snapshot not being available for 16.04 was not a situation I thought I would run into, yet here we are, disappointed by Canonical once again.

In the meantime, I have ported the package from Trusty (14.04.5) to work on Xenial (16.04.1) and Debian Jessie.

You can download the Ubuntu 16.04 package here, and install it by running:
# dpkg -i zfs-auto-snapshot-trustyport.deb

You can download the Debian package here, and install it by running:
# dpkg -i zfs-auto-snapshot-debian.deb

(edit: I borked the permissions on the files in the Ubuntu .deb. You will need to chown them after install. The Debian package is correct.)

Romaco Radio EP4 – How to Get Started With Open Source

A bit of information as to how to get your foot in the door on an open source project.

Romaco Radio EP4 – How to Get Started With Open Source [26:15]

Romaco Radio EP3 – Why HD Audio is Useless

A little technical explanation about why 24bit/96kHz audio isn’t all it’s talked up to be.

Romaco Radio EP3 – Why HD Audio is Useless [34:02]

Romaco Radio EP2 – How I Got Started with Microcontrollers

A tale of the decisions I made in choosing what microcontroller architecture to learn first and how I got started with embedded programming.

Romaco Radio EP2 – How I Got Started with Microcontrollers [22:16]

A Tale of ZFS’ Success

ZFS is great. We hear it all the time, but you hope you will never need to utilize any of it’s advanced data-protecting features. I recently went through a series of events which makes me truly grateful for ZFS. This is a real world example that would have gone horribly different if I wasn’t using ZFS.

It’s Wednesday April 29th, 2015. I get home from work at about 6pm, fire up my workstation and tend to some emails. I look down and see the HDD LED on my file server, tesla, is solidly lit. All normal; today is a Wednesday and tesla started a scheduled ZFS scrub at 2am, which probably won’t complete until Thursday morning. I SSH in to check to see it’s progress and notice something new in the output of zpool status (serial numbers have been redacted in all the screenshots):

Screenshot from 2015-04-29 22:38:28-2

Some data was silently corrupted and repaired on one disk. To some this might not seem very odd, this happens a lot on some systems, but in over 2 years with this server, I’ve never seen this before. I immediately checked the SMART on the disk with corrupt data and found that 36 sectors were unrecoverably errored and the disk had 1091 pending sectors for reallocation. I was a little concerned, but in the past I’ve had several disks in other machines get a few bad sectors without any further issues for the rest of their lives.

I monitored the disk over the next few days and watched as the stats got worse:

  • 04/30/2015, reallocated sector count: 592, pending reallocation count: 7640.
  • 05/02/2015, reallocated sector count: 592, pending reallocation count: 7744.
  • 05/04/2015, reallocated sector count: 592, pending reallocation count: 7816.
  • 05/05/2015, reallocated sector count: 600, pending reallocation count: 7848.

My first thought was that this could be one of two cases. The first is that there is a bad portion of the disk which is slowly getting flushed out. The second is that there is a mechanical failure which is manifesting itself as read errors. I made a simple decision; if the disk has more serious issues during next week’s scrub, I will replace the disk. I had gone out earlier that week and bought a new disk because they were on sale, so I could replace the hot spare if need be.

Wednesday comes along and after work I come home to find this:

Screenshot from 2015-05-06 17:29:10r

Well, I guess my decision is made. Time to replace the drive now and investigate later. I replaced the drive with the hot spare and the resilver began.

Screenshot from 2015-05-06 18:26:06r


Screenshot from 2015-05-07 18:26:01r

After detaching the bad drive, the pool looks good again.

Screenshot from 2015-05-07 18:26:10r

The job isn’t over yet though. Time to replace the bad drive with a new one to act as a hot spare. Doing this requires shutting down the server because the case does not have hot-swap bays. After replacing the drive and logging in, I’m faced with a terrifying sight. I don’t have a screenshot of this one because I was a little in shock. 3 vdevs were unavailable, and the pool was degraded. The drives were on different controllers, so the only thing I could think of was that some cables had come loose in the drive replacement. So I promptly shut down the server and opened it up again. On one drive, the power connector was slightly askew, but the other 2 had no signs of issue, so I just unplugged and plugged them back in at both ends.

At this point I was unsure of what would happen when I rebooted the server. Would ZFS be able to reassemble the pool now that the vdevs were reattached? Would I have to resilver the entire pool again? The only way to know was to power it up.

Screenshot from 2015-05-10 15:05:36r

Now that’s impressive. ZFS resilvered only the changed data and the pool is back online again.

This series of events isn’t that uncommon, but the outcome would have likely been very different if I was not using ZFS. First of all, without ZFS I likely would not have been able to detect the silent corruption on the disk when I did. I would have eventually noticed as there were errors in the kernel log and SMART logs, but that could have been days or weeks later, after more damage had been done. Even if I knew there were errors, other implementations of RAID wouldn’t have been able to correct them. My best bet would have been to remove the disk entirely and rebuild the array from parity and hope that the parity was valid. If upon replacing the bad drive the cables had come loose as they did in my adventure, the array would most likely be destroyed. Most RAID implementations will want to resync a disk completely upon it disappearing for any period of time. The loss of 3 drives would mean that almost any RAID implementation would not have enough data to rebuild.

Experimental Differences in Audio Compression Formats

I think everyone knows that lossy audio compression formats in the likes of MP3 and AAC sacrifice audio quality for a smaller file size. Some people like myself can distinguish between lossless and lossy copies of the same song just by ear, but most cannot, or simply do not know what to listen for.

You can easily take a diff of 2 text files and see the differences, so why can’t you do that with audio?

For my experiment, I began with Jillette Johnson’s song “Torpedo” stored as lossless 16bit/44.1kHz CD audio in the FLAC format. For copyright reasons, I won’t provide a download link to the file, but the Internet is a thing, so you can find it if you want.
I began by transcoding this original into several different formats with varying parameters:

  1. 320kbps CBR MP3 – 10.9MB
  2. 256kbps CBR MP3 – 9.1MB
  3. 256kbps VBR MP3 – 8.4MB
  4. 128kbps CBR MP3 – 5.6MB
  5. 256kbps CBR AAC – 5.4MB
  6. 500kbps VBR OGG – 15.5MB
  7. 256kbps VBR OGG – 9.6MB

CBR stands for constant bitrate, and VBR is variable bitrate.

I took one channel from each converted file and the original FLAC file and aligned them to the sample. I then took the difference of the 2 waveforms. This resulting difference is the error induced by the compression algorithm. I saved 3 data points from each comparison: the waveform as an image, the frequency spectrum versus amplitude plot, and a rendered audio file of the error. The waveforms plot the relative amplitude at each moment in time from 0 to 1. The frequency spectrums show the average amplitude in decibels (dB) of each frequency from 0Hz to 22,050Hz (the maximum of CD audio) over the entire signal. 0dB is the highest amplitude that can be stored in the audio file. All amplitudes are taken relative to this maximum and are expressed in negative decibels, so the more negative the amplitude, the quieter is is. I have chosen to render the error signals as uncompressed WAV files in order to accurately represent them in their entirety and still allow them to be played in a web browser (most don’t have FLAC support yet). I have not amplified any of the error signals to keep everything as accurate as possible. If you intend to listen along, try seeking the track to the times listed along the top of the waveform images to hear how different parts of the song sound, like the chorus at 1:40. Also note that I’m not really interested in the exact error versus filesize figures, so feel free to figure them out on your own.

Just a disclaimer; I am not an audio engineer and these tests are in no way scientifically accurate. This is just a home-grown experiment.

Let’s take a look at the 320kbps CBR MP3 first. You can click any image to enlarge it in another tab.

320kbps CBR MP3 Error

320kbps CBR MP3 Error

320 CBR MP3 Spectrum

320 CBR MP3 Spectrum

Listen to the 320kbps CBR MP3 Error

As you can see (and hear), the error is clearly there. If you know the song, you can probably even make out lyrics. The first thing to note is that the highest amplitude error is in the high frequencies, specifically above 16kHz. This is not surprising, because this is how MP3 was designed. MP3 takes in to account how we perceive sound. Frequencies in the range of the human voice for example are most important, and high frequency content which is for some people not even audible at all, is least important. The highest error amplitude peaks at -53dB with the average around -61dB.

In order to make some comparisons, let’s take a look at 256kbps CBR MP3.


256kbps CBR MP3 Error

256kbps CBR MP3 Error

256kbps CBR MP3 Spectrum

256kbps CBR MP3 Spectrum

Listen to the 256kbps CBR MP3 Error

Notice that the highest peak is at -52dB, not much higher than the 320kbps CBR file, however the other frequencies are significantly higher, by about 3 to 4 dB. Remember that dB is a logarithmic scale, so an increase by 3dB is roughly 10 times louder.

I have heard that VBR is “better” than CBR, so I decided to test that as well.

256kbps VBR MP3 Error

256kbps VBR MP3 Error

256kbps VBR MP3 Spectrum

256kbps VBR MP3 Spectrum

Listen to the 256kbps VBR MP3 Error

Going from 256kbps CBR to VBR, the spectrum is very similar with the exception of the high frequencies. The peak that was very prominent at 320kbps CBR, and slightly less at 256kbps CBR, is now actually lower in amplitude than the lower frequencies. At a glance it looks like the error has actually decreased, but look at the scale. The overall error has actually increased by 1 to 2dB. Now given that the file size is 700kB lower, it’s debatable whether VBR is “better”.

Just to round out this part of the experiment, I tested 128kpbs CBR MP3 as well.

128kbps CBR MP3 Error

128kbps CBR MP3 Error

128kbps CBR MP3 Spectrum

128kbps CBR MP3 Spectrum

Listen to the 128kbps CBR MP3 Error

The average error is a whopping 10dB higher than the 256kbps CBR, and follows an almost linear drop as the frequency increases. This is consistent with the gradual drop off of average amplitude in the original file, so statistically that is expected. At this point the song is becoming scarily audible; making out lyrics is not difficult at all. Just remember that what you hear in the error signals is what is missing from the transcoded file.

Who says that MP3 is the only format used for lossy compression? I thought I’d start off with the lesser known OGG Vorbis audio codec. The following is the result of transcoding at 256kbps VBR.

256kbps VBR OGG Error

256kbps VBR OGG Error

256kbps VBR OGG Spectrum

256kbps VBR OGG Spectrum

Listen to the 256kbps VBR OGG Error

This is quite different than what we saw with the MP3 spectrum, and it sounds very different too. The OGG codec has higher error at low frequencies, which is easier to hear than see, however the overall average error is less than that of MP3 over the rest of the spectrum, by about 2dB. Being 1.2MB larger in size than the 256kbps VBR MP3, this should be expected.

Unlike MP3, OGG is not limited to 320kbps. I ran the test at a bitrate of 500kbps VBR.

500kbps VBR OGG Error

500kbps VBR OGG Error

500kbps VBR OGG Spectrum

500kbps VBR OGG Spectrum

Listen to the 500kbps VBR OGG Error

The filesize is 5.9MB larger than the 256kbps VBR OGG file, but the error is a massive 15dB lower on average. Surprisingly, the error at the low end is still worse than that of 320kbps CBR MP3. Also note that just because the spectrum plot bottoms out around 17kHz does not mean that there is no error there, it is just at -82dB, which is near the edge of the zoomed in area.

The last file type I compared, is 256kbps CBR AAC, the same format used by Apple for most iTunes downloads.

256kbps CBR AAC Error

256kbps CBR AAC Error

256kbps CBR AAC Spectrum

256kbps CBR AAC Spectrum

Listen to the 256kbps CBR AAC Error

I was kind of horrified at how poor the output was. The error is only about 2dB less than the 128kbps CBR MP3 file. The only saving grace for AAC is that the resulting filesize was 5.4MB, 200kB less than the 128kbps CBR MP3. The error also sounds very different than the other formats.

A common scenario is the conversion of one lossy format to another. In my library, all files are either FLAC or MP3, so when purchasing music off iTunes for example, it would make sense for me to transcode the files to 320kbps CBR MP3.

256kbps CBR AAC to 320kbps CBR MP3 Error Difference

256kbps CBR AAC to 320kbps CBR MP3 Difference Spectrum

256kbps CBR AAC to 320kbps CBR MP3 Difference Spectrum

Listen to the 256kbps CBR AAC to 320kbps CBR MP3 Error

The error doesn’t seem that bad, however, remember that this is the error from the second transcode. This error compounds on the error from the original transcode. The following is a diff taken from the original FLAC to the new MP3.

256kbps CBR AAC to 320kbps CBR MP3 Error Difference from FLAC

256kbps CBR AAC to 320kbps CBR MP3 Error Difference from FLAC

256kbps CBR AAC to 320kbps CBR MP3 Error Difference from FLAC

256kbps CBR AAC to 320kbps CBR MP3 Error Difference from FLAC

Listen to the 256kbps CBR AAC to 320kbps CBR MP3 versus FLAC Error

The total error is almost no different from the original AAC. This is not surprising, as the relative amplitude of the new error in the AAC to MP3 conversion is several orders of magnitude lower than the original error in the AAC file.

In a future post I’ll explain more about errors in audio reproduction more mathematically, including quantization error and bit depth, sample rate, and noise floors. Until then, I hope this was interesting and provided a little more insight into audio compression quality.

Fix NFS Stalls on Linux with Fast Networking (Like InfiniBand)

For the last 6 months, I’ve had the most frustrating issue with my 20Gbps Infiniband home network. Copying files from my file server would ultimately cause NFS to lockup for an indeterminate period of time. During that time, which could be anywhere from 3 seconds to 3 hours, there is no traffic on the IB link whatsoever. I spent several weekends chasing red herrings – everything from hardware issues like bad cables and overheating HCAs, to needless kernel driver changes and firmware upgrades. From my testing I determined some interesting properties of the failure mode:

  • The failure was not limited to just NFS, it also happened with Samba (SMB/CIFS) and would even affect iperf.
  • When the connection was stalled, the IB link still works. I could SSH over it just fine.
  • The issue only affected the outgoing direction of iperf from my workstation. It could receive fine, just not send.
  • When NFS stalls, it takes 200% CPU. When anything else stalls, it uses no extra CPU.
  • The last packet sent from the client is always an ACK. TCP keepalive packets are sent every 60 seconds while the connection hangs. The transfer(s) are otherwise normal, with the only exception being the huge delay in between packets. (I’m using TCP rather than RDMA due to a kernel bug affecting 3.15 and earlier.)

I was convinced that this was a kernel bug, and my suspicions were confirmed, but not in the way I’d expected. It turns out that the issue is caused by the Linux kernel swapper daemon, kswapd. I noticed that every time NFS would lockup, the CPU would spike to 200%. One of those processes was a kworker, the other was kswapd.

Wait a second. I don’t have any swap.

This immediately struck me as odd. What would the kernel be trying to swap? I was only using 50% of 16GiB of RAM, so there shouldn’t be any issues regarding OOM conditions. A quick Google search returned some interesting info. Apparently this has been a known issue for a while, and according to some, kswapd is in need of a rewrite for modern systems.

From what I understand, during the file transfer(s), some data is cached in RAM as usual. As the transfers progress, the RAM slowly fills to capacity with cached data. I’m also running ZFS, so that can’t help with the caching, as its ARC will try to use 7/8th of the available memory for itself. At some point, kswapd scans through all the memory looking for what to evict to disk. This process takes 100% CPU and blocks NFS while it does… something?

This leaves the question as to what we can do. The short answer is that there isn’t a proper fix at the moment of writing this, but there are a few workarounds.

The first is to set vm.swappiness = 0 to reduce the likelihood that kswapd will want to ruin your day. This change will not solve the issue on its own, but it does seem (at least in my case) to reduce the severity. You can set this parameter on the fly by running sudo sysctl -w vm.swappiness=0, and you can make it permanent by adding vm.swappiness=0 to /etc/sysctl.conf.

The real workaround is to clear the cache completely. This frees up a massive amount of memory and immediately removes anything for kswapd to do. You can clear the cache by running as root: echo 3 > /proc/sys/vm/drop_caches. My solution was to create a bash script which runs that command once every 10 minutes. In my cases that is enough to prevent kswapd from ever doing anything, and my NFS transfers running. I just start the script and keep it running while I’m doing intensive transfers. I kill it when I don’t need it, as there will be an increase in I/O in general, it is purging cache after all.

I hope that this issue gets addressed soon in the mainline kernel.

Romaco Radio EP1 – ZFS Intro

I was supposed to start a podcast long ago, and never got around to it. Thought I’d start off the new year by getting things rolling with a completely off the cuff impromptu intro to ZFS.

Future podcasts will most likely follow some sort of schedule, and will include guests and real audio production.

Romaco Radio EP1 – ZFS Intro [46:37]

Fix Volume Control Jitter in Ubuntu

A while ago when I switched from GNOME Shell to Unity, the first thing to bug me was the volume control in the top panel. When I would drag it side to side, it wouldn’t slide smoothly, instead it would jitter and jump back and forwards. This made it almost impossible to select the correct volume. I was convinced that this was an issue with Unity as I never had such an issue with GNOME or KDE.

As it turns out, by random prodding, that the volumes for each channel of audio were different by 3%. By installing pavucontrol and setting the levels equal in pulseaudio, the slider works correctly now. It seems that Unity is not equipped to handle unbalanced channels. The stranger thing is why they were unbalanced in the first place. This isn’t the first time I’ve had to set the balance on startup, but in all the other cases, the balance was usually something obvious like 95% to one side or the other. My audio card is a modified ASUS Xonar Essence STX on which Ubuntu uses the Virtuoso 100 driver by default. Driver bug? Maybe.