iowait on Linux

This article may come in handy. For some reason, I’ve had a number of requests over the past week about iowait on Linux.

The big problem is that people don’t seem to understand the relationship between iowait and load. They see a very high load and very few processes and they can’t figure out why the load is so high.

When running ‘top’, look to the iowait percent number. If it’s high, the CPU is spinning its tires waiting on io. The referenced article can tell you how to find the culprit. You then need to figure out how to smooth out the bottleneck. If it’s temporary, you may have to just ride it out. If it’s a permanent process, such as MySQL, you may have to redesign your disk.

Linux, netbooks, Intel and Moblin

I have a netbook. A Lenovo S10, as a matter of fact. My S10 is split into two partitions–one running Windows 7, the other various flavours of Linux.

A netbook is a pretty useful little device. It’s small enough that it can go pretty much anywhere. It’s just big enough to get work done, within a pretty narrow definition of “work”. As a net device, it’s excellent.

The biggest shortcoming of these devices is the screen size–more specifically, the resolution of the screen. 1024×600 is just barely enough to run a web browser in full screen mode (and I recommend Chrome in kiosk mode for this). Multiple windows? I don’t think so. More than 1 or 2 windows introduces a clutter that’s difficult to manage, especially if you’re using the trackpad rather than a mouse (it’s trivial to grab a window and flick it out of the way with a mouse–not so trivial with a trackpad).

In order to combat issues with screen size, I think we need to rethink the user interface. In Windows 7, I move the task bar to be vertical on the left and autohiding, and that helps a lot. It’s still not ideal, though, because the windows are still windows and still have the same properties that make them ideal for larger displays but not so much for smaller ones.

Several Linux distributions attempt to resolve this by modifying the UI elements to optimize them for the small screen. Jolicloud and Ubuntu Netbook Remix are two examples. Both are very nice Linux distros, both are optimized for netbooks, including the display. Both install and run on my S10 with no issues. The way they manage the “window management” issue is simple–each window is essentially a full screen, and you switch between windows as you switch between applications. You don’t ever (or hardly ever) see multiple windows on the screen, though you certainly do have multiple applications.

Which brings us to Moblin. Moblin rethinks the whole netbook package, not just the UI. UI changes are similar to other netbook Linux distros–one window, one screen. But the rest of the OS shows how much can be done when you go all the way back to the roots of the OS. On my S10, Moblin goes from power off to fully functioning in 30 seconds. Battery life on Moblin is better than Windows 7. Screen effects are crisp and sharp. Moblin takes advantage of every hardware tweak netbooks have to offer, and works around most of the hardware compromises. It’s crisp and clean and, after a little tweaking, ideal for my needs.

Moblin can optimize netbook hardware so well because it’s very foundation is created by folks at Intel. These are folks that have intimate knowledge of the Atom processor and the Intel GMA video chips that make up most notebooks. These folks know how to wring every drop of performance out of their hardware, which is as it should be.

Sadly, though, they cannot create a Linux distro. Sure, Moblin rocks. It’s very pretty and it works very well on my S10. But that’s after two days of dicking around with it in ways that your average consumer won’t, and shouldn’t have to, do.

First I had to hardwire my net connection so that I could download, build and install the Broadcomm drivers for the wifi card in the S10. Why? Moblin ships without support for these drivers because they are proprietary. That’s a decision the Moblin team made and I respect that. But there’s no reason why they couldn’t make a package available for people to install, rather than forcing them through hoops. Ubuntu does this with, for example, proprietary nVidia drivers.

Next, in order to get media support, I had to download and install gstreamer and it’s codecs. From source. This is unacceptable. The Linux Foundation knows that people listen to MP3′s. Again, there are licensing issues that make it difficult to include the codec support in the distro, but, again, organizations like Ubuntu have worked around this. Making Joe Consumer download and compile source code is cutting off your nose to spite your face.

If I install Jolicloud or the Ubuntu Netbook Remix, I get a similar experience to Moblin. But I’m good to go out of the box–there’s no need to hook up a wired connection to get wireless working. There’s no downloading and compiling source code for functionality that should be there from the get go.

I still think that, so far, Moblin is a better experience. I like the way app switching is done, for example: alt-tab shows all open windows in a miniature but live view. I can take a look at a mini version of a terminal and see if the job that was running has completed. Release alt-tab and I’m back where I started.

I respect Intel for getting into Moblin. The assistance they bring to the Linux community for optimizing their hardware is invaluable. I hope that other organizations, like Canonical, take advantage of this. I also hope that Moblin can grow into something much more consumer friendly and Linux-newbie ready. They’re at version 2.1 now. Perhaps with version 3…

On wizards…

Ocassionally, I like to pooh-pooh Windows admins and their “wizards”. Relying on wizards to get work done can, in my humble opinion, stunt your growth as an admin. You should know how to use the command line and edit config files by hand. The day your wizards fail you is the day you regret that lack of knowledge.

So. Today I went to install WordPress on a FusedNetwork account and the installation wizard failed me. It turns out the wizard relies on DNS working the way it should, but at this point we’ve not yet pointed the existing domain to the new site. We need to test before we do that. Completely forgetting the above, I emailed the support guys at Fused. They, of course, reminded me politely that a manual install would suffice.

I fell into my own trap.

An hour later, WordPress was installed and we’re ready to go with site testing. WordPress’s famous “5 Minute Install” takes an hour when you’re bouncing things off a client machine in Viet Nam. If I had full net access with decent bandwidth, it probably would have been pretty close to 5 minutes.

As an aside, FusedNetwork rocks. Their support guys are great. They are incredibly responsive. They’re not the cheapest hosts, but their customer service is what makes them stand out. It is, truly, second to none.

Dynamic vs. Static web pages

A discussion tonight about dynamic vs. static web pages made me think about how some people believe dynamic web pages are inherently “better” than static. Often, dynamic web pages are prettier, but they are not necessarily better.

Take Bing, Microsoft’s search engine, for example. It dynamically render’s beautiful images on the background of the page. It looks terrific, but it also takes about 5 times longer to load than Google.

If I want to search the net, I don’t need pretty pictures. I don’t need to see great background images. I need a search box, ang Google gives me that as fast as it can.

Sites like Bing and Google actually blur the distinction between dynamic and static–parts of Google’s home page, for example, are static and parts are dynamic. But really, the discussion here is not about static vs. dynamic, it’s about user interface design.

Make your page static. Make it dynamic. Either way–it just doesn’t matter. What matters is something called User Experience. UI design is one element of that. Site performance is another. If your User Experience is crappy, your users don’t come back.

And that’s what you should be concerned about.

Note to FSF: Nobody cares…

The Free Software Foundation is running a campaign highlighting the “evils” of Windows 7. They highlight 7 deadly sins (of course) and they really make a compelling case against Windows 7. It’s a major shift for the FSF–they usually don’t get all political at this level.

Here’s the bad news: nobody cares.

Corporations that have adopted Windows and Office aren’t about to jump ship to Linux or the ever-elusive “GNU Operating System”. Too much has been invested in infrastructure and training. License fees have been paid. Sys admins have been hired and trained. That ship has sailed.

Small business may switch, but not based on this political action. That decision will be based on one thing and one thing only: impact to the bottom line. Any small business owner that makes this kind of decision without considering the bottom line isn’t going to be in business long.

I am a firm advocate and supporter of Linux. I use Linux every day in my job, both on the desktop and on servers. I have Windows machines, and I own a copy of Office. Why? Because it’s what my clients want. No matter how much OpenOffice touts itself as “Office compatible” it’s not 100% compatible. The differences may be minor and subtle, but when I create a report for a client and the formatting is messy because of the differences, it reflects badly on me. Not on OpenOffice. My client doesn’t care how I create the document. They only care that they can open the document in Word.

When a client asks for full functionality in Outlook, I can’t just tell them they should be using Thunderbird and Lightning. They’ve made a decision at the desktop level to use Outlook and I have to respect that. Now that they’ve made that decision, what’s the best way to give them the infrastructure to support it? Exchange must be considered because, while it may not be the cheapest or easiest answer, it is still arguably the best way to provide full functionality to the Outlook client. Linux based “free” alternatives to Exchange are still largely a variety of different services that have been integrated into a unified platform. If you can get them tweaked out properly, they work and work well. But they require a support infrastructure that’s often foreign to folks like my clients.

It could be argued that situations like the one above support the FSF’s contention that Microsoft locks users in and violates standards. They’re right, of course, but again the decision to go down that road has been made and altering course is often challenging and not cost effective.

I admire and respect the work of the Free Software Foundation (except for the whole GNU/Linux thing, but that’s a different rant). I wish them luck in changing the world, but trying to alter the course of ocean liners isn’t the easiest thing to do and may not be the best way to spend resources.

Next up: the kitchen sink!

For all you Unix geeks:

Yes, Android can do that!

People don’t want free beer…

The Free and Open Source Software (FOSS) community defines two types of freedom:

1. free as in speech
2. free as in beer

When consulting, keep in mind that your customer doesn’t want free beer and probably doesn’t care about free speech. Your job is to downplay the free beer aspect and educate about the free speech aspect.

Why not free beer? Free beer is fine, but giving away anything else of value, like software, plays to our “you get what you pay for” mentality. That’s a dichotomy that’s not easily broken, especially in the business world.

When undertaking a new project, smart business people will set out a budget and work diligently to keep the project within that budget. If I come in and bid a FOSS-based solution, I don’t start by heralding the fact that the software is free. Software, even free-as-in-beer software, is never free. Even if the code itself is free, there’s still the need for building an infrastructure to run the code, installing and configuring the code, and ongoing maintenance of the code. It would be irresponsible to even hint that there’s no associated cost with the software. Breaking the software out from all the other pieces and attaching a $0 price tag to it is disingenuous.

So what to do?

For starters, calculate the cost of the software. Separate the parts of building out the infrastructure, installation and configuration, etc. as the software cost. Add the ongoing maintenance costs as just that—ongoing maintenance. Make your consulting fees about consulting: requirements analysis, recommendations, project management, usability and acceptance testing.

Second, highlight the free-as-in-speech element of FOSS-based solutions:

1. no vendor lock in
2. support for open standards
3. community development
4. availability of source code

Each of these elements is important, but they can mean nothing to the client without your explanations. Educating your client is part of your job. Don’t teach them to expect something for nothing, or nothing is what they’ll offer you.

By highlighting free-as-in-beer, you run the risk of having the client think: “Hmmm… I could have my bosses nephew install that. The software’s free. What could go wrong?” Down that road lies madness for the client, but they often don’t know that. They only see the chance to save the money they had budgeted for the project.

When performing the job, you have to differentiate yourself. Illustrate why simply having the nephew do it doesn’t make sense. Show that you will complete all localizations, including branding. Don’t show screen shots of the raw software—show screenshots after the localization and branding is complete. Even if you have to mock these screenshots up, there impact will be much greater. Demonstrate the ways in which you’ll document the setup and use of the software. Have a stock acceptance test plan that you can show the client. Keep your toolbox full of little things like this. These are the things that separate the highreasonably priced consultant from the bosses nephew.

Well. These things and years of experience…

Jaunty Jackalope

The latest version of Ubuntu has hit, and lucky for me FTP Telecom in Viet Nam is mirroring. This means I was able to download and install in a pretty reasonable time.

How’s it look? So far, it’s rock solid. It recognized all of my hardware (Thinkpad T61 with nVIDIA card) out of the box. The install was painless–even resizing my Windows 7 partition worked. I’ve changed my Appearance settings to Dust as I prefer the darker widget look. It tends to focus my attention on the large fields I need to fill out… The look and feel are typical Ubuntu–very polished and smooth.

Of course, I did have to do a bunch of things like install the Adobe Flash plugin and decoders for MP3. Silly that, in this day, we still need to do this. I understand the reasoning but surely there’s some detente that can be reached.

Anyway, it’s great to be back in a pure Linux environment. When administering Linux systems, a Linux desktop really is the best tool for the job, at least for me. As good as tools like PuTTY are, they can’t compete with the efficiency I gain from just running at the command line.

Two big thumbs up to the Ubuntu crew for getting yet another release out the door.

Disk Space or DNS

Whenever I have a client call me up to tell me that a service that’s been working fine for ages has suddenly stopped, two prime culprits immediately spring to mind. One is very subtle, the other not so much.

When first logging in to the afflicted system, I do `df -h` to check available disk space on all volumes. It’s amazing how many of these incidents come down to a disk running out of space. Usually, a monitoring system warns when disk space hits 90%. Sometimes, though, that last 10% can be churned through very quickly, catching even the monitoring service off guard. In a number of cases, I’ve had monitoring systems miss full disk conditions because the full disk condition stopped the agent from accurately reporting the current state (a topic for another entry). Bottom line: check your disk space.

DNS issues are much more subtle and can manifest themselves in different ways. Often, you’ll see a sudden decrease in performance for some inexplicable reason. Things that used to work suddenly break. In many cases, this is because systems that refer to hosts by name can either no longer find those hosts or there’s a sudden delay in the process.

The `host` command is your friend here:

chris@utc-pdc1:~$ host www.dollmont.net
www.dollmont.net is an alias for dollmont.net.
dollmont.net has address 216.86.156.11

This tells me that www.dollmont.net points to dollmont.net, which in turn has an ip address of 216.86.156.11.

chris@utc-pdc1:~$ host 216.86.156.11
11.156.86.216.in-addr.arpa domain name pointer server1.fusednetwork.com.

Doing a `host` command on the IP address will tell me if reverse DNS is working. In this case, it’s pointing me at a Fused Network server, which is correct—Fused Networks does my site hosting. In a perfect world, I would have a DNS PTR record for the IP address that pointed back to dollmont.net, but this is more than good enough. Forward and reverse DNS both resolve.

If there’s an error in either one of these checks, it means that DNS is broken somewhere down the line. Trouble shooting that can be a nightmare, and is beyond the scope of this article.

Disk space and DNS are the first two things you should look at whenever things start behaving in a wildly different manner. Often the problem will be one or the other and usually involves a quick fix.

The perils of bit rot…

Bit rot is a concept familiar to a lot of people. Usually, it refers to the slow degradation of data storage. Magnetic devices, over time, can lose their charge in places that can cause data holes to appear. Lose a few bits from a file and the file can become useless.

But there’s another form of bit rot that just as insidious and can be far more dangerous. It’s the bit rot that comes from keeping old systems on line and in production with no plan for an upgrade path. Let enough time pass and it could become impossible to upgrade the system.

I have a client that has a production system running an older version of Linux. The version of this particular distro is so old that file repositories are no longer maintained and security patches are no longer offered. Furthermore, this particular server is running a commercial piece of software that has been end-of-life’d by the vendor. Not only does the vendor no longer support the software, it’s impossible to get distribution media for the software. If this system were to blow up somehow, the client is basically screwed. We cannot reload the OS because we no longer have the original install media. We cannot reload the application for the same reason. The vendor might offer an upgrade to the software or we could just pay for the package all over again, but there’s no guarantee that the latest version will be backward-compatible with the existing system. Like I said—screwed.

The good news in this case is that we were able to uplift the entire system into a virtual machine. We can now back up the virtual machine image. In a worst case scenario, we can recover from backup and restore the virtual machine to be the same as the current hardware system. The process of uplifting such an old system was not trivial, and we were lucky that we were able to get it done. The client is now “protected” much more than he would have been had he continued down the same path.

Whenever you put a system in place, you need to plan for the eventual upgrade of that system. The reality is that software, and especially operating systems like Linux, matures. You need to plan for that. Distribution choice matters. Any of the Ubuntu LTS distributions are guaranteed to be supported for 5 years. Commercial distributions like RedHat and their Open derivatives like CentOS have long term support options available, too. Using Open Source tools makes the software path more favourable to upgrade.

Don’t let systemic bit rot happen to you…