Tuesday, October 27, 2009

Dear Mr. Shuttleworth, Canonical, and Ubuntu developers.

I am writing this article today in hopes that I may compel you to reconsider your release cycle. There is far more value to users, developers, vendors, and OEMs to know that the software product that you provide to them is stable, and mature. While there is absolutely value in a release cycle I believe that more harm is being done than good with your current process as bugs remain unfixed for extended periods of time and in my opinion the quality of your releases have never been lower.

I have spent many years using your product, and as a proponent of Linux in general, and now I find myself questioning my ability to really say that Ubuntu is a mature stable operating system for the masses. I see bug after bug after bug sitting in launchpad for months on end which directly impact users ability to perform simple functions like turning on bluetooth or WIFI. Problems like this give your operating system black eyes in the form of users spreading word that your operating system is no good, that it doesn't work right, and that it is not ready for prime time. Not only that, but it gives end users a distaste for Linux distributions in general because they don't know any better.

You know, they aren't wrong. They are absolutely right to complain when an end user turns off their WIFI only to find that their system halts with a kernel panic and they have lost anything that they were working on that was unsaved. This level of performance would kill a company like Microsoft or Apple if it was met with the types of answers that we receive from members of your teams whom are responsible for repairing these problems.

Currently users are expected to know how to get to bug reports, file bug reports, and install kernel packages from PPAs to fix or workaround issues. These are not functions for consumers, and until Ubuntu matures to the point where an end user can rest assured that when they suspend their computers that they will be returned to the work they were performing before closing their lids, they will never trust Linux on their consumer systems at home or for business functions.

You have recently made statements that your mission is to deliver benefits of free software to the consumer ecosystem, however without a complete change in process shifting to quality being your first priority you can only be expected to fail. In the eyes of the end users that you are targeting, bugs that impact users ability to enjoy and use their computers are not acceptable even to the smallest degree.

I would like to provide a small set of examples of a few of the bugs that I have been following personally which impact thousands of users. These bugs remain unrepaired in almost all cases in Jaunty, and yes in some cases they will exist in Karmic on release day because your release schedule is more important than end user experience.

Bug #404626; Turning wifi "off" using Fn+F2 on Eee PC with Ralink rt2860 results in kernel panic

This bug reported July 25, 2009 impacts at a minimum all Eee PC 901 users. When a user turns off their WIFI using any of the available utilities including those integrated into Karmic, the user receives a kernel panic. The user is unable to save any data from open applications, and they are forced to reboot their computer. Unfortunately, this bug will still exist in Karmic upon release. When asked by a user if it could possibly be fixed in the release, it was stated in the bug: "No, it doesn't work that way. An "exception" here would set back the release by a whole week, due to the time involved in integrating kernel changes and validating the resulting images. We aren't going to do that for a single piece of hardware that's supported on a best effort basis - this is entirely suitable for fixing in a post-release update."

Expect thousands of kernel panics upon release day because users computers don't work properly. Expect them to erase Ubuntu immediately afterwards because "Ubuntu sucks".

Update: [ 2009-10-28 13:23:01 errata for bug #404626 ]

Bug #371434; PCI ExpressCard hotplug requires pciehp.pciehp_force=1

This bug reported May 3, 2009 impacts a wide variety of systems. With this bug, the kernel hotplug driver is unable to hotplug certain devices forcing a user to have to reboot their computer and in some cases enter the BIOS to re-enable a device that has been turned off in the operating system in order to turn it back on. There are multiple work-arounds however all of them are more complex than should be expected of an end user. This issue will impact Karmic users on release day.

Bug #337199; eeepc-laptop patch missing in Jaunty Alpha 5

This bug reported March 5, 2009 impacted all Eee PCs. In May after waiting several months for a patch to be released I issued a patch for Eeebuntu repairing the Eee PC Laptop kernel module and released it in a DKMS package which overwrote your module. This corrected the problem for thousands of my users, and any of yours that stumbled across my website. This is simply unacceptable as it still is not fixed in Jaunty and only marked as repaired because the kernel your team pulled downstream for Karmic corrected it. Jaunty users are left without a fix even though you still support their platform. This isn't even basic O&M support in my opinion.

Bug #349314; Slow performance and tiling issues on i915

This bug reported March 26, 2009 was fortunately flagged as repaired in Jaunty August 17th, 2009 long after most users fumbled through forum posts and upgraded or downgraded their Intel drivers to drivers that worked. This bug fortunately does not exist in Karmic, however users were left to figure out how to fix it themselves (even if they read the release notes) for several months. A consumer would never except this.

I strongly urge you and your team to reconsider your release cycles and to integrate better with upstream maintainers while releasing patches faster for your users to minimize their pain with your operating system. Mature Linux distributions issue patches for issues to correct them for users quickly rather than sitting on them in hopes that they will be fixed upstream for the next release. They then work to merge their changes upstream after solving issues for their customers improving the Linux ecosystem as a whole. End users will never accept the current process in which patches are ignored, categorized as low priority when they impact multiple users, or the difficulty in finding resolution and guidance to issues that they continue to face. Continuing down this path only further blemishes my favorite platform. So, please do us all a favor. Shift your focus to quality, integrate better with upstream, and please remember that you can't sell something to common consumers that does not work right.


Thank you for your time,

-Fewt

WARNING: Do not install 9.10 with EXT4, there are known data corruption issues with it [ 453579 ]. They released with it as the default filesystem though it has been known to be an issue for 2 weeks. Be sure to switch to EXT3 during the installation to be safe. There's an Ubuntu one bug too that causes data loss. WOW, really great stuff there Canonical.

Saturday, October 17, 2009

Eeebuntu Evolution

The Eeebuntu core team met today to discuss the evolution of Eeebuntu, and what we can do to improve stability in the future while maintaining an award winning operating system.

As an output there are a few things that hopefully all Eeebuntuers will be happy with.

1 .. In the next release, Eeebuntu will no longer be based upon Ubuntu but rather Debian Unstable.

2 .. We are moving to a rolling release schedule, with quarterly ISO rollups.

3 .. We are moving away from being an Eee specific Operating System into a model where we target any desktop or laptop.

In addition, there have been multiple news articles claiming that I have quit work with the Eeebuntu team. These articles are inaccurate, and were written without my or the Eeebuntu teams consultation.

What is true is that I am no longer supporting Ubuntu or Ubuntu variants with any software that I write. It is also true that I am no longer supporting Eee PC Utilities as I previously stated, though I may update it periodically for my own personal use and it will remain on SourceForge and in my personal software repository.

Another inaccuracy spreading in the news is that I am killing Eeebuntu, this couldn't be more inaccurate as I don't own Eeebuntu, I am simply a member of the team. If I or any other single developer left the distribution it would be a setback for sure, but would not "kill" the distribution. As I stated in a comment the other day, I remain comitted to working on software for the Eeebuntu distribution.

If you would like to read more on the team decision concerning the evolution of Eeebuntu, click [ HERE ] to go directly to the team post in the Eeebuntu forum.

Tuesday, October 6, 2009

I give up..

I've been fighting an OS (kernel and xorg mostly) that changes with nearly every patch or release to support the Eee PCs for almost 12 months to try to make Linux better for Eee users.

Every time something in the OS changes, Eee PC Utils gets the blame for "breaking" people's computers even though whatever I release typically doesn't even alter the code being reported. Even though I've published work-around after work-around on the Wiki for Linux bugs like the famous bluetooth bug in 2.6.28, the WIFI hotplug bug (pciehp force), or linking to the fix for the Intel video driver bugs, its still considered my fault in the eyes of the users when the screen won't turn off and on (xrandr reports a protocol error that it didn't a few weeks ago) or their WIFI won't come on after they turn it off (because the kernel pciehp module is broken).

Now that Karmic is coming Eee users are looking forward to having 2 Radio Kill devices for Bluetooth, and 2 for WIFI. Yes, this is in a production released kernel, yes, the interfaces are provided by different modules (eeepc-laptop exposes them as do the driver modules), yes their naming and function is of course not named using a reasonably grep-able syntax, and yes they do actually work differently somehow.

Not only that, but you also get to look forward to even more bugs in the Intel drivers like the one causing xrandr to report success even when it does nothing, or the super awesome kernel mode setting bug that leaves many Eee 900 users with nothing but a blank screen at startup. This is real PROGRESS people!

Try for yourself. Run these two commands on Jaunty and Karmic. If you have the Intel driver you should see nothing happen in Karmic with the first command, and both commands work in Jaunty (note: Change LVDS to LVDS1 for it to work in Karmic, but then it doesn't work in Jaunty, but the point is WHY DOES IT RETURN 0!).

xrandr --output LVDS --mode 800x600 --scale 1.0x1.0
xrandr -s 800x600

To return to your default resolution, type this:
xrandr -s -

Why not have a little fun with it! Yes, unfortunately this really works. xrandr returns 0 and "YEP" is echoed as if it set output UBUNTU to mode SUCKS. :(
xrandr --output UBUNTU --mode SUCKS && echo YEP

I've been hearing a lot of users posting (by a lot I mean .002% of the 15,000 Eee PC ACPI users that download the utilities every release) that they wish I would continue working on the utilities, some have even posted that they were completely successful following the advice and posts that are well documented.

I just can't do it, because Ubuntu sucks. Instead of moving forward with every release, they have the uncanny ability to take Linux back in time by piling code that doesn't work on top of more code that doesn't work until they have turned their OS into a garbage salad. 8.10 was GREAT, and for the most part everything worked. Some things were missing concerning support of newer Eees, but then they released Ubuntu Jaunty 9.04. 9.04 could only have been released completely untested, as they missed something as stupid as the Intel tiling kernel bug which caused every Intel card out there to crawl during any OpenGL function. There is no excuse for their release of alpha grade drivers and less than alpha grade kernel code into their release distribution.

In order to work around all of the bugs that 9.04 released as "gold" "stable" code, one must spend at a minimum 30 minutes replacing packages and patching mtrr bugs to get a reasonable level of performance out of their Eees. We spent days working on documenting all of the fixes that need to be applied to 9.04 to get it to some semblance of stable. Even then, don't turn compiz on and suspend because your super cool new "stable" version of xorg will consistently segfault leaving you at a GDM screen without your data being saved in any of the applications it unloads out from underneath itself.

Now we have Karmic coming down the pike in just a few weeks, and xrandr can't even take you from 1024x600 to 800x600 even though it thinks it did. Not only that but since your rfkill devices are re-ordered at boot, the only way to get the right one is to trigger them both and hope for the best unless you like nasty greps in something that should respond instantly. When you consider WIFI if you don't trigger one, sleep, then trigger the other it's not going to work anyway even though it will think it did. If you trigger rfkill, but you see that NetworkManager still has your WIFI disabled, yeah, that's because it's looking at the other one. Oh and don't put away the force pciehp edit that you put in your menu.list for Jaunty because even though there's a cool kernel mode setting feature, they still haven't fixed that bug that's existed since *2.6.28*. No, there is nothing that I can do about the notifications clogging up your screens, Ubuntu's shiny new notification system STILL ignores the expiration time option from notify-send and apparently now defaults to more than 10 seconds before expiring a notification. Oh and the scripts that live in /etc/pm/power.d yeah they only trigger sometimes now. So, if you have my utilities installed in 9.10 and you are wondering why it doesn't trigger to powersave when you unplug power, you can thank the wonderful Ubuntu team for creating even more bugs in this "great" operating system by switching to devkit rather than choosing to stick with something stable and mature.

The release tonight (1.1.47) will be my last for Eee PC ACPI, as I don't have the time or the desire to keep working toward a goal that continually moves out of reach, or support an operating system that isn't even as stable as Windows XP. Provided you have patched your Ubuntu / Eeebuntu install with all of the non-standard hotfixes (2.6.30, multiple configuration files, mtrr fix, Intel Driver switch, etc) it should hopefully remain functional for the duration of Jaunty's lifecycle.

Maybe with luck someone at Ubuntu will figure out that they are ruining Linux with their release schedules and constant bleeding edge cruft and will slow down to focus on bugs that are making their OS suck. Maybe the kernel team will actually learn how to write modules that work and don't duplicate code and functions like rfkill, and perhaps the xorg team will focus on real issues like making drivers that actually work rather than changing how you configure it 3 times in 3 releases. Oh wow cool, it goes from terminal to GUI in 1 second, too bad I can't suspend with composite mode on without the driver segfaulting. Who needs suspend anyway, not like its a useful feature and all on an ultraportable.

Then again, maybe I'm just irritated because I released an update that ONLY alters how notifications are displayed and once again I immediately start getting hate mail that I broke people's WIFI when it's really the fault of whomever decided that eeepc-laptop AND ath5k needed to expose rfkill and the fault of the kernel for not hotplugging the device when rfkill is called unless you force it at boot.

Well don't worry folks, I won't "break your WIFI" anymore.

Maybe I should buy a copy of Windows 7, I hear that it actually works. How can we expect non-techical users to use this pile of garbage that is "Linux"?

/rant

UPDATE: Eeebuntu team votes to shift distribution base to debian. [ LINK ]