For a while now I wanted to begin writing a series of posts on the subject of RackTop’s BrickstorOS, specifically various design philosophies, engineering choices we made, things that we chose to avoid, etc. The main purpose behind the series is to share with others what we have learned and how that shaped the core of BrickstorOS that exists as we know it today.
Very early on we went back and forth quite a bit about just how to handle all software that we need to have on any Brickstor machine to make it useful. This is by no means a trivial conversation. There are millions of various files that all have different degrees of importance, and even more critically, unless you chose to build an OS quite literally from scratch you will have the baggage of the existing operating system, or kernel, if you decide to build an operating system with an open source kernel, like the Linux Kernel for example, and nothing else, effectively building your own userspace, however small it might be.
It is not exactly practical to try to build your very own OS without leveraging work of the great communities like that maintaining Linux or Illumos, and those supporting various distros simply because there are already great choices out there which give you an outstanding rock-solid system with a lot of good, and maybe a little bad.
First of all we did not choose Linux, and this is a discussion for perhaps a number of other posts. Suffice it to say, we chose to use Illumos, which is effectively a complete Operating System, and one which is very customizable letting one build anything from a general purpose desktop to a server, to a switch, to a network edge appliance, and well at least in our case a network storage system. There are in fact examples of all of these in existence already, and we were not the first to decide to build an “appliance operating system” with Illumos, nor will we be last!
Back to software and package management. Convenience is important and I think over the last decade or so package management has come into its own. Unfortunately, most package management suites still really leave a lot to be desired. The Deb system, RHEL’s RPM system, pkgsrc system and a number of others, all tend to solve one problem. Well, maybe 3 problems. These are: 1) how to easily add new software to a system, knowing that OS versions and hardware vary widely, 2) how to maintain that software, i.e. saving user configurations, non-destructive updates, backwards compatible configuration files, etc., and 3) how to clean-up after itself when software is being removed. These seem like obvious problems, and maybe at first glance fairly uncomplicated and straight-forward. They are not.
Just making sure that some software as built on one system runs on another is hard enough. Today’s complex software is almost never built in a way where it stands on its own two feet and needs nothing else. Libraries, various dependencies, etc., etc. are all a necessity for almost every piece of software written. We write code leveraging libraries in efforts to maintain compatibility, reusability, reduced duplication of bytes, etc. Yet, all that software is still difficult to manage on any system. Package management does not suck because nobody has the skills/vision to do it right, it is because the best you can ever hope for is that most of the edge-cases are handled. This is not enough sadly, but that is all you can realistically expect. Complexity is the nature of the beast.
One problem stems from the fact that depending upon what language something is built-in, it will need more or less stuff to run and its dependencies may be delivered in ways that others don’t do. Looking at most any modern language, there is a package manager for software written in that language. Python has pip, Node.js has npm and bower, Ruby has a host, rvm being one, etc. Point is, there are so many of these you never have enough with just one. Is there one that is better than others? Maybe, if all you needed was X, then Y and Z are left in the cold, etc. There is not one right choice, if you are to have packages and you want ultimate control you have to decide to build your own, even if you choose to use some existing suite. And well, this is complexity and opportunity for error.
One theme that I hope will become more and more clear, eventually is that we are shooting for simplicity. This is not simple, nor is it elegant. We rapidly came to the conclusion that if we had package management we’d have a number of problems on our hands. Complexity, was the word that kept on rising above the rest. When one builds an appliance one also comes to a realization that an appliance unlike a multi-function system has to do far less, but those few things that it has to do must be streamlined, as robust as possible in the face of failure and must be as user-friendly and in our case admin-friendly as possible. What’s more, software does not change a lot, other than fixes, and improvements as a result of growth and code’s evolution. There is a host of reasons due to which we decided to drop all package management, and we will talk about them in more detail in future posts.