Ranting, Technically Speaking

Windows Server 8 Proudly Joins 5 Years Ago

noreply@blogger.com (Dominic Lepiane) — Sat, 07 Apr 2012 18:18:00 +0000

I've been poking around a bit to try to get some idea of Windows 8 might be an enticing upgrade in the workplace. I haven't been following too closely so there may be features I'm missing but here's what I've found so far.

On the desktop - no benefit. New Metro UI is the biggest change and it's primarily a touch-screen friendly interface suitable for tables or smartphones. Not at all for a "working" desktop where you might want to do more advanced tasks such as using a word processor or a spreadsheet. And since Microsoft's plan to compete in the smart phone market is to first surpass Blackberry mostly by waiting for RIM's demise as BlackBerry use hits 0, Metro is irrelevant before even getting out the door.

On the server - a lot of big benefits and well worth the upgrade. Windows Server 8 looks to be a big step forward from the late 20th century and into the early part of the 21st century. This PC Mag article from the fall gives a pretty good breakdown. I won't re-hash the author's well written piece and just go for the jugular here.

"Intellisense Powershell" lets administrators auto-complete in PowerShell. Big benefit, must-have, and has been readily available to bash and zsh users in Linux-based operating systems for a long long time. Seriously - get real! Microsoft has only just started down the road of a headless server OS path where automation can truly scale out operations and they have a lot of ground to cover. This is our first example of the Microserfs pulling their heads out of <the ground> and look at what's going on outside their <world>.
"Live Migration" lets Hyper-V guests be moved without disruption to new hosts. I can't honestly say I'd touch Hyper-V without some sort of hazmat suit on. Seriously, this is a "new" feature for Microsoft? VMware vmotion has be been doing this for vmware customers for a while. Yes, Hyper-V is free and VMware is paid, but with Hyper-V you're not getting your money's worth. Maybe if VMware doesn't innovate at all for the next decade, Hyper-V will catch up enough to make it a viable option for anything other than a test lab or party tricks.
"NIC Teaming" oddly I don't consider a "must-have", but this would be a feature possibly 20 years or more behind the times. Hardware independent NIC teaming for bandwidth agreggation and fault tolerance has been the norm on any network operating system outside Microsoft Windows since, well, forever. Where MS admins have historically depended on NIC vendors' drivers to provide this functionality to date, there at least is a path to do this in Winblows so though this is an important feature, I wouldn't buy Windows Server 8 specfically for it.
"Claim Definitions" is a feature that allows sensitive files to be tagged as confidential, for example and access can be based on these "claim definitions". I have no gripe here - sounds like "access control lists" based on tags. I'd like to see how flexible this functionality is but even as-is can be an important tool under Windows 8.
"Flexible Deployment" means that you can install Windows Server "core" (the stupid headed "headless" install we know from Windows Server 2008) and then, wait for this shocker.... Upgrade to full at a later time. #facepalm I mean, seriously? You've got Ubuntu users who do in-place one-click upgrades across major versions, RedHat Enterprise Linux admins who will generally install headlessly just to get a box up and then add in all the features including the GUI in their default software package manager tool, but Windows Server users are only now going to be able to add the full Windows install into core without reformatting? Maybe with Windows 25 in the year 2050, Microsoft will shock and amaze us by letting their users get software updates for their application all from a common update utility rather getting random prompts every other day to update all the plethora of third-party applications and utilities they have installed just to make their computer usable (actually, this will never happen...).

In summary - looking forward to Windows Server 8. Maybe get myself a pager again to relive life in the pre-iPhone era.

Rolling out of nested shells

noreply@blogger.com (Dominic Lepiane) — Thu, 01 Mar 2012 00:04:00 +0000

I just realized that if I'm really lazy, I can stick && exit after everything to dump me out of all my nested shells after a program completes.

ssh <whateverhost> && exit
sudo -i && exit
for item in list ; do someprocessing ; done && exit

PC apps are dead?

noreply@blogger.com (Dominic Lepiane) — Mon, 27 Feb 2012 22:32:00 +0000

I've been looking around from time to time for an app which would let me scan books from our collection at home and build a digital library - most useful for loaning books. I never found much on a PC, I did find

https://market.android.com/details?id=com.eleybourn.bookcatalogue&feature=also_installed#?t=W251bGwsMSwxLDEwNCwiY29tLmVsZXlib3Vybi5ib29rY2F0YWxvZ3VlIl0.

In short - yes, apps for desktop seem to be pretty much dead. I can't think of the last time I found a usable desktop application. At most, it's browser plugins like Nagios Checker. There are some "rich" applications or system management applications with rich clients, like InterMapper, but generally it's all web UI.

As Martha says, "It's a Good Thing".

Charting Systems Using Cacti

noreply@blogger.com (Dominic Lepiane) — Sun, 29 Jan 2012 19:22:00 +0000

There are a lot of great monitoring tools out there. I've posted many times before about Nagios and I could post still more on this great tool, but it's not the only tool I use. Another one is Cacti which is an excellent tool I've also mentioned before and it is mostly for graphing system resources.

Out of the box, Cacti will give you a lot of the basics especially when combined with SNMP. Disk usage, network interface usage, CPU, and memory. But what I really like about these great Open Source tools is that there are extensions readily available from the F/OSS community. With Cacti, you can extend by getting new host templates and data queries (and more). Here are some examples.

Disk IO - this is a new data query that tracks disk IO usage either in IOPS or MB/s. This is one of the simplest examples of how you can extend cacti. It comes as a xml file defining an SNMP query which you copy into your resrouces/snmp_query installation folder and as a data query template which you import through the Cacti UI. Once you've done this quick installation, you can add the disk io checks to any SNMP enabled host you are already tracking.

Dell PowerEdge Environment - this is another simple example which is the same as the Disk IO in that it is an SNMP query plus a data query template but there's 3 checks it adds. System ambient temperature, fan speeds, and system voltages. Its a great example of how Cacti as a generic tool can be tuned to target your specific operating environment whether you're a Dell shop, HP shop, or otherwise.

APC UPS Daemon - Another example of an application specific example. This one comes as a host template so its a collection of checks you can use to capture all the data queries on a host using APC UPS Daemon. A great example of where F/OSS tools *far* exceed the stock or closed-source tools provided by vendors. Rather than these cheesy brief inflexible views of how your system works as provided by APC that require overly large utilities to be installed, its quick, lightweight, and much more flexible to use the F/OSS tools.

Cacti is another of these great tools that works well in conjunction with other tools to give system administrators great insight into the operation of their network.

Storing Passwords

noreply@blogger.com (Dominic Lepiane) — Sat, 31 Dec 2011 01:09:00 +0000

The most effective way to manage your passwords for personal or professional use us to use a password manager. This allows you to manage unique logins for all the different resources you access (bank vs email vs general forums vs ...) and only have to maintain one master password. Pick a reputable password manager, like KeePass, and remember that backing up and restoring your password database is critical.

Keeping electronic copies is fine, but also consider keeping a hard-copy as well in a relatively secure location. One suggestion is that you print off your passwords every time you change your master password (annually is pretty minimal) but write that master password down on the print out so you can recover it if you forget it! Useful if you do cycle your master password frequently.

WiFi Routers and NAS

noreply@blogger.com (Dominic Lepiane) — Fri, 16 Dec 2011 14:27:00 +0000

The last time I bought a new router was when the Linksys WRT54G was "the king" of home WiFi routers - and mostly because you can replace the useless stock firmware with DD-WRT. Otherwise, it was "a router". At the time, 4 years ago, which is like many generations in Internet time, you had to manually setup security on your WiFi AP still so you saw lots of open WiFi hot-spots like "Linksys" or "Dlink" around. Then the WiFi router manufactuers started providing security setup as part of their setup wizard so you see more SSID customization and security enabled. Now, apparently, everyone auto-configures security with a magic button called "WPS". Then you've got other features USB ports so you can run a file-server from a USB drive or print server and "guest networking" so you can isolate isolate your workstations from other users.

"WPS" - WiFi Protected Setup is definately a cool feature. It comes as a button on the router so when you press the button, its like the router goes into a sort of "security auto-config mode". WPS, if its supported on your client (I assume it's a software install), will then automatically configure your client and your router with strong security settings. It means no more default passwords and streamlining the security options for users who frankly don't need to have "WEP" as an option.

[Edit: WPS is broken and should be disabled on all routers that support it according to SANS.]

Guest networking is another cool feature on some routers. It is a separate SSID for, well, guests to use your WiFi from. It is isolated from your main network so that guests won't have access to, for example, your network attached printer or to your media collection you stream from your laptop to your television. This is just so cool for people who may be sharing their Internet connection with their neighbours or roommates but just don't want their surfing habbits to infect their own systems :)

And the USB ports. Many routers seem to have one or two USB ports on them which is interesting, but what's more interesting is what you can do with them. A lot of new routers have built-in file servers so as soon as you attach some storage, you can share files and folders from it to the PCs on your network. How convenient is that? Some routers have more sophisticated web interfaces than others and let you specify which folders are or aren't shared - but either way, if you're buying a new WiFi router anyhow and you get this feature, it means you get a functional NAS for the cost of a USB key or USB attached hard drive! *And* some routers are starting to come out with USB 3 - SuperSpeed USB which if you consider these routers have not only 802.11n speed on the WiFi but also Gigabit speed for the network ports, is an awesome feature.

And that's not the only thing you can do with the USB port - some routers will also act as a print server! So you attach your generic USB printer to the router, and it's now a network printer you can print to from any laptop or PC in the house. Talk about great value-added feature! I love it!

And did I mention that new routers are all now wireless N with Gigabit LAN interfaces? WiFi is still garbage and a ways away from being reliable outside very small deployments, but N is an improvement over previous specs. Interestingly, I found out the other day as well that if you run your router in "dual band" to support both N and G clients, your wireless speeds on both N and G suffer. So ironically if you have any wireless G clients, unless you really need your N devices to run at "slightly faster than G but nowhere near N speeds", you should still run G only.

Cool beans! I'm liking some of the features I'm seeing on the box these days from some of the WiFi routers. A nice change from the utter crap they used to shlep out where the only smart thing to do was check if you you run a custom firmware on the device and replace the junk software sold with it.

Source Control for Server Admin

noreply@blogger.com (Dominic Lepiane) — Wed, 19 Oct 2011 16:58:00 +0000

So you manage a server, or a lot of servers alone or in a team, however you are doing this, you are going to be tweaking configuration files often and creating custom scripts for automation. There are two tools I use for revision control - RCS for configuration files (generally) and SVN for scripts (generally).

RCS. The classic. All the documentation you will ever need is in the man pages. Well that and some context for how to use it. RCS creates revision files in place. So if you change /etc/dhcpd.conf, it will create /etc/dhcpd.conf,v. This is a very useful setup when controlling local files in arbitrary locations - like most of /etc on most of your servers. There are a few caveats to keep in mind:

RCS will put revision files (the ,v files) in an RCS folder if present
The default behaviour is to remove a file from its current path on check-in

Keeping these in mind, this is my general pattern for working with files under /etc.

If there is no RCS folder (e.g. /etc/RCS), create it first

mkdir -m 700 ./RCS
Assuming your working folder is where the file in question is, this will create an RCS folder and protect it from other users (typically non-root)

If a file doesn't exist in RCS, check it in first

ci -u dhcpd.conf && co -l dhcpd.conf
ci is short for "check-in", unlike SVN or CVS, "ci" is the command and not an argument to "rcs"
The -u "unlocks" the file leaving it in place (so dhcpd can read it)
co is "check-out" and -l "locks" the file for editing

I always leave files checked out to capture changes by other users or by the system (like rpm)

If the file does exist in RCS, check for any un-committed changes

rcsdiff dhcpd.conf
This does a diff against the last checked-in version by default but you can specify a version if you want to compare against earlier changes
Check-in any un-committed changes or find the person who made the changes and make them do it

The file should always be left checked-out (per above comment), otherwise check it out
Make changes
Check-in changes, and check-out the file for the next user

ci -u dhcpd.conf && co -l dhcpd.conf
Give a brief log message indicating what the changes were and again, leave the file checked-out to capture changes by the system or other users

Now the last useful command I'll mention there is rlog which lets you read the revision history log.

Now SVN is a proper centralized source control system. They have excellent documentation on setting up a repository. This is very useful for system admin scripts.

Although most system administration related scripts won't ever have "releases" or "branches", you probably still want to follow the SVN guide and create at least a trunk in case you ever do need to tag a specific version. There's no cost, so I use a trunk even though I've never used it because changing later is a problem.

With SVN you'll want to keep an updated local working copy ("tip") either on a shared NFS location or locally on each server. How you do it is up to you, just create a cronjob to run "svn update /path/to/tip" and then you can always run scripts from that path.

RapidSVN is a great tool, well maybe not great, but works very well for sys admin anyhow and its readily available. So check out your own working copy of the trunk with RapidSVN. I configured RapidSVN to use gedit as my standard editor and meld as my diff tool.

This gives you everything you need for day-to-day creating and maintain system configuration files and your toolbox of scripts for automated system maintenance.

Debugging Python Scripts

noreply@blogger.com (Dominic Lepiane) — Sat, 15 Oct 2011 16:02:00 +0000

This is really just props for a site I found with a nice walk-through of using the Python Debugger - pdb.

http://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/

pdb your built-in step-through debugger allowing you to inspect objects and all the usual things you need in developing a program.

Running the numbers

noreply@blogger.com (Dominic Lepiane) — Fri, 09 Sep 2011 16:13:00 +0000

Two interesting tools popped up recently.

Good old Linux Counter has been passed down to a new maintainer. This is a classic project which attempts to get Linux usage data from user input. Its hard to tell if its particularly relevant, but it is interesting to see relative usage across platforms and by region. As for estimating global Linux use? Hard to be convinced this provides a good enough sampling to be very convincing. Nevertheless, I keep my machines at home registered there. Or at least some of them :P

Another one I really like is Debian Popcon which tracks popularity of Debian packages by installs and by "votes". Popcon is actually just a Debian package which phones home your installed package list and it is installed by default on some distros while not others. What I like about popcon is that when there are a wide variety of F/OSS tools available, you can check the list to see which tools are ranked highest so you can at least start by trying the most used tool rather than taking a total wild guess. For example, in looking for a SVN GUI tool, I did a "yum search svn" and there were a lot of hits. So I opened up popcon, search the list top to bottom for "svn" and took the highest hit which was a GUI tool which was RapidSVN. Well, then I checked with Dante which tool he used, but lo and behold, it was RapidSVN :)

Reorganizing Ubuntu Partitions

noreply@blogger.com (Dominic Lepiane) — Thu, 07 Jul 2011 13:06:00 +0000

My personal PC at home died. It was an old PC no matter which way you look at it. Every part had been replaced or upgraded over time (case, PSU, optical drive, hard drive, memory, CPU, mainboard, NIC, video card) so knowing it's actual age is pretty hard, but it looks like "Friday" as a PC existed for 7 years. Checking my blog, the first reference I found was October 31, 2004 indicating Friday was the new name for an old PC called Michael.

Time for a new new PC. I've reused the optical drive but everything else is new in Agnes (from Immortality by Milan Kundera). I did your basic "install Windows first, Ubuntu second" so pretty much just a mommy-install. Until I realized I really hadn't made a big enough Windows partition.

I figured it would be a pain, moving the first Ubuntu partition back on the drive so I backed everything up and booted from the Live CD. "gparted" is included on the live CD and it was painless to shrink the Ubuntu partition, move it "right" and extend the Windows partition. I didn't have to reinstall grub or do anything else, it pretty much just worked - for both OSes. It's always so nice when things just work.

But I will say, it's pretty dumb that Ubuntu doesn't use LVM. As I have posted before, LVM is very useful. What would be nice is if I could have just lumped most of the free space into LVM and then just carved out an LV for home and another for media so I could grow them as needed. Rather than fiddle too much with that though, I ended up just going with a relatively large /home partition and will just grow that as needed and then if I need space for other things - like more storage under the 'doze, I can put a partition at the end of the disk.

Heartbeat

noreply@blogger.com (Dominic Lepiane) — Thu, 16 Jun 2011 12:00:00 +0000

I recently have tested out running Heartbeat (finally, took too long to get to this, but that's another story). This is a cluster resource manager (CRM) which polls nodes in a cluster and brings resources up when a node failure is detected.

It's interesting. I wouldn't call it elegant really, maybe the newer Pacemaker would seem cleaner. But it is simple and at least in testing it is effective especially when combined with DRBD which I posted on earlier. The thing is where DRBD really seems built for top-notch resiliency and flexibility, Heartbeat seems it will work, but it's not obvious that you'll get what you expected - maybe it's just the documentation on DRBD was really well done.

At any rate, there is great documentation on getting Heartbeat up with DRBD both from the CentOS wiki and from DRBD. I used heartbeat with drbd83 in CentOS.

What Heartbeat will do is listen for heartbeats from a peer node in a cluster and if a peer goes down, it will bring up services on the working node. There's a handful of important things about this to keep in mind.

First is the heartbeat - this is just a stand-alone network connection between two nodes so if that connection goes down or the heartbeats get choked out by competing traffic, Heartbeat may well decide you have a node failure. This is not a trivial problem because now that Heartbeat can kill services on an active node, this is potentially an new point of failure. And this is common to many HA configurations including DRBD itself though as we know, it will identify split-brain and gives you some recourse for repairs. So the suggestions here are to use a dedicated connection, preferably a point-to-point connection with a cross-over cable or a serial port - and this is not uncommon for clusters (the point-to-point connection) like in this white paper for Microsoft Storage Server.

Then there is the issue of resource management - when the CRM is managing the resources, the usual OS procedures should not. If Heartbeat is in charge of bringing up MySQL, you shouldn't be starting MySQL from the init scripts when the OS boots. Now the nice thing with DRBD is that it's behaviour is consistent with this paradigm - when DRBD resources start up, they are in "secondary" mode and cannot be accessed by the OS. So if you have a file share protected by DRBD, Samba wouldn't be started by the OS, and likewise, that file system would be unavailable when the OS starts (by default at least). So here, Heartbeat makes a lot of sense. You take a 2 node cluster for example, when the nodes start up, Heartbeat looks for the peer, picks someone to become active, and then would make "primary" the DRBD resource on that peer, mount the file system, start smb. On the stand-by node, you would both have 'smb' off and the file system would not be writeable which helps ensure consistency.

I guess I could go on about Heartbeat quite a bit, but there's one last thing to mention specifically here and that's the style of cluster. There are "R1" style clusters which are simple but limited to 2 resources (and other limitations) and then there are CRM enabled clusters which are more robust but more complicated to configure. I have only used R1 because it was sufficient for my needs - 2 nodes, one was known "preferred", keeping cluster configuration in sync "manually" wasn't onerous. But CRM enabled clusters are more interesting because you can add more nodes and it will distribute the cluster configuration automatically, etc.

The one thing I haven't really touched on is the quorum which others who are more familiar with cluster management will be more familiar with than I. Basically with Heartbeat in an R1 style cluster, there isn't going to be a quorum. Your configuration is maintained pretty much manually, services are only running on one node, etc. In CRM style Heartbeat or other application clusters, the quorum is what all the nodes agree on and typically stored in a file. On Windows Storage Server and other clusters, the quorum is stored on the shared disk which means any problem there means the cluster fails. With Heartbeat, the quorum file is copied among the nodes but this is susceptible to becoming out of sync like if there is a communication failure on the heartbeat channel leading to a split brain. Or this is my limited understanding of this. At any rate, it is a problem and it isn't trivial in working with active/active or multi-node configurations.

Cache in Openfire

noreply@blogger.com (Dominic Lepiane) — Fri, 13 May 2011 14:18:00 +0000

In the course of troubleshooting the office Jabber server the other day, I came across some interesting info about the various caches that Openfire has. If you log on to the admin console of your Openfire server and go to the cache summary page, you can see what the usage and effectiveness of your various caches are. Specifically, I found that a couple caches were full - Roster and VCard. The Roster cache was limited to 0.50MB by default it seemed and it's effectiveness was less than 20% at the time.

It is a fairly common issue and it has been discussed in the Ignite Realtime forums. The solution posted is to set a couple system properties to override the default:

cache.username2roster.size
cache.vcardCache.size

Both of these are given in bytes. The post in that thread says to go to 5MB, I found that my Vcard cache didn't need to be much bigger than the default and the roster cache only needed 2 or 3 MB.

After changing this, both cache hit rates are closer to 90%.

Our system is very small (less than a couple hundred users total), so the effect is not big on regular usage. But well worth checking on your server as it is a quick and easy optimization.

Again with the tapes

noreply@blogger.com (Dominic Lepiane) — Thu, 05 May 2011 09:58:00 +0000

In a previous post I said that to get around devices changing their numbering, it was useful to use the /dev/tape/by-id instead of the generic /dev/nst0. Unfortunately, this is also imperfect I've just learned as the device which was previously "scsi-35000e11138aa0001-nst" this time came up as "scsi-35000e11138aa0001". And you can guess how gracefully the software handled that (not at all). Now I don't know if was a driver update (possible) or if the device was switched to a different SAS interface (also possible), or maybe just the gremlins. Whatever it was, once again, I had to reconfigure the software to find the new device ID.

Flexible Storage Replication

noreply@blogger.com (Dominic Lepiane) — Thu, 10 Mar 2011 21:06:00 +0000

I have recently been looking quite a lot at different storage setups including storage replication and have been so far mostly relying on running rsync to copy a file system to an appropriate secondary host. For large file systems - either with a lot of files or simply a lot of changing data, this is slow and resource intensive. Not really a problem in some cases, but very problematic if you want your secondary system to have very current data. If you want to cobble something together yourself from commodity hardware, DRBD is an excellent tool and very feature-rich.

First of all, I can't recommend the DRBD User Guide enough. It really lays out the features and usage not just of DRBD but also some common applications you would use alongside like LVM for storage management and Pacemaker and Heartbeat (and others) for clustering.

What DRBD is going to do is basically copy writes to a block device over the network to a replica device - this storage set is called a "resource". Generally, you will expect to have two nodes for each resource. During normal operation, you will have one "Primary" node and one "Secondary" node for each resource which logically indicates that one node is writing changes to the resource while the other is making a copy. DRBD is generally very slick in handling replication and the status of the nodes. First of all, when you configure the resource, you specify an IP address for the replication target and generally you are going to want this to be a separate network interface from your general data plane - for example a cross-over cable for point-to-point connection between the two nodes. If the replication path goes down, DRBD is basically going to mark at what point in time it happened and then keep track of which blocks changed since that point so when the path comes back up, it has a list of which blocks need to be transferred instead of having to resync the whole device. That's another thing - it does the whole device sync for you too when you create the device. And also, you get basically the same behaviour if your secondary node tanks, or if both nodes tank for that matter, or even the primary node.

Unless both nodes end up in a "primary" state during some overlapping time. So if you automatically bring up the secondary node in case of a primary failure with Pacemaker, for example, but the issue was a path failure and not a node failure, then both nodes may end up in "primary" state. Since DRBD is tracking when communication is disrupted, it will detect this problem - a "split brain". You get several options for manual resolution (I think automatic as well) including taking the changes of one node or the other, the node with the "most" changes, the node with the "least" changes, the oldest primary, the youngest primary... You may still be stuck losing some data - but you can keep both nodes in split brain and consolidate externally (e.g. if you have critical data like financial data where you can never drop a transaction).

DRBD supports three replication "protocols" called, intuitively, A, B, and C. "A" is asynchronous so writes to local storage device unblock after the local device finishes writing. "B" is "semi-synchronous" which unblocks after the data has reached the peer. And "C" which is "synchronous" so the write operation is only complete once the data is written to both devices. I was finding that "A" and "B" got me similar speeds and "C" was slower - but this is not very rigorous testing and my replication link was 100Mbps through a shared data plane.

One of the things about any of these replication options compared to rsync is that they are going to generally be much nicer on your memory. I find that when rsync scrapes the file system, this effectively nukes the OS's disk cache such that after rsync runs, users may notice it takes a while to "warm up" again. But, replication is not a backup - if a virus eats your files on your primary node, it will eat them on the secondary node synchronously or asynchronously - your choice.

If you are using LVM (and you should be, I've posted about LVM before, so have others), you'll wonder whether you layer DRBD on top of LVM or vise-versa. As Chef would say: Use DRBD on top of your LVs. Dramatic over-simplification aside, it does depend on what you are doing. If you are using LVM to carve up a pool of storage for example for virtualization and then want the storage layer to replicate your VMs, it may make more sense to create your DRBD volume from physical storage, then it will replicate the whole LVM structure to your replica node. But there's complications like ensuring LVM will even look at DRBD devices for PVs and managing size changes, etc. There's a time and a place for everything, and that's college.

Um, what else is awesome about DRBD? Offline initialization, "truck based replication" (a.k.a. sneakernet), replicate the node locally, ship it to the remote site, turn-up from there. DRBD Proxy (paid feature) for when you need to buffer replication for slow or unreliable network links. Dual-primary (for use with something like GFS) operation. 3 node operation by layering DRBD on top of DRBD.

Yeah, it's cool. It's Free and free. You can get it stock with Fedora and CentOS (probably Ubuntu and others, but haven't tried it yet).

And one last thing - you cannot mount a resource that is "Secondary". So if you are getting crazy error messages that you can neither mount nor even fsck your file system, it's probably in Secondary - don't bang your head against the wall, just do "drbdadm primary <resourcename>". Is clear?

Tape Devices for Amanda

noreply@blogger.com (Dominic Lepiane) — Wed, 26 Jan 2011 16:15:00 +0000

I've found a few times now that my tape server can be a bit of a pain about tape devices. Generally, I have Amanda configured to use /dev/nst0 but the tape drive isn't always this device if I attach other devices (at least other drives). So rather than configuring the "nst0" and then changing it to "nst1" after a few days of realizing the backups aren't working for some reason, I've started using the "tape/by-id" device instead. So my amanda.conf now shows:

changerdev 	"/dev/tape/by-id/scsi-1IBM_3573-TL_00X2U78M1255_LL0"	# tape device controlled by mtx
tapedev 	"/dev/tape/by-id/scsi-35000e11138aa0001-nst"	# the non-rewind

Is clear?

No more mail

noreply@blogger.com (Dominic Lepiane) — Sat, 22 Jan 2011 08:01:00 +0000

One more service down - no more mail. All "real" email has been offloaded or canceled except for uro.mine.nu which has basically just been sacked. I've closed the ports for SMTP, POP, and IMAP. So now this is it, I'm down to just web applications that I'm hosting from home.

What I'd like to do is find a dirt-cheap web-host for this stuff. None of it is high volume - the old URO forums which is still used by some of my gamer buddies (I think - haven't checked in a while) and some personal blogs including this one, and I have a couple personal site type things up. iweb.ca is still offering hosting for $1.67 / mo so I'd like to give them a shot. We shall see, I'll try a couple different services over the next couple months.

One Less Service on Alia

noreply@blogger.com (Dominic Lepiane) — Sun, 26 Dec 2010 10:31:00 +0000

Alia, the latest in the line of servers hosted at home has one less service to host today. I've sacked the DNS service which had in the past provided primary DNS for some the public domains I had used. However, those are all now hosted by the DNS providers. I cleaned up the Bind configuration and closed that port so that it no longer forwards in from the Internet.

The last thing it was doing was DNS for local LAN - the internal DNS to lookup the printer (mostly). This is easily handled by DNSMasq in DD-WRT which is basically a tick-box to replace everything that Alia was doing for DNS. And it automatically adds the lookups for statically configured DHCP hosts so I don't have to setup a host once on the router for DHCP and then again on Alia for DNS.

At this point, it looks like Alia will be the last server I host at home. I've offloaded Jabber and now DNS leaving SMTP and HTTP. SMTP is almost ready to go already as there's only one personal domain for one user using that and that user may retire the domain otherwise we can move to Google Apps along with the other email. And that will leave HTTP which, since I can get shared hosting for less than $2 a month, is an easy one to offload. Not free, but shutting off Alia, even as an energy efficient system (low-power CPU and everything), will save just over $2 / mo in electricity consumption.

So we're coming to the end of an era. It really goes to show just how greatly improved hosted services are today and also the breadth of features you can get from consumer products for home. To have all the trappings of a full network that is so easy to use and so cheap, it is really amazing.

Patch Your #$%^!

noreply@blogger.com (Dominic Lepiane) — Mon, 06 Dec 2010 21:08:00 +0000

According to SANS, the top security threat right now is *drum roll* unpatched applications! *gasp* *shock* Yes, it's blindingly obviously, but organizations (and individuals) are downright negligent in patching desktop applications. Applications that are highly targeted, again no surprise here, Adobe Flash, Adobe Acrobat Reader, Apple Quicktime, and Microsoft Office. And furthermore, "On average, major organizations take at least twice as long to patch client-side vulnerabilities as they take to patch operating system vulnerabilities. In other words the highest priority risk is getting less attention than the lower priority risk."

So patch your #$%^ or else Walter is going to come beat the #$%^ out of your new car while shouting "This is what happens when you find a stranger in the Alps!" .

Or block Flash, Acrobat Reader, and Quicktime - can't say I'd shed any tears for those apps myself ;)

Disk management with Logical Volume Manager (LVM)

noreply@blogger.com (Dominic Lepiane) — Sat, 13 Nov 2010 11:31:00 +0000

There is a lot of documentation on how to use Logical Volume Manager (LVM) Online but I'd like to just go over how I've been using LVM to illustrate some of the strengths and weaknesses.

The initial driving issue which made LVM a killer app was for handling large disks. This one system had an older SCSI RAID attached which only supported 2TB drives (a limitation of 32bit LBA, I think) but the sum of the disks (14 x 300GB) was, well, bigger. The equipment basically let me carve the array into 2TB disks. Using LVM, I can add those Physical Volumes (PVs) to a Volume Group (VG) and create Logical Volumes (LV) of any size desired including, ultimately, the total capacity of the RAID.

Another great feature of LVM is snapshots. Generally, a snapshot means you get a temporally fixed view of the file system for special purposes while general use continues unimpeded by storing the subsequent changes separately. So I can take a snapshot and then backup the snapshot which will assure that the filesystem (in the snapshot) is consistent from the time the backup starts to the time the backup finishes. Snapshots can also be used as a facility to simply roll-back files to a previous state. For example, I take a snapshot, run a test application which modifies a file, then restore that file from the snapshot to revert back.

However, LVM snapshots aren't as elegant as they are on some platforms. To create a snapshot, you must first have some unallocated space in your VG. You then allocate that space to the "snapshot" where disk changes since the snapshot can be stored. The bummer, man, is that this is a fixed amount of space you have to have on-hand and if it fills up, your "snapshot" device fails and if you had say a long backup running, you have to restart that backup. Even with this limitation, however, snapshots are still pretty useful. You can sortof figure out what the minimum size you need for a snapshot and ultimately, if you have snapshot space equal to the live system space, you're snapshot will never fill up.

The last feature I'd like to rant about is Online filesystem resizing. Now this is just absolutely great and very useful especially in concert with handling large volumes and managing snapshots. First of all, if you have a hardware RAID controller which lets you add drives and expand existing arrays as an Online operation, LVM is the layer which will let you expand your volumes to suit. There's two ways of doing this and first is to expand an existing block device (e.g. grow your sda from 1TB to 1.5TB) and you have to do this by modifying the partition table. This is slightly tricky but can be done online. The other way is by adding additional devices. Some RAID controllers (good ones) would let you add a second "logical disk" (or "virtual disk" depending on your vendor's jargon). If you add that additional disk, you simply initialize it as a new PV, add it to your VG and then add whatever you want to your LV.

Take the first example I had where the equipment would only allow 2TB devices. So first, you put all your disks in an array, and because you've got a lot of disks, maybe reserve 1 as a hot spare. So your total capacity is (14 disks - 1 hot spare - 1 for RAID-5 parity) * 300GB = 3600GB. You carve out your first LD and it's 2TB and appears in the OS as /dev/sda. Now generally, you should be putting a partition on your drives, to my knowledge, it's not required, but generally accepted that most disk applications will behave saner if they see a partition. Anyhow, so you've got /dev/sda1, so you initialize it (pvcreate /dev/sda1), you create a volume group (vgcreate myvgblah /dev/sda1), and you spin out your first LV (lvcreate -l 100%FREE -n mylv myvgblah). Hooray, you create your filesystem (mke2fs -j -L bigfs /dev/myvgblah/mylv) and mount it for regular use. Now sometime later you fill up that 2TB and realize that there's a pile of unused space. Well, you carve out another LD with the remaining 1.6TB which appears to the OS as /dev/sdb. Generally, I would expect this device to just show up, no rebooting or any crap like that. So you throw a partition on there, initialize the PV (pvcreate /dev/sdb1), add it to the existing volume group (pvextend myvgblah /dev/sdb1). With this free space, you can either add it all (lvextend -l 100%FREE /dev/myvgblah/mylv) or you could add it incrementally (lvextend -L +100G /dev/myvgblah/mylv) reserving free space for snapshots, additional LVs, and future growth.

Very handy to have all your disks in a pool (your VG) and be able to add logical drives (LVs), snapshot your drives, and incrementally expand your drives.

- Arch

Tab Mix Plus Trick

noreply@blogger.com (Dominic Lepiane) — Fri, 10 Sep 2010 08:40:00 +0000

I had been using a Firefox plugin called New Tab Jumpstart which for new tabs shows like a splash of recently used pages much like you get with Chrome. I found that it was rarely useful and I was only using a single page from it, if anything. So I removed that plugin and found the feature I needed in Tab Mix Plus. You can control what appears in a new tab including a specific URL. Since my "home page" is 3 pages, the "home page" isn't quite what I need, but a specific URL does just the trick.

So there, now I use 2 features of Tab Mix Plus, but it was already #1 in my Essential Plugins simply for the mouse-wheel tab scrolling.

Access Control Lists and Ubuntu

noreply@blogger.com (Dominic Lepiane) — Tue, 03 Aug 2010 07:39:00 +0000

Basic UNIX permissions: Owner, Group, Others and each with Read, Write, Execute, plus a handful of special permissions (setuid, sticky bits, etc). Covers 90% maybe say 99.9%, but not 100%. Sometimes, you really just want to grant more than just the "owner", "group", "everyone" permissions so you need Access Control Lists (ACL).

To get ACL support, your file system must support ACLs. If you're using a file system created this century, it probably supports ACLs. ACL support is usually an option for the file system which can either be set to default on (with tune2fs for example) or can be turned on at mount time with the "acl" option (e.g. in fstab). Some distros simply default the file systems to have acl on (Fedora, RedHat EL) and others don't (Debian, Ubuntu).

To view or manipluate ACLs you also need acl tools: getfacl and setfacl. Distros usually have a package called "acl" available which provides these utilities and with the distros that have ACL defaulting on for file systems (RedHat etc), the package is pre-installed.

First thing you'll want to know is how to read an ACL. The utility "getfacl" (Get File ACL) can show you the ACL. This is what a file looks like that doesn't have an ACL:

getfacl torrentflux 
# file: torrentflux
# owner: www-data
# group: www-data
# flags: -s-
user::rwx
group::r-x
other::r-x

For files that have ACLs, you will see they have a "+" in their permissions list when using your regular ls -l and then you can view the ACL again with getfacl:

$ ls -l
drwxr-s---+  7 www-data  www-data   4096 2009-11-21 15:06 torrentflux

$ getfacl torrentflux 
# file: torrentflux
# owner: www-data
# group: www-data
# flags: -s-
user::rwx
user:archangel:r-x
user:aandrea:r-x
group::r-x
mask::r-x
other::---

As you can see, this is the same directory, but rather than granting global read/execute as under UNIX permissions, we've granted instead read/execute to two specific users with ACLs. These ACLs were created with setfacl (Set File ACL):

$ setfacl -m user:archangel:rx torrentflux
$ setfacl -m user:aandrea:rx torrentflux

If you get some error trying to use "setfacl", it's because the file system does not have the ACL option turned on. Add "acl" to the mount point in fstab and then remount the file system.

The last handy thing you may want to know is that getfacl and setfacl can be used to dump and restore ACLs. With getfacl, you can recursively pull all ACLs and skip files that have only base ACLs (UNIX permissions only). This dump can then be re-applied with setfacl. You will find this useful as not all tools that handle files handle ACLs - specifically tar.

That's Access Control Lists for you. There's no reason not to use them - they're widely supported and very useful.

Enjoy!
- Arch

DSL Speeds

noreply@blogger.com (Dominic Lepiane) — Sun, 01 Aug 2010 08:08:00 +0000

Just came across this article on the BBC:

http://www.bbc.co.uk/news/technology-10774406
"The survey found that for DSL services advertised as being "up to" 20Mbps, only 2% of customers got speeds in the range of 14-20Mbps. Of the others, 32% were getting a 8-14Mbps service and 65%, 8Mbps or less."

2% of users get 75% (or better) of advertised speeds? That's pretty damned harsh. That's the kind of thing that your customers ought to know up front.

But that's DSL for you. The article gives a fairly good explanation of some of the reasons why DSL sucks. What we need is fiber-to-the-home and none of this DSL crap:

http://www.newswire.ca/en/releases/archive/February2010/04/c6687.html
http://seekingalpha.com/article/197137-competition-is-starting-to-weigh-on-rogers-communications?

Upgrade from Ubuntu Server 8.04 to 10.04

noreply@blogger.com (Dominic Lepiane) — Thu, 01 Jul 2010 13:55:00 +0000

Well, decided that today was the day to do the upgrade of my server, Alia, from 8.04 to 10.04. And, since I'm able to post, you can guess that it went generally fine.

It was quite brilliant really. I just ran the following command and followed the prompts:

do-release-upgrade --proposed

So far, everything looks good. New kernel (2.6.32 from 2.6.24), MySQL (5.1 from 5.0), Apache, Postfix, slapd, etc etc. The one that looks like needs some babysitting is Dovecot which requires an updated config file.

Everything else worked "out of the box". And I'd consider this system fairly customized in the sense that a wide variety of applications have been installed but where possible (and almost entirely), taken from the Ubuntu repositories.

So if there's anyone else out there still waffling, do it! Do the upgrade!

- Arch

Keeping Copies of Group Emails

noreply@blogger.com (Dominic Lepiane) — Wed, 23 Jun 2010 15:05:00 +0000

One of the things that's a bit ghetto of groups in Google Apps is that groups are really just a glorified alias file. Users cannot manage their subscription, get emails delivered in batches, and there's no message archive unlike Google Groups or a Mailman managed list. And this is the same problem with Microsoft Exchange (at least up to 2007, probably 2010 too).

Okay, so ranting aside, here's a couple quick hacks to squeeze a couple features out of groups in GA.

Archiving. Create a mailbox, add it to the group. Shazzam! This is better in Exchange were you can share that mailbox easily with many users and limit them to read-only access so people aren't deleting your archive.

Mailing list features. Well, you're only answer for now is going to be to forward messages to a mailing list. So point mylist@example.com to mylist-example-com@googlegroups.com and members should subscribe directly to the Google Group instead.

Aliases. Now this is one feature I would have preferred in the face of the above limitations of GA groups. That is, if I've got a group called "hibuddy@example.com", I also want to have "heybuddy@example.com" and other variations. So here, create a mailbox called "hibuddy@example.com" and rename (or create) a group called "hibuddy-group@example.com". You can add as many aliases as you want to the mailbox, and then configure that mailbox to just forward to the group.

Ciao
- Arch

Clonezilla Good! Fire Bad!

noreply@blogger.com (Dominic Lepiane) — Wed, 12 May 2010 09:36:00 +0000

Clonezilla, quite simply, is tha bomb. It's really fast, very flexible, it will do everything including your laundry.

You get basically two styles of cloning systems (or disks in general). Either one at a time with the LiveCD or many at a time with a multicasting server. I've only tried the liveCD method since I was simply doing two hosts. And in my case, I was dealing with the 'doze which is always more of a pain than it should be. So here's what I did to clone a Windows Server 2003 install to two hosts.

Get the Windows host installed and setup with all the desired applications but not joined to the domain

Create an unattended install file for Sysprep (it's a quick wizard)

SAVE THAT SYSPREP FILE (for some reason, sysprep will destroy this as incriminating evidence?)

Sysprep the host - this will strip the Security ID (SID), computer name, and remove it from the domain (if you had it on one) and it shuts down the host

Get the Clonezilla LiveCD and something for external storage

Boot the sysprepped host from the liveCD

Basically defaults all the way, it will ask what the storage media for system images is, what disk or partition to copy (I did it by partition, though you could do disk if you wanted to keep the partition info)

It ripped a 5.4GB base server install into a ~2GB image in about 5 minutes

Reboot, reconfigure PC with a name, join it to the domain, etc

Then on each target host,

Boot from the Clonezilla LiveCD

Attach the external storage

Follow the wizard

It restored the above partition for me in 2 minutes, 17 seconds

Reboot, give the PC a name, put it in the domain, etc

Repeat for each host you are cloning

The crazy thing I was finding was that "proprietary" cloning tools were hard to find. Basically, Symantec has been buying up everyone in the field, killing the products, and then telling everyone to use Ghost which at least since when they acquired Norton and until recently, did not take offline disk copies. Instead, you have to install the application in the OS (which you'll note with Sysprep is impossible since the host is SHUT OFF) and it does a "hot backup". It just doesn't work for cloning at all. WTH?

But apparently, between some more sophisticated usage of sysprep and using a "clonezilla server", you could have your PCs, say in a lab, all doing PXE boot, re-imaging themselves, and picking up their name and domain information simultaneously. Once setup, you could do a lab of, I don't know what size, but whatever the max number of clients is (presumably dozens or hundreds) in less time than it takes to get a Starbucks.

- Arch