Viking Geek Y'All

A Little Learning

Hi Marvin

About three months ago I took a new job at DrillingInfo as a Systems Engineer. A lot of my job is working as a bridge between systems administration and the development teams. As a result I’ve been working with a bunch of tools I hadn’t previously worked with. Two of these tools operate in a very similar space.

Ansible is a configuration management, and service orchestration tool. The default configuration is for managed servers to run without agents at all, but be managed entirely via SSH from the Ansible server. Configuration of Ansible jobs is written in a combination of YAML, with Jinja2 which allows for a fairly powerful set of options, while also remaining easy to read and understand. Jobs executed on the servers typically resemble simple shell scripts, which as a systems person, make a lot of sense to me. Prior to coming to DrillingInfo I was aware of Ansible but hadn’t done any work with it. So far I’m impressed. It reminds me of SaltStack, with which I have prior experience, in a lot of ways (Jinja2/Yaml) however it’s also completely different.

Another tool we are using is Chef. Chef is one of the most widely used configuration management utilities. Companies like Facebook,, and Riot Games all use Chef as an important part of their Continuous Integration/Continuous Delivery tool chain. Chef uses Ruby for it’s configuration syntax. As a result I don’t understand it as well. Chef also has an agent called “Chef-Client” loaded on all managed servers. There is also a utility loaded on your workstation called “Knife” which is used to interact with the Chef server. In general Chef is a very powerful configuration management utility. However the learning curve is far steeper than that of Ansible.

In any case Monday September 8th, I will be going to the first of a two day Chef Fundamentals class. After that I will be going to an Austin DevOps Meetup for an Ansible Tech Talk & QA. I love learning new things and am very much looking forward to next week.

If you have any perspective on Chef, Ansible, or SaltStack please let me know in the comments below!


100 Days of Whole Foods – Week 1 Update

My last update was a quick intro to the 100 days of whole foods challenge my wife and I are doing.

This is a super brief update to say it’s very hard to meet the requirements when you oversleep which seems to be a real problem for me this week. I’ve only had breakfast once since starting the challenge. I’ve also skipped lunch most days this week.

Hopefully next week will be better.


100 Day Whole Food Challenge

My wife and I have committed ourselves to the 100 days of Real Food challenge. This is technically the second time we’ve started this, I changed jobs at the start of our last 100 day stint, and as a result we slipped. We’re back up and running again.

Today August 4th, 2014 marks day 1 of 100. So far my breakfast consisted of two large cups of black coffee and two fried eggs. I have some smoked chicken I’ll be eating for lunch.

I’ll be updating this page frequently with progress and my thoughts on the day. One of the hardest things is going to be cutting out sugar. I cut almost all soda out of my diet the last time we started the challenge, and have managed to keep that limited quite a bit. My wife has a meal plan she’s working from which includes a grocery shopping list. Hopefully this will help keep us on target and on plan.

More updates to come.


Fun With Foreman

One of the projects I have going on at work involves setting up an auto-discovery & auto-provisioning system. This should help on two fronts. First we will have a database of facts surrounding the hardware in our datacenter. Second we will be able to remotely install operating systems via pre-defined provisioning templates. This should cut the time it takes to set new equipment up in the datacenter from days or weeks down to hours.

The tool we’re using for this is called Foreman which claims to be “A complete lifecycle management tool for physical and virtual servers.” Foreman is written in Ruby and includes a number of additional tools including Puppet, which is used for fact gathering while booting a micro kernel on otherwise unknown systems.

While this auto discovery feature of Foreman is amazing and useful. The PXE + Preseed components in Foreman unfortunately are far from fully baked. This ultimately lead to our pulling the plug on Foreman and mothballing it.

We’re going to re-evaluate Cobbler as the provisioning component is arguably more important than the discovery components.

With luck a follow up post outlining Cobbler and our installation with follow shortly. Foreman looks like a promising project, with some great features. Unfortunately we were simply unable to meet one of our must have requirements. Sometimes finding out what doesn’t work is as important as getting things working.


PFSense Part 2

Previously when I wrote about PFSense I was using a dual core Opteron system with a couple of NICs as my router. My good friend Amos started using PFSense as a result of some discussions we had that week. He didn’t like using an ancient Pentium 4 system as his router. Or perhaps it was his wife who wasn’t so sure about the giant tower in the middle of his desk. In any case he went out and picked up a couple of Thin Client computers to replace the giant tower he was using.

For an exceptionally reasonable price he picked up a pair of MaxSpeed MaxTerm 8300 units. These have an 800mhz Via C3 processor, and 256mb of SD Ram 133mhz. Which for a computer is practically unusable. However as a router this is still quite a bit more hardware than anything Linksys, Buffalo, or similar companies sell.

These devices consume about 12w of power while operating, and have no moving parts in them. Which makes them completely silent, cool, and inexpensive to operate. I suspect the Maxterm takes more power to run than a Linksys router would, but the other benefits of PFSense outweigh the small change in power consumption.

After taking some time to tune the configuration files last night, I now have the new MaxTerm router in place and running the show in my home. This will allow me to re-task the dual core Opteron system as a FreeNAS file server for the house.


Customer Service Done Right

I have read several articles over the last several years that customer service is dead. Or at least I’ve read articles asking if customer service is dead. It however doesn’t take reading about customer service online to know that there are many examples of terrible customer service out there. Rather than complain about companies that suck at customer service, I’d like to take time today to talk about a few companies that don’t.

First we have Buffalo Technology. When my wife and I moved to Austin, TX from Indianapolis, IN I had to move out about three months before she did. In that time she packed up the house and shipped everything to me. unfortunately the power adapter for our second wireless router didn’t make the trip. I had reached out to Buffalo via their contact form asking if it was possible to order a power adapter, and if so how much it would cost. Later that same day I had a response saying that they didn’t sell power adapters by themselves, but he would be happy to mail me one free of charge. When I provided my address he let me know that “since you’re in Austin, TX if you like I can just have one sitting at our front desk with your name on it if you would like to drive by and pick it up. This was well above and beyond my expectations, and it had the exact effect I think Buffalo was looking for. The next time I need to buy a router/access point/commodity NAS, I’m likely to purchase one of their products.

Second, is a company called Kershaw. Kershaw makes pocket knives. They have some fairly inexpensive knives made in China, and some very high quality pocket knives made in the United States. I owned a middle of the road, made in the USA Kershaw Leek. When I experienced a hand full of minor problems with the knife I messaged Kershaw’s customer service via their website at something like 2 in the morning. Not only did they send me replacement parts with no questions asked. They sent me three or four of each part I needed, shipped next day, so they arrived on Thursday. All folding knives I’ve purchased since then have been made by Kershaw.

Finally, and most recently I had a fantastic experience with not one, but TWO companies related to the same problem. My TCX X-Cubed motorcycle boots, purchased in December broke. Not in a way that would have prevented me from riding in them, but it was annoying none the less. I reached out to TCX via their customer service form online, and received a response the following morning from their office in Italy. TCX opened a trouble ticket for me, but then directed me to talk to the company I had purchased my boots from, which in my opinion was acceptable. So I reached out to the second company in this event Revzilla and shared the same information I had previously shared with TCX. Within ten minutes of me reaching out to Revzilla, TCX American partner/importer contacted me saying that “Italy had forwarded my trouble ticket to them” and wanted to know how they could help. I let them know I had followed their instructions and reached out to Revzilla. I mentioned that if I didn’t hear anything from them I would get back in touch. The American partner had high praise for Revzilla, and assured me I’d be taken care of. In the time it took me to read the response from the American Partner, Revzilla had provided me an RMA number, and a pre-paid UPS shipping label. I couldn’t have asked for more!

It’s now one week from when I originally contacted TCX, and I have confirmation from Revzilla that TCX has accepted my return, and I’ve been granted in store credit for a full replacement of my boots. Sure it’s a little annoying to be without something for about two weeks when it’s all said and done, but you know what. Both companies met or exceeded my expectations at every turn.

The moral of this story is, just because many companies suck at customer service doesn’t mean they all do. I’d recommend rather than complain about the companies that get it wrong. Take your dollars to the companies that get it right. Stop rewarding bad behavior, and do reward the good. One of those articles I linked at the beginning of this post says that almost universally bad customer service is our own fault for continuing to purchase products at places that get it wrong.

I can’t recommend the companies listed above any more highly! If you’re in the market for motorcycle boots do yourself a favor and shop at Revzilla, and take a long hard look at TCX. If you need a good high quality pocket knife it’s impossible to go wrong with a Kershaw. Finally all home routers really are roughly the same. Do yourself a favor and don’t overlook Buffalo just because they don’t have the same kind of flashy, high dollar marketing a company like Linksys has.

If you’ve got a killer customer service success story, please leave it below in the comments!


LastPass Security Tools

In my last post I mentioned LastPass while discussing how to resolve a problem I was having. In this post I want to highlight the “Security Check” feature which makes LastPass a fantastic tool for managing your passwords.

If you click the LastPass plugin Icon, and then tools, the top option is “Security Check”. This will then prompt you for your LastPass password. After which it will analyze your password vault and give you an overall score which looks something like this.

Scrolling down from the overall score will give you details on which sites, if any, have a duplicate paassword. It will highlight the overall strength of each password in your vault. And it provides details on how recently your passwords have been updated. I’ve made it a practice to run the Security check about once a quarter. It’s amazing how much random junk you collect over the course of three months that can be cleaned up when you’re aware of it.

In any case, if you’ve been looking for a password management system that helps you make good decisions regarding your online security, I highly recommend taking a look at LastPass.


Fixing Lastpass & Firefox

Let’s start out by discussing a little bit about what LastPass is. LastPass is a hosted online password management utility. It provides a facility to both generate random passwords for every site you access on the internet, as well as the ability to store text based notes. This is coupled with strong encryption, preventing LastPass from ever sending or receiving decrypted data from their servers. Also included are browser plugins for all of the major browsers, on all major platforms. This allows LastPass to detect if the site is already known from the store, and auto-fill your login and password information. Or detect that a new site is detected, and offer to generate a password, and save it to your vault. In short LastPass is an very valuable tool in improving your overall online security as you can have a unique password for every site you visit. Finally LastPass includes tools to audit your vault for duplicate or weak passwords. This allows you to little by little go through each of the known sites and update/replace weak or duplicate passwords.

However LastPass isn’t without some problems. Over the last three weeks I’ve noticed the most recent version of LastPass on the most recent version of Firefox leads to the browser hanging whenever you enter a password into a site not currently tracked by LastPass. This morning I found a solution to the problem that allows you to continue to use LastPass, while also preventing the browser from locking up. If you are having problems with your browser locking up whenever you’re entering a password perform the following change. Click the LastPass icon in your browser, and select preferences. Then click on Notifications in the left hand menu. Finally un-select the “Click Icons in Fields” option. The interactive bits in the fields seems to be the core cause of the browser slow down / hangs.

Let me know below if you find a larger scale underlying problem. For now this is a pretty solid workaround.


Hosting Improvements With Docker

In conjunction with my new Static Files Blogging I’m also working on re-designing how the hosting environment itself is set up. The model I’m currently working on in a test environment is using Docker to drop groups of sites into individual containers. Each of these containers will have its own instance of Apache running mod_php & MySQL. In front, running in yet another container will be nginx. This will manage named virtual hosting for all of the sites, and will then proxy_pass out of its container and into the appropriate container for the site being requested.

This should help mitigate most security concerns of having a completely exploited system if a single web application is compromised. Rather only the container in which that site is running would be exploited, thus limiting the surface area and scope of the attack.

I need to put a little more effort into exactly how I’m going to split up the individual sites. I also need to determine what the least privilege I can assign to each Docker container will be while still providing a rich web application environment for the users. I don’t expect to run into resource problems on the new dedicated server even though I’ll be running several distinct instances of Apache and MySQL. However if problems arrise I should be able to make adjustments pretty easily. I’ll update this again once I have more to report, including some of the implimentation notes.


Networking With PFSense

I’ve been meaning to upgrade the network in our house since we moved in. I had intended to take care of setting things up the right way immediately after our Internet was migrated from the apartment we moved out of. Of course you know what they say.

The best-laid plans of men and mice often go awry

When I say I’ve been meaning to upgrade the network, I specifically am referring to using something a little more powerful than a Buffalo router running DD-WRT to take care of the networking. It’s not that there’s anything exactly wrong with DD-WRT, it’s just I want and expect more from my router/firewall appliance. This week I finally acquired a new power supply for an older dual core AMD Opteron system I had laying around and installed PFSense which is a FreeBSD based firewall distribution. My background includes a lot of working with FreeBSD. As a result I’m fairly comfortable with the way they do things. PFSense allows for the easy installation of additional packages and features, which is possible on DD-WRT, but because of the hardware you’re very limited in what you can usually add.

Just getting PFSense running on a machine isn’t enough to warrant replacing a functional solution based on DD-WRT. The first big change I made is devices wired to the router are on a different network than wireless devices. I have routing in place between the two networks so it’s a seamless separation, however it opens the door to do more things in the future. The next change I made was installing HAVP – HTTP Anti-Virus Proxy. This is configured as a transparent web proxy, which integrates with ClamAV to scan web pages and download attachments for viruses before they ever reach your desktop or laptop computer. I don’t expect this to be a silver bullet that protects the two Windows computers from thoughtless browsing, but it might help a little. By default like most home routers function as a DNS Forwarder, I’ve been meaning to run my own caching name server for a while. As a result I installed the Unbound package, which is a validating, recursive, and caching DNS resolver. Unbound also supports dnssec – DNS Security Extensions.

DNSSEC was designed to protect the Internet from certain attacks, such as DNS cache poisoning [0]. It is a set of extensions to DNS, which provide: a) origin authentication of DNS data, b) data integrity, and c) authenticated denial of existence. Finally I’ve wanted to install an IDS for some time. Again PFSense makes this easy by including Snort an open source IDS system.

I don’t expect to see any significant changes with how my network runs and operates with PFSense at the edge of my network over DD-WRT. DD-WRT is a powerful home router, able to do far more than most home users would ever demand of it. I made the move to PFSense for the following reasons:

  • I’m a giant nerd who can’t leave things alone.
  • I have a soft spot for FreeBSD and things powered by it.
  • I hate having hardware just sitting around not being used

After I’ve been on this system for a while I’ll write a follow-up that includes some of the statistics for what Snort and HAVP helped protect us from.