Category Archives: Uncategorized

A tale of a small business environment refresh – part 1

During my spare time I support the IT infrastructure for a small company located in the east part of Iceland. Late last year I decided it was time to refresh the infrastructure so I spent some time figuring out what would be the best way.

We wanted to have the system hosted locally since they have, in the past, lost connectivity so moving everything into a cloud-hosted environment wasn’t an option this time (although I expect that the next time we do a refresh we will move away from the on-premises setup). And since they are located as far away from where I live I wanted to move away from the single server setup that has been in place for a long time. I might have gone a little bit overboard on overdesigning the environment for such a small business but the results have over my expectations.

We did a cost analysis of the current setup, and calculated the cost for the new infrastructure for 5 years and found out that it came down to about the same cost as to host the main application used by the business for three years with the application provider. If we had decided to go that way we would always have had to buy some infrastructure to host basic monitoring tools/supporting applications to monitor the network environment anyway, but instead we can host it on the new environment as well.

I ended up going with the following specifications for the hardware and network infrastructure:

Network:

  • 2x Mikrotik CCR2004-1G-12S+2XS (Core routers)
  • 2x Mikrotik CRS518-16XS-2XQ-RM (Core switches)

Servers:

  • 3x SuperMicro CSE-116AC10-R706WB3 chassis, each with the following specs
    • Supermicro MBD-H12SSW-INR-B motherboard
    • AMD EPYC 7313 16C CPU
    • 128GB RAM
    • 2x Samsung PM9A1 512GB boot disks
    • 3x Mikron 7450 Pro 1.92TB NVMe disks for CEPH
    • Supermicro AOC-S25GC-I4S-O (4x 25G ethernet adapter)
    • Dual PSU

The hypervisor I decided to use was Proxmox 7.4 (latest release when I did the setup of the environment). For the backups I used the Proxmox Backup Server, and am running it on a old HPE Proliant ML350 Gen8 server.

For the network I decided to setup the Mikrotik CRS518 in a MLAG setup. Although the MLAG functionality of the RouterOS software isn’t perfect (connectivity is lost for about a minute if one of the switches goes down since the system id of the switches doesn’t stay static like it does on all enterprise switches, but I am sure that Mikrotik will fix that in a later release). The routers then have a 25G link to each switch, setup in a LACP configuration. I created new VLANs for all networks (workstations, servers, infrastructure) and setup VRRP between the core routers for each VLAN for redundancy. A simple access list is configured for the infrastructure VLANs limits access to the infrastructure. I have thought about adding a firewall running on the virtualization cluster, but at this time I haven’t set one up yet.

The main access switch has 1G link to each core switch configured as a LACP port-channel.

Backups are kept locally, but also replicated to Tuxis, which provides a PBS instance where you can replicate your backups to, meaning that if we have to restore files/VMs quickly (if we need data in the short term) but if we have a disaster we do have a copy of the data with longer retention at Tuxis. But if you feel like it you can also host your own PBS instance anywhere and store your backups there – it is amazing how easy it is to manage PBS instead of some of the backup solutions I’ve seen in the past!

For Internet redundancy we have connections from different providers – the main Internet connection is fronted by a Fortigate 40F firewall which advertises the default route through OSPF to the core routers. Then we have a Mikrotik L41G-2AXD&FG621-EA 4G router with a connection through a different provider that has a static route with a high priority on the core routers which acts as a backup. This has proven to be very stable setup for the Internet connectivity so far.

Here is a high level drawing of the infrastructure:

So far the performance has been great. My biggest worry was that the Ceph performance would be bad enough so that I would have to refactor everything and use ZFS with replication instead. A very limited testing has shown about ~1-2GB/s in writes, and 2+ GB/s in reads. Each node has only 3 OSDs (I thought about partitioning the disks and use two OSDs per disk, but after my initial testing I was more than happy with the performance) so things are kept as simple as possible.

There is a old APC SmartUPS 1000 in place that can run the environment for about 7 minutes before it looses power. So far we have only had a single incident where all of the hosts lost power (the power can be somewhat unstable in the area). During the bootup process there was a issue where two out of three hosts didn’t detect at least one out of the three NVMe disks for Ceph so the servers didn’t have the minimal amount of OSDs to boot up and I had to manually restart the hosts to get the disks to appear again. This seems to be a bug in the SuperMicro BIOS, but since then I have upgraded to a newer version and so far I haven’t seen this before (I had already seen it during the setup phase so I wasn’t all that worried when we had the issue). If we see this over and over again I will consider adding a PCIe adapter to handle the NVMe disks.

For the money, I think this environment is great, and with the exception of issue with the NVMe disks, and the MLAG issue with the Mikrotik switches I could not be happier with the result for the money. I rarely have to touch the environment, as of now I still do manual patching of the Proxmox hosts, and the Mikrotik infrastructure. All of the server patching has been automated and I don’t think I will need to touch that any time in the future.

All of the environment is monitored by CheckMK, which is running in a container on a Linux VM. CheckMK monitors the virtual guests, the Proxmox infrastructure, Ceph, hardware and the network infrastructure.

At last I have been playing around with Security Onion to monitor the environment for security events but I am still in the evaluation phase – it looks good as a open source product and seems to have most of the features I would want for such a small environment – the only thing I feel like I would want to add is to have Qualys + Kenna for vulnerability scanning for both OS updates and third party applications.

RC!

Last year I happened to stumble up on some videos of electric RC cars. I watched couple of those and started reminiscing about the time I had spent back in ~2004 when me and a friend ordered couple of Traxxas Revo’s with the TRX 3.3 engine. It was a blast, but the bad thing about living on this lovely island we call Iceland is that the temperature here is pretty low, and the engine settings really often need some modifications so we spent the better half of all sessions doing tuning before we could start bashing.

But watching those videos of those electric cars was pretty interesting as you could just charge up and start to bash right away! I started researching and ended up ordering a Traxxas Maxx v2. I had a blast bashing it couple of times, but a little later I got the chance to get my hands on a Traxxas TRX4 crawlers (Bronco 2021 body). Now….this is where things started to get interesting. I took it along with me when the family went on a hike and man….I realized I had so much more fun crawling then I did bashing.

So – since then I have built a Vanquish Phoenix VS4-10, and I just finished a Axial SCX10 Pro build (well, I still need to do some work on it, but it is in a driveable condition). Building those kits has been the best hobby I can think of. I’ve searched for a hobby to spend my free time on for a long long time and I think I have finally found it.

There are two issues though. Number 1 – this hobby is a money dump!…..however I am pretty sure that this is a lot cheaper than if I had gone into hobbies like fly fishing or hunting. Number 2 – living on a island in the middle of nowhere with a population of ~400.000 means that access to crawler parts that are not made by Traxxas is very much non-existent. So I need to order pretty much everything except original parts for my TRX4 from abroad. But even if I can find cheap stuff then shipping along with the duty fees always adds a premium to every part so I have to think carefully before making any orders.

But never the less I am lucky enough that there are stores in Germany and Asia that do ship things pretty cheap (and some even ship things pretty fast, thank god for cheap Fedex shipping!) so if I make sure to put together a sizeable order it won’t be anything crazy. I am going to create a page here sharing the stores I primarily use along with a list of my parts just for fun if it helps anyone that is in a similar situation.

Yet another summer is coming to an end…

First post in long time!

Yet another summer is coming to an end. Work starts again tomorrow and things get back to normal.

During the summer I saw that I needed to rebuild a small SMB environment for a friend and I decided on using Mikrotik for the networking (switches, routers), SuperMicro for the servers and Proxmox for the virtualization layer. I’m going to document my process here and find out the good, the bad and the ugly around those three vendors. Can’t wait to get that started but I expect to get the equipment in my hands in the next ~4 weeks or so.

This is going to be somewhat over-designed environment but I am excited to see how those vendors stack up against the enterprise vendors I work with most of the time.

On-premise Kubernetes

For the better part of the year I have been playing around with Kubernetes on-premise. While testing random solutions I didn’t realize what can of worms I just opened! ……Don’t get me wrong – the whole Kubernetes ecosystem is extremely fun to “play” in.

But after trying multiple solutions a colleague of mine pointed me to a project called Rancher. This project is pretty cool!

The project makes the installation extremely easy (yes yes, I sound like a sales person) but this was the most straight-forward product I had seen (and yes, I have seen a few) in this space.

Out of the box the project offers multi-cluster management, support for AKS, EKS and support for other managed solutions as well as a on-premise installation using either RancherOS (a custom Linux distro for running Kubernetes) or using roll-your-own VMs/bare metal instances (using for example CentOS). It can integrate with vSphere to spin up instances…..and they have a decent Active Directory integration for authentication/authorization.

Rancher is deployed on a dedicated Kubernetes cluster (if it is set up for HA) that should just be used for Rancher. Then you can go ahead and add your own clusters from AKS/EKS or on-premise. It is a nice single pane of glass for operating your Kubernetes clusters. If you have environments all over the place it can help you gain better control of the environments as well as offer a single place to interact against for things like deployments.

While I won’t go into details (the documentation simply speaks for itself) I recommend you take a look at this project if you plan to start using Kubernetes for your organization, or even just to play with your own stuff.

And the best part? The project is fully open source. Rancher are also working on a persistent storage solution (Longhorn) and they offer professional services/support if you need some help along the way.

They also have a mini Kubernetes distro called K3s – it is a (very) small instance of Kubernetes that you can run on pretty much anything that can boot Linux and be managed in the same way.

Simply put, this is an amazing project! 🙂

Openconnect and GlobalProtect VPN!

Hi!

Just tried the globalprotect support in openconnect 8 (8.02 in Fedora 29).

Very simplified version:

sudo openconnect --protocol=gp your.vpn.gw.com

Worked liked a treat! Hopefully I can stop using the offical Linux client now.

Now – hopefully NetworkManager-openconnect drops in support for connecting to globalprotect VPN soon! 🙂

Bgrds,
Finnur

Palo Alto GlobalProtect on Fedora

After spending some serious time trying to get GlobalProtect 4.1.2 to work on Fedora 28 (and probably 27 earlier this year) I finally managed to get it working. It is almost embarrassing how easy it was…

  1. Replace /etc/redhat-release and /etc/os-release with info from RHEL 7 or CentOS 7
  2. Profit.

Yep….it’s sucky….but at least it shows that this works. Maybe it is possible to modify some file that lists supported operating systems……will have to look into that later on.

Always read the release notes….and the supported OS lists…..and the error logs. Even better if you do it all in the same evening to puzzle this amazing solution together……

FYI: The error I was getting was: Error: Gateway my.gateway.hostname: The server certificate is invalid. Please contact your IT administrator.

Neat RSS reader for MacOSX

Howdy all,

If you are still hurting after the closure of Google Reader and have not found a replacement for it you could check out Vienna. It has everything that I have been looking for in a RSS reader and it’s also open source !

Bgrds,
Finnzi

Back to Mac OS X :O

Howdy,

Finally my old trusty Thinkpad T61 decided to leave this world. I went a month or two without having a private laptop (I try not to use my work machine for private matters) but finally I could not go a minute longer ! I searched quite a bit for a nice machine I could use for running Fedora, but I have been using Fedora for few years ony my trusty T61. However, after spending few hours searching for a good machine (The T430 with some high end specs was quite interesting) I settled for a Macbook Pro Retina 13″.

This machine is amazing ! The last time I was using Mac OS X was on a 2007 Macbook with 13″ screen (1366×768 resolution if I remember correctly). That was quite sad experience since it was unable to do things like scroll a flash-heavy site without feeling sluggish. However the new machine has nothing I could call sluggish. Everything runs quite smoothly. If I could complain about anything it would be the size of the SSD and the write speed after doing a full disk encryption. However it is not trivial and the disk still feels a lot faster then my old 7.2K 320GB disk in the T61 🙂

At first I disliked the whole idea of making Mac OS X feel a bit “iOS” like. Applications are delivered through the App Store etc. I however quickly realized that it quite nice to have the apps delivered this way. Updates all go through Apple. If there is a security bug etc the users get updates delivered centrally. Just that feature is brilliant (and quite old if you look at how Linux distributions deliver apps/updates:)).

Well, let’s see if I keep on praising Mac OS X after using it for more then a week 🙂

Bgrds,
Finnzi

Bye Gnome 3, Hello Cinnamon !

Howdy all!

Few weeks ago I was once again crying about how I do not like Gnome 3 (normal and fallback modes). Once again I went searching for a shoulder to cry on but suddenly remembered reading something about MATE and Cinnamon in some Linux magazine.

Found some screenshot of both and saw that both had some potential to them so I decided to try them both.

Well, to make a long story short I now have Cinnamon configured on my Laptop and Workstation. I love it !

Large shout out to the Cinnamon team, Cinnamon truly rocks ! 🙂

Bgrds,
FOG

First post!

Howdy all,

After a “small” mistake I found out that I did not have backups of my WordPress installation so all my posts are gone…..which is “awesome”…..*shruugggg*

However this gives me a opportunity to redesign the site and maybe even put something useful here which was always the plan anyway 🙂

Lets see how it goes….

Bgrds,
Finnzi