CentOS 7 grub virtio error migrating to KVM

When migrating CentOS7 from physical servers or vmware / hyperv, it typically does not have virtio drivers built into it. This will often give grub or initramfs errors.

Once the image has been migrated, you have three choices:

1. Boot using Rescue mode from a latest CentOS or RockyOS (v8+), and choose to mount the existing machine (option 1). Version 8+ has the virtio drivers built in, so will see the drive no problem.
2. Boot using recovery mode. This is usually a grub menu option. We find this typically has virtio built in
3. Change emulation for the drive to IDE and boot as normal

Once booted into the os (or mounted via rescue cd and chroot /mnt/sysimage), change to root user if you aren’t already.

Run the following command to rebuild the initramfs:

mkinitrd -f --allow-missing --with=virtio_blk --preload=virtio_blk --with=virtio_net --preload=virtio_net --with=virtio_console --preload=virtio_console /boot/initramfs-$(uname -r).img $(uname -r)

Then just shut down the VM and boot it back up. new initramfs will now have the virtio drivers and be able to see the disk.

chronyd NTP server for local network

Configuration on Redhat / CentOS / Rocky Linux / Almalinux

yum install chrony

These are the important bits in your /etc/chrony.conf file:

local stratum 10
manual
allow 192.168.0.0/16
allow 10.10.0.0/16
ratelimit interval 3 burst 16

local stratum is a bit like a trust score, lower is more trusted. 10 is high enough that you wont affect much if your particular server goes horribly wrong.

manual keyword specifies that you’re able to use chronyc on the command line to manually set the time. I always leave this enabled but you can choose to not include this if you prefer.

allow directive specifies the networks that should be allowed. specify multiple times to allow multiple networks. Alternatively you can just say allow any, but please do read about dns reflection ddos attacks first.

ratelimit allows rate limiting replies on a per-ip address basis. I always specify this just in case some client software goes haywire. interval is not in seconds, but actually 2 to the power of X seconds. so interval of 3 actually means 8 seconds. burst is how many responses are allowed above the threshold before enforcing this interval.

Dont forget to restart chronyd:

systemctl restart chronyd

Example chrony.conf configuration file

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (https://www.pool.ntp.org/join.html).
pool 2.rocky.pool.ntp.org iburst

# Use NTP servers from DHCP.
sourcedir /run/chrony-dhcp

# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift

# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second.
makestep 1.0 3

# Enable kernel synchronization of the real-time clock (RTC).
rtcsync

# Enable hardware timestamping on all interfaces that support it.
#hwtimestamp *

# Increase the minimum number of selectable sources required to adjust
# the system clock.
#minsources 2

# Rate limit responses
ratelimit interval 3 burst 6

# Allow NTP client access from local network.
allow 10.0.0.0/8

# Serve time even if not synchronized to a time source.
local stratum 10
manual
# Require authentication (nts or key option) for all NTP sources.
#authselectmode require

# Specify file containing keys for NTP authentication.
keyfile /etc/chrony.keys

# Save NTS keys and cookies.
ntsdumpdir /var/lib/chrony

# Insert/delete leap seconds by slewing instead of stepping.
#leapsecmode slew

# Get TAI-UTC offset and leap seconds from the system tz database.
leapsectz right/UTC

# Specify directory for log files.
logdir /var/log/chrony

# Select which information is logged.
#log measurements statistics tracking

nftables installation

You can choose your own firewall policy implementation, but we use nftables:

yum install nftables

I usually edit this file:

[root@XXXXXXXXXXXXXXXXX admin]# cat /etc/sysconfig/nftables.conf
# Uncomment the include statement here to load the default config sample
# in /etc/nftables for nftables service.

#include “/etc/nftables/main.nft”

I swap out the commented out include line for the following:

include "/etc/nftables/nftables.nft"

And then inside that config file I put all my rules:

[root@XXXXXXXXXXXXXXXXX admin]# cat /etc/nftables/nftables.nft
table inet filter {
chain INPUT {
type filter hook input priority 0; policy accept;
iif "lo" accept
ct state established,related accept
ip protocol icmp icmp type echo-request accept
ip6 nexthdr ipv6-icmp icmpv6 type 1 counter accept comment "accept ICMPv6 dest unreachable"
ip6 nexthdr ipv6-icmp icmpv6 type 2 counter accept comment "accept ICMPv6 packet too big"
ip6 nexthdr ipv6-icmp icmpv6 type 3 counter accept comment "accept ICMPv6 time exceeded"
ip6 nexthdr ipv6-icmp icmpv6 type 4 counter accept comment "accept ICMPv6 parameter problem"
ip6 nexthdr ipv6-icmp icmpv6 type 128 icmpv6 code 0 counter accept comment "accept ICMPv6 echo request"
ip6 nexthdr ipv6-icmp icmpv6 type 129 icmpv6 code 0 counter accept comment "accept ICMPv6 echo reply"
ip6 nexthdr ipv6-icmp icmpv6 type 133 icmpv6 code 0 counter accept comment "accept ICMPv6 router solicitation"
ip6 nexthdr ipv6-icmp icmpv6 type 134 icmpv6 code 0 counter accept comment "accept ICMPv6 router advertisement"
ip6 nexthdr ipv6-icmp icmpv6 type 135 icmpv6 code 0 counter accept comment "accept ICMPv6 neighbor solicitation"
ip6 nexthdr ipv6-icmp icmpv6 type 136 icmpv6 code 0 counter accept comment "accept ICMPv6 neighbor advertisement"

tcp dport 22 ip saddr X.X.X.X accept
udp dport 123 accept
drop
}

chain OUTPUT {
type filter hook output priority 0; policy accept;
}
}

This will allow SSH port 22 access to your system from a predefined X.X.X.X IP, and open access to NTP. This could be dangerous if you’re putting this on a public network, so either restrict this to your local IPs by adding a ip saddr X.X.X.X/X accept on the end, or just know what you’re opening yourself up for by reading up on NTP software compromises and NTP reflection ddos attacks.

Testing using ntpdate

And ofcourse we need to do some testing….

Testing NTP server using ntpdate:

ntpdate -q 103.43.119.204
server 103.43.119.204, stratum 3, offset 0.000072, delay 0.02623
29 Oct 08:03:48 ntpdate[14770]: adjust time server 103.43.119.204 offset 0.000072 sec

As long as the offset is tiny it should be good to go.

Cisco ASA 5512-X Allow outside/external ssh access

To enable external SSH access to a brand new Cisco ASA 5512-X, follow these example instructions:
X.X.X.X is the IP address you wish to allow access from.
yourusername is the username you want to use
XXXXXXXXXXXXXXXXX is your password

ciscoasa# conf t
ciscoasa(config)# ssh X.X.X.X 255.255.255.255 outside
ciscoasa(config)# crypto key generate rsa general-keys modulus 2048
ciscoasa(config)# username yourusername password XXXXXXXXXXXXXXXXX
ciscoasa(config)# aaa authentication ssh console LOCAL
ciscoasa(config)# ssh version 2

You may also want to set an enable password:
ciscoasa(config)# enable password XXXXXXXXXXXXXXXXX
(You should probably make this different to your user password)

CVE-2015-0235 “Ghost” Linux glibc vulnerability

For those that missed it, CVE-2015-0235 (aka Linux Ghost) was announced today which details a glibc library bug that is still on many Linux distributions. glibc is used by many applications including webservers, mail servers, php applications etc.

The specific bug was in the gethostbyname() and gethostbyname2() functions (hence “ghost” name!), so only applications that call these are potentially vulnerable. Even then, there is limited scope for exploitation, but there already has been a PoC for the Exim mail server developed so it certainly is possible (given the right conditions). Luckily, these two functions lack IPv6 support, so many newer applications and services have chosen to stop using these functions, and instead use IPv6-enabled functions instead. As has been seen however, some popular ones such as Exim do still use the older IPv4-only functions.

The bug itself has been around since 2000, and was inadvertently patched in August 2014 without realising the implications. Unfortunately since the security issues were not detected at the time, many Linux distributions didn’t back-port the patch into Linux distributions. This is what has occurred today.

Accordingly, we have now taken the following actions:

All standard-level webservers globally and chroot environments have been patched and restarted between 6:30AM and 7:15AM this morning.

All mail servers were patched and restarted between 6:30AM and 7:15AM this morning.

We will be taking the following actions that may result in a few minutes downtime for some sites tonight:

All protected-level webservers globally and chroot environments will be patched and restarted overnight at varying times (critical maintenance alert will have been received by all affected customers). As these are all behind load balancers, this shouldn’t have any end-user affect.

RackCorp monitoring services will be restarted throughout the day. This may result in some performance graphs being slightly skewed at times.

In addition:

VM Hosts will have no noticeable impact.

Load Balancer services will have no noticeable impact.

RackCorp API services will have no noticeable impact (however some unrelated database maintenance is scheduled for tonight that may result in queries taking a few seconds longer than usual).

Content delivery services are unaffected.

Network services are unaffected.

In terms of customer patch cycles, we are treating this as a critical bug for some customers, and moderate (normal patch cycle) for others depending upon the attack vector surface area. All affected customers will have received an email accordingly. If you are unsure of the impact for your specific services, please raise a support ticket accordingly.

Additional useful resources:
Ars Technica Writeup on Linux Ghost
gethostbyname() vs getaddrinfo() by Erratasec

RackAdmin – RackCorp’s in-house developed web hosting and network management platform

One of the key principles behind our company is emphasis on in-house custom developed solutions based on open source software.

A bespoke open source foundation provides us with

• Maximum reliability as bugs can be patched before official fixes are available
• Maximum performance thanks to streamlined and lean code
• Maximum flexibility and extensibility to meet our customers and our own specific needs
• Minimising cost and licensing which helps keeps prices competitive

These principles form the basis on which is why we built RackAdmin, the backbone of our company since foundation. A custom, in house developed management platform which provides end-user account management, option and service provisioning including reseller management and geo-location and automated billing. All within a slick and easy to use interface and an API.

Read more »