ZFS convert stripe to striped-mirror

I’m a huge fan of ZFS because of its performance and other features like snapshots, transparent compression. In fact I had switched to FreeBSD for servers just because it had native ZFS support. But as of Ubuntu 16.04, ZFS is officially supported for non-root partitions.

Now I’m migrating a FreeBSD server to Ubuntu 16.04 with ZFS for data storage – this is happening because I need support for some special hardware which has drivers only for Linux and I do not have a spare server machine of same capacity in terms of memory/disk/processor.

My case –
Here’s the zpool layout on my existing FreeBSD server:

        zroot              ONLINE       0     0     0
          mirror-0         ONLINE       0     0     0
            diskid/DISK-1  ONLINE       0     0     0
            diskid/DISK-2  ONLINE       0     0     0
          mirror-1         ONLINE       0     0     0
            diskid/DISK-3  ONLINE       0     0     0
            diskid/DISK-4  ONLINE       0     0     0

zroot ONLINE 0 0 0

mirror-0 ONLINE 0 0 0

diskid/DISK-1 ONLINE 0 0 0

diskid/DISK-2 ONLINE 0 0 0

mirror-1 ONLINE 0 0 0

diskid/DISK-3 ONLINE 0 0 0

diskid/DISK-4 ONLINE 0 0 0

Each of those disks are 1TB in size and the layout here is something known as RAID 10, or striped mirroring. Striped mirroring can be extended to more than four disks but in my case, I have two pairs of disks. Each pair is mirrored and the each such mirror is striped, illustrated as in the image below:

Image taken from techtarget.com, their trademark/copyright holds.

The advantage of this layout is that you get read speed of four disks, and write speed of two disks and a failure tolerance of two disks (but in different mirrors) at the same time.

I have a spare 1TB disk which I can use for preparing a new server using a low-end machine for migration. I remove one of the disks from the live server so the pool there runs in a degraded state. The removed disk is used in the new server. So I create this zpool in Ubuntu:

zpool create zdata /dev/disk/by-id/disk1-part3 /dev/disk/by-id/disk2-part3
zpool status

        zdata          ONLINE       0     0     0
          disk1-part3  ONLINE       0     0     0
          disk2-part3  ONLINE       0     0     0

zpool create zdata /dev/disk/by-id/disk1-part3 /dev/disk/by-id/disk2-part3

zpool status

zdata ONLINE 0 0 0

disk1-part3 ONLINE 0 0 0

disk2-part3 ONLINE 0 0 0

The pool created here is a plain simple stripe. To convert this into a striped-mirror, the zpool attach command has to be used:

zpool attach zdata disk1-part3 disk3-part3
zpool attach zdata disk2-part3 disk4-part3

1 2	zpool attach zdata disk1-part3 disk3-part3 zpool attach zdata disk2-part3 disk4-part3

With this, the pool now becomes a striped mirror:

	zdata            ONLINE       0     0     0
	  mirror-0       ONLINE       0     0     0
	    disk1-part3  ONLINE       0     0     0
	    disk3-part3  ONLINE       0     0     0
	  mirror-1       ONLINE       0     0     0
	    disk2-part3  ONLINE       0     0     0
	    disk4-part3  ONLINE       0     0     0

zdata ONLINE 0 0 0

mirror-0 ONLINE 0 0 0

disk1-part3 ONLINE 0 0 0

disk3-part3 ONLINE 0 0 0

mirror-1 ONLINE 0 0 0

disk2-part3 ONLINE 0 0 0

disk4-part3 ONLINE 0 0 0

Perfect! 😀

Group based HTTP basic authentication using Nginx and MySQL with help of Lua

Recently I moved from Apache to Nginx on one of my servers due to increase in traffic. But I was using HTTP Basic authentication with group based authorization on Apache in this manner:

<Location /foo>
  AuthType Basic
  AuthName Restricted
  AuthBasicProvider file
  AuthUserFile /etc/apache2/htpasswd
  AuthGroupFile /etc/apache2/groups
  Require group somegroup
</Location>

AuthType Basic

AuthName Restricted

AuthBasicProvider file

AuthUserFile /etc/apache2/htpasswd

AuthGroupFile /etc/apache2/groups

Require group somegroup

</Location>

However, there’s no AuthGroupFile in nginx. But LUA, a programming language is supported in nginx. So here’s how I used LUA and MySQL for achieving this:

location ~ ^/restricted {
  set $user_group 'somegroup';
  access_by_lua_file '/etc/nginx/authenticate.lua';
}

location ~ ^/restricted {

set $user_group 'somegroup';

access_by_lua_file '/etc/nginx/authenticate.lua';

}

Now the real magic comes in the authenticate.lua script, I’m posting the code below which is available in Github as well:

-- basic configuration of the script

local cookie_domain = ".yourdomain.com"
local db_username = "dbuser"
local db_password = "dbpasswrod"
local db_socket = "/tmp/mysql.sock"
local db_name = "dbname"

-- end configuration

local session = require "resty.session".open{ cookie = { domain =  cookie_domain } }
local remote_password

if ngx.var.http_authorization then
	local tmp = ngx.var.http_authorization
	tmp = tmp:sub(tmp:find(' ')+1)
	tmp = ngx.decode_base64(tmp)
	remote_password = tmp:sub(tmp:find(':')+1)
end

function authentication_prompt()
	session.data.valid_user = false
	session.data.user_group = nil
	session:save()
	ngx.header.www_authenticate = 'Basic realm="Restricted"'
	ngx.exit(401)
end

function authenticate(user, password, group)
	local mysql = require "resty.mysql"
	local db, err, errno, sqlstate, res, ok
	
	db = mysql:new()
	if not db then
		ngx.log(ngx.ERR, "Failed to create mysql object")
		ngx.exit(500)
	end

	db:set_timeout(2000)
	ok, err, errno, sqlstate = db:connect{
		path = db_socket,
		database = db_name,
		user = db_username,
		password = db_password
	}

	if not ok then
		ngx.log(ngx.ERR, "Unable to connect to database: ", err, ": ", errno, " ", sqlstate)
		ngx.exit(500)
	end

	user = ngx.quote_sql_str(user)
	password = ngx.quote_sql_str(password)
	local query = "select 1 from http_users where username = %s and password = SHA2(%s, 224) and (find_in_set('superadmin', groups) > 0 or find_in_set('%s', groups) > 0)"
	query = string.format(query, user, password, group);
	res, err, errno, sqlstate = db:query(query)

	if res and res[1] then
		session.data.valid_user = true
		session.data.user_group = group
		session:save()
	else
		authentication_prompt()
	end
end

if session.present and (session.data.valid_user and session.data.user_group == ngx.var.user_group) then
	return
elseif ngx.var.remote_user and remote_password then
	authenticate(ngx.var.remote_user, remote_password, ngx.var.user_group)
else
	authentication_prompt()
end

-- basic configuration of the script

local cookie_domain = ".yourdomain.com"

local db_username = "dbuser"

local db_password = "dbpasswrod"

local db_socket = "/tmp/mysql.sock"

local db_name = "dbname"

-- end configuration

local session = require "resty.session".open{ cookie = { domain = cookie_domain } }

local remote_password

if ngx.var.http_authorization then

local tmp = ngx.var.http_authorization

tmp = tmp:sub(tmp:find(' ')+1)

tmp = ngx.decode_base64(tmp)

remote_password = tmp:sub(tmp:find(':')+1)

end

function authentication_prompt()

session.data.valid_user = false

session.data.user_group = nil

session:save()

ngx.header.www_authenticate = 'Basic realm="Restricted"'

ngx.exit(401)

end

function authenticate(user, password, group)

local mysql = require "resty.mysql"

local db, err, errno, sqlstate, res, ok

db = mysql:new()

if not db then

ngx.log(ngx.ERR, "Failed to create mysql object")

ngx.exit(500)

end

db:set_timeout(2000)

ok, err, errno, sqlstate = db:connect{

path = db_socket,

database = db_name,

user = db_username,

password = db_password

}

if not ok then

ngx.log(ngx.ERR, "Unable to connect to database: ", err, ": ", errno, " ", sqlstate)

ngx.exit(500)

end

user = ngx.quote_sql_str(user)

password = ngx.quote_sql_str(password)

local query = "select 1 from http_users where username = %s and password = SHA2(%s, 224) and (find_in_set('superadmin', groups) > 0 or find_in_set('%s', groups) > 0)"

query = string.format(query, user, password, group);

res, err, errno, sqlstate = db:query(query)

if res and res[1] then

session.data.valid_user = true

session.data.user_group = group

session:save()

else

authentication_prompt()

end

if session.present and (session.data.valid_user and session.data.user_group == ngx.var.user_group) then

return

elseif ngx.var.remote_user and remote_password then

authenticate(ngx.var.remote_user, remote_password, ngx.var.user_group)

else

authentication_prompt()

end

The group authentication script looks for users and groups in a table called http_users. Since this is a script you can modify the way users are searched for in the database or change the database altogether!
The lua modules required to run this script are: resty.mysql, resty.session, resty.string and cjson. Though the passwords are stored in the database as a SHA224 hash, the comparison of the password is done by the database itself. I did not convert the password to hash before sending it to database, so you may want to review this in case you are using remote database. I’m using local database over Unix socket so it doesn’t matter much.

The table and triggers I have for the same:

CREATE TABLE `http_users` (
  `username` varchar(15) NOT NULL DEFAULT '',
  `password` varchar(56) NOT NULL DEFAULT '',
  `groups` set('superadmin','group1','group2','group3','group3') NOT NULL DEFAULT '',
  PRIMARY KEY (`username`),
  UNIQUE KEY `password` (`password`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

DELIMITER $$
CREATE trigger http_user_ins before insert on http_users for each row
begin
SET NEW.password = SHA2(NEW.password, 224);
end;
DELIMITER $$

DELIMITER $$
CREATE trigger http_user_upd before update on http_users for each row
begin
if LENGTH(NEW.password) != 56 then
SET NEW.password = SHA2(NEW.password, 224);
end if;
end;
DELIMITER $$

CREATE TABLE `http_users` (

`username` varchar(15) NOT NULL DEFAULT '',

`password` varchar(56) NOT NULL DEFAULT '',

`groups` set('superadmin','group1','group2','group3','group3') NOT NULL DEFAULT '',

PRIMARY KEY (`username`),

UNIQUE KEY `password` (`password`)

) ENGINE=InnoDB DEFAULT CHARSET=latin1;

DELIMITER $$

CREATE trigger http_user_ins before insert on http_users for each row

begin

SET NEW.password = SHA2(NEW.password, 224);

end;

DELIMITER $$

CREATE trigger http_user_upd before update on http_users for each row

begin

if LENGTH(NEW.password) != 56 then

SET NEW.password = SHA2(NEW.password, 224);

end if;

end;

DELIMITER $$

The triggers are required to convert the INSERT or UPDATE statements into SHA224. I’m using MySQL’s SET data type to ensure that the group value is fixed. The same values can be used by Nginx in $user_group variable before specifying the access_by_lua_file directive.

March 8, 2016 2

FreeBSD IPFW NAT and Jails

IPFW in FreeBSD has built-in support for NATing and the configuration syntax is same as that of natd. It took me quite some time to figure out how to NAT for jails while ensuring that certain jails can have public IPs.

Configure the nat on one of the IP addresses:

ipfw nat 123 config ip a.b.c.d

1	ipfw nat 123 config ip a.b.c.d

When using stateful firewall, the NAT rule for incoming traffic must appear before check-state:

ipfw add 100 nat 123 from any to a.b.c.d in
ipfw add 101 check-state

1 2	ipfw add 100 nat 123 from any to a.b.c.d in ipfw add 101 check-state

Other rules (service ports) can be placed below this:

index=300
for port in $tcp_service_ports; do
	ipfw add $index allow tcp from any to me $port in
	ipfw add $index allow tcp from me $port to any out
	index=$((index+1))
done

index=400
for port in $udp_service_ports; do
	ipfw add $index allow udp from any to me $port in
	ipfw add $index allow udp from me $port to any out
	index=$((index+1))
done

index=300

for port in $tcp_service_ports; do

ipfw add $index allow tcp from any to me $port in

ipfw add $index allow tcp from me $port to any out

index=$((index+1))

done

index=400

for port in $udp_service_ports; do

ipfw add $index allow udp from any to me $port in

ipfw add $index allow udp from me $port to any out

index=$((index+1))

done

Then the NAT rule for outgoing traffic:

ipfw add 800 nat 123 ip4 from 10.0.0.0/8 to any out

1	ipfw add 800 nat 123 ip4 from 10.0.0.0/8 to any out

Notice above, I am NATing only traffic that comes from 10.0.0.0/8 . I allocate jails an IP on that subnet (unless I need a public IP for the jail). If the source is not mentioned in the rule, it will NAT even public IPs!

And finally, the outgoing ports:

index=500
for port in $out_tcp_ports; do
	ipfw add $index skipto 800 tcp from 10.0.0.0/8 to any $port out setup keep-state
	ipfw add $index allow tcp from me to any $port out setup keep-state
	index=$((index+1))
done

index=600
for port in $out_udp_ports; do
	ipfw add $index skipto 800 udp from 10.0.0.0/8 to any $port out keep-state
	ipfw add $index allow udp from me to any $port out keep-state
	index=$((index+1))
done

index=500

for port in $out_tcp_ports; do

ipfw add $index skipto 800 tcp from 10.0.0.0/8 to any $port out setup keep-state

ipfw add $index allow tcp from me to any $port out setup keep-state

index=$((index+1))

done

index=600

for port in $out_udp_ports; do

ipfw add $index skipto 800 udp from 10.0.0.0/8 to any $port out keep-state

ipfw add $index allow udp from me to any $port out keep-state

index=$((index+1))

done

The catch here is that we jump to the NAT rule only if the traffic comes from 10.0.0.0/8 . If the traffic is coming from somewhere else (for example, a public IP allocated to one of the jails), it will hit the second rule and directly allow it.

Make sure you have the rule to allow loX traffic if you have separate clone interfaces for each jail.

Final touches:

echo firewall_enable=YES >> /etc/rc.conf
echo firewall_nat_enable=YES >> /etc/rc.conf
echo firewall_script=/etc/ipfw.rules >> /etc/rc.conf
echo gateway_enable=YES >> /etc.rc.conf
service routing restart
service ipfw restart

echo firewall_enable=YES >> /etc/rc.conf

echo firewall_nat_enable=YES >> /etc/rc.conf

echo firewall_script=/etc/ipfw.rules >> /etc/rc.conf

echo gateway_enable=YES >> /etc.rc.conf

service routing restart

service ipfw restart

The firewall script ipfw.rules must to contain other rules for services, icmp, etc not mentioned here.
Everything working smoothly now – ip4 from private jails, ip4 and ip6 from others 😀

December 7, 2014 4

FreeBSD ipfw: add_dyn_rule: Cannot allocate rule

One of the servers I run has FreeBSD 10. It hosts a high traffic Magento site. Magento being a very heavy application, requires a dedicated server. The site’s performance is very bad when it is hosted on VPS — or perhaps that depends on provider / needs tuning. Not my site. My task was to move it to dedicated server so I don’t have to consider all that stuff.

As someone new to FreeBSD, I try to stick to tools and utilities that are provided by FreeBSD itself and do not rely on those provided by other BSDs. This rule is quite flexible, but I can’t cite examples of relying on tools by other BSDs that I’m using right now. So, naturally, for firewall I chose IPFW which is FreeBSD’s own firewall. The other firewalls supported by FreeBSD are PF (which comes from OpenBSD) and IPFilter (which comes from NetBSD).

Continue reading “FreeBSD ipfw: add_dyn_rule: Cannot allocate rule” →

September 7, 2014 1

The move from Linux to FreeBSD

About 2 months ago, I had a spare VPS at my host, Hetzner. So I decided to play with FreeBSD which was being offered for Hetzner servers and VPSes.
That’s how the whole thing started. I didn’t have much problems getting the concepts because it belongs to *nix family of OSes and I have been a pure Linux user since 2008.

First of all the basic difference between FreeBSD and GNU/Linux is that Linux is just the kernel and GNU is the userland. In layman’s terms, the hardware interface is called Linux, while the rest of the part: the shell, core tools, etc are GNU.It’s a piece from there, another from somewhere else and merging the whole thing into one collectively known as GNU/Linux. Linux itself cannot boot without GNU and GNU will not work without Linux (Yes, there is a GNU kernel project called GNU Hurd, but I don’t how far that went).
In FreeBSD, the whole thing is a complete unit. FreeBSD was derived from the original AT&T Unix and open sourced. You can read more about the differences at over-yonder.

Continue reading “The move from Linux to FreeBSD” →

June 7, 2013 74

Securing FreeBSD server with Fail2Ban and IPFW

I’ve been playing with a FreeBSD machine for a while now and my primary server now runs FreeBSD 😀
So I came across this problem: installing Fail2Ban with IPFW.

FreeBSD has three different firewalls, so it’s difficult for any upstream application to decide on what kind of setup it should advocate. There is no one-size-fits-for-all. I read about various firewalls, and since I wanted to stick with FreeBSD only, I decided to use IPFW.

Continue reading “Securing FreeBSD server with Fail2Ban and IPFW” →

April 18, 2013 0

Category: FreeBSD

ZFS convert stripe to striped-mirror

Like this:

Group based HTTP basic authentication using Nginx and MySQL with help of Lua

Like this:

FreeBSD IPFW NAT and Jails

Like this:

FreeBSD ipfw: add_dyn_rule: Cannot allocate rule

Like this:

The move from Linux to FreeBSD

Like this:

Securing FreeBSD server with Fail2Ban and IPFW

Like this:

Categories

Connect with me

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Categories

Tags

Connect with me