NFS cannot allocate memory and No such file or directory errors

NFS cannot allocate memory seems to be common issue a lot of people face while mounting NFSv4 shares, as revealed from search results on Google. Even I faced it when trying to mount a NFS share exported from my desktop on my Raspberry Pi. It didn’t give the error on the Pi, though. I discovered this issue when debugging this problem and tried to mount on localhost.

Initially it appears to be a kernel issue, but it turned out not to be one. After researching a lot about the problem I understood that it is a error in exports configuration!

Basically, in NFSv4, there is a / (root) under which all shares are exported and this must have the fsid=0 parameter specified in the configuration.
At the client, while mounting you are not supposed to specify the full path to the resource, instead it must be relative to the NFS Root.

The following example would clear it up –

I’ll try to export /home and mount it on localhost with following configuration:

# /etc/exports: NFS file systems being exported.  See exports(5).
/home *(rw,no_root_squash,no_all_squash,no_subtree_check)

And when I run the mount command:

Linux ~ # mount localhost:/home /mnt/nfs -v
mount.nfs: timeout set for Tue Jan 29 11:59:27 2013
mount.nfs: trying text-based options 'vers=4,addr=::1,clientaddr=::1'
mount.nfs: mount(2): Cannot allocate memory
mount.nfs: Cannot allocate memory

Now I’ll change the exportfs configuration to:

Linux ~ # cat /etc/exports 
# /etc/exports: NFS file systems being exported.  See exports(5).
/home *(fsid=rw,no_root_squash,no_all_squash,no_subtree_check)

Running the mount command again,

Linux ~ # exportfs -rv
exporting *:/home
Linux ~ # mount localhost:/home /mnt/nfs -v
mount.nfs: timeout set for Tue Jan 29 12:01:58 2013
mount.nfs: trying text-based options 'vers=4,addr=::1,clientaddr=::1'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=::1'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying ::1 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying ::1 prog 100005 vers 3 prot UDP port 60646
localhost:/home on /mnt/nfs type nfs (rw)

Now, in the above output, you can see that while it failed to mount with NFSv4, but it succeeded in mounting NFSv3. This comes from the newer way of exports which is partially explained in the manual exports(5).

The solution to the No such file or directory is simple, you have to use / as the path to mount the root and if you are exporting some subdirectories, they need to be relative to / in the mount command instead of the full path.

Now modifying the mount command accordingly:

Linux ~ # mount localhost:/ /mnt/nfs -v
mount.nfs: timeout set for Tue Jan 29 12:15:11 2013
mount.nfs: trying text-based options 'vers=4,addr=::1,clientaddr=::1'
localhost:/ on /mnt/nfs type nfs (rw)

Now if I export a subdirectory inside /home, say my home folder /home/nilesh then it would be available via NFSv4 to clients as /nilesh and not /home/nilesh.

There seems to be a big problem controlling access in this model, you can’t have different permissions for /home and /home/nilesh. If you export /home as read-only, /home/nilesh will also get mounted read-only no matter what.

If you export /home as read-write and /home/nilesh as read-only, then clients would be able to write into all directories on the share, when the share root is mounted, i.e. /, but if the client mounts /home/nilesh it will be read-only.

I’m not sure about how to control access in NFSv4 for subdirectories, since you cannot have more than one entry for fsid=0. If you know the answer, do comment.

Routing for multiple ISP links on Linux

Many of us have multiple ISP uplinks for different reasons like reliability, download limit, etc. but there is no clear guide [at least, I couldn’t find any] on routing for multiple links.

I came across the same problem — I got a new fiber connection, but it is a capped one and I have a slow ADSL link which is not capped. So I obviously would want to use the ADSL link for low priority downloads while fiber for browsing.
After a lot of researching around, I found the solution. Probably a specific case, since my computer has just one Ethernet card and I want to use both the links simultaneously on the same computer.

The network setup scenario:

  • There’s a main router (NAT, DHCP Enabled), which has WAN port, WiFi and LAN ports with IP Address 192.168.0.1
  • There’s another WiFi router (which is used in WDS Bridge mode) with IP Address 192.168.0.2
  • The ADSL Router (NAT Enabled, DHCP disabled) has IP Address 192.168.0.3  and is connected to main router’s LAN port, since fiber must be connected to WAN port for PPPoE connection.
  • An Ethernet Switch is wired to the second WiFi router.
  • The Linux box is connected to the Ethernet Switch.

I setup two IP Addresses on the Linux box: 192.168.0.4 and 192.168.0.5 as follows:

# ifconfig eth0 192.168.0.4
# ifconfig eth0:1 192.168.0.5

Now, I want to route all traffic originating from 192.168.0.4 (the default IP Address for eth0) via fiber, and all traffic originating from 192.168.0.5 via ADSL Router.
To accomplish this, I use policy based routing (CONFIG_ADVANCED_ROUTER must be enabled in kernel configuration) using iproute2.

Routing:

We need to create a new table which will be looked up if traffic is from the second IP and otherwise use the default route.
And then the policy routing rule which specifies that if traffic is from second IP, lookup the other table.

# echo 10 adsl >> /etc/iproute2/rt_tables
# ip route add default via 192.168.0.1
# ip route add default via 192.168.0.3 table adsl
# ip rule add from 192.168.0.5 table adsl

And you’re done. Traffic from applications using 192.168.0.4 (it will be used as default, if you don’t specify anything) will get routed through 192.168.0.1 and those who use 192.168.0.5 will get routed through 192.168.0.3.
As a simple test, you can do this:

$ curl text.whatisyourip.org
$ curl --interface 192.168.0.5 text.whatisyourip.org

That should give you two different IP Address, the first one your Fiber’s public address and the second one your ADSL’s public address.
If that works, you can add the routing code to run at startup or may be even in system network configuration. Every Linux distribution has it’s own way of doing that, so I won’t cover that. If you’re a Gentoo user, you can add it in /etc/conf.d/net as follows:

config_eth0="192.168.0.4/24 192.168.0.5/24"
dns_servers_eth0="192.168.0.1"
routes_eth0="default via 192.168.0.1
default via 192.168.0.3 table adsl"

postup() {
        if [ "${IFACE}" == "eth0" ]; then
                ip rule | grep 192.168.0.5 &> /dev/null
                [ $? -ne 0 ] && ip rule add from 192.168.0.5 table adsl
        fi
}

Now, many applications like Firefox, Chrome, etc. don’t support listening/using to a specific interface/IP Address. There’s a simple solution for the problem I found on Daniel Lange’s Blog.
All you have to do is, download this libc wrapper and compile it as follows:

$ gcc -nostartfiles -fpic -shared bind.c -o bind.so -ldl -D_GNU_SOURCE

Once you have done that, you can use it as follows:

$ LD_PRELOAD=bind.so BIND_ADDR=192.168.0.5 firefox

If you visit text.whatisyourip.org from there, you should see your ADSL public IP. For applications which support binding to a specific address, just configure it do so.

Blog converted to WordPress

There’s no doubt that WordPress is the most popular blogging platform. I used to be a WordPress hater since I had a disaster with the same in 2007 when I had jumped into the blogging bandwagon (that blog does not exist anymore).
And after that incident, I always used Drupal, and in my opinion, Drupal is a very powerful and great Content Management System, suitable for nearly any and every task.

But, when I started my new personal blog, I was in no mood to use WordPress and Drupal was just too heavy for a simple personal blog. So I started out with Octopress.
While most people move from WordPress to Octopress, I did the opposite and I have reasons for it.

Read More

The Minimum Attendance Requirement

One day, I was studying for one of my exams, and this thought came to my mind – Why do colleges keep the minimum required attendance level so high? The first answer that comes to anyone’s mind is: Because students bunk classes.

Okay, but why do students bunk classes?

The reason is simple: Because the lectures are boring. The hard fact is, very few people take up teaching out of dedication. Most of the people are those who for some reasons, could not get a job in their industry and are in the teaching job.

My college specifies a minimum attendance requirement of 75%. And this is almost same across major colleges in India. I don’t know what’s the case in IITs, NITs, etc. I have a few friends from those but nobody ever complained about attendance.

Read More

The Flawed Evaluation System

It’s semester end again at college and we’ve got hell lots of journal-writing work. The journal-writing work is a indespsible part of every semester in Engineering, at least in my University and possibly every other University out there.

The general trend in my college, a lot different from others is – A student (usually the topper) is given the format and sample by the professors. They are then instructed to write it (we do it by hand, unfortunately), distribute their copies to friends, and the chain goes on. So basically, every student has the same journal, differing by some extent due to unavoidable errors that creep in the copy-chain and laziness.

The job of writing journals isn’t very hard that one cannot write it by using their brain, but it is a very boring task to do. Hence everyone (that includes me) refers to be somewhere in the middle of the copy chain, and never in the front. Everyone mindlessly copies the things without even thinking what they are writing in their journals – this means they even copy nonsense. But there’s one thing for sure – nobody would try (exceptions do exist, and they are of the order of 0.01%) to even understand or alter the code which was originally written.

Read More

Getting CherryPy Working With uWSGI

Since Python 3 is the future, I directly started with Python 3 for my projects. While, There are some frameworks for Python 3, but I’m not a fan of frameworks. I prefer to glue the components myself and write the application – even if it involves more work and boilerplate code, and that’s because of freedom.

When you use a particular framework, you’re bound to a few rules and and modifying the behavior of the rules gets quite difficult, unless you know the framework you’re using from head to tail completely in detail. Half knowledge is very bad.

I knew about CherryPy since Python 2, and when I studied how to write applications using that, it became my favorite framework. CherryPy is basically a minimal framework, or more specifically a server which handles the common headaches that are required to do when writing a web application in Python. The common headaches are like mapping a request into Controller/Action, handling HTTP errors, authentication, etc. No, these tasks are not difficult, but are rather time consuming because of the size of the code involved.

So, why not use a pre-built framework? I’m negating my own statement, eh? Yes. In this case, freedom is not a strict requirement as much as it is required in the application logic, and CherryPy supports Python 3 – Awesomeness.

CherryPy can be deployed by various methods – It has an inbuilt HTTP server, it can be used as a FastCGI, SCGI and CGI server as well. The point is, HTTP parsing is slow, and doing a slow task using an interpreted language – doesn’t get me. I could very well use CherryPy’s HTTP server as the server for website and stay quiet. That doesn’t work because we’ll be running a relatively slower code (as compared to C/C++, in which web servers are usually written) even for static resources! This is a big waste of resources.

CherryPy uses the Flup1 module for FastCGI/SCGI/CGI implementation and unfortunately, there’s isn’t a working release of the same for Python 3. It’s development seems to have stalled. I was able to install and use the Flup module in Python 3. It installed successfully, but it seems some of the code in it wasn’t ported to Python 3, running 2to3 fixed it. What about any Bugs in it? – The biggest problem. It doesn’t have a public release! If the author had released a version, I would have just used it.

uWSGI is a application container written in pure C. It has a lot of features like multiple protocol support, process management, easy configuration and a lot. You can learn more about it at the official website.

As per the Python PEP-33332, the WSGI application (a callable) should be named as application, should accept two parameters and return bytes after calling the function that comes as second parameter (usually named as start_response).

The same specification is used by uWSGI. But the problem here is, how to get CherryPy running with uWSGI, since the default method is to spawn a HTTP server if you use cherrypy.quickstart() or, use the cherryd command. You can spawn a FastCGI/SCGI/CGI server with the cherryd command (which requires the Flup module).

In the uWSGI case, the server processes are handled by uWSGI, so you need not spawn any processes in your application. After a lot of searching around, reading manuals and experimenting I finally found a workaround to get this thing working.

Here’s a simple code which deploys two apps (basically two classes) using cherrypy:

#!/usr/bin/python3

# Script written by NileshGR (http://nileshgr.com)
# This script is under BSD license

import cherrypy

class One:
  @cherrypy.expose
  def index(self):
    r = cherrypy.response
    r.headers['Content-Type'] = 'text/plain'
    return "Hello World"

class Two:
  @cherrypy.expose
  def default(self, *args, **kwargs):
    r = cherrypy.response
    r.headers['Content-Type'] = 'text/plain'
    content = "Positional arguments\n\n"
    for k in args:
      content += k + "\n"
    content += "\nKeyword arguments\n\n"
    for k in kwargs:
      content += k + ": " + kwargs[k] + "\n"
    return content

def application(environ, start_response):
  cherrypy.tree.mount(One(), '/', None)
  cherrypy.tree.mount(Two(), '/par', None)
  return cherrypy.tree(environ, start_response)

The first two classes are CherryPy applications. Notice the last part, after line #28, we’re defining a function named application(environ, start_response) as specified by PEP-33332.

In the function, we mount the first application at mount point / and the second application at mount point /par and finally return cherrypy.tree(environ, start_response which is transfers control to CherryPy. The secret here is in the fact that, cherrypy.tree is a WSGI compatible application and which is why this works!

Quoting the text from cherrypy.tree doc page (pydoc):

cherrypy.tree = class Tree(builtins.object)
 |  A registry of CherryPy applications, mounted at diverse points.
 |  
 |  An instance of this class may also be used as a WSGI callable
 |  (WSGI application object), in which case it dispatches to all
 |  mounted apps.

Starting the uWSGI server to run our CherryPy application:

uwsgi --http :8080 --wsgi-file cherrypy_uwsgi.py

I passed --http to uWSGI because I don’t have a WSGI capable HTTP server on my PC where I do all this development and test work. Anyway, if HTTP with uWSGI works, then the other protocols like WSGI, FastCGI, etc. that uWSGI implements should also work (isn’t that obvious?).

See uwsgi --help for more information on how to spawn a server using the protocol you need.

Screenshot of the app running in my browser:

cherrypy_uwsgi

The credit to what made me try this out and get it working goes to @unbit who helped me out on Twitter and pointed me to this article on CherryPy wiki.

Trac and Git: The Right Way

This post is about configuring Trac and Git to work together properly. As you might be knowing, Git is a DVCS – Distributed Version Control System and Trac is a project management system. By default, Trac supports SVN, but there are plugins for Git and Mercurial. I don’t know if there are plugins for other repositories like Bazaar, etc.

In the default configuration in Trac’s Git plugin, the repository is not cached and setting repository_sync_per_request to blank doesn’t stop Trac from syncing repositories on each request. This is a big performance disadvantage and can big a real big trouble if you have lots of repositories with quite a lot of commits.

Read More

SQLite Replication

SQLite is probably the world’s simplest Relational Database Management System. Basically it’s a C library which can be embedded in programs easily. There is no server/client mechanism, the database is a single file. For small work loads, it often makes no sense to use a big RDBMS package like MySQL or PostgreSQL, unless of course you need the special features they provide.

So, I came across this situation where I need replicate SQLite database. The problem arised because, data redundancy was needed across multiple servers and the program in question was supporting SQLite, MySQL and PostgreSQL; but one of the servers had only workload for PostgreSQL database and installing MySQL for the small amount of data the program handled wasn’t sensible. The other two servers had pure MySQL workload. Also, the updates needed to be propagated. So there is the deadlock.

I searched around to find nothing useful, but I remembered there is a cron called incrond which can watch files and directories for events using INOTIFY and execute commands when specific events occur. The solution is almost there. All I need is a script to copy the database file to other servers when data was written to it. I wrote a simple script which would copy the files to other servers; at first I tried rsync with incremental updates, but it didn’t work, because SQLite doesn’t delete data actually when rows are deleted as written in this FAQ at official website. The data is simply marked for deletion and reused during future inserts.

So I guess you now know why incremental updates won’t work: even if I delete a row, the size of database is going to remain same. Actually it should work with rsync’s checksum method, but it didn’t for me. Nevertheless, since the data size was pretty small, I used scp to transfer the database.

The script is ready, now you need a incrontab entry for IN_MODIFY event i.e. to run the script when the file is modified. Here’s a small example:

/var/lib/database.sqlite IN_MODIFY /scripts/copy_database.sh

That’s it. Whenever the file is modified, it will be transferred to the other servers. Wait, the story doesn’t end there.

I tried modifying the database on primary server and it did properly replicate to other servers, but the new changes weren’t visible neither on the primary server nor on the other servers. In the sense, the program could see the deleted row. This is mainly because of caching in memory. The solution was to reload the program (which didn’t cost much in this case). So I just added the commands for reloading the program in the script before syncing the database to other servers on the primary server and then two ssh commands in the script which execute the reload command on the other two servers.

/etc/init.d/program reload
rsync
rsync
ssh server1 '/etc/init.d/program reload'
ssh server2 '/etc/init.d/program reload'