Monday, September 24, 2007

Out of inodes

This article shows you how I dealt with an "out of inodes" problem.
I was installing the entire ports tree on a new box. It froze. So I restarted it. Then it froze again. OK, this time I noticed the following messages on the console:
Jul  5 14:25:48 little /kernel: pid 431 (cpio), uid 0 on /usr2:
out of inodes
Jul 5 14:26:19 little last message repeated 2550 times
Jul 5 14:28:20 little last message repeated 10276 times
Jul 5 14:28:25 little last message repeated 451 times

inode background
The problem is that we didn't have enough inodes. This is a common problem when you have many files of a small size. If you look at the man page for newfs, you'll see the following:
-f frag-size
The fragment size of the file system in bytes. It must be a pow­
er of two ranging in value between blocksize/8 and blocksize. The
default is 1024 bytes.
-i number of bytes per inode
Specify the density of inodes in the file system. The default is
to create an inode for every (4 * frag-size) bytes of data space.
If fewer inodes are desired, a larger number should be used; to
create more inodes a smaller number should be given. One inode
is required for each distinct file, so this value effectively
specifies the average file size on the file system.

As you can read, with lots of small files, you'll quickly use up the inodes. When you install the ports system, you are adding lots of small files.


Create a new file system
Here's how I created a new file system. Note that this removes all files from the device. Make a backup first if you wish to retain anything.

In the message log, it mentions /usr2. If you look the output from mount, you can see what device is out of inodes. The default value for inodes is 4096. So by specifying 1024 below, I have effectively increased the number of inodes by a factor of 4. In order words, we can now have four times as many files as we used to have. We'll use this as a starting point and see how it goes.

# mount
/dev/wd0s1a on / (local, writes: sync 424 async 2307)
/dev/wd0s1f on /usr (local, writes: sync 3 async 2158)
/dev/wd0s1e on /var (local, writes: sync 2086 async 7972)
procfs on /proc (local)
/dev/wd1s1e on /usr2 (local, writes: sync 8360 async 12055)

You can see that /usr2 is actually /dev/wd1s1e. So that's the device which needs to be modified. On my file system, /usr2 contained only the ports directory.

# umount /dev/wd1s1e
# newfs -i 1024 /dev/wd1s1e
newfs: /dev/wd1s1e: not a character-special device
Warning: Bytes per inode restrict cylinders per group to 12.
Warning: 2448 sector(s) in last cylinder unallocated
/dev/wd1s1e: 415344 sectors in 102 cylinders of 1 tracks, 4096 sectors
202.8MB in 9 cyl groups (12 c/g, 24.00MB/g, 20672 i/g)
super-block backups (for fsck -b #) at:
32, 49184, 98336, 147488, 196640, 245792, 294944, 344096, 393248,
# mount /dev/wd1s1e
# cd /usr

The next step is to recreate my symbolic links back to /usr/ports:

# cd /usr2
# mkdir ports
# ln -s /usr2/ports /usr/ports

http://www.freebsddiary.org/inodes.php

No comments: