User Tools

Site Tools


network_file_system_for_lxc_nfsv4

NFS server and client setup (NFSv4)

http://www.itp.uzh.ch/~dpotter/howto/kerberos I will only use the new syntax of the /etc/exportfs file. You will probably find other pages using the NFSv3 syntax (without the 'fsid=0' line in the '/etc/exports' file). Don't just copy-paste them in your '/etc/exports'-file.

Contents:

  • [Setting the system time =====
    • Kernel server mode vs usermode nfs server
    • Write permissions on NFS shares → kerberos + ??
    • Mapping usernames between NFS server and clients → kerberos?
  • Using LDAP ? & Kerberos for security & (?mapping users?) (optional, but highly recommended for reasons yet to explain)
  • Troubleshooting NFS issues

Correcting the system time

Modern computer clocks are pretty accurate, but often they are not set to the right time. Kerberos, which we'll need for nfs, will have problems if the time differs more than 5 minutes. I'll show you how to painlessly set the correct time for your computers (both NFS server and NFS client) using very accurate “time servers” on the internet (very accurate = max. 1.4 thousandths of a second deviation of the official time!).

Setting the correct timezone

Because not everyone lives on 'Greenwich Mean Time' (even the people in Greenwich don't, at least 6 months they don't), the computer must be told where (=in which timezome) you live. I live in Eindhoven, The Netherlands, Europe, so I selected 'Europe/Amsterdam' which is pretty close to where I live:

dpkg-reconfigure tzdata
  Current default time zone: 'Europe/Amsterdam'
  Local time is now:      Thu Dec  2 20:26:34 CET 2010.
  Universal Time is now:  Thu Dec  2 19:26:34 UTC 2010.

/* Files: /etc/timezone /etc/localtime */

Synchronizing the time with time servers on the internet using ntp

The command we'll be using to correct the time is called ntp (after the Network Time Protocol). Install it (will also install 'libopts25') and manually run the command to synchronize the time (in the future, it will do this automatically):

apt-get install ntp
ntpd -gq

Check the result:

date
  Thu Dec  2 20:22:12 CET 2010

NFS server setup

Install the required packages

on the vm-mgr (the machine I wish to install the nfs server on):

apt-get install nfs-kernel-server nfs-common portmap

Creating and filling the /export folder

mkdir /export
mkdir /export/maildir
mkdir /export/www

My e-mail is stored in the /mnt/data/e-mail/ folder, and my websites are in '/mnt/data/websites'. 'Bind'-mount these folders into the '/export' folder

#EXPORT BINDS FOR NFS4:
/mnt/data/e-mail/      /export/maildir       bind    defaults,bind   0 0
/mnt/data/websites/    /export/www           bind    defaults,bind   0 0

Updating the /etc/exports file

nano /etc/exports
/export       10.76.0.0/24(rw,fsid=0,insecure,no_subtree_check,async)
/export/maildir <<vm-mgr_ip>>(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/maildir <<vm-mail_ip>>(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/www <<vm-mgr_ip>>(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/www <<vm-web_ip>>(rw,nohide,insecure,no_subtree_check,async,no_root_squash)

Remember to replace «vm-mail_ip», «vm-mgr_ip» and «vm-web_ip» with the correct ip adresses.

Everytime after editing /etc/exports, enter the following to make the nfs server aware of the changes:

exportfs -rav

nfs-kernel-server vs. usermode nfs (nfs-user-server)

http://www.debianhelp.co.uk/nfs.htm Use nfs-kernel-server package if you have a fairly recent kernel (2.2.13 or better) and you want to use the kernel-mode NFS server. The user-mode NFS server in the “nfs-user-server” package is slower but more featureful and easier to debug than the kernel-mode server.

NFS client setup

mount -t nfs4 -o proto=tcp,port=2049 10.76.0.3:/download /mnt/download mount -t nfs4 -o proto=tcp,port=2049 10.76.0.3:/www /var/www

NFS Firewall settings

iptables rules for a statefull firewall:

#NFS:
iptables -A INPUT -p tcp -i eth0 --dport 32771 -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -i eth0 --dport 111 -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -i eth0 --dport 2049 -m state --state NEW -j ACCEPT

(if your firewall isn't stateful, perhaps removing '-m state –state NEW' will do)

?TODO: Portmap Lockdown in firewall, nfs port 4000-4004?

LXC specific NFS issues

Troubleshooting NFS issues

LXC: If you can't get it working, you can also use bind mounts for the time being, bind mounts however are a PITA when you want to 'tar' it up into a nice small package, as the contents of the mounted folders will also be copied (resulting in a very large backup file, depending on your mount sizes)

  • In /etc/exports on the NFS server, make sure that both the client IP AND the NFS server IP have access to the NFS share
  • If you 'exported' /export/
  • Check firewalls -on both sides-
  • Check if you can ping and telnet(port 22) to the other host, I've had my share of troubles with the bridging not going as planned
  • To my (current) knowledge, you can't run the nfs-kernel-server in a lxc container, I believe the nfs-user-server can.
  • Restart the NFS server (or even reboot the machine… :-( )
  • make sure the kernel re-reads the /etc/exports file:
    exportfs -rav
  • /var/lib/nfs/etab
  • in the lxc config file, add a mount entry to 'bind mount' an extra file (fstab) into the containers file system:
lxc.mount.entry=/lxc/vm-template/fstab /lxc/vm-template/rootfs/etc/fstab none bind 0 0

The '/lxc/vm-template/fstab' file contains the information on what to mount upon boot:

<<NFS_server_ip>>:/www /var/www nfs4 defaults,_netdev 0 0

Replace '/var/www' and '/www' with the correct directory names.

Optionally, add 'noauto'(?) to disable the container from mounting it at boot, and add to /lxc/vm-template/etc/rc.local

mount /var/www

to mount it at boot.

Write permissions on NFS shares

Well, 1 simple advice for starters: make sure the uid+gid's on the server and client are the same You can check the uid & gid of a file/folder with ls:

ls -ln
Mapping users between nfs kernel server and client

FIXME Exploring methods for mapping user IDs between client and server http://linuxmafia.com/linux/suse-linux-internals/chapter20.html

http://fedorasolved.org/post-install-solutions/nfsv4-fedora ??After you edit /etc/idmapd.conf, type “service rpcidmapd restart”. That will make it re-read the config file. root@vm-mgr:/etc# /etc/init.d/nfs-common restart

http://linux.die.net/man/8/idmapd http://www.troubleshooters.com/linux/nfs.htm#_Mounting_an_NFS_Share_on_a_Client

Fix for broken nfs client in lxc containers

root@vm-mgr:# rmdir /vm-template/rootfs/var/run/networkmountnfs/ Restart the container, or restart nfs-common + portmap inside the container.

Restarting the nfs kernel server

root@vm-mgr:/export/download# /etc/init.d/nfs-kernel-server restart Stopping NFS kernel daemon: mountd nfsd. Unexporting directories for NFS kernel daemon…. Exporting directories for NFS kernel daemon…. Starting NFS kernel daemon: nfsd

dmesg: [18577.048022] rpcbind: server localhost not responding, timed out

root@vm-mgr:/export/download# ps auxww|grep rpc root 713 0.0 0.0 0 0 ? D 11:49 0:00 [rpciod/0] statd 5203 0.0 0.0 1936 776 ? Ss 16:57 0:00 /sbin/rpc.stat root 5212 0.0 0.0 2272 556 ? Ss 16:57 0:00 /usr/sbin/rpc.dmapd root 5257 0.0 0.0 0 0 pts/0 Z 16:58 0:00 [rpc.nfsd] <deunct> root 5293 0.0 0.0 2108 364 ? Ss 16:59 0:00 /usr/sbin/rpc.ountd –manage-gids –no-nfs-version 3 root 5317 0.0 0.0 3304 752 pts/0 S+ 17:05 0:00 grep rpc root@vm-mgr:/export/download#

Killing a defunct nfsd nfs server in Linux

I noticed in the ps list: [rpc.nfsd] <defunct>

I tried restarting it:

root@vm-mgr:# /etc/init.d/nfs-kernel-server start
Exporting directories for NFS kernel daemon....
Starting NFS kernel daemon: nfsdrpc.nfsd: Setting version failed: errno 16 (Device or resource busy)

That didn't work.

google:defunct linux proc → http://mirror.hamakor.org.il/archives/linux-il/11-2004/12560.html defunct linux proc http://mirror.hamakor.org.il/archives/linux-il/11-2004/12560.html http://www.cts.wustl.edu/~allen/kill-defunct-process.html You should look for the process' parent (using ps -l's PPID column) and try to see why it doesn't do a wait(2), usually it's a bug in the parent.

root@vm-mgr:# ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 D     0  5005  5002  0  80   0 -  1134 -      pts/0    00:00:00 bash
4 S     0  5014  5011  0  80   0 -  1391 -      pts/0    00:00:00 bash
0 T     0  5228  5014  0  80   0 -   437 -      pts/0    00:00:00 nfs-kernel-serv
0 T     0  5246  5228  0  80   0 -   437 -      pts/0    00:00:00 nfs-kernel-serv
4 Z     0  5257  5246  0  80   0 -     0 -      pts/0    00:00:00 rpc.nfsd <defunct>
4 R     0  5333  5014  0  80   0 -   915 -      pts/0    00:00:00 ps
root@vm-mgr:# kill -9 5246
root@vm-mgr:# ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 D     0  5005  5002  0  80   0 -  1134 -      pts/0    00:00:00 bash
4 S     0  5014  5011  0  80   0 -  1391 -      pts/0    00:00:00 bash
0 T     0  5228  5014  0  80   0 -   437 -      pts/0    00:00:00 nfs-kernel-serv
0 Z     0  5246  5228  0  80   0 -     0 -      pts/0    00:00:00 nfs-kernel-serv <defunct>
4 R     0  5337  5014  0  80   0 -   915 -      pts/0    00:00:00 ps
root@vm-mgr:# kill -9 5228
root@vm-mgr:# ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 D     0  5005  5002  0  80   0 -  1134 -      pts/0    00:00:00 bash
4 S     0  5014  5011  0  80   0 -  1391 -      pts/0    00:00:00 bash
4 R     0  5338  5014  0  80   0 -   915 -      pts/0    00:00:00 ps
[1]+  Killed                  /etc/init.d/nfs-kernel-server restart
root@vm-mgr:#

http://www.google.nl/search?sourceid=chrome&ie=UTF-8&q=automounter+nfs http://docstore.mik.ua/orelly/networking_2ndEd/nfs/ch09_01.htm Your exposure to hanging a process when an NFS server crashes is greatly reduced. The automounter unmounts all filesystems that are not in use, removing dependencies on fileservers that are not currently referenced by the client.

http://www.linux-tutorial.info/modules.php?name=MContent&pageid=153

nfs write permissions for users: http://forums.gentoo.org/viewtopic-t-748472.html

other nfs stuff

root@vm-mgr:# showmount -e root@vm-mgr:# rpcinfo -p

http://www.vanemery.com/Linux/NFS-Van.html

Configure idmapd, NFSv4, and autofs http://staff.washington.edu/high/doc/fedora/kerb_client/ar01s12.html http://forums.gentoo.org/viewtopic-p-5611037.html#5611037

http://tldp.org/HOWTO/NFS-HOWTO/client.html http://tldp.org/HOWTO/NFS-HOWTO/security.html

/*

linux nfs rpc.nfsd daemon / LXC http://comments.gmane.org/gmane.linux.kernel.containers.lxc.general/977 http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg01006.html http://sourceforge.net/mailarchive/forum.php?thread_name=AANLkTin3vLZQFobnW8R%3DvkTzXPdtrgn9z_LdbUUGGSLr%40mail.gmail.com&forum_name=lxc-users

nfs werkt nog niet vanuit een container https://help.ubuntu.com/community/NFSv4Howto Making your computer an NFS server or client is very easy.A Debian NFS client needs

# apt-get install nfs-common portmap

while a Debian NFS server needs

[nfs] NFSv4:

time sync (required for kerberos auth): apt-get –no-install-recommends install ntp # =libcap2 libopts25 ntp sync time:

  ntpd -gq

optionally, edit /etc/ntp.conf

[nfs server] apt-get install nfs-kernel-server Not starting NFS kernel daemon: no support in current kernel. … (warning).

reboot vm-mgr (!)??

http://osdir.com/ml/debian-bugs-dist/2009-11/msg01267.html nfs-kernel-server: init script incompatible with kernel 2.6.32 ??

grep NFSD /boot/config-`uname -r` cat /proc/net/rpc/nfs nfsstat -s

[nfs server configuratie] kerberos If the host name does not match the reverse DNS lookup, Kerberos authentication will fail. nfsd: NFS4 en userrechten mount /var/www → /export/www en via nfs naar .60 /var/www

top - 11:45:28 up  7:44,  1 user,  load average: 1.00, 2.59, 30.47

Setting up the NFS client mounts

Editor note: should be linked from vm-template.

mount -t nfs4 -o proto=tcp,port=2049 <NFS_SERVER_IP>:/download /mnt/download mount -t nfs4 -o proto=tcp,port=2049 <NFS_SERVER_IP>:/www /var/www

auto mounting nfs4 in an lxc container

cat 'config' file (partly, relevant section):

# system devices mounts (proc,pts,sys):
lxc.mount.entry=proc /lxc/vm-template/rootfs/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=devpts /lxc/vm-template/rootfs/dev/pts devpts defaults 0 0
lxc.mount.entry=sysfs /lxc/vm-template/rootfs/sys sysfs defaults  0 0
# Additional mounts:
lxc.mount.entry=/lxc/vm-template/fstab_nfs /lxc/vm-template/rootfs/etc/fstab none bind 0 0

Remember to replace 'vm-template' with the correct name of your vm. The last line mounts a 'fstab' file into the vm's file system, this file contains the instructions how to mount the NFS share. In this case the fstab file contains instructions on how to mount the 'www' export of the NFSv4 server on the '/var/www' folder in the vm:

cat /lxc/vm-template/fstab_nfs
<NFS_SERVER_IP>:/www /var/www nfs4 defaults,_netdev 0 0

Remember to replace <NFS_SERVER_IP> with the correct ip address of the NFS server.

/* <NFS_SERVER_IP>:/dhcp /var/www/ip/phpdhcpd/leases nfs4 defaults,_netdev 0 0 */

Recreate container (just stop/start won't work, as the config files have changed it should be recreated) log into running container, and enter:

mount

If everything is correct, you will see the NFS mount (among the other mounts)

...
/export/www <NFS_SERVER_IP>(rw,nohide,insecure,no_subtree_check,async)
...

TODO:

  • explain nosync, secure, no_root_squash etc using 'conf/acronyms.conf.local'

Please leave feedback or questions at the main page.

network_file_system_for_lxc_nfsv4.txt · Last modified: 2023-02-28 16:44 by 127.0.0.1