Lustre file system – simple deploy

OS: CentOS 6.3
MGS server: lustre-mgs IP 192.168.0.1
OSS server: lustre-oss-01 IP 192.168.0.2
Storage: local drive on OSS server

Download Lustre packages: http://downloads.whamcloud.com/public/lustre/latest-maintenance-release/
Download modified e2fsprogs packages: http://downloads.whamcloud.com/public/e2fsprogs/latest/

Deploy MGS server

rpm -ivh kernel-2.6.32-279.14.1.el6_lustre.x86_64.rpm kernel-firmware-2.6.32-279.14.1.el6_lustre.x86_64.rpm
rpm -Uvh e2fsprogs-1.42.3.wc3-7.el6.x86_64.rpm libcom_err-1.42.3.wc3-7.el6.x86_64.rpm e2fsprogs-libs-1.42.3.wc3-7.el6.x86_64.rpm libss-1.42.3.wc3-7.el6.x86_64.rpm
rpm -ivh lustre-modules-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64.rpm lustre-ldiskfs-3.3.0-2.6.32_279.14.1.el6_lustre.x86_64.x86_64.rpm
yum install net-snmp-libs
rpm -ivh lustre-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64.rpm

Disable SELinux
Add selinux=0 to /boot/grub/grub.conf at the end of kernel line

title CentOS (2.6.32-279.14.1.el6_lustre.x86_64)
    root (hd0,0)
    kernel /vmlinuz-2.6.32-279.14.1.el6_lustre.x86_64 ro root=/dev/mapper/vg_lustremgs01-lv_root rd_LVM_LV=vg_lustremgs01/lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_lustremgs01/lv_swap rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet selinux=0
    initrd /initramfs-2.6.32-279.14.1.el6_lustre.x86_64.img

Disable firewall and reboot

chkconfig iptables off
reboot

Create file /etc/modprobe.d/lustre.conf, add line:

options lnet networks=tcp0(eth0)

Create a combined MGS/MDT file system on a block device

mkfs.lustre --fsname=lustre --mgs --mdt --index=0 /dev/mapper/vg_lustremgs01-lv_lustre
mkdir /mdt
mount -t lustre /dev/mapper/vg_lustremgs01-lv_lustre /mdt

Add line to /etc/fstab

/dev/mapper/vg_lustremgs01-lv_lustre /mdt lustre defaults 0 0

Deploy OSS server

The same steps as for MGS

  1. Install the same packages
  2. Disable SELinux and firewall
  3. Create /etc/modprobe.d/lustre.conf
  4. Add line to /etc/fstab

Create and mount filesystem

mkfs.lustre --fsname=lustre --mgsnode=192.168.0.1@tcp --ost --index=1 /dev/mapper/vg_lustreoss01-lv_lustre
mkdir /lustre
mount -t lustre /dev/mapper/vg_lustreoss01-lv_lustre /lustre

Deploy client

rpm -ivh lustre-client-2.1.4-2.6.32_279.14.1.el6.x86_64.x86_64.rpm lustre-client-modules-2.1.4-2.6.32_279.14.1.el6.x86_64.x86_64.rpm
mkdir /lustre
mount -t lustre 192.168.0.1@tcp:/lustre /lustre

If client has another kernel version, you need compile client

Download client source: http://downloads.whamcloud.com/public/lustre/latest-maintenance-release/

yum install lustre-client-source-2.1.4-2.6.32_279.14.1.el6.x86_64.x86_64.rpm
cd /usr/src/lustre-2.1.4
./configure
make
make install
mkdir /lustre
reboot
mount -t lustre 192.168.0.1@tcp:/lustre /lustre

10 thoughts on “Lustre file system – simple deploy

  1. Ajeet

    I just completed lustre installation on RHEL 6.3. The steps looks pretty straightforward and worked like a charm.
    Query: – Did you tried using it with Hadoop replacing HDFS?
    Query:- Did you tried this setup with any application?

    Do share your comments.

    Reply
    1. Pavlo Khmel Post author

      I have not tried to use with Hadoop.
      I’m making benchmark test with IOzone now (single and multi-client IOPS). I want to understand what is best Lustre configuration for my applications. Like: OSS storage size, OSS amount, stripe_count, stripe_size, …

      Reply
  2. Cong

    Hi, my kernel version is: Linux ucs14 2.6.32-358.6.1.el6.x86_64.
    So my client has another kernel version and I need to compile client. However, when I am trying to
    # ./configure
    It shows error information: “configure: error: Kernel source /usr/src/linux could not be found.”
    How to solve this problem?

    Thanks!

    Reply
    1. Pavlo Khmel Post author

      If you use RHEL/Fedora/CentOS:
      # yum install kernel-devel
      # ln -sf /usr/src/kernel/2.6.32 /usr/src/linux

      Reply
      1. Cong

        Yes, you are right, the problem has been solved.
        Creating link or adding “–with-linux=/usr/src/kernels/2.6.32-358.6.1.el6.x86_64”:
        # ./configure –with-linux=/usr/src/kernels/2.6.32-358.6.1.el6.x86_64
        either way works.
        Again, thanks a lot for your reply. 🙂

        Reply
  3. Sunny

    I followed all your steps for lustre 2.5.3 on Centos 6.5. Things are smooth until I try to mount on the client. I get the following error.
    mount.lustre: myip@tcp:lustre has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool.

    However I did run mkfs.lustre on both MDT/MGS and OSS.
    Can you please let me know if there is anything I missed.

    Thanks.

    Reply
    1. Pavlo Khmel Post author

      I have not seen such error before. Is /lustre successfully mounted on OSS server?
      Run command on you OSS server: mount | grep lustre

      Reply
      1. Sunny

        Thanks for you comment. Actually I found the issue, there was a bug in lustre-2.5.3 rpms related to ldiskfs. When I used lustre-2.6.0 my issue was resolved.

        Reply
  4. ashish nimje

    Sir i am using lustre 2.6. I need to test benchmarking on SAS drive . i am using bonniee++ tool for benchmarking.Will you plz give some tips regarding bonnie++ tool and Benchmarking in lustre.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *