Mogile on Tyan GT86C-B5630 Servers

Tyan GT86C-B5630 1U LFF 12 bay (3.5“) storage server

• Chassis Documentation (Note: Use BIOS/FW updates from Motherboard link below instead!)
• Motherboard: S5630GMRE-CGN
  • Docs/Downloads
• Intel C621 SATA controller
• 500W PSU: S14-500P1AA
• Xeon Gold 5118 2nd Gen Scalable, 12-core 2.3Ghz
• 64GB (2x32GB) DDR4
• X520-DA2 2x 10GbE SFP+ OCP w/ 2x 10Gtek ASF-GE-T (1000Base-T SFP Copper RJ45)
• 1x Micron 60GB SATA SSD
• 4x Seagate Exos X16 ST14000NM005G 14TB 7.2K RPM SATA 6Gb/s 512e CMR 3.5in

Firmware update steps:

• Make a FAT32 usb stick.
• Load UEFI Shell using steps: https://github.com/KilianKegel/Howto-create-a-UEFI-Shell-Boot-Drive
  • UEFI files: efi.zip
• BIOS Upgrade: s5630gmr_v202.zip
• BMC/IPMI Upgrade: s5630-bmc_v700.zip

BIOS Changes:

• Boot performance to Max Efficiency
• Fan control to Manual/30%

IPMI Notes:

• Default login: root/superuser (will prompt for change on first login)
• Set PW using internal mnemonic scheme: soup
• Also create user dar with same PW

Ubuntu 22.04.5 custom configs:

• To disable cloud-init, create the empty file /etc/cloud/cloud-init.disabled

  sudo touch /etc/cloud/cloud-init.disabled

• If you need to do an offline install of ubuntu 22.04, look at https://askubuntu.com/questions/1409135/22-04-offline-installation

Config log:

apt -y install vim
# temporary allow root access for setup
vi /etc/ssh/sshd_config
PermitRootLogin yes
systemctl restart ssh
sudo passwd root

cd /etc/netplan/
vi 50-cloud-init.yaml

network:
    ethernets:
        ens3f0:
            dhcp4: false
            addresses: [10.10.22.181/24]
            gateway4: 10.10.22.1
            nameservers:
              addresses: [8.8.8.8,8.8.4.4]
    version: 2

Turn off kernel audit messages

vi /etc/default/grub

Add “audit=0” to GRUB_CMDLINE_LINUX_DEFAULT

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="audit=0"
GRUB_CMDLINE_LINUX=""

If the server has nvme, proactively do the following to avoid a potential NVMe power saving mode bug.

GRUB_CMDLINE_LINUX_DEFAULT="audit=0 pcie_aspm=off pcie_port_pm=off nvme_core.default_ps_max_latency_us=0 apparmor=0 security=apparmor"

Update grub and reboot

update-grub

Setup Disks

sgdisk -o -n 1:0:0 -t 1:8300 /dev/sdb
sgdisk -o -n 1:0:0 -t 1:8300 /dev/sdc
sgdisk -o -n 1:0:0 -t 1:8300 /dev/sdd
sgdisk -o -n 1:0:0 -t 1:8300 /dev/sde
mkfs.ext4 -m 0 /dev/sdb1
mkfs.ext4 -m 0 /dev/sdc1
mkfs.ext4 -m 0 /dev/sdd1
mkfs.ext4 -m 0 /dev/sde1

mkdir /data

#create a unique device id e.g. dev900
mkdir /data/dev900
mkdir /data/dev901
mkdir /data/dev902
mkdir /data/dev903

vi /etc/fstab
# add the following lines
/dev/sdb1       /data/dev900 ext4       defaults 1 2
/dev/sdc1       /data/dev901 ext4       defaults 1 2
/dev/sdd1       /data/dev902 ext4       defaults 1 2
/dev/sde1       /data/dev903 ext4       defaults 1 2

Grab Mogile packages

scp -oHostKeyAlgorithms=+ssh-dss -P 75 user@server:/S76usrlocalsrc.tar .

move stuff to /usr/local/src

Load dependencies

apt -y install build-essential
apt -y install net-tools
apt -y install less
apt -y install libssl-dev
apt -y install libz-dev
apt -y install mysql-server
apt -y install libio-aio-perl
apt -y install mysql-client
apt -y install libmysqlclient-dev

cpan install CPAN
cpan install Perlbal 
cpan install Danga::Socket 
cpan install IO::Socket
cpan install Net::SSLeay
cpan install IO::Socket::SSL
cpan install Gearman::Client 
cpan install Gearman::Server
cpan install Gearman::Client::Async 
cpan install Net::Netmask 
cpan install IO::WrapTie
cpan install DBI
cpan install BSD::Resource
cpan install Perlbal::XS:HTTPHeaders
cpan install YAML
cpan install DBD:mysql

Add Users:

groupadd mogile
useradd -m -g mogile mogile

Install mogile

cd MogileFS-Client-1.07
perl Makefile.PL
make
make test
make install

cd MogileFS-Utils-2.11
perl Makefile.PL
make
make install


cd mogilefs-server-2.17
perl Makefile.PL
make
make test
make install


mkdir /etc/mogilefs
cp mogstored.conf /etc/mogilefs/

cat /etc/mogstored.conf

daemonize=1
docroot=/data

cat /etc/mogilefs/mogilefsd.conf

daemonize = 1
db_dsn = DBI:mysql:mogilefs:host=10.2.2.19:port=3306
db_user = mogile
db_pass = mogile
listen = 0.0.0.0:7001
conf_port = 7001
listener_jobs = 10
delete_jobs = 5
replicate_jobs = 0
reaper_jobs = 3
old_repl_compat = 0
rebalance_ignore_missing = 1

SU to user mogile and start daemon

mogilefsd -c /etc/mogilefs/mogilefsd.conf

timedatectl list-timezones
timedatectl set-timezone PST8PDT