Mongodb Production servers hardware recommendations for better performace

i) Disable NUMA -- Running MongoDB on a system with Non-Uniform Access Memory (NUMA) can cause a number of operational problems, including slow performance for periods of time and high system process usage. So,it is recommended to turn off NUMA

ii) Allocate swap space -- Allocating swap space can avoid issues with memory contention and can prevent the OOM Killer on Linux systems from killing mongod. The method mongod uses to map memory files to memory ensures that the operating system will never store MongoDB data in swap space.

iii) RAID10 -- RAID-5 and RAID-6 do not typically provide sufficient performance to support a MongoDB deployment.MongoDB deployments should use disks backed by RAID-10.

iv) Noop scheduler -- Local block devices attached to virtual machine instances via the hypervisor should use a noop scheduler for best performance. The noop scheduler allows the operating system to defer I/O scheduling to the underlying hypervisor.

v) ulimit -- Set the file descriptor limit, -n, and the user process limit (ulimit), -u, to 64000.

vi) Disable transparent huge pages -- MongoDB performs better with normal (4096 bytes) virtual memory pages.

vii) Turnoff atime : Operating systems maintain file system metadata that records when each file was last accessed, this is atime. The importance of the noatime setting is that it eliminates the need by the system to make writes to the file system for files which are simply being read. Since writes can be somewhat expensive, this can result in measurable performance gains.Turn off atime for the storage volume containing the database files.

viii)read-ahead  Ensure that read-ahead settings for the block devices that store the database files are appropriate. For random access use patterns, set low read ahead values. A read ahead of 32 (16 kB) often works well.

Few more:

1. If we use userId(primary key) as _id, we can avoid _id for primary key duplication.
2. Key field length should be short.
3. Date order has to be taken care while creating index.
4. Disable diagnostic logging due to performance issue.

5.Monitor replica health(nagios).

6.Have to verify mongodb snapshot restoration.

7.Use diff storage location for Journal, loging and data.

8. Log-stash for log analysis(candidates specific).

9.Error log rotation for all mongo env.

10.TCP keep alive time need to be set properly.

1.File system type should be  XFS 

With the WiredTiger storage engine, use of XFS is strongly recommended to avoid performance issues that may occur when using EXT4 with WiredTiger.

2. Disable Transparent Huge Pages.

MongoDB performs better with normal (4096 bytes) virtual memory pages 

Temporary  disable the Transparent Huge Pages by executing the below commands .

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

To disable the Transparent Huge Pages permanently .

cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
cat /sys/kernel/mm/transparent_hugepage/defrag
always madvise [never]

add the transparent_hugepage=never in /etc/grub.conf ans scripts

###########
vi /etc/init.d/disable-transparent-hugepages
#!/bin/sh
### BEGIN INIT INFO
# Provides: disable-transparent-hugepages
# Required-Start: $local_fs
# Required-Stop:
# X-Start-Before: mongod mongodb-mms-automation-agent
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description: Disable Linux transparent huge pages, to improve
# database performance.
### END INIT INFO

case $1 in start)
if [ -d /sys/kernel/mm/transparent_hugepage ]; then
thp_path=/sys/kernel/mm/transparent_hugepage
elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
thp_path=/sys/kernel/mm/redhat_transparent_hugepage
else
return 0
fi

echo 'never' > ${thp_path}/enabled
echo 'never' > ${thp_path}/defrag

unset thp_path
;; 
esac

chmod 755 /etc/init.d/disable-transparent-hugepages 
chkconfig --add disable-transparent-hugepages

###########

3.Disable NUMA in your BIOS 

add the below numa=off in /etc/grub.conf 
to check the numa enable or not by using the cat /var/log/dmesg | grep -i numa.

in rhel 7 vi /etc/default/grub

add the numa off on GRUB_CMDLINE_LINUX="numa=off".

example 

GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200 rootdelay=300 net.ifnames=0,numa=off"

4. DIsable the SELINUX 

Disable SELinux by setting the SELINUX setting to disabled in /etc/selinux/config.

SELINUX=disabled

5.Turnoff atime

add the noatime in /etc/fstab for the data partition.

to verify the noatime disabled or not execute mount command.

4. tcp_keepalive_time 

to check the keep alive time execute cat /proc/sys/net/ipv4/tcp_keepalive_time

adjust the tcp_keepalive_time temporarily 

 tcp_keepalive_time echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time

permanently  add the net.ipv4.tcp_keepalive_time=300 in /etc/sysctl.conf

5.Noop scheduler

check the no scheduler value cat /sys/block/xvdg/queue/scheduler

check pvs command to find the device names

should be noop for VM and deadline for physical server

For centos 7

grubby --update-kernel=ALL --args="elevator=noop"

6.Setting of the readahead setting for random I/O 'S

to verify the readhead execute the command:  blockdev --report

For the WiredTiger storage engine:

Read-Ahead

Set the readahead setting to 0 . Setting a higher readahead benefits sequential I/O operations. However, since MongoDB disk access patterns are generally random, setting a higher readahead provides limited benefit. As such, for most workloads, a readahead of 0  provides optimal MongoDB performance.

For the MMAPv1 storage engine:

Ensure that readahead settings for the block devices that store the database files are appropriate. For random access use patterns, set low readahead values. A readahead of 32 (16 kB) often works well.
For a standard block device, you can run sudo blockdev --report to get the readahead settings and sudo blockdev --setra <value> <device> to change the readahead settings. Refer to your specific operating system manual for more information.

adjust the read add count blockdev --setra 0 /dev/xvdf need add in /etc/rc.local

Mongodb VM recomendations:

1.Disable the memroy compresion cache 

2.Enabling the enhanced the networking i.e Elastic network adapter

3.Disable the Dvfs and CPU power  saving modes.

4.Disabling the hyperthreading