Find broken symlinks and delete them

Look for the broken link, and  list them:

#find -L /path -type l

Delete broken link

#find -L /path -type l -delete

To delete with confirmation do:

#find -L /path -type l -exec rm -i {} +

L flag follows symlinks, so the -type l test only returns true if the link can’t be followed, or is a symlink to another broken symlink.

Nagios is an awesome Open Source monitoring tool, its provides you more comprehensive monitoring environment to always keep an eye on your all machines / networks whether you are in a your data center or just your small labs.

With Nagios, you can monitor your remote hosts and their services remotely on a single window. It shows warnings and indicates if something goes wrong in your servers which eventually helps us to detect some problems before they occur. It helps us to reduce downtime and business losses.

Recently, Nagios released its latest versions Nagios Core 4.3.4 and its latest stable release of Nagios plugins 2.2.1 on April 19, 2017.

This article is intended to guide you with easy instructions on how to install latest Nagios Core 4.3.4 from source (tarball) on RHEL 7.x/6.x/5.x, CentOS 7.x/6.x/5.x  distributions.

Within 30 minutes you will be monitoring your local machine, no any advanced installation procedure only basic installation that will work 100% on most of the today’s Linux servers.

Please Note: The installation instructions were shown in here are written based on CentOS 7.2 Linux distribution.

Installing Nagios 4.3.4 and Nagios Plugin 2.2.1

If you follow these instructions correctly, you will end up with following information.

1. Nagios and its plugins will be installed under /usr/local/nagios directory.
2. Nagios will be configured to monitor few services of your local machine (Disk Usage, CPU Load, Current Users, Total Processes, etc.)
3. Nagios web interface will be available at http://localhost/nagios

Step 1: Install Required Dependencies:

We need to install Apache, PHP and some libraries like gcc, glibc, glibc-common and GD libraries and its development libraries before installing Nagios 4.3.4 with source. And to do so, we can use yum default package installer.

# yum install -y httpd httpd-tools php gcc glibc glibc-common gd gd-devel make net-snmp

Step 2: Create Nagios User and Group

Create a new nagios user and nagcmd group account and set a password.

# useradd nagios

# groupadd nagcmd

Next, add both the nagios user and the apache user to the nagcmd group.

# usermod -G nagcmd nagios

# usermod -G nagcmd apache

Step 3: Download Nagios Core 4.3.4 and Nagios Plugin 2.2.1

# mkdir /root/nagios

# cd /root/nagios

Now download latest Nagios Core 4.3.4 and Nagios plugins 2.2.1 packages with wget.

# wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.3.4.tar.gz

# wget https://nagios-plugins.org/download/nagios-plugins-2.2.1.tar.gz

Step 4: Extract Nagios Core and its Plugins

We need to extract downloaded package with tar command as follows.

# tar -xvf nagios-4.3.4.tar.gz

# tar -xvf nagios-plugins-2.2.1.tar.gz

When you extract these tarballs with tar command, two new folders will appear in that directory.
Dir:nagios-4.3.4 / nagios-plugins-2.2.1

Files : nagios-4.3.4.tar.gz / nagios-plugins-2.2.1.tar.gz

Configure Nagios Core

Now, first we will configure Nagios Core and to do so we need to go to Nagios directory and run configure file and if everything goes fine, it will show the output in the end as sample output. Please see below.

# cd nagios-4.3.4/

# ./configure --with-command-group=nagcmd

Sample output:

Creating sample config files in sample-config/ ...
*** Configuration summary for nagios 4.3.4 2017-08-24 ***:
General Options:
-------------------------
Nagios executable:  nagios
Nagios user/group:  nagios,nagios
Command user/group:  nagios,nagcmd
Event Broker:  yes
Install ${prefix}:  /usr/local/nagios
Install ${includedir}:  /usr/local/nagios/include/nagios
Lock file:  /run/nagios.lock
Check result directory:  ${prefix}/var/spool/checkresults
Init directory:  /etc/rc.d/init.d
Apache conf.d directory:  /etc/httpd/conf.d
Mail program:  /usr/bin/mail
Host OS:  linux-gnu
IOBroker Method:  epoll
Web Interface Options:
------------------------
HTML URL:  http://localhost/nagios/
CGI URL:  http://localhost/nagios/cgi-bin/
Traceroute (used by WAP):  /usr/bin/traceroute
Review the options above for accuracy.  If they look okay,
type 'make all' to compile the main program and CGIs.

After configuring, we need to compile and install all the binaries with make all and make install command, it will
 install all the needed libraries in your machine and we can proceed further.

# make all

# make install

Sample output:

*** Exfoliation theme installed ***
NOTE: Use 'make install-classicui' to revert to classic Nagios theme
make[1]: Leaving directory `/root/nagios/nagios-4.3.4'
make install-basic
make[1]: Entering directory `/root/nagios/nagios-4.3.4'
/usr/bin/install -c -m 775 -o nagios -g nagios -d /usr/local/nagios/libexec
/usr/bin/install -c -m 775 -o nagios -g nagios -d /usr/local/nagios/var
/usr/bin/install -c -m 775 -o nagios -g nagios -d /usr/local/nagios/var/archives
/usr/bin/install -c -m 775 -o nagios -g nagcmd -d /usr/local/nagios/var/spool/checkresults
chmod g+s /usr/local/nagios/var/spool/checkresults
*** Main program, CGIs and HTML files installed ***
You can continue with installing Nagios as follows (type 'make'
without any arguments for a list of all possible options):
make install-init
- This installs the init script in /etc/rc.d/init.d
make install-commandmode
- This installs and configures permissions on the
directory for holding the external command file
make install-config
- This installs sample config files in /usr/local/nagios/etc
make[1]: Leaving directory `/root/nagios/nagios-4.3.4'

Following command will install the init scripts for Nagios.

# make install-init
/usr/bin/install -c -m 755 -d -o root -g root /etc/rc.d/init.d
/usr/bin/install -c -m 755 -o root -g root daemon-init /etc/rc.d/init.d/nagios

*** Init script installed ***

To make nagios work from command line we need to install command-mode

# make install-commandmode
/usr/bin/install -c -m 775 -o nagios -g nagcmd -d /usr/local/nagios/var/rw
chmod g+s /usr/local/nagios/var/rw

*** External command directory configured ***

Next, install sample nagios files, please run following command.

# make install-config

Sample output:

/usr/bin/install -c -m 775 -o nagios -g nagios -d /usr/local/nagios/etc
/usr/bin/install -c -m 775 -o nagios -g nagios -d /usr/local/nagios/etc/objects
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/nagios.cfg /usr/local/nagios/etc/nagios.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/cgi.cfg /usr/local/nagios/etc/cgi.cfg
/usr/bin/install -c -b -m 660 -o nagios -g nagios sample-config/resource.cfg /usr/local/nagios/etc/resource.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/templates.cfg /usr/local/nagios/etc/objects/templates.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/commands.cfg /usr/local/nagios/etc/objects/commands.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/contacts.cfg /usr/local/nagios/etc/objects/contacts.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/timeperiods.cfg /usr/local/nagios/etc/objects/timeperiods.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/localhost.cfg /usr/local/nagios/etc/objects/localhost.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/windows.cfg /usr/local/nagios/etc/objects/windows.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/printer.cfg /usr/local/nagios/etc/objects/printer.cfg
/usr/bin/install -c -b -m 664 -o nagios -g nagios sample-config/template-object/switch.cfg /usr/local/nagios/etc/objects/switch.cfg
*** Config files installed ***
Remember, these are *SAMPLE* config files.  You'll need to read
the documentation for more information on how to actually define
services, hosts, etc. to fit your particular needs.

Step 5: Customizing Nagios Configuration

Open the “contacts.cfg” file with your choice of editor and set the email address associated
 with the nagiosadmin contact definition to receiving email alerts.

# vi /usr/local/nagios/etc/objects/contacts.cfg

Sample Output:

###############################################################################
###############################################################################
#
# CONTACTS
#
###############################################################################
###############################################################################
# Just one contact defined by default - the Nagios admin (that's you)
# This contact definition inherits a lot of default values from the 'generic-contact'
# template which is defined elsewhere.
define contact{
contact_name                    nagiosadmin             ; Short name of user
use                             generic-contact         ; Inherit default values from generic-contact template (defined above)
alias                           Nagios Admin            ; Full name of user
email                           admin@yourdomain.com     ; *** CHANGE THIS TO YOUR EMAIL ADDRESS ****
}

Step 6: Install and Configure Web Interface for Nagios

We are done with all configuration in the backend, 
now we will configure Web Interface For Nagios with following command.
The below command will Configure Web interface for Nagios and a web admin user will be created “nagiosadmin”.

# make install-webconf

In this step, we will be creating a password for “nagiosadmin”. 
After executing this command, please provide a password twice and keep it remember because this password will be used
when you login in the Nagios Web interface.

# htpasswd -s -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
New password:
Re-type new password:
Adding password for user nagiosadmin

Restart Apache to make the new settings take effect.

# service httpd start 
Starting httpd:                                            [  OK  ]

Step 7: Compile and Install Nagios Plugin

We have downloaded nagios plugins in /root/nagios, Go there and configure and install it as directed below.

# cd /root/nagios
# cd nagios-plugins-2.2.1/
# ./configure --with-nagios-user=nagios --with-nagios-group=nagios

Sample Output:

.......
configure: creating ./config.status
config.status: creating gl/Makefile
config.status: creating nagios-plugins.spec
config.status: creating tools/build_perl_modules
config.status: creating Makefile
config.status: creating tap/Makefile
config.status: creating lib/Makefile
config.status: creating plugins/Makefile
config.status: creating lib/tests/Makefile
config.status: creating plugins-root/Makefile
config.status: creating plugins-scripts/Makefile
config.status: creating plugins-scripts/utils.pm
config.status: creating plugins-scripts/utils.sh
config.status: creating perlmods/Makefile
config.status: creating test.pl
config.status: creating pkg/solaris/pkginfo
config.status: creating po/Makefile.in
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands
config.status: executing po-directories commands
config.status: creating po/POTFILES
config.status: creating po/Makefile


# make

Sample Output:

........
make[2]: Leaving directory `/root/nagios/nagios-plugins-2.2.1/plugins-root'
Making all in po
make[2]: Entering directory `/root/nagios/nagios-plugins-2.2.1/po'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/nagios/nagios-plugins-2.2.1/po'
make[2]: Entering directory `/root/nagios/nagios-plugins-2.2.1'
make[2]: Leaving directory `/root/nagios/nagios-plugins-2.2.1'
make[1]: Leaving directory `/root/nagios/nagios-plugins-2.2.1'


# make install

Sample Output:

installing fr.gmo as /usr/local/nagios/share/locale/fr/LC_MESSAGES/nagios-plugins.mo
installing de.gmo as /usr/local/nagios/share/locale/de/LC_MESSAGES/nagios-plugins.mo
if test "nagios-plugins" = "gettext-tools"; then \
 /bin/mkdir -p /usr/local/nagios/share/gettext/po; \
 for file in Makefile.in.in remove-potcdate.sin Makevars.template; do \
 /usr/bin/install -c -o nagios -g nagios -m 644 ./$file \
 /usr/local/nagios/share/gettext/po/$file; \
 done; \
 for file in Makevars; do \
 rm -f /usr/local/nagios/share/gettext/po/$file; \
 done; \
 else \
 : ; \
 fi
make[1]: Leaving directory `/root/nagios/nagios-plugins-2.2.1/po'
make[1]: Entering directory `/root/nagios/nagios-plugins-2.2.1'
make[2]: Entering directory `/root/nagios/nagios-plugins-2.2.1'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/root/nagios/nagios-plugins-2.2.1'
make[1]: Leaving directory `/root/nagios/nagios-plugins-2.2.1'

Step 8: Verify Nagios Configuration Files

Now we are all done with Nagios configuration and its time to verify it and to do so please insert following command.
If everything goes smooth it will show up similar to below output.

# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Sample Output:

Nagios Core 4.3.4
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2017-08-24
License: GPL
Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...
Running pre-flight check on configuration data...
Checking objects...
Checked 8 services.
Checked 1 hosts.
Checked 1 host groups.
Checked 0 service groups.
Checked 1 contacts.
Checked 1 contact groups.
Checked 24 commands.
Checked 5 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 1 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 5 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...
Total Warnings: 0
Total Errors:   0
Things look okay - No serious problems were detected during the pre-flight check

Step 9: Add Nagios Services to System Startup

To make Nagios work across reboots, we need to add nagios and httpd with chkconfig and systemctl command.

# chkconfig --add nagios
# chkconfig  nagios on

# chkconfig --add httpd
# chkconfig  httpd on

Restart Nagios to make the new settings take effect.

# service nagios start  
Starting nagios: done.

Step 10: Login to the Nagios Web Interface

username: nagiosadmin

Password: yourpassword

 

Nagios Web Dashboard

 

How to roll back to a patched server to an earlier patch level

1.) Edit the /etc/grub.conf file to make the kernel boot to an eariler kernel. Inside of /etc/grub.conf, file the line “default=0” and change to “default=1”

#vi /etc/grub.conf

Change: default=0
To: default=1
<save and quit>
2.) Reboot the server so you can boot into the second, pervious, kernel

#reboot
3.) After reboot, find out when the patching took place.

#yum history list

4.) After finding out when the patching took place. undo the patching by the ID number.

#yum history undo <ID # of patch update>

5.) Check for patch duplicates

#yum check duplicates

• Please use the following steps to create Oracle ASM disks with either disks or dm-multipath:
  1. Identify Device(s): determine self-identifier for the disk or multipath device using an environmental (E:) variable from udevadm command output.
  2. Create Rule(s): Create a udev rule file /etc/udev/rules.d/96-storage-asm.rules and add a udev rule for each disk or multipath device.
  3. Apply Rule(s): Reload the udev service for the above changes to take effect and create the ASM disks.
• It is highly recommended to also review the following Best Practices guide which covers udev rules in greater detail, as well as various other important settings recommend for running Oracle
• Identify Device(s): determine self-identifier for the disk or multipath device using an environmental (E:) variable from udevadm command output. The identifier is required to be used for creating Oracle ASM devices.
  • A “self-identifier” is an identifier value provided directly by the disk or created multipath device that uniquely identifies the device independent of any device name assigned by the host. That is, while assigned sdX and mpathN names may change across boots, the identifier provided by the device will not change. Typically this identifier is based upon the disk’s World Wide IDentifer (WWID).
  • The command below will need to be repeated for each disk and/or multipath device that is to be used with Oracle ASM. Please note down the E: environmental value retrieved for each device. This value will be used in creating the ASM disk udev rule.
  • For disks, use ID_WWN (identifier, world wide name) — or if that is not available then ID_SERIAL, the disks serial number. Replace sdXbelow with the scsi disk name to be used as ASM disk

$ udevadm info --query=all --name=/dev/sdX |  egrep "WWN|SERIAL"
E: ID_SERIAL=ST6000NM0004-1FFFFF_ZZZZAAAA
E: ID_SERIAL_SHORT=Z4D0HDFG
E: ID_WWN=0x5000c50000000000                              <<== Use this id
E: ID_WWN_WITH_EXTENSION=0x5000c50000000000

For dm-multipath devices, use DM_UUID (dm device Universally Unique IDentifier). 
This identifier is nominally based up the underlying device's WWID or serial number. 
Replace mpathX below with the dm-multipath name to be used as ASM disk.

$ udevadm info --query=all --name=/dev/mapper/mpathX |  grep -i "DM_UUID"
E: DM_UUID=mpath-36001405a02467a627a24e62afa3d506c         <<== Use this id

$ udevadm info --query=all --name=/dev/sdX | grep -i "ID_WWN"
E: ID_WWN=0x6001405a02467a62
E: ID_WWN_VENDOR_EXTENSION=0x7a24e62afa3d506c
E: ID_WWN_WITH_EXTENSION=0x6001405a02467a627a24e62afa3d506c
# sg_inq -p 0x83 /dev/sdX
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 20
    designator_type: NAA,  code_set: Binary
    associated with the addressed logical unit
      NAA 6, IEEE Company_id: 0x1405
      Vendor Specific Identifier: 0xa02467a62
      Vendor Specific Identifier Extension: 0x7a24e62afa3d506c
      [0x6001405a02467a627a24e62afa3d506c]

• NOTE: The DM_UUID above is derived from the underlying device’s WWN as shown. Within the above, sdX is one of the paths of the mpathX device. The ID_WWN_WITH_EXTENSION is the full NAA WWID of the disks under the multipath device.
• NOTE: Using the environmental variables of udev is much more efficient than using Program= within udev rules. Therefore using the E:variables already available for a device is the recommended practice when creating udev rules.

 

Create Rule(s): Create a udev rule file /etc/udev/rules.d/96-storage-asm.rules and add a udev rule for each disk or multipath device.

• For disks, add the following rule within the above file:

  ACTION=="add|change", ENV{ID_WWN}=="{id-wwn-from-udevadm}", SYMLINK+="oracleasm/{asm-disk-name}", GROUP="dba", OWNER="oracle", MODE="0660"
  
  for example, using the information from step 1 to create ASM disk named asm01:
  
  ACTION=="add|change", ENV{ID_WWN}=="0x5000c50000000000", SYMLINK+="oracleasm/asm01", GROUP="dba", OWNER="oracle", MODE="0660" For dm-multipath devices, add the following rule within the above file:

  ACTION=="add|change", ENV{DM_UUID}=="{dm-uuid-from-udevadm}", SYMLINK+="oracleasm/{asm-disk-name}", GROUP="dba", OWNER="oracle", MODE="0660
  
  for example, using the information from step 1 to create ASM disk named asm01:
  
  ACTION=="add|change", ENV{DM_UUID}=="mpath-36001405a02467a627a24e62afa3d506c", SYMLINK+="oracleasm/asm01", GROUP="dba", OWNER="oracle", MODE="0660"

  • Similarly, a udev rule for each device would need to be added into the /etc/udev/rules.d/96-storage-asm.rulesfile.
  • Modify the Group, Owner, and MODE values as needed for matching your environment.

   

Apply Rule(s): As root, reload the udev service for the above changes to take effect and create the ASM disks. See "How to reload "udev rules" without restarting udev service? " for more complete steps.

# udevadm control --reload-rules
# udevadm trigger --type=devices --action=change