Wednesday, 30 September 2015

Oracle cx_Oracle python nagios DB check

I used this python plugin for nagios to find delay in a database. Database is used for etl processes. You need to create a readonly user at Oracle DB.

It takes arguments ip of db, delay , table name holds delay data, and sid
import sys,os

import cx_Oracle
ip = str(sys.argv[1])
port = 1521
SID = str(sys.argv[4])
maxcount = sys.argv[2]
TABLE = str(sys.argv[3])

dsn_tns = cx_Oracle.makedsn(ip, port, SID)
connection = cx_Oracle.connect('dbuser', 'dbpass', dsn_tns)
cursor = connection.cursor()
cursor.execute("SELECT max(m5_id) FROM  " + TABLE )

m5_id = cursor.fetchall()[0][0]
delay = cursor.fetchall()[0][0]

if int(delay) < int(maxcount):
        print "OK | "  + str(delay) 
        print "NOK | "  + str(delay) 

Nagios definitions
define command{
        command_name    check_dwhtime3
        command_line    $USER1$/   $HOSTADDRESS$   $ARG1$  $ARG2$ $ARG3$

define service{
        use                     generic-service
        host_name               servername
        service_description     DWH time
        check_command           check_dwhtime3!90!DWH.DWH_TIME_TB!SID
        contact_groups          admins

Tuesday, 29 September 2015

log trace send application logs to remote syslog server

I aimed send log file (not /var/log/* all kind of application log files) from a system to remote syslog server. Nice part ; using multiple cpu, runs as a deamon , and it is configurable :) - you can specify any file,  you can search specific keywords or you can send all lines. all syslog proprieties are also configurable .

Probably you cant see all code so please select all start from to
#!/usr/bin/python to the end and then paste. You will see all codes.

Description: This threaded python script checks text files which defined in parameter file and (if you want you can specify search keywords) send lines to syslog server (with server, port, facility,priority) or any local file. It runs in daemon mode.

#Name:          Logtrace
#Release:       v0.2    03.June.2007
#Description:   This threaded python script checks text files which defined in parameter file and
#               (if you want you can specify search keywords) send lines to syslog
#               server (with server, port, facility,priority) or any local file.
#               It runs in daemon mode. 
#               Threads depends on logfile count*2.
#Written by:    ANIL ERCAN SONMEZ 
#Modified by:
#Notice:        You will see many processes, if you check with ps command.
#Modules:       ConfigParser,os,re,socket,sys,threading,time
#Parameter file content:
#;Log parameters for logtrace python script
#;Put your parameter file (logparam.ini) and script in the same directory
#;Each section [] starts with logfile_??
#;logfilename:   Put your file name here search for logtext_?? and send each line to syslog server or any local file.
#;logfacility:   auth,authpriv,cron,daemon,kern,lpr,mail,news,security,syslog,user,uucp
#;               local0,local1,local2,local3,local4,local5,local6,local7
#;               You can configure your syslog server which facility will be written to files.
#;               Check syslog documentation for priority and facility.
#;logpriority:   alert,crit,debug,emerg,err,error,info,notice,panic,warn,warning
#;               You can configure syslog server which priority  will be written to files.
#;               Check syslog documentation for priority and facility
#;logserver:     Ip adress of your syslog server.
#;               Please check syslog server has started with -r option and syslog.conf
#;               is configured to write to relative log file (messages etc.).
#;               You can find further information in syslog documentation.
#;               If you want to send logs another file in local system
#;               (except syslog controlled files messages,cron,boot etc)
#;               leave empty logport option leave empty logport option
#;logport:       syslog port, default 514.
#;timeout:       frequeny of log file control.
#;logtext_??:    search text for you log file. Use '' for sending all of new records.You can append more search.

def daemonize():
        import os,sys
        if os.fork(): os._exit(0)
        sys.stdin  = sys.__stdin__  = open('/dev/null','r')
        sys.stdout = sys.__stdout__ = open('/dev/null','w')
        sys.stdout = sys.__stderr__ = os.dup(sys.stdout.fileno())

def log_watcher(logfilename,logfacility,logpriority,logserver,logport,logtimeout,search_keywords):
        import time, os, re
        file = open(logfilename, 'r')
        watcher = os.stat(logfilename)
        this_modified = last_modified = watcher[8]

        """ Go to the end of the file """,2)

        """ Main Loop """
        while 1:
                if this_modified > last_modified:
                        last_modified = this_modified
                        """ File was modified, so read new lines, look for error keywords """
                        while 1:
                                line = file.readline()
                                if not line: break
                                for keyword in search_keywords:
                                        if, line):
                                                if logport=='':
                                                        lgrfile.write(logfilename+' ' +line)
                                                        lgr = syslog_client((logserver,int(logport)))
                watcher = os.stat(logfilename)
                this_modified = watcher[8]

def configread():
        config = ConfigParser.ConfigParser()
 confpath =  os.path.dirname(sys.argv[0]) + '/logparam.ini'
        thr = []
        for section in config.sections():
                if'logfile',section) :
                        logfilename= config.get(section,'logfilename')
                        logfacility= config.get(section,'logfacility')
                        logpriority= config.get(section,'logpriority')
                        logserver= config.get(section,'logserver')
                        logport=  config.get(section,'logport')
                        logtimeout= config.get(section,'logtimeout')
                        for option in config.options(section):
                                        keyword = config.get(section,option)
                        thr= threading.Thread(target=log_watcher,kwargs={"logfilename":logfilename,"logfacility":logfacility,"logpriority":logpriority,"logserver":logserver,"logport":logport,"logtimeout":logtimeout,"search_keywords":search_keywords})

#-----This part belong to Sam Rushing
# ======================================================================
# Copyright 1997 by Sam Rushing
#                         All Rights Reserved
# priorities (these are ordered)

LOG_EMERG               = 0             #  system is unusable
LOG_ALERT               = 1             #  action must be taken immediately
LOG_CRIT                = 2             #  critical conditions
LOG_ERR                 = 3             #  error conditions
LOG_WARNING             = 4             #  warning conditions
LOG_NOTICE              = 5             #  normal but significant condition
LOG_INFO                = 6             #  informational
LOG_DEBUG               = 7             #  debug-level messages

#  facility codes
LOG_KERN                = 0             #  kernel messages
LOG_USER                = 1             #  random user-level messages
LOG_MAIL                = 2             #  mail system
LOG_DAEMON              = 3             #  system daemons
LOG_AUTH                = 4             #  security/authorization messages
LOG_SYSLOG              = 5             #  messages generated internally by syslogd
LOG_LPR                 = 6             #  line printer subsystem
LOG_NEWS                = 7             #  network news subsystem
LOG_UUCP                = 8             #  UUCP subsystem
LOG_CRON                = 9             #  clock daemon
LOG_AUTHPRIV    = 10    #  security/authorization messages (private)
#  other codes through 15 reserved for system use
LOG_LOCAL0              = 16            #  reserved for local use
LOG_LOCAL1              = 17            #  reserved for local use
LOG_LOCAL2              = 18            #  reserved for local use
LOG_LOCAL3              = 19            #  reserved for local use
LOG_LOCAL4              = 20            #  reserved for local use
LOG_LOCAL5              = 21            #  reserved for local use
LOG_LOCAL6              = 22            #  reserved for local use
LOG_LOCAL7              = 23            #  reserved for local use

priority_names = {
        "alert":        LOG_ALERT,
        "crit":         LOG_CRIT,
        "debug":        LOG_DEBUG,
        "emerg":        LOG_EMERG,
        "err":          LOG_ERR,
        "error":        LOG_ERR,                #  DEPRECATED
        "info":         LOG_INFO,
        "notice":       LOG_NOTICE,
        "panic":        LOG_EMERG,              #  DEPRECATED
        "warn":         LOG_WARNING,            #  DEPRECATED
        "warning":      LOG_WARNING,

facility_names = {
        "auth":         LOG_AUTH,
        "authpriv":     LOG_AUTHPRIV,
        "cron":         LOG_CRON,
        "daemon":       LOG_DAEMON,
        "kern":         LOG_KERN,
        "lpr":          LOG_LPR,
        "mail":         LOG_MAIL,
        "news":         LOG_NEWS,
        "security":     LOG_AUTH,               #  DEPRECATED
        "syslog":       LOG_SYSLOG,
        "user":         LOG_USER,
        "uucp":         LOG_UUCP,
        "local0":       LOG_LOCAL0,
        "local1":       LOG_LOCAL1,
        "local2":       LOG_LOCAL2,
        "local3":       LOG_LOCAL3,
        "local4":       LOG_LOCAL4,
        "local5":       LOG_LOCAL5,
        "local6":       LOG_LOCAL6,
        "local7":       LOG_LOCAL7,

import socket

class syslog_client:
        def __init__ (self, address='/dev/log'):
                self.address = address
                if type (address) == type(''):
                        self.socket = socket.socket (socket.AF_UNIX, socket.SOCK_STREAM)
                        self.socket.connect (address)
                        self.unix = 1
                        self.socket = socket.socket (socket.AF_INET, socket.SOCK_DGRAM)
                        self.unix = 0

        # curious: when talking to the unix-domain '/dev/log' socket, a
        #   zero-terminator seems to be required.  this string is placed
        #   into a class variable so that it can be overridden if
        #   necessary.

        log_format_string = '<%d>%s\000'

        def log (self, message, facility=LOG_USER, priority=LOG_INFO):
                message = self.log_format_string % (
                        self.encode_priority (facility, priority),
                if self.unix:
                        self.socket.send (message)
                        self.socket.sendto (message, self.address)

        def encode_priority (self, facility, priority):
                if type(facility) == type(''):
                        facility = facility_names[facility]
                if type(priority) == type(''):
                        priority = priority_names[priority]
                return (facility<<3 data-blogger-escaped-br="" data-blogger-escaped-priority="">
        def close (self):
                if self.unix:

#-----Sam Rushing finished here.

if __name__=='__main__':
        import re
        import os
        import sys
        import ConfigParser
        import threading

Nagios python plugin

I used below plugin for creating easy services in Nagios. Main idea behind this without adding new commands for everycheck i just use same command for many service definition
check_command_big!find /var/ | grep test  |wc -l!5   . Here you can change only OS commands and check anything you want greater than 5 or any value you state. another example of check can be check_command_big!ps -ef | grep aprocess | wc -l !5


#import sys, os, base64, getpass, socket, traceback, termios, tty, select
import paramiko, getpass
import os,sys,re
import signal


t = paramiko.Transport((server,22))
        print server + ": Bad password or login!"
        ch = t.open_channel(kind = "session")
        if (ch.recv_ready):
                if x <= maxcount:
                        print "OK " + str(x) + " command=" + re.sub(r'\|', "",sys.argv[2]) + " | " +  str(x)
                        print "NOK " + str(x) + " command=" + re.sub(r'\|', "",sys.argv[2]) + " | " +  str(x)

define command{
        command_name    check_command_big
        command_line    $USER1$/  $HOSTADDRESS$ '$ARG1$' $ARG2$

define service{
        hostgroup_name                  testservers
        use                             generic-service
        check_interval                  10
        service_description             TEST
        check_command                   check_command_big!find /var/ | grep test  |wc -l!5

Sunday, 27 September 2015

Cacti plugin installation

You need at least plugin achitecture, settings plugin. Here i used thold plugin for test.
Download plugin architecture cacti-plugin-0.8.7g-PA-v2.8.tar.gz
Download plugins settings-v0.7-1.tar.gz and thold-latest.tar.gz

# tar -zxvf settings-v0.7-1.tar.gz
# tar -zxvf cacti-plugin-0.8.7g-PA-v2.8.tar.gz
# tar -zxvf thold-latest.tar.gz
# cp -R /root/cacti-plugin-arch/files-0.8.7g/ /var/www/html/cacti/
# cp -R /root/settings /var/www/html/cacti/plugins
# cp -R /root/thold-0.41 /var/www/html/cacti/plugins

Under /root/cacti-plugin-arch
# mysql cacti -u root -p < pa.sql

edit /var/www/html/cacti/include/config.php and add lines below

/* load up old style plugins here */
$plugins = array();
//$plugins[] = 'thold';

   Edit this to point to the default URL of your Cacti install
   ex: if your cacti install as at http://serverip/cacti/ this
   would be set to /cacti/
$url_path = "/cacti/";

User Management enable plugin management
Plugin management enable plugins
User Management configure view and configure settings again.

Calculate timezone difference for bash epoch time

Lets find difference of time from UTC with epoh time.
tzdifference=$(($(TZ=Europe/Istanbul date +%:::z)*60*60*1000))

And add this to UTC for showing time local.
All time information was epoch time.
cat $filename | sed  's/\\,//g' |  awk -F"," '{print $1","$2","$3","$4","strftime("%F %T",($5+'$tzdifference')/1000)","$6","$7","$8","$9","$10","$11","$12","$13","$14    

Bonding linux

Cables are connected to eth2 and eth3 , check switch lacp is enabled. 

Edit file /etc/sysconfig/network-scripts/ifcfg-eth2 
and ifcfg-eth3


Add this /etc/modprobe.conf
alias bond0 bonding

Create /etc/sysconfig/network-scripts/ifcfg-bond0
IPADDR=your ip address
GATEWAY=your gatweay 
BONDING_OPTS="miimon=1000 mode=balance-rr"

check for other options 

/etc/init.d/network restart 

Bandwith Calculation

I need to find bandwith between two server. I used iperf for this.
Just compile it run on one instance as server 
# iperf -s

On another server 
# /usr/local/bin/iperf -c ipaderssofinstance -f M

It generates outputs like below
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   830 MBytes  83.0 MBytes/sec