Cron Job : To automate the service down up restart

1. Write one shell script
2. Add that into cron job
3. Restart the cron service

1. /opt/folder/filename.sh

#!/bin/bash
# Grabs and kill a process from the pidlist that has the word mq

pid=`ps aux | grep api.jar | awk '{print $1}'`
kill -9 $pid


pid=`ps aux | grep api.jar | awk '{print $2}'`
kill -9 $pid

#!/bin/bash
nohup java -jar /opt/folder/api.jar & \n


2. Open  vi /etc/crontab

 > crontab -e                 //to edit cronfile

>  crontab -l                    //to list cron settings

>  crontab -d                    //to delete cron settings

# m h dom mon dow user  command

10 6,14 * * * root sh /opt/folder/filename.sh

#to store logs in custom folder

*/1 * * * * /usr/bin/python3 /opt/drvijay/applications/backend/testcorn.py >> /opt/drvijay/logs/testcorn.log 2>&1

3. Run
service cron stop
service cron start

To see system log or custom log

grep CRON /var/log/syslog
tail -f /var/log/syslog | grep CRON

tail -f /opt/drvijay/logs/testcorn.log

it will run 6:10 AM, 2:10 PM.

Cluster - Rabbit MQ Master Slave

RabbitmQ Master & Slave Installation Documents:

apt-get update
apt-get install rabbitmq-server
Enable the plugin - cd /etc/rabbitmq - rabbitmq-plugins enable rabbitmq_management

Master:

vi /etc/rabbitmq/rabbitmq-env.conf
    NODENAME=rabbit@rabbitmaster (must be unique, can be arbitrary)
chown rabbitmq:rabbitmq /etc/rabbitmq/*
chmod 400 /etc/rabbitmq/*
 vi /etc/hosts - add the following line (must match the content after "@" in the rabbitmq-env.conf NODENAME
    127.0.0.1   rabbitmaster
systemctl restart rabbitmq-server - a success message is displayed when the restart completes successfully.

Slave:

vi /etc/rabbitmq/rabbitmq-env.conf
   NODENAME=rabbit@rabbitslave
vi /etc/hosts - add the following lines:
Note:  192.168.7.* is the local IPv4 address of the master server (e.g., 192.x.x.x)

192.168.7.*  rabbitmaster

127.0.0.1   rabbitslave


sudo chown rabbitmq:rabbitmq /etc/rabbitmq/*

ls -a - show the hidden files & folders
ls -la - show the hidden files & folders with permission

execute below commands in master to slave

before copying please take .erlang.cookie in slave server.
scp -r /var/lib/rabbitmq/.erlang.cookie root@192.168.7.189:/var/lib/rabbitmq/
chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
chmod 600 /var/lib/rabbitmq/.erlang.cookie

systemctl restart rabbitmq-server

execute below commands in master

systemctl restart rabbitmq-server
rabbitmqctl start_app

Before this step, you have to reboot your slave servers. Run this at all slave nodes;

systemctl status rabbitmq-server
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@rabbitmaster
rabbitmqctl start_app

Execute the below commands in master server

Verify the cluster with running this command at master node;
rabbitmqctl cluster_status
Adding new administrator user
Add a new/fresh user, say user ‘test’ and password ‘test’
rabbitmqctl add_user test test
Give administrative access to the new user
rabbitmqctl set_user_tags test administrator
Set permission to newly created user
rabbitmqctl set_permissions -p / test ".*" ".*" ".*"
HA Queues
To make all queues HA run this command at master. With this policy enabled RabbitMQ sync all queues to all nodes.
rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

Haproxy Load Balace - Both TCP & HTTP [ Rabbitmq and Rest Webservice ]

Example- 
1. HAProxy installed in             192.168.7.11
MQ installed in                          192.168.7.12,            192.168.7.13
REST API also running in        192.168.7.12,             192.168.7.13

2. http://192.168.7.11:8080
3. Run your queue sender to 192.168.7.11:5672
     //MQ send receiver code = http://drvijayy2k2.blogspot.com/2019/07/java-rabbitmq-sample-send-receiver.html


haproxy.cfg

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client  50000
timeout server  50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend haproxy_in_ws
    bind *:8080
    mode http
    default_backend haproxy_http_ws

frontend haproxy_in_mq
    bind *:8890
    mode tcp
    default_backend haproxy_tcp_mq

backend haproxy_http_ws
    balance roundrobin
    mode http
    server rabbitmaster 192.168.1.12:8080 check
    server rabbitslave  192.168.1.13:8080 check

backend haproxy_tcp_mq
    balance roundrobin
    mode tcp
    server rabbitmaster 192.168.1.12:5672 check
    server rabbitslave  192.168.1.13:5672 check

Java RabbitMQ - Sample Send Receiver

//path lib setting
set CLASSPATH=.;lib\amqp-client-5.4.1.jar;lib\*.jar;lib\slf4j-api-1.7.26.jar;

//compile
javac TestSender.java

//run
java TestSender


TestSender.java

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;

public class TestSender
{

private final static String QUEUE_NAME = "rmq";

public static void main ( String [] argv ) throws Exception
{
ConnectionFactory factory = new ConnectionFactory ();
factory.setHost ( "192.168.7.11" );
factory.setPort ( 8890 ); //always give HA port to test if you want to do HA testing. else give queue port
factory.setUsername ( "test" );
factory.setPassword ( "test" );
try (Connection connection = factory.newConnection ();
Channel channel = connection.createChannel ())
{
channel.queueDeclare (QUEUE_NAME, true, false, false, null );
String message = "Hello World111!";
channel.basicPublish ( "", QUEUE_NAME, null, message.getBytes ( "UTF-8" ) );
System.out.println ( " [x] Sent '" + message + "'" );
}
}
}

TestReceiver.java

import com.rabbitmq.client.Channel;

import com.rabbitmq.client.Connection;

import com.rabbitmq.client.ConnectionFactory;

import com.rabbitmq.client.DeliverCallback;

public class TestReceiver

{

private final static String QUEUE_NAME = "rmq";

    public static void main(String[] argv) throws Exception {

        ConnectionFactory factory = new ConnectionFactory();

        factory.setHost ( "192.168.7.13" );

factory.setPort ( 5672 ); //always receive with your default queue port and not HA port

factory.setUsername ( "test" );

factory.setPassword ( "test" );

        Connection connection = factory.newConnection();

        Channel channel = connection.createChannel();

        channel.queueDeclare(QUEUE_NAME, true, false, false, null);

        System.out.println(" [*] Waiting for messages. To exit press CTRL+C");

        DeliverCallback deliverCallback = (consumerTag, delivery) -> {

            String message = new String(delivery.getBody(), "UTF-8");

            System.out.println(" [x] Received '" + message + "'");

        };

        channel.basicConsume(QUEUE_NAME, true, deliverCallback, consumerTag -> { });

    }

}

NOTE: channel.queueDeclare (QUEUE_NAME, true, false, false, null ); //if durable else give false instead of true.

Python - Construct Json in loop

data={}
for  weightagesObj in weightagesAllObject:
        item = {"weight":weightagesObj.weight, "rank":weightagesObj.rank}
        data[weightagesObj.scrip] = item
nseJson=json.loads ( json.dumps(data) )
print ( nseJson["scrip name"] )



{
"RELIANCE": {
"weightage": 9.04,
"rank": 1
},
"TATAMOTORS": {
"weightage": 9.01,
"rank": 2
}
}

Python - Object of type decimal is not json serializable

Simply add your header like below

import simplejson as json

#import json #remove this line

Python - Expecting property key name enclosed in double quotes - String to JSON

import demjson 
result = demjson.decode(' { key: "value" }' )


demjson plugin is very good on this

- Enjoy