Kafka - Multi broker Cluster like Master Slave

Source : https://kafka.apache.org/quickstart
Thanks


Setting up a multi-broker cluster

So far we have been running against a single broker, but that's no fun. For Kafka, a single broker is just a cluster of size one, so nothing much changes other than starting a few more broker instances. But just to get feel for it, let's expand our cluster to three nodes (still all on our local machine).

First we make a config file for each of the brokers (on Windows use the copy command instead):
   
> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties



Now edit these new files and set the following properties:   
config/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dirs=/tmp/kafka-logs-1

config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dirs=/tmp/kafka-logs-2



The broker.id property is the unique and permanent name of each node in the cluster. We have to override the port and log directory only because we are running these all on the same machine and we want to keep the brokers from all trying to register on the same port or overwrite each other's data.

We already have Zookeeper and our single node started, so we just need to start the two new nodes:
   
> bin/kafka-server-start.sh config/server-1.properties &
...
> bin/kafka-server-start.sh config/server-2.properties &
...

Now create a new topic with a replication factor of three:
> bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 1 --topic my-replicated-topic

Okay but now that we have a cluster how can we know which broker is doing what? To see that run the "describe topics" command:
   
> bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic my-replicated-topic
Topic:my-replicated-topic   PartitionCount:1    ReplicationFactor:3 Configs:
    Topic: my-replicated-topic  Partition: 0    Leader: 1   Replicas: 1,2,0 Isr: 1,2,0

Here is an explanation of output. The first line gives a summary of all the partitions, each additional line gives information about one partition. Since we have only one partition for this topic there is only one line.

    "leader" is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.
    "replicas" is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
    "isr" is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.

Note that in my example node 1 is the leader for the only partition of the topic.

We can run the same command on the original topic we created to see where it is:

> bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic test
Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: test Partition: 0    Leader: 0   Replicas: 0 Isr: 0

So there is no surprise there—the original topic has no replicas and is on server 0, the only server in our cluster when we created it.

Let's publish a few messages to our new topic:
   
> bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-replicated-topic
...
my test message 1
my test message 2
^C

Now let's consume these messages:
   
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
...
my test message 1
my test message 2
^C

Now let's test out fault-tolerance. Broker 1 was acting as the leader so let's kill it:
   
> ps aux | grep server-1.properties
7564 ttys002    0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java...
> kill -9 7564
On Windows use:
   
> wmic process where "caption = 'java.exe' and commandline like '%server-1.properties%'" get processid
ProcessId
6016
> taskkill /pid 6016 /f

Leadership has switched to one of the followers and node 1 is no longer in the in-sync replica set:
   
> bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic my-replicated-topic
Topic:my-replicated-topic   PartitionCount:1    ReplicationFactor:3 Configs:
    Topic: my-replicated-topic  Partition: 0    Leader: 2   Replicas: 1,2,0 Isr: 2,0

But the messages are still available for consumption even though the leader that took the writes originally is down:
   
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
...
my test message 1
my test message 2
^C

Cluster - Rabbit MQ Master Slave

RabbitmQ Master & Slave Installation Documents:

apt-get update
apt-get install rabbitmq-server
Enable the plugin - cd /etc/rabbitmq - rabbitmq-plugins enable rabbitmq_management

Master:

vi /etc/rabbitmq/rabbitmq-env.conf
    NODENAME=rabbit@rabbitmaster (must be unique, can be arbitrary)
chown rabbitmq:rabbitmq /etc/rabbitmq/*
chmod 400 /etc/rabbitmq/*
 vi /etc/hosts - add the following line (must match the content after "@" in the rabbitmq-env.conf NODENAME
    127.0.0.1   rabbitmaster
systemctl restart rabbitmq-server - a success message is displayed when the restart completes successfully.

Slave:

vi /etc/rabbitmq/rabbitmq-env.conf
   NODENAME=rabbit@rabbitslave
vi /etc/hosts - add the following lines:
Note:  192.168.7.* is the local IPv4 address of the master server (e.g., 192.x.x.x)

192.168.7.*  rabbitmaster

127.0.0.1   rabbitslave


sudo chown rabbitmq:rabbitmq /etc/rabbitmq/*

ls -a - show the hidden files & folders
ls -la - show the hidden files & folders with permission

execute below commands in master to slave

before copying please take .erlang.cookie in slave server.
scp -r /var/lib/rabbitmq/.erlang.cookie root@192.168.7.189:/var/lib/rabbitmq/
chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
chmod 600 /var/lib/rabbitmq/.erlang.cookie

systemctl restart rabbitmq-server

execute below commands in master

systemctl restart rabbitmq-server
rabbitmqctl start_app

Before this step, you have to reboot your slave servers. Run this at all slave nodes;

systemctl status rabbitmq-server
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@rabbitmaster
rabbitmqctl start_app

Execute the below commands in master server

Verify the cluster with running this command at master node;
rabbitmqctl cluster_status
Adding new administrator user
Add a new/fresh user, say user ‘test’ and password ‘test’
rabbitmqctl add_user test test
Give administrative access to the new user
rabbitmqctl set_user_tags test administrator
Set permission to newly created user
rabbitmqctl set_permissions -p / test ".*" ".*" ".*"
HA Queues
To make all queues HA run this command at master. With this policy enabled RabbitMQ sync all queues to all nodes.
rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

Haproxy Load Balace - Both TCP & HTTP [ Rabbitmq and Rest Webservice ]

Example- 
1. HAProxy installed in             192.168.7.11
MQ installed in                          192.168.7.12,            192.168.7.13
REST API also running in        192.168.7.12,             192.168.7.13

2. http://192.168.7.11:8080
3. Run your queue sender to 192.168.7.11:5672
     //MQ send receiver code = http://drvijayy2k2.blogspot.com/2019/07/java-rabbitmq-sample-send-receiver.html


haproxy.cfg

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client  50000
timeout server  50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend haproxy_in_ws
    bind *:8080
    mode http
    default_backend haproxy_http_ws

frontend haproxy_in_mq
    bind *:8890
    mode tcp
    default_backend haproxy_tcp_mq

backend haproxy_http_ws
    balance roundrobin
    mode http
    server rabbitmaster 192.168.1.12:8080 check
    server rabbitslave  192.168.1.13:8080 check

backend haproxy_tcp_mq
    balance roundrobin
    mode tcp
    server rabbitmaster 192.168.1.12:5672 check
    server rabbitslave  192.168.1.13:5672 check

JBoss 6 clustering with Apache Modjk [ Load balance ]

Download the software required

  Apache Server :- httpd-2.2.21-win32-x86-no_ssl.msi
  mod_jk :- tomcat-connectors-1.2.32-windows-i386-httpd-2.0.x.zip
  Jboss Application Server :- jboss-as-distribution-6.0.0.Final.zip

  unzip the mod_jk zip file and copy mod_jk.so and paste it under APACHE_HOME/modules/

Configure Apache to load mod_jk

  1)  Modify APACHE_HOME/conf/httpd.conf  and add a single line at the end of the file
   
     # Include mod_jk's specific configuration file 
  Include conf/mod-jk.conf
 

  and modify these line in httpd.conf file

  Listen localhost:80

  ServerName localhost

  
 2) Next, create a new file named APACHE_HOME/conf/mod-jk.conf:

   # Load mod_jk module
 # Specify the filename of the mod_jk lib
 LoadModule jk_module modules/mod_jk.so

 # Where to find workers.properties
 JkWorkersFile conf/workers.properties

 # Where to put jk logs
 JkLogFile logs/mod_jk.log

 # Set the jk log level [debug/error/info]
 JkLogLevel info

 # Select the log format
 JkLogStampFormat  "[%a %b %d %H:%M:%S %Y]"

 # JkOptions indicates to send SSK KEY SIZE
 JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

 # JkRequestLogFormat
 JkRequestLogFormat "%w %V %T"
              
 # Mount your applications
 JkMount /helloworld/* loadbalancer

     (Note :- here you have to give you war file name)

 # You can use external file for mount points.
 # It will be checked for updates each 60 seconds.
 # The format of the file is: /url=worker
 # /examples/*=loadbalancer
 JkMountFile conf/uriworkermap.properties              

 # Add shared memory.
 # This directive is present with 1.2.10 and
 # later versions of mod_jk, and is needed for
 # for load balancing to work properly
 JkShmFile logs/jk.shm
             
 # Add jkstatus for managing runtime data
 
     JkMount status
        Order deny,allow
     Deny from all
     Allow from 127.0.0.1
 

   3) Create a uriworkermap.properties file in the APACHE_HOME/conf directory.
 # Simple worker configuration file
 # Mount the Servlet context to the ajp13 worker
 /jmx-console=loadbalancer
 /jmx-console/*=loadbalancer
 /web-console=loadbalancer
 /web-console/*=loadbalancer
 /helloworld/*=loadbalancer

Configure worker nodes in mod_jk
   4) Next, you need to configure mod_jk workers file conf/workers.properties
 # Define list of workers that will be used
 # for mapping requests
 worker.list=loadbalancer,status
 #worker.list=node1

 # Define Node1
 # modify the host as your host IP or DNS name.
  worker.node1.port=8009
  worker.node1.host=192.168.7.59
  worker.node1.type=ajp13
  worker.node1.lbfactor=1
  worker.node1.cachesize=10

 # Define Node2
 # modify the host as your host IP or DNS name.
  worker.node2.port=8009
  worker.node2.host= 192.168.7.50
  worker.node2.type=ajp13
  worker.node2.lbfactor=1
  worker.node2.cachesize=10

 # Define Node3
 # modify the host as your host IP or DNS name.
  worker.node3.port=8009
  worker.node3.host= 192.168.7.58
  worker.node3.type=ajp13
  worker.node3.lbfactor=1
  worker.node3.cachesize=10

 # Load-balancing behaviour
  worker.loadbalancer.type=lb
  worker.loadbalancer.balance_workers=node1,node2,node3
  worker.loadbalancer.sticky_session=0
  worker.list=loadbalancer

 # Status worker for managing load balancer
  worker.status.type=status

Configuring HTTP session state replication
     1) Enabling session replication in your application
 To enable replication of your web application sessions, you must tag the application as  distributable in the web.xml descriptor(web.xml of your application war file).

<distributable/>
 

Deploy the application in JBoss
 Deploy the application war file in server\all\deploy
Start the Jboss different nodes
 Then run the Jboss in different machines by using command
      run.bat -b -c all -D jboss.messaging.ServerPeerID= -D  jboss.service.binding.set=ports-default

Example
 run.bat -b 192.168.7.59 -c all -D jboss.messaging.ServerPeerID=1 -D  jboss.service.binding.set=ports-default
 run.bat -b 192.168.7.50 -c all -D jboss.messaging.ServerPeerID=2 -D   jboss.service.binding.set=ports-default

Run the application using the URL
 http://localhost/helloworld/hi.jsp

Cluster Load Balance

step 1:

     download jdkx.x, tomcat 6.0 ,  apache http server [latest], jmeter [optional]

step2 :
       install jdk

    install  2 or 3 or n copies of tomcat 6.0 as tomcatA, tomcatB, tomcatC

step3:
     open server.xml in tomcatx.x\conf

change tomcatA

port="8115" shutdown="SHUTDOWN">

port="8111" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

port="8119" protocol="AJP/1.3" redirectPort="8443" />

jvmRoute="tomcatA">


   



change tomcatB

port="8225" shutdown="SHUTDOWN">

port="8221" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

port="8229" protocol="AJP/1.3" redirectPort="8443" />

jvmRoute="tomcatA">

   

change tomcatC [ports]


step 4

Add distributed tag in web.xml

Next step to do add distributed tag in web.xml to switching session among the tomcat clustering instance
Make any application, e.g we are making cluster as application folder in webapps
1. TomcatA - >  webapps - > cluster -> WEB-INF -> web.xml
2. TomcatB - >  webapps - > cluster -> WEB-INF -> web.xml
3. TomcatC - >  webapps - > cluster -> WEB-INF -> web.xml
Add  in all three web.xml file

"http://java.sun.com/xml/ns/javaee"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
   version="2.5">
 
 

step 5

Install mod_jk in module

Mod_jk is connector that communicates with apache web server and tomcat. We need to download and copy that connector in module folder of apache
http://tomcat.apache.org/connectors-doc/ we are using latest connector of mod_jk
Make mod_jk.log file in logs folder otherwise apache will throw error
RENAME MOD_JK.SO OR MOD_JK.[WINDOWS]


server.xml

Include c:/apache-tomcat/conf/auto/mod_jk.conf
[httpd.conf]


step 6

Edit in httpd.conf

Open apache -> httpd.conf
Open in any text editor and add

LoadModule jk_module modules/mod_jk-apache-2.2.4.so
JkWorkersFile "C:\cluster\Apache\conf\workers.properties"
JkLogFile "logs/mod_jk.log"
JkLogLevel error
JkMount /cluster loadbalancer
JkMount /cluster/* loadbalancer

Worker.properties file

Make a file with name of workers.properties in conf folder. This file tells properties of all tomcat instances. We have to specify all tomcat properties here. Apache will forword request to tomcat through this file
Vertical tomcat clustering this file like

workers.tomcat_home=/tomcatA
workers.java_home=$JAVA_HOME
ps=/
worker.list=tomcatA,tomcatB,tomcatC,loadbalancer

worker.tomcatA.port=8119
worker.tomcatA.host=localhost
worker.tomcatA.type=ajp13
worker.tomcatA.lbfactor=1

worker.tomcatB.port=8229
worker.tomcatB.host=localhost
worker.tomcatB.type=ajp13
worker.tomcatB.lbfactor=1
worker.tomcatC.port=8339
worker.tomcatC.host=localhost
worker.tomcatC.type=ajp13
worker.tomcatC.lbfactor=1
worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tomcatA,tomcatB,tomcatC
worker.loadbalancer.sticky_session=1

step 7



worker.tomcatC.lbfactor=100
increase and decrease request to this tomcatC instance

Restart apache, tomcatA, tomcatB, tomcatC

Check clustering if apache start properly, It is fine without any errors otherwise check problems,basically mod_jk problems come. So download different mod_jk for your machine. Start all tomcats, tomcatA, tomcatB, and tomcatC. If all tomcats is started this means tomcat is working fine.
Open test.jsp on browser and check session id. Check which tomcat on test.jsp is running tomcatB or tomcatC. Close that tomcat, reload test.jsp. Check session id,if session id is same. Then tomcat clustering is working fine.

Test jsp file

Make test.jsp in cluster folder of webapps
tomcatA

<%
  session.setAttribute("a","a");
%>







 
   
   
 
 
   
   
 

TomcatA Machine
Session ID :	<%=session.getId()%>

tomcatB

<%
  session.setAttribute("a","a");
%>







 
   
   
 
 
   
   
 

TomcatB Machine
Session ID :	<%=session.getId()%>

TomcatC
<%
  session.setAttribute("a","a");
%>







 
   
   
 
 
   
   
 

TomcatC Machine
Session ID :	<%=session.getId()%>

step 8


[httpd.conf]
Alias /docs /var/www/html/docs

JkMount /docs loadbalancer
JkMount /docs/* loadbalancer

JkUnMount /docs/*.gif loadbalancer
JkUnMount /docs*.png  loadbalancer


or

If the Tomcat Host appBase (webapps) directory is accessible by the Apache web server, Apache can be configured to serve web application context directory static files instead of passing the request to Tomcat.
Caution: For security reasons is is strongly recommended that JkMount is used to pass all requests to Tomcat by default and JkUnMount is used to explicitly exclude static content to be served by httpd. It should also be noted that content served by httpd will bypass any security constraints defined in the application's web.xml.
Use Apache's Alias directive to map a single web application context directory into Apache's document space for a VirtualHost:

# Static files in the examples webapp are served by apache
  Alias /examples /vat/tomcat3/webapps/examples
  # All requests go to worker1 by default
  JkMount /* worker1
  # Serve html, jpg and gif using httpd
  JkUnMount /*.html worker1
  JkUnMount /*.jpg  worker1



  JkUnMount /*.gif  worker1

links

http://www.easywayserver.com/implementation-tomcat-clustering.htm#server-xml

http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html