18-Kafka01

字数 1157阅读 16

Kafka: 消息中间件 -->分布式流式平台

MQ Redis

Kafka    Flume

生产者  source

Broker channel

消费者 sink

正常部署的是: Broker进程

Flume-->Kafka-->Spark streaming 实时流式

Kafka Streaming

发布/订阅

实时app

分布式  副本数 一个

SCALA编写

部署:

kafka_2.11-0.10.0.0.tgz

scala version:2.11

准备环境:

1.JDK

java -version

2.Scala

scala -version

3.Zookeeper

[root@yws85 software]# ln -s zookeeper-3.4.6 zookeeper

[root@yws85 software]# ll

total 462992

lrwxrwxrwx  1 hdfs hdfs        33 Jan  5  2018 flink -> flink1.4.0-hadoop2.6.0-cdh5.12.0/

drwxr-xr-x  11 hdfs hdfs      4096 Jan  5  2018 flink1.4.0-hadoop2.6.0-cdh5.12.0

-rw-r--r--  1 root root 424555111 Aug 20  2017 hadoop-2.8.1.tar.gz

-rw-r--r--  1 root root  31831257 Aug 25  2018 kafka_2.11-0.10.0.1.tgz

drwxrwxr-x.  6 root root        46 Sep 12  2017 scala-2.11.8

lrwxrwxrwx  1 root root        15 Aug 17 23:38 zookeeper -> zookeeper-3.4.9

drwxr-xr-x  10 1000 1000      4096 Feb 20  2014 zookeeper-3.4.6  bug

zookeeper-3.4.9

-rw-r--r--  1 root root  17699306 Jun 20  2017 zookeeper-3.4.6.tar.gz

[root@yws85 software]#

软连接

ln -s 物理文件夹/文件 快捷的文件夹/文件

1.删除 快捷的文件夹/文件 ,增加安全系数

2.多版本管理

3.硬连接

常用命令:

kafka-topics.sh

bin/kafka-topics.sh --create \

--zookeeper hadoop001:2181/kafka \

--replication-factor 1 \

--partitions 3 \

--topic test

高并发写/读 

bin/kafka-topics.sh --list \

--zookeeper hadoop001:2181/kafka

生产者:

bin/kafka-console-producer.sh \

--broker-list yws85:9092,yws86:9092,yws87:9092 \

--topic test

消费者

bin/kafka-console-consumer.sh \

--zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \

--topic test \

--from-beginning

bin/kafka-topics.sh \

--describe \

--zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \

--topic test

bin/kafka-topics.sh \

--alter \

--zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \

--topic test \

--partitions 4

干净删除:

bin/kafka-topics.sh --delete \

--zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \

--topic test

rmr /kafka/admin/delete_topics/test

rmr /kafka/config/topics/test

rmr /kafka/brokers/topics/test

rm -rf logs/test-*

概念:

topic:主题

ERP系统log-->erp_topic

OA系统log-->oa_topic

partitions:分区

--partitions 3  3个分区  下标从0开始

--replication-factor 3  一个分区3个副本

单个分区:有序

全局分区: 无序

如何保证生产上 全局有序?

MySQL BINLOG 日志文件 按顺序----有序-----》Kafka test 3个分区

ruozedata.stu

id  name age

insert into stu values(1,'jepson',18);

insert into stu values(2,'ruoze',28);

update stu set age=26 where id=1;

delete from stu where id=1;

test-0:

insert into stu values(1,'jepson',18);

test-1:

insert into stu values(2,'ruoze',28);

delete from stu where id=1;

test-2:

update stu set age=26 where id=1;

消费时

insert into stu values(2,'ruoze',28);

delete from stu where id=1;

insert into stu values(1,'jepson',18);

update stu set age=26 where id=1;

核心点:  共性数据发送到同一个topic的1个分区

拼装的Key: ruozedata_stu_id=1  代码  hash(ruozedata_stu_id=1 ) 取模 0,1,2

    value: sql

test-0:

insert into stu values(1,'jepson',18);

update stu set age=26 where id=1;

delete from stu where id=1;

test-1:

insert into stu values(2,'ruoze',28);

test-2:

[root@yws86 kafka]# bin/kafka-topics.sh \

> --describe \

> --zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \

> --topic test

Topic:test      PartitionCount:3        ReplicationFactor:3    Configs:

Topic: test    Partition: 0    Leader: 1      Replicas: 1,2,3 Isr: 1,2,3

Topic: test    Partition: 1    Leader: 2      Replicas: 2,3,1 Isr: 2,3,1

Topic: test    Partition: 2    Leader: 3      Replicas: 3,1,2 Isr: 3,1,2

[root@yws86 kafka]#

第一行: 哪个topic 多少个分区  每个分区多少个副本数

Topic: test    Partition: 0    Leader: 1      Replicas: 1,2,3 Isr: 1,2,3

分区-0

Replicas: 1,2,3 三个副本位于

Leader: 1  读写

Isr: 1,2,3 

[root@yws86 kafka]# bin/kafka-topics.sh \

> --describe \

> --zookeeper yws85:2181,yws86:2181,yws87:2181/kafka \

> --topic test

Topic:test      PartitionCount:3        ReplicationFactor:3    Configs:

        Topic: test    Partition: 0    Leader: 2      Replicas: 1,2,3 Isr: 2,3

        Topic: test    Partition: 1    Leader: 2      Replicas: 2,3,1 Isr: 2,3,1

        Topic: test    Partition: 2    Leader: 3      Replicas: 3,1,2 Isr: 3,1,2

[root@yws86 kafka]# bin/kafka-topics.sh --describe --zookeeper yws85:2181,yws86:2181,yws87:2181/kafka --topic test

Topic:test      PartitionCount:4        ReplicationFactor:3    Configs:

        Topic: test    Partition: 0    Leader: 2      Replicas: 1,2,3 Isr: 2,3,1

        Topic: test    Partition: 1    Leader: 2      Replicas: 2,3,1 Isr: 2,3,1

        Topic: test    Partition: 2    Leader: 3      Replicas: 3,1,2 Isr: 3,2,1

        Topic: test    Partition: 3    Leader: 1      Replicas: 1,3,2 Isr: 1,3,2

[root@yws86 kafka]#

1 /3 1

2    2

3    0

4    1

5    2

6    0

7

8

9

10  1

HBASE Phoniex 部署

推荐阅读更多精彩内容