Rocketmq选型

Rocket是一个专业的队列服务,性能优于Rabbitmq,优势是性能和并发,源于Kafka的扩展版,增强了数据的可靠性。

Rocketmq的队列类型

普通队列,广播队列、顺序队列,分区顺序

2、同步机制

Rocketmq使用主从同步模式,同步分为同步和异步模式,这和mysql类似。

3、Rocketmq管理命令

rocketmq也可以通过web管理,坑中有说 

创建topic

bin/mqadmin updateTopic -n '192.168.1.64:9876;192.168.1.107:9876' -c DefaultCluster -t tt -w 5 -r 5 -o=true   #-o=true是有序队列

Rocketmq坑

1、rocketmq 在本地搭建环境启动时,内存溢出报了内存错误:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.

解决方法:修改bin/runbroker.sh,bin/runserver.sh

-server -Xms1g -Xmx1g -Xmn1g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m

内存大的虚拟机没问题,像我这渣渣笔记本,得节省点啊!

 

2、Rocketmq管理web

https://github.com/apache/rocketmq-externals/tree/master/rocketmq-console

下载安装Console项目

 

Rocketmq的普通队列、顺序队列、分区顺序队列、广播消息实现、事务队列

 

 

我遇到的坑

阿里云上面买的RocketMq和开源版本的Rocketmq的sdk不同,有着兼容问题。我们有一个自己机房和阿里的共用运行的服务,这时候发现只能在阿里机房上布,而在本地机房看来只能再写一套抽象层了,这非常不友好。使用了阿里的 Rocketmq也就意味着你的程序移植不到其它云或服务器上了。

slave 修改:

1、内存优化

bin/runbroker.sh MaxDirectMemorySize =15g  改成1g

 

2、host解析失败

在rocketmq的主从是通过获取hostname然后解析host到ip上进行互相访问的,所以需要将各机器的hosts增加解析

 

192.168.1.102       localhost.ceontsPHP

192.168.1.101       localhost.bobby

 

hostname可以通过各服务器的/etc/hostsname查看

rocket会报错:rocketmq Failed to obtain the host name

 

 

3、锁文件冲突原因

因为rocketmq设计一个用户运行一个示例,一个用户锁定了一个用户store存储目录,所以在一起运行会冲突,提示:Lock failed,MQ already startedz

解决方法是每个示例使用一个用户

broker.conf 指定 storePathRootDir目录

 

 

4、端口会冲突

不同broker需使用不同端口,如果不是root用户,端口号应该是1024+

一个broker不仅仅使用你指定的listenPort,上下最好空几个,比如指定1081端口,他还占用了1079 1081 1082

 

5、配置文件坑

我直接拿官方配置文件2m-2s-async下的配置,不行,没有提示,就是执行clusterList时,没有集群,这很坑,然后我自己建配置文件成功

 

bin/mqbroker -c /root/rocketmq/release/conf/2m-2s-async/broker-a-s.properties > a.log 2>&1
 
bin/mqadmin clusterList -n centos:9876
 
bin/mqadmin updateTopic -n 'centos:9876' -c mycluster -t tt -w 5 -r 5 -o=true
 
# broker a
namesrvAddr=centos:9876
brokerIP1=192.168.56.101
brokerName=broker-a
brokerClusterName=mycluster
brokerId=0
listenPort=1081
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
rejectTransactionMessage=false
fetchNamesrvAddrByAddressServer=false
storePathRootDir=/root/storeA
storePathCommitLog=/root/store/commitlog
flushIntervalCommitLog=500
commitIntervalCommitLog=200
flushCommitLogTimed=false
deleteWhen=04
fileReservedTime=48
maxTransferBytesOnMessageInMemory=262144
maxTransferCountOnMessageInMemory=32
maxTransferBytesOnMessageInDisk=65536
maxTransferCountOnMessageInDisk=8
accessMessageInMemoryMaxRatio=40
messageIndexEnable=true
messageIndexSafe=false
haMasterAddress=
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
cleanFileForciblyEnable=true
transientStorePoolEnable=false
#broker a-s
 
namesrvAddr=centos:9876
brokerIP1=192.168.56.102
brokerName=broker-a
brokerClusterName=mycluster
brokerId=1
listenPort=1091
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
rejectTransactionMessage=false
fetchNamesrvAddrByAddressServer=false
storePathRootDir=/root/storeAs
storePathCommitLog=/root/store/commitlog
flushIntervalCommitLog=500
commitIntervalCommitLog=200
flushCommitLogTimed=false
deleteWhen=04
fileReservedTime=48
maxTransferBytesOnMessageInMemory=262144
maxTransferCountOnMessageInMemory=32
maxTransferBytesOnMessageInDisk=65536
maxTransferCountOnMessageInDisk=8
accessMessageInMemoryMaxRatio=40
messageIndexEnable=true
messageIndexSafe=false
haMasterAddress=
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
cleanFileForciblyEnable=true
transientStorePoolEnable=false
 
#broker b
namesrvAddr=centos:9876
brokerIP1=192.168.56.101
brokerName=broker-b
brokerClusterName=mycluster
brokerId=0
listenPort=1085
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
rejectTransactionMessage=false
fetchNamesrvAddrByAddressServer=false
storePathRootDir=/root/storeB
storePathCommitLog=/root/store/commitlog
flushIntervalCommitLog=500
commitIntervalCommitLog=200
flushCommitLogTimed=false
deleteWhen=04
fileReservedTime=48
maxTransferBytesOnMessageInMemory=262144
maxTransferCountOnMessageInMemory=32
maxTransferBytesOnMessageInDisk=65536
maxTransferCountOnMessageInDisk=8
accessMessageInMemoryMaxRatio=40
messageIndexEnable=true
messageIndexSafe=false
haMasterAddress=
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
cleanFileForciblyEnable=true
transientStorePoolEnable=false
#broker b-s
namesrvAddr=centos:9876
brokerIP1=192.168.56.102
brokerName=broker-b
brokerClusterName=mycluster
brokerId=1
listenPort=1095
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
rejectTransactionMessage=false
fetchNamesrvAddrByAddressServer=false
storePathRootDir=/root/storeBs
storePathCommitLog=/root/store/commitlog
flushIntervalCommitLog=500
commitIntervalCommitLog=200
flushCommitLogTimed=false
deleteWhen=04
fileReservedTime=48
maxTransferBytesOnMessageInMemory=262144
maxTransferCountOnMessageInMemory=32
maxTransferBytesOnMessageInDisk=65536
maxTransferCountOnMessageInDisk=8
accessMessageInMemoryMaxRatio=40
messageIndexEnable=true
messageIndexSafe=false
haMasterAddress=
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
cleanFileForciblyEnable=true
transientStorePoolEnable=false

 

rocketmq sdk使用newFixThreadPool线程池用于获取server中的数据(线程数获取的cpu核心数),拿到消息后,submit 给->ThreadPoolExecutor线程池进行执行消费,执行消费可以指定最大线程数和最小线程数。

rocketmq多消费端是需要通过tag指定的,如果你想和使用rabbitmq一样使用,可能会疯,百度搜出来的鸡八狗屁文章,根本狗屁不通。

 

rocketmq 存储结构:

IO调度方式: