最近阅读了Rocketmq关于pullmessage的实现方式,分享出来
众所周知,Rocketmq在consumer端是拉取消息的方式,它会在客户端维护一个PullRequestQueue,这个是一个阻塞队列(LinkedBlockingQueue),内部的节点是PullRequest,每一个PullRequest代表了一个消费的分组单元
PullRequest会记录一个topic对应的consumerGroup的拉取进度,包括MessageQueue和PorcessQueue,还有拉取的offset
(代码片段一)
public class PullRequest {
private String consumerGroup;
private MessageQueue messageQueue;
private ProcessQueue processQueue;
private long nextOffset;
private boolean lockedFirst = false;
}
其中MessageQueue记录元信息:
(代码片段二)
public class MessageQueue implements Comparable<MessageQueue>, Serializable {
private String topic;
private String brokerName;
private int queueId;
}
PorcessQueue记录一次拉取之后实际消息体和拉取相关操作记录的快照
(代码片段三)
public class ProcessQueue {
private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock();
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();
private final AtomicLong msgCount = new AtomicLong();
private final AtomicLong msgSize = new AtomicLong();
private final Lock lockConsume = new ReentrantLock();
private final TreeMap<Long, MessageExt> consumingMsgOrderlyTreeMap = new TreeMap<Long, MessageExt>();
private final AtomicLong tryUnlockTimes = new AtomicLong(0);
private volatile long queueOffsetMax = 0L;
private volatile boolean dropped = false;
private volatile long lastPullTimestamp = System.currentTimeMillis();
private volatile long lastConsumeTimestamp = System.currentTimeMillis();
private volatile boolean locked = false;
private volatile long lastLockTimestamp = System.currentTimeMillis();
private volatile boolean consuming = false;
private volatile long msgAccCnt = 0;
}
PullMessageService负责轮询PullRequestQueue,并进行消息元的拉取
(代码片段四)
public class PullMessageService extends ServiceThread {
private final InternalLogger log = ClientLogger.getLog();
private final LinkedBlockingQueue<PullRequest> pullRequestQueue = new LinkedBlockingQueue<PullRequest>();
private final MQClientInstance mQClientFactory;
@Override
public void run() {
while (!this.isStopped()) {
try {
PullRequest pullRequest = this.pullRequestQueue.take();
this.pullMessage(pullRequest);
} catch (InterruptedException ignored) {
} catch (Exception e) {
log.error("Pull Message Service Run Method exception", e);
}
}
}
}
在发送的时候会维护一个PullCallback,这是拉取收到消息后的回调处理
(代码片段五)
public interface PullCallback {
void onSuccess(final PullResult pullResult);
void onException(final Throwable e);
}
这里的实现逻辑就不贴了,本质上就是把消息丢给消费线程池来处理
pullMessage分为同步拉取和异步拉取两种模式,先解读异步拉取,然后再解读同步拉取,再说明两者的区别
其实从这里已经大概可以看出来,异步的方式,这个方法返回值是null,同步的方式必须要返回PullResult,后续说明区别
(代码片段六)
public PullResult pullMessage(
final String addr,
final PullMessageRequestHeader requestHeader,
final long timeoutMillis,
final CommunicationMode communicationMode,
final PullCallback pullCallback
) throws RemotingException, MQBrokerException, InterruptedException {
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.PULL_MESSAGE, requestHeader);
switch (communicationMode) {
case ONEWAY:
assert false;
return null;
case ASYNC:
this.pullMessageAsync(addr, request, timeoutMillis, pullCallback);
return null;
case SYNC:
return this.pullMessageSync(addr, request, timeoutMillis);
default:
assert false;
break;
}
return null;
}
先介绍异步拉取
可以看到,把PullCallback传进去,并封装了InvokeCallback,
(代码片段七)
private void pullMessageAsync(
final String addr,
final RemotingCommand request,
final long timeoutMillis,
final PullCallback pullCallback
) throws RemotingException, InterruptedException {
this.remotingClient.invokeAsync(addr, request, timeoutMillis, new InvokeCallback() {
@Override
public void operationComplete(ResponseFuture responseFuture) {
RemotingCommand response = responseFuture.getResponseCommand();
if (response != null) {
try {
PullResult pullResult = MQClientAPIImpl.this.processPullResponse(response);
assert pullResult != null;
pullCallback.onSuccess(pullResult);
} catch (Exception e) {
pullCallback.onException(e);
}
} else {
if (!responseFuture.isSendRequestOK()) {
pullCallback.onException(new MQClientException("send request failed to " + addr + ". Request: " + request, responseFuture.getCause()));
} else if (responseFuture.isTimeout()) {
pullCallback.onException(new MQClientException("wait response from " + addr + " timeout :" + responseFuture.getTimeoutMillis() + "ms" + ". Request: " + request,
responseFuture.getCause()));
} else {
pullCallback.onException(new MQClientException("unknown reason. addr: " + addr + ", timeoutMillis: " + timeoutMillis + ". Request: " + request, responseFuture.getCause()));
}
}
}
});
}
接下来进入NettyRemotingAbstract这个类中,使用netty的Chanle发送
(代码片段八)
public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
final InvokeCallback invokeCallback)
throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
final int opaque = request.getOpaque();
boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
if (acquired) {
final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, invokeCallback, once);
this.responseTable.put(opaque, responseFuture);
try {
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
responseFuture.setSendRequestOK(true);
return;
} else {
responseFuture.setSendRequestOK(false);
}
responseFuture.putResponse(null);
responseTable.remove(opaque);
try {
executeInvokeCallback(responseFuture);
} catch (Throwable e) {
log.warn("excute callback in writeAndFlush addListener, and callback throw", e);
} finally {
responseFuture.release();
}
log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
}
});
} catch (Exception e) {
responseFuture.release();
log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
}
} else {
if (timeoutMillis <= 0) {
throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
} else {
String info =
String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
timeoutMillis,
this.semaphoreAsync.getQueueLength(),
this.semaphoreAsync.availablePermits()
);
log.warn(info);
throw new RemotingTimeoutException(info);
}
}
}
这里需要详细解读一下:
RemotingCommand是request和response的载体,先从request中取出opaque,这是一个自增的操作id,然后将传进来的opaque和invokeCallback封装成一个ResponseFuture,再put到一个叫
responseTable的map中,这个map是一个核心的map,维护着opaque和对应的ResponseFuture
/**
* This map caches all on-going requests.
*/
protected final ConcurrentMap<Integer /* opaque */, ResponseFuture> responseTable =
new ConcurrentHashMap<Integer, ResponseFuture>(256);
从注释中可以看出,它缓存着正在执行的request
再回到刚刚的(代码片段八)中,channel.writeAndFlush(request).addListener(new ChannelFutureListener(){...}),netty在writeAndFlush发送完之后会回调我们ChannelFutureListener的operationComplete方法:如果发送成功则responseFuture.setSendRequestOK(true); 并且就return了;如果发送失败,则从responseTable中移除,并且起一个异步线程执行responseFuture中的InvokeCallback,在(代码片段七)中,可以看到当responseFuture.isSendRequestOK()是false的时候,执行了onException,这里就不多介绍了。
那么此时发送的逻辑就全部结束了,整个过程没有任何的阻塞,当Broker收到拉取请求后,会按照queueOffset等信息封装好返回consumer端,
会经过NettyRemotingServer上注册的NettyServerHandler
(代码片段九)
class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> {
@Override
protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
processTunnelId(ctx, msg);
processMessageReceived(ctx, msg);
}
public void processTunnelId(ChannelHandlerContext ctx, RemotingCommand msg) {
if (nettyServerConfig.isValidateTunnelIdFromVtoaEnable()) {
if (null != msg && msg.getType() == RemotingCommandType.REQUEST_COMMAND) {
Vtoa vtoa = tunnelTable.get(ctx.channel());
if (null == vtoa) {
vtoa = VpcTunnelUtils.getInstance().getTunnelID(ctx);
tunnelTable.put(ctx.channel(), vtoa);
}
msg.addExtField(VpcTunnelUtils.PROPERTY_VTOA_TUNNEL_ID, String.valueOf(vtoa.getVid()));
}
}
}
}
最终会调用到NettyRemotingAbstract的processResponseCommand,RemotingCommand中根据opaque从responseTable中获取ResponseFuture,然后同样也是执行callback,这样,就实现了整个pullmessage的异步模式
(代码片段十)
public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
final int opaque = cmd.getOpaque();
final ResponseFuture responseFuture = responseTable.get(opaque);
if (responseFuture != null) {
responseFuture.setResponseCommand(cmd);
responseTable.remove(opaque);
if (responseFuture.getInvokeCallback() != null) { //异步
executeInvokeCallback(responseFuture); //执行回调
} else { //同步
responseFuture.putResponse(cmd); //为了解除阻塞
responseFuture.release();
}
} else {
log.warn("receive response, but not matched any request, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
log.warn(cmd.toString());
}
}
我们再看下同步的方式是如何实现的
回顾下代码片段六,同步的方式是需要返回PullResult的,换句话说,这种方式是需要在发送的线程中来处理返回结果的
我们从代码片段六跟下去,跟到NettyRemotingAbstract的invokeSyncImpl
(代码片段十一)
public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
final long timeoutMillis)
throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
final int opaque = request.getOpaque();
try {
final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, null, null);
this.responseTable.put(opaque, responseFuture);
final SocketAddress addr = channel.remoteAddress();
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
responseFuture.setSendRequestOK(true);
return;
} else {
responseFuture.setSendRequestOK(false);
}
responseTable.remove(opaque);
responseFuture.setCause(f.cause());
responseFuture.putResponse(null);
log.warn("send a request command to channel <" + addr + "> failed.");
}
});
RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
if (null == responseCommand) {
if (responseFuture.isSendRequestOK()) {
throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
responseFuture.getCause());
} else {
throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
}
}
return responseCommand;
} finally {
this.responseTable.remove(opaque);
}
}
和异步发送的代码片段八对比一下,可以看到,同步方式也要放到responseTable中,这里就有个疑惑了,既然都同步了,还要放到responseTable中干什么呢,继续往下看,
ChannelFutureListener都是一样的,如果发送成功就返回了,然后到了最关键的一行:
RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
这一行顾名思义就是阻塞了,但是也不能一直阻塞住(因为PullMessageService是单线程的,如果因为一个异常就阻塞那就跪了),所以是一个设置了超时时间的阻塞,看下是如何阻塞的
ResponseFuture中有这两个方法,当putResponse的时候,把RemotingCommand赋值,并且countDownLatch.countDown,而在waitResponse的时候countDownLatch.await
(代码片段十二)
public RemotingCommand waitResponse(final long timeoutMillis) throws InterruptedException {
this.countDownLatch.await(timeoutMillis, TimeUnit.MILLISECONDS);
return this.responseCommand;
}
public void putResponse(final RemotingCommand responseCommand) {
this.responseCommand = responseCommand;
this.countDownLatch.countDown();
}
这样就清晰多了,剩下的疑问就是,在什么时候putResponse的,有两个地方:
第一个地方,当拉取消息回来的时候,回顾下代码片段十,有一句是(responseFuture.getInvokeCallback() != null),通过刚刚的流程已经知道,只有异步的时候invokeCallback才不为null,因此走到else,看到在这个时候responseFuture.putResponse(cmd)和responseFuture.release(),也就是说同步方式也是通过responseTable存储的方式,来获取结果,并且通过CountDownLatch来阻塞发送的线程,当收到消息之后再countDown,发送端最终返回PullResult来处理消息
第二个地方,回顾下代码片段十一,在ChannelFutureListener中当发送失败了以后,也会put一个null值:responseFuture.putResponse(null),这里只是为了将阻塞放开
至此,Rockmq关于pullmessage的同步和异步方式就已经说明白了,总结一下,同步和异步本质上都是“异步”的,因为netty就是一个异步的框架,Rockmq只是利用了CountDownLatch来阻塞住发送端线程来实现了“同步”的效果,
通过一个responseTable来缓存住发送出去的请求,等收到的时候从这个缓存里按对应关系取出来,再去做对应的consumer线程的消息处理