Netty 数据流处理 - write

Netty Data Stream Handling - write

上篇文章中介绍了Netty是读数据的流程:EventLoop不停的select IO;一旦发现OP_READ可用则利用Channel.Unsafe读取数据,并把数据传给Pipeline;Pipeline拿到数据,并交由内部的Handler处理。还有上篇文章也说了Pipeline中的两种数据流向:inbound和outbound。read操作符合inbound流向;而write则符合outbound刘翔。这些知识对我们理解write操作大有裨益,因为write操作在Pipeline中是read的反向操作。

那我们开始介绍Netty的write写数据流程。先找到EchoServerHandler的channelRead方法,在这个示例中,它会把读到的数据再写回客户端:

public void channelRead(ChannelHandlerContext ctx, Object msg) {
    ctx.write(msg);
}

这个方法直接调用了ChannelHandlerContext的write方法:

@Override
public ChannelFuture write(Object msg) {
    return write(msg, newPromise());
}

没什么好解释的,给出write的调用链:

@Override
public ChannelFuture write(final Object msg, final ChannelPromise promise) {
    if (msg == null) {
        throw new NullPointerException("msg");
    }

    try {
        if (isNotValidPromise(promise, true)) {
            ReferenceCountUtil.release(msg);
            // cancelled
            return promise;
        }
    } catch (RuntimeException e) {
        ReferenceCountUtil.release(msg);
        throw e;
    }
    write(msg, false, promise);

    return promise;
}

private void write(Object msg, boolean flush, ChannelPromise promise) {
    AbstractChannelHandlerContext next = findContextOutbound();
    final Object m = pipeline.touch(msg, next);
    EventExecutor executor = next.executor();
    if (executor.inEventLoop()) {
        if (flush) {
            next.invokeWriteAndFlush(m, promise);
        } else {
            next.invokeWrite(m, promise);
        }
    } else {
        AbstractWriteTask task;
        if (flush) {
            task = WriteAndFlushTask.newInstance(next, m, promise);
        }  else {
            task = WriteTask.newInstance(next, m, promise);
        }
        safeExecute(executor, task, promise, m);
    }
}

private void invokeWrite(Object msg, ChannelPromise promise) {
    if (invokeHandler()) {
        invokeWrite0(msg, promise);
    } else {
        write(msg, promise);
    }
}

private void invokeWrite0(Object msg, ChannelPromise promise) {
    try {
        ((ChannelOutboundHandler) handler()).write(this, msg, promise);
    } catch (Throwable t) {
        notifyOutboundHandlerException(t, promise);
    }
}

invokeWrite0接下来会调用HeadContext中的write方法(没什么好解释的了,read的反向操作):

@Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
    unsafe.write(msg, promise);
}

这个方法调用了Unsafe的write方法,至此write操作从Pipeline中走完了,接下来才是重头戏。我们来看这个AbstractUnsafe实现的write方法:

@Override
public final void write(Object msg, ChannelPromise promise) {
    assertEventLoop();

    // 1
    ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
    if (outboundBuffer == null) {
        safeSetFailure(promise, WRITE_CLOSED_CHANNEL_EXCEPTION);
        ReferenceCountUtil.release(msg);
        return;
    }

    int size;
    try {
        // 2
        msg = filterOutboundMessage(msg);
        size = pipeline.estimatorHandle().size(msg);
        if (size < 0) {
            size = 0;
        }
    } catch (Throwable t) {
        safeSetFailure(promise, t);
        ReferenceCountUtil.release(msg);
        return;
    }

    // 3
    outboundBuffer.addMessage(msg, size, promise);
}

直接调用write写数据的时候,并不是直接写到channel中,而是先写到缓冲区里,也就是ChannelOutboundBuffer。当调用调用flushflush才开始向channel写数据。ChannelOutboundBuffer是一个无界链表,如果不停的向缓冲区写入数据可能会导致内存溢出。因此ChannelOutboundBuffer检测当newWriteBufferSize > channel.config().getWriteBufferHighWaterMark()时,buffer便不可写。同时通知Pipeline Writablity Changed:pipeline.fireChannelWritabilityChanged();。我们在实现自己的Handler时可以重新实现该方法。

我们接着看下filterOutboundMessage这个方法:

@Override
    protected final Object filterOutboundMessage(Object msg) {
        if (msg instanceof ByteBuf) {
            ByteBuf buf = (ByteBuf) msg;
            if (buf.isDirect()) {
                return msg;
            }

            return newDirectBuffer(buf);
        }

        if (msg instanceof FileRegion) {
            return msg;
        }

        throw new UnsupportedOperationException(
                "unsupported message type: " + StringUtil.simpleClassName(msg) + EXPECTED_TYPES);
    }

在讲解ByteBuf的文章中提到过关于使用直接内存还是堆内存的最佳实践:在IO通信的线程中操作ByteBuf应使用DirectBuffer(省去内存拷贝的成本),在后端业务逻辑中操作ByteBuf应使用HeapBuffer(不用担心内存泄露)

这个方法首先检查msg类型是不是ByteBuf,如果是ByteBuf则检查是不是使用了直接内存,如果没有则把基于堆的内存换成直接内存。

我们接着分析ChannelOutboundBuffer的addMessage方法:

public void addMessage(Object msg, int size, ChannelPromise promise) {
    Entry entry = Entry.newInstance(msg, size, total(msg), promise);
    if (tailEntry == null) {
        flushedEntry = null;
        tailEntry = entry;
    } else {
        Entry tail = tailEntry;
        tail.next = entry;
        tailEntry = entry;
    }
    if (unflushedEntry == null) {
        unflushedEntry = entry;
    }

    incrementPendingOutboundBytes(entry.pendingSize, false);
}

这个方法做了两件事情:把msg加入到链表中;调用incrementPendingOutboundBytes方法。我们来看这个方法:

private void incrementPendingOutboundBytes(long size, boolean invokeLater) {
    if (size == 0) {
        return;
    }

    long newWriteBufferSize = TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, size);

    if (newWriteBufferSize > channel.config().getWriteBufferHighWaterMark()) {
        setUnwritable(invokeLater);
    }
}

首先,先更新缓冲区大小,接着判断缓冲区是否大于最高水位,如果是则设置buffer为unWritable(默认的高水位线大小是64K)。setUnwritable方法内通知了Pipeline Writablity Changed,这里不贴代码了自己看去吧。

到这的话write方法是执行完了,但是数据仍然留在内存中。接下来我们看flush方法。跳过Pipeline直接来看AbstractUnsafe的flush:

@Override
public final void flush() {
    assertEventLoop();

    ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
    if (outboundBuffer == null) {
        return;
    }

    outboundBuffer.addFlush();
    flush0();
}

这个方法首先调用了outboundBuffer的addFlush方法,我们先看下addFlush做了什么:

public void addFlush() {
    Entry entry = unflushedEntry;
    if (entry != null) {
        if (flushedEntry == null) {
            // there is no flushedEntry yet, so start with the entry
            flushedEntry = entry;
        }
        do {
            flushed ++;
            if (!entry.promise.setUncancellable()) {
                int pending = entry.cancel();
                decrementPendingOutboundBytes(pending, false, true);
            }
            entry = entry.next;
        } while (entry != null);

        unflushedEntry = null;
    }
}

unflushedEntry表示缓冲区链表中第一个未被flush的元素。如果这个变量为null的话,表示当前缓冲区已被flush。for循环中首先把entry的promise设置为Uncancellable(Promise继承自Future,Future是可以cancel的),然后增加flushed计数。decrementPendingOutboundBytes方法中有逻辑检查目前Buffer是否低于低水位,如果是则重置Buffer为可写。

我们接着看flush0方法,flush0主要是调用NioSocketChannel的doWrite方法:

protected void flush0() {
    ///...
    doWrite(outboundBuffer);
    ///...
}

protected void doWrite(ChannelOutboundBuffer in) throws Exception {
    for (;;) {
        int size = in.size();
        if (size == 0) {
            clearOpWrite();
            break;
        }
        long writtenBytes = 0;
        boolean done = false;
        boolean setOpWrite = false;

        // 1
        ByteBuffer[] nioBuffers = in.nioBuffers();
        int nioBufferCnt = in.nioBufferCount();
        long expectedWrittenBytes = in.nioBufferSize();
        SocketChannel ch = javaChannel();

        switch (nioBufferCnt) {
            case 0:
                super.doWrite(in);
                return;
            case 1:
                ByteBuffer nioBuffer = nioBuffers[0];
                for (int i = config().getWriteSpinCount() - 1; i >= 0; i --) {
                    // 2
                    final int localWrittenBytes = ch.write(nioBuffer);
                    if (localWrittenBytes == 0) {
                        setOpWrite = true;
                        break;
                    }
                    expectedWrittenBytes -= localWrittenBytes;
                    writtenBytes += localWrittenBytes;
                    if (expectedWrittenBytes == 0) {
                        done = true;
                        break;
                    }
                }
                break;
            default:
                ///...
        }

        in.removeBytes(writtenBytes);

        if (!done) {
            // Did not write all buffers completely.
            incompleteWrite(setOpWrite);
            break;
        }
    }
}

doWrite方法首先获取Buffer中缓存的消息,并将Netty ByteBuf转成Nio ByteBuffer;然后把数据写入Nio SocketChannel。写数据的操作放到了一个for循环中。因为Netty的一个缓冲数据可能不会一次性的刷到Channel中,如果只作一次write操作就返回,那么很有可能余下的数据要等到下次OP_WRITE Selection Key可用才能全部写完。这期间经历了一次昂贵的IO操作,所以这个for循环又是CPU时间和IO时间的一个精心分配。

至此,Netty写数据的流程分析完毕。

推荐阅读更多精彩内容