Commit c462f86a authored by gejun's avatar gejun

reviewed iobuf.md

parent 65d05389
brpc使用[butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobuf.h)作为存储附件或http body的数据结构,它是一种非连续零拷贝缓冲,在其他项目中得到了验证并有出色的性能。IOBuf的接口和std::string类似,但不相同。 [English version](../en/iobuf.md)
如果你之前使用Kylin中的BufHandle,你将更能感受到IOBuf的便利性:前者几乎没有实现完整,直接暴露了内部结构,用户得小心翼翼地处理引用计数,极易出错。BufHandle是很多bug的诱因。 brpc使用[butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobuf.h)作为一些协议中的附件或http body的数据结构,它是一种非连续零拷贝缓冲,在其他项目中得到了验证并有出色的性能。IOBuf的接口和std::string类似,但不相同。
如果你之前使用Kylin中的BufHandle,你将更能感受到IOBuf的便利性:前者几乎没有实现完整,直接暴露了内部结构,用户得小心翼翼地处理引用计数,极易出错。
# IOBuf能做的: # IOBuf能做的:
...@@ -9,7 +11,7 @@ brpc使用[butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobu ...@@ -9,7 +11,7 @@ brpc使用[butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobu
- 可以append另一个IOBuf,不拷贝数据。 - 可以append另一个IOBuf,不拷贝数据。
- 可以append字符串,拷贝数据。 - 可以append字符串,拷贝数据。
- 可以从fd读取,可以写入fd。 - 可以从fd读取,可以写入fd。
- 可以和protobuf相互转化。 - 可以解析或序列化为protobuf messages.
- IOBufBuilder可以把IOBuf当std::ostream用。 - IOBufBuilder可以把IOBuf当std::ostream用。
# IOBuf不能做的: # IOBuf不能做的:
...@@ -18,13 +20,13 @@ brpc使用[butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobu ...@@ -18,13 +20,13 @@ brpc使用[butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobu
# 切割 # 切割
切下16字节的IOBuf: 从source_buf头部切下16字节放入dest_buf:
```c++ ```c++
source_buf.cut(&heading_iobuf, 16); // 当source_buf不足16字节时,切掉所有字节。 source_buf.cut(&dest_buf, 16); // 当source_buf不足16字节时,切掉所有字节。
``` ```
跳过16字节: 从source_buf头部弹掉16字节:
```c++ ```c++
source_buf.pop_front(16); // 当source_buf不足16字节时,清空它 source_buf.pop_front(16); // 当source_buf不足16字节时,清空它
...@@ -32,13 +34,13 @@ source_buf.pop_front(16); // 当source_buf不足16字节时,清空它 ...@@ -32,13 +34,13 @@ source_buf.pop_front(16); // 当source_buf不足16字节时,清空它
# 拼接 # 拼接
append另一个IOBuf: 在尾部加入另一个IOBuf:
```c++ ```c++
buf.append(another_buf); // no data copy buf.append(another_buf); // no data copy
``` ```
append std::string 在尾部加入std::string
```c++ ```c++
buf.append(str); // copy data of str into buf buf.append(str); // copy data of str into buf
...@@ -46,7 +48,7 @@ buf.append(str); // copy data of str into buf ...@@ -46,7 +48,7 @@ buf.append(str); // copy data of str into buf
# 解析 # 解析
解析IOBuf为protobuf 解析IOBuf为protobuf message
```c++ ```c++
IOBufAsZeroCopyInputStream wrapper(&iobuf); IOBufAsZeroCopyInputStream wrapper(&iobuf);
...@@ -71,7 +73,7 @@ IOBufAsZeroCopyOutputStream wrapper(&iobuf); ...@@ -71,7 +73,7 @@ IOBufAsZeroCopyOutputStream wrapper(&iobuf);
pb_message.SerializeToZeroCopyStream(&wrapper); pb_message.SerializeToZeroCopyStream(&wrapper);
``` ```
把可打印数据送入IOBuf 用可打印数据创建IOBuf
```c++ ```c++
IOBufBuilder os; IOBufBuilder os;
...@@ -81,15 +83,21 @@ os.buf(); // IOBuf ...@@ -81,15 +83,21 @@ os.buf(); // IOBuf
# 打印 # 打印
可直接打印至std::ostream. 注意这个例子中的iobuf必需只包含可打印字符。
```c++ ```c++
std::cout << iobuf; std::cout << iobuf << std::endl;
std::string str = iobuf.to_string(); // or
std::string str = iobuf.to_string(); // 注意: 会分配内存
printf("%s\n", str.c_str());
``` ```
# 性能 # 性能
IOBuf有很高的综合性能: IOBuf有不错的综合性能:
- 从文件读入->切割12+16字节->拷贝->合并到另一个缓冲->写出到/dev/null这一流程的吞吐是240.423MB/s或8586535次/s | 动作 | 吞吐 | QPS |
- 从文件读入->切割12+128字节->拷贝->合并到另一个缓冲->写出到/dev/null这一流程的吞吐是790.022MB/s或5643014次/s | ---------------------------------------- | ----------- | ------- |
- 从文件读入->切割12+1024字节->拷贝->合并到另一个缓冲->写出到/dev/null这一流程的吞吐是1519.99MB/s或1467171次/s | 文件读入->切割12+16字节->拷贝->合并到另一个缓冲->写出到/dev/null | 240.423MB/s | 8586535 |
| 文件读入->切割12+128字节->拷贝->合并到另一个缓冲->写出到/dev/null | 790.022MB/s | 5643014 |
| 文件读入->切割12+1024字节->拷贝->合并到另一个缓冲->写出到/dev/null | 1519.99MB/s | 1467171 |
brpc uses [butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobuf.h) as data structure for attachment storage and HTTP body. It is a non-contiguous zero copy buffer, which has been proved in other projects as excellent performance. The interface of `IOBuf` is similar to `std::string`, but not the same. [中文版](../cn/iobuf.md)
If you used the `BufHandle` in Kylin before, you should notice the difference in convenience of `IOBuf`: the former hardly had any encapsulation, leaving the internal structure directly in front of the user. The user must carefully handle the reference count, which is very error prone, leading to lots of bugs. brpc uses [butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobuf.h) as data structure for attachment in some protocols and HTTP body. It's a non-contiguous zero-copied buffer, proved in previous projects, and good at performance. The interface of `IOBuf` is similar to `std::string`, but not the same.
# What IOBuf can: If you've used the `BufHandle` in Kylin before, you should notice the convenience of `IOBuf`: the former one is badly encapsulated, leaving the internal structure directly in front of users, who must carefully handle the referential countings, very error prone and leading to bugs.
- Default constructor doesn't involve copying. # What IOBuf can:
- Explicit copy doesn't change source IOBuf. Only copy the management structure of IOBuf instead of the data.
- Append another IOBuf without copy.
- Append string involves copy.
- Read from/Write into fd.
- Convert to protobuf and vice versa.
- IOBufBuilder可以把IOBuf当std::ostream用。
# What IOBuf can't: - Default constructor does not allocate memory.
- Copyable. Modifications to the copy doesn't affect the original one. Copy the managing structure of IOBuf only rather the payload.
- Append another IOBuf without copying payload.
- Can append string, by copying payload.
- Read from or write into file descriptors.
- Serialize to or parse from protobuf messages.
- constructible like a std::ostream using IOBufBuilder.
- Used as general storage structure. IOBuf should not keep a long life cycle to prevent multiple memory blocks (8K each) being locked by one IOBuf object. # What IOBuf can't:
# Slice - Used as universal string-like structure in the program. Lifetime of IOBuf should be short, to prevent the referentially counted blocks(8K each) in IOBuf lock too many memory.
Slice 16 bytes from IOBuf: # Cut
Cut 16 bytes from front-side of source_buf and append to dest_buf:
```c++ ```c++
source_buf.cut(&heading_iobuf, 16); // cut all bytes of source_buf when its length < 16 source_buf.cut(&dest_buf, 16); // cut all bytes of source_buf when its length < 16
``` ```
Remove 16 bytes: Just pop 16 bytes from front-side of source_buf:
```c++ ```c++
source_buf.pop_front(16); // Empty source_buf when its length < 16 source_buf.pop_front(16); // Empty source_buf when its length < 16
``` ```
# Concatenate # Append
Append to another IOBuf Append another IOBuf to back-side
```c++ ```c++
buf.append(another_buf); // no data copy buf.append(another_buf); // no data copy
``` ```
Append std::string Append std::string to back-sie
```c++ ```c++
buf.append(str); // copy data of str into buf buf.append(str); // copy data of str into buf
...@@ -46,14 +48,14 @@ buf.append(str); // copy data of str into buf ...@@ -46,14 +48,14 @@ buf.append(str); // copy data of str into buf
# Parse # Parse
Parse protobuf from IOBuf Parse a protobuf message from the IOBuf
```c++ ```c++
IOBufAsZeroCopyInputStream wrapper(&iobuf); IOBufAsZeroCopyInputStream wrapper(&iobuf);
pb_message.ParseFromZeroCopyStream(&wrapper); pb_message.ParseFromZeroCopyStream(&wrapper);
``` ```
Parse IOBuf as user-defined structure Parse IOBuf in user-defined formats
```c++ ```c++
IOBufAsZeroCopyInputStream wrapper(&iobuf); IOBufAsZeroCopyInputStream wrapper(&iobuf);
...@@ -64,14 +66,14 @@ coded_stream.ReadLittleEndian32(&value); ...@@ -64,14 +66,14 @@ coded_stream.ReadLittleEndian32(&value);
# Serialize # Serialize
Serialize protobuf into IOBuf Serialize a protobuf message into the IOBuf
```c++ ```c++
IOBufAsZeroCopyOutputStream wrapper(&iobuf); IOBufAsZeroCopyOutputStream wrapper(&iobuf);
pb_message.SerializeToZeroCopyStream(&wrapper); pb_message.SerializeToZeroCopyStream(&wrapper);
``` ```
Append printable data into IOBuf Built IOBuf with printable data
```c++ ```c++
IOBufBuilder os; IOBufBuilder os;
...@@ -81,17 +83,21 @@ os.buf(); // IOBuf ...@@ -81,17 +83,21 @@ os.buf(); // IOBuf
# Print # Print
Directly printable to std::ostream. Note that the iobuf in following example should only contain printable characters.
```c++ ```c++
std::cout << iobuf; std::cout << iobuf << std::endl;
std::string str = iobuf.to_string(); // or
std::string str = iobuf.to_string(); // note: allocating memory
printf("%s\n", str.c_str());
``` ```
# Performance # Performance
IOBuf has excellent performance in general aspects: IOBuf is good at performance:
| Action | Throughput | QPS | | Action | Throughput | QPS |
| ---------------------------------------- | ----------- | ------- | | ---------------------------------------- | ----------- | ------- |
| Read from file -> Slice 12+16 bytes -> Copy -> Merge into another buffer ->Write to /dev/null | 240.423MB/s | 8586535 | | Read from file -> Cut 12+16 bytes -> Copy -> Merge into another buffer ->Write to /dev/null | 240.423MB/s | 8586535 |
| Read from file -> Slice 12+128 bytes -> Copy-> Merge into another buffer ->Write to /dev/null | 790.022MB/s | 5643014 | | Read from file -> Cut 12+128 bytes -> Copy-> Merge into another buffer ->Write to /dev/null | 790.022MB/s | 5643014 |
| Read from file -> Slice 12+1024 bytes -> Copy-> Merge into another buffer ->Write to /dev/null | 1519.99MB/s | 1467171 | | Read from file -> Cut 12+1024 bytes -> Copy-> Merge into another buffer ->Write to /dev/null | 1519.99MB/s | 1467171 |
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment