iobuf.md 3.13 KB
Newer Older
gejun's avatar
gejun committed
1
[中文版](../cn/iobuf.md)
2

gejun's avatar
gejun committed
3
brpc uses [butil::IOBuf](https://github.com/brpc/brpc/blob/master/src/butil/iobuf.h) as data structure for attachment in some protocols and HTTP body. It's a non-contiguous zero-copied buffer, proved in previous projects, and good at performance. The interface of `IOBuf` is similar to `std::string`, but not the same.
4

gejun's avatar
gejun committed
5
If you've used the `BufHandle` in Kylin before, you should notice the convenience of `IOBuf`: the former one is badly encapsulated, leaving the internal structure directly in front of users, who must carefully handle the referential countings, very error prone and leading to bugs.
6

gejun's avatar
gejun committed
7
# What IOBuf can:
8

gejun's avatar
gejun committed
9 10 11 12 13 14 15
- Default constructor does not allocate memory.
- Copyable. Modifications to the copy doesn't affect the original one. Copy the managing structure of IOBuf only rather the payload.
- Append another IOBuf without copying payload.
- Can append string, by copying payload.
- Read from or write into file descriptors.
- Serialize to or parse from protobuf messages.
- constructible like a std::ostream using IOBufBuilder.
16

gejun's avatar
gejun committed
17
# What IOBuf can't:
18

gejun's avatar
gejun committed
19
- Used as universal string-like structure in the program. Lifetime of IOBuf should be short, to prevent the referentially counted blocks(8K each) in IOBuf lock too many memory.
20

gejun's avatar
gejun committed
21 22 23
# Cut

Cut 16 bytes from front-side of source_buf and append to dest_buf:
24 25

```c++
gejun's avatar
gejun committed
26
source_buf.cut(&dest_buf, 16); // cut all bytes of source_buf when its length < 16
27 28
```

gejun's avatar
gejun committed
29
Just pop 16 bytes from front-side of source_buf:
30 31 32 33 34

```c++
source_buf.pop_front(16); // Empty source_buf when its length < 16
```

gejun's avatar
gejun committed
35
# Append
36

gejun's avatar
gejun committed
37
Append another IOBuf to back-side:
38 39 40 41 42

```c++
buf.append(another_buf);  // no data copy
```

gejun's avatar
gejun committed
43
Append std::string to back-sie
44 45 46 47 48 49 50

```c++
buf.append(str);  // copy data of str into buf
```

# Parse

gejun's avatar
gejun committed
51
Parse a protobuf message from the IOBuf 
52 53 54 55 56 57

```c++
IOBufAsZeroCopyInputStream wrapper(&iobuf);
pb_message.ParseFromZeroCopyStream(&wrapper);
```

gejun's avatar
gejun committed
58
Parse IOBuf in user-defined formats
59 60 61 62 63 64 65 66 67 68

```c++
IOBufAsZeroCopyInputStream wrapper(&iobuf);
CodedInputStream coded_stream(&wrapper);
coded_stream.ReadLittleEndian32(&value);
...
```

# Serialize

gejun's avatar
gejun committed
69
Serialize a protobuf message into the IOBuf
70 71 72 73 74 75

```c++
IOBufAsZeroCopyOutputStream wrapper(&iobuf);
pb_message.SerializeToZeroCopyStream(&wrapper);
```

gejun's avatar
gejun committed
76
Built IOBuf with printable data
77 78 79 80 81 82 83 84 85

```c++
IOBufBuilder os;
os << "anything can be sent to std::ostream";
os.buf();  // IOBuf
```

# Print

gejun's avatar
gejun committed
86 87
Directly printable to std::ostream. Note that the iobuf in following example should only contain printable characters.

88
```c++
gejun's avatar
gejun committed
89 90 91 92
std::cout << iobuf << std::endl;
// or
std::string str = iobuf.to_string(); // note: allocating memory
printf("%s\n", str.c_str());
93 94 95 96
```

# Performance

gejun's avatar
gejun committed
97
IOBuf is good at performance:
98 99 100

| Action                                   | Throughput  | QPS     |
| ---------------------------------------- | ----------- | ------- |
gejun's avatar
gejun committed
101 102 103
| Read from file -> Cut 12+16 bytes -> Copy -> Merge into another buffer ->Write to /dev/null | 240.423MB/s | 8586535 |
| Read from file -> Cut 12+128 bytes -> Copy-> Merge into another buffer ->Write to /dev/null | 790.022MB/s | 5643014 |
| Read from file -> Cut 12+1024 bytes -> Copy-> Merge into another buffer ->Write to /dev/null | 1519.99MB/s | 1467171 |