stream.md 12.7 KB
Newer Older
miloyip's avatar
miloyip committed
1
# Stream
Milo Yip's avatar
Milo Yip committed
2

3
In RapidJSON, `rapidjson::Stream` is a concept for reading/writing JSON. Here we first show how to use streams provided. And then see how to create a custom stream.
Milo Yip's avatar
Milo Yip committed
4

5 6 7
[TOC]

# Memory Streams {#MemoryStreams}
Milo Yip's avatar
Milo Yip committed
8 9 10

Memory streams store JSON in memory.

11
## StringStream (Input) {#StringStream}
Milo Yip's avatar
Milo Yip committed
12 13 14

`StringStream` is the most basic input stream. It represents a complete, read-only JSON stored in memory. It is defined in `rapidjson/rapidjson.h`.

15
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
16 17 18 19 20 21 22 23 24 25
#include "rapidjson/document.h" // will include "rapidjson/rapidjson.h"

using namespace rapidjson;

// ...
const char json[] = "[1, 2, 3, 4]";
StringStream s(json);

Document d;
d.ParseStream(s);
26
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
27 28 29

Since this is very common usage, `Document::Parse(const char*)` is provided to do exactly the same as above:

30
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
31 32 33 34
// ...
const char json[] = "[1, 2, 3, 4]";
Document d;
d.Parse(json);
35
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
36 37 38

Note that, `StringStream` is a typedef of `GenericStringStream<UTF8<> >`, user may use another encodings to represent the character set of the stream.

39
## StringBuffer (Output) {#StringBuffer}
Milo Yip's avatar
Milo Yip committed
40 41 42

`StringBuffer` is a simple output stream. It allocates a memory buffer for writing the whole JSON. Use `GetString()` to obtain the buffer.

43
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
44 45 46 47 48 49 50
#include "rapidjson/stringbuffer.h"

StringBuffer buffer;
Writer<StringBuffer> writer(buffer);
d.Accept(writer);

const char* output = buffer.GetString();
51
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
52 53 54

When the buffer is full, it will increases the capacity automatically. The default capacity is 256 characters (256 bytes for UTF8, 512 bytes for UTF16, etc.). User can provide an allocator and a initial capacity.

55
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
56 57
StringBuffer buffer1(0, 1024); // Use its allocator, initial size = 1024
StringBuffer buffer2(allocator, 1024);
58
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
59 60 61 62 63

By default, `StringBuffer` will instantiate an internal allocator.

Similarly, `StringBuffer` is a typedef of `GenericStringBuffer<UTF8<> >`.

64
# File Streams {#FileStreams}
Milo Yip's avatar
Milo Yip committed
65 66 67 68 69

When parsing a JSON from file, you may read the whole JSON into memory and use ``StringStream`` above.

However, if the JSON is big, or memory is limited, you can use `FileReadStream`. It only read a part of JSON from file into buffer, and then let the part be parsed. If it runs out of characters in the buffer, it will read the next part from file.

70
## FileReadStream (Input) {#FileReadStream}
Milo Yip's avatar
Milo Yip committed
71 72 73

`FileReadStream` reads the file via a `FILE` pointer. And user need to provide a buffer.

74
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
75 76 77
#include "rapidjson/filereadstream.h"
#include <cstdio>

78 79
using namespace rapidjson;

Milo Yip's avatar
Milo Yip committed
80 81 82 83 84 85 86 87 88
FILE* fp = fopen("big.json", "rb"); // non-Windows use "r"

char readBuffer[65536];
FileReadStream is(fp, readBuffer, sizeof(readBuffer));

Document d;
d.ParseStream(is);

fclose(fp);
89
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
90 91 92 93 94

Different from string streams, `FileReadStream` is byte stream. It does not handle encodings. If the file is not UTF-8, the byte stream can be wrapped in a `EncodedInputStream`. It will be discussed very soon.

Apart from reading file, user can also use `FileReadStream` to read `stdin`.

95
## FileWriteStream (Output) {#FileWriteStream}
Milo Yip's avatar
Milo Yip committed
96 97 98

`FileWriteStream` is buffered output stream. Its usage is very similar to `FileReadStream`.

99
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
100 101 102
#include "rapidjson/filewritestream.h"
#include <cstdio>

103 104
using namespace rapidjson;

Milo Yip's avatar
Milo Yip committed
105 106 107 108 109 110 111 112 113 114 115 116 117
Document d;
d.Parse(json);
// ...

FILE* fp = fopen("output.json", "wb"); // non-Windows use "w"

char writeBuffer[65536];
FileWriteStream os(fp, writeBuffer, sizeof(writeBuffer));

Writer<FileWriteStream> writer(os);
d.Accept(writer);

fclose(fp);
118
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
119 120 121

It can also directs the output to `stdout`.

122
# Encoded Streams {#EncodedStreams}
Milo Yip's avatar
Milo Yip committed
123 124 125 126 127 128 129 130 131 132 133

Encoded streams do not contain JSON itself, but they wrap byte streams to provide basic encoding/decoding function.

As mentioned above, UTF-8 byte streams can be read directly. However, UTF-16 and UTF-32 have endian issue. To handle endian correctly, it needs to convert bytes into characters (e.g. `wchar_t` for UTF-16) while reading, and characters into bytes while writing.

Besides, it also need to handle [byte order mark (BOM)](http://en.wikipedia.org/wiki/Byte_order_mark). When reading from a byte stream, it is needed to detect or just consume the BOM if exists. When writing to a byte stream, it can optionally write BOM.

If the encoding of stream is known in compile-time, you may use `EncodedInputStream` and `EncodedOutputStream`. If the stream can be UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE JSON, and it is only known in runtime, you may use `AutoUTFInputStream` and `AutoUTFOutputStream`. These streams are defined in `rapidjson/encodedstream.h`.

Note that, these encoded streams can be applied to streams other than file. For example, you may have a file in memory, or a custom byte stream, be wrapped in encoded streams.

134
## EncodedInputStream {#EncodedInputStream}
Milo Yip's avatar
Milo Yip committed
135 136 137

`EncodedInputStream` has two template parameters. The first one is a `Encoding` class, such as `UTF8`, `UTF16LE`, defined in `rapidjson/encodings.h`. The second one is the class of stream to be wrapped.

138
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156
#include "rapidjson/document.h"
#include "rapidjson/filereadstream.h"   // FileReadStream
#include "rapidjson/encodedstream.h"    // EncodedInputStream
#include <cstdio>

using namespace rapidjson;

FILE* fp = fopen("utf16le.json", "rb"); // non-Windows use "r"

char readBuffer[256];
FileReadStream bis(fp, readBuffer, sizeof(readBuffer));

EncodedInputStream<UTF16LE<>, FileReadStream> eis(bis);  // wraps bis into eis

Document d; // Document is GenericDocument<UTF8<> > 
d.ParseStream<0, UTF16LE<> >(eis);  // Parses UTF-16LE file into UTF-8 in memory

fclose(fp);
157
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
158

159
## EncodedOutputStream {#EncodedOutputStream}
Milo Yip's avatar
Milo Yip committed
160 161 162

`EncodedOutputStream` is similar but it has a `bool putBOM` parameter in the constructor, controlling whether to write BOM into output byte stream.

163
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182
#include "rapidjson/filewritestream.h"  // FileWriteStream
#include "rapidjson/encodedstream.h"    // EncodedOutputStream
#include <cstdio>

Document d;         // Document is GenericDocument<UTF8<> > 
// ...

FILE* fp = fopen("output_utf32le.json", "wb"); // non-Windows use "w"

char writeBuffer[256];
FileWriteStream bos(fp, writeBuffer, sizeof(writeBuffer));

typedef EncodedOutputStream<UTF32LE<>, FileWriteStream> OutputStream;
OutputStream eos(bos, true);   // Write BOM

Writer<OutputStream, UTF32LE<>, UTF8<>> writer(eos);
d.Accept(writer);   // This generates UTF32-LE file from UTF-8 in memory

fclose(fp);
183
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
184

185
## AutoUTFInputStream {#AutoUTFInputStream}
Milo Yip's avatar
Milo Yip committed
186 187 188 189 190

Sometimes an application may want to handle all supported JSON encoding. `AutoUTFInputStream` will detection encoding by BOM first. If BOM is unavailable, it will use  characteristics of valid JSON to make detection. If neither method success, it falls back to the UTF type provided in constructor.

Since the characters (code units) may be 8-bit, 16-bit or 32-bit. `AutoUTFInputStream` requires a character type which can hold at least 32-bit. We may use `unsigned`, as in the template parameter:

191
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209
#include "rapidjson/document.h"
#include "rapidjson/filereadstream.h"   // FileReadStream
#include "rapidjson/encodedstream.h"    // AutoUTFInputStream
#include <cstdio>

using namespace rapidjson;

FILE* fp = fopen("any.json", "rb"); // non-Windows use "r"

char readBuffer[256];
FileReadStream bis(fp, readBuffer, sizeof(readBuffer));

AutoUTFInputStream<unsigned, FileReadStream> eis(bis);  // wraps bis into eis

Document d;         // Document is GenericDocument<UTF8<> > 
d.ParseStream<0, AutoUTF<unsigned> >(eis); // This parses any UTF file into UTF-8 in memory

fclose(fp);
210
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
211 212 213 214 215

When specifying the encoding of stream, uses `AutoUTF<CharType>` as in `ParseStream()` above.

You can obtain the type of UTF via `UTFType GetType()`. And check whether a BOM is found by `HasBOM()`

216
## AutoUTFOutputStream {#AutoUTFOutputStream}
Milo Yip's avatar
Milo Yip committed
217 218 219

Similarly, to choose encoding for output during runtime, we can use `AutoUTFOutputStream`. This class is not automatic *per se*. You need to specify the UTF type and whether to write BOM in runtime.

220 221 222
~~~~~~~~~~cpp
using namespace rapidjson;

Milo Yip's avatar
Milo Yip committed
223 224 225 226 227 228 229 230 231 232
void WriteJSONFile(FILE* fp, UTFType type, bool putBOM, const Document& d) {
    char writeBuffer[256];
    FileWriteStream bos(fp, writeBuffer, sizeof(writeBuffer));

    typedef AutoUTFOutputStream<unsigned, FileWriteStream> OutputStream;
    OutputStream eos(bos, type, putBOM);
    
    Writer<OutputStream, UTF8<>, AutoUTF<> > writer;
    d.Accept(writer);
}
233
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
234 235 236

`AutoUTFInputStream` and `AutoUTFOutputStream` is more convenient than `EncodedInputStream` and `EncodedOutputStream`. They just incur a little bit runtime overheads.

237
# Custom Stream {#CustomStream}
Milo Yip's avatar
Milo Yip committed
238 239 240 241 242

In addition to memory/file streams, user can create their own stream classes which fits RapidJSON's API. For example, you may create network stream, stream from compressed file, etc.

RapidJSON combines different types using templates. A class containing all required interface can be a stream. The Stream interface is defined in comments of `rapidjson/rapidjson.h`:

243
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271
concept Stream {
    typename Ch;    //!< Character type of the stream.

    //! Read the current character from stream without moving the read cursor.
    Ch Peek() const;

    //! Read the current character from stream and moving the read cursor to next character.
    Ch Take();

    //! Get the current read cursor.
    //! \return Number of characters read from start.
    size_t Tell();

    //! Begin writing operation at the current read pointer.
    //! \return The begin writer pointer.
    Ch* PutBegin();

    //! Write a character.
    void Put(Ch c);

    //! Flush the buffer.
    void Flush();

    //! End the writing operation.
    //! \param begin The begin write pointer returned by PutBegin().
    //! \return Number of characters written.
    size_t PutEnd(Ch* begin);
}
272
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
273 274 275 276 277

For input stream, they must implement `Peek()`, `Take()` and `Tell()`.
For output stream, they must implement `Put()` and `Flush()`. 
There are two special interface, `PutBegin()` and `PutEnd()`, which are only for *in situ* parsing. Normal streams do not implement them. However, if the interface is not needed for a particular stream, it is still need to a dummy implementation, otherwise will generate compilation error.

278
## Example: istream wrapper {#ExampleIStreamWrapper}
Milo Yip's avatar
Milo Yip committed
279 280 281

The following example is a wrapper of `std::istream`, which only implements 3 functions.

282
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312
class IStreamWrapper {
public:
    typedef char Ch;

    IStreamWrapper(std::istream& is) : is_(is) {
    }

    Ch Peek() const { // 1
        int c = is_.peek();
        return c == std::char_traits<char>::eof() ? '\0' : (Ch)c;
    }

    Ch Take() { // 2
        int c = is_.get();
        return c == std::char_traits<char>::eof() ? '\0' : (Ch)c;
    }

    size_t Tell() const { return (size_t)is_.tellg(); } // 3

    Ch* PutBegin() { assert(false); return 0; }
    void Put(Ch) { assert(false); }
    void Flush() { assert(false); }
    size_t PutEnd(Ch*) { assert(false); return 0; }

private:
    IStreamWrapper(const IStreamWrapper&);
    IStreamWrapper& operator=(const IStreamWrapper&);

    std::istream& is_;
};
313
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
314 315 316

User can use it to wrap instances of `std::stringstream`, `std::ifstream`.

317
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
318 319 320 321 322 323
const char* json = "[1,2,3,4]";
std::stringstream ss(json);
IStreamWrapper is(ss);

Document d;
d.Parse(is);
324
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
325 326 327

Note that, this implementation may not be as efficient as RapidJSON's memory or file streams, due to internal overheads of the standard library.

328
## Example: ostream wrapper {#ExampleOStreamWrapper}
Milo Yip's avatar
Milo Yip committed
329 330 331

The following example is a wrapper of `std::istream`, which only implements 2 functions.

332
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354
class OStreamWrapper {
public:
    typedef char Ch;

    OStreamWrapper(std::ostream& os) : os_(os) {
    }

    Ch Peek() const { assert(false); return '\0'; }
    Ch Take() { assert(false); return '\0'; }
    size_t Tell() const {  }

    Ch* PutBegin() { assert(false); return 0; }
    void Put(Ch c) { os_.put(c); }                  // 1
    void Flush() { os_.flush(); }                   // 2
    size_t PutEnd(Ch*) { assert(false); return 0; }

private:
    OStreamWrapper(const OStreamWrapper&);
    OStreamWrapper& operator=(const OStreamWrapper&);

    std::ostream& os_;
};
355
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
356 357 358

User can use it to wrap instances of `std::stringstream`, `std::ofstream`.

359
~~~~~~~~~~cpp
Milo Yip's avatar
Milo Yip committed
360 361 362 363 364 365 366 367
Document d;
// ...

std::stringstream ss;
OSStreamWrapper os(ss);

Writer<OStreamWrapper> writer(os);
d.Accept(writer);
368
~~~~~~~~~~
Milo Yip's avatar
Milo Yip committed
369 370 371

Note that, this implementation may not be as efficient as RapidJSON's memory or file streams, due to internal overheads of the standard library.

372
# Summary {#Summary}
Milo Yip's avatar
Milo Yip committed
373 374

This section describes stream classes available in RapidJSON. Memory streams are simple. File stream can reduce the memory required during JSON parsing and generation, if the JSON is stored in file system. Encoded streams converts between byte streams and character streams. Finally, user may create custom streams using a simple interface.