Skip to content

Commit 058eb91

Browse files
lemirelemire
committed
Better doc.
1 parent d61aa21 commit 058eb91

3 files changed

Lines changed: 73 additions & 35 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -68,18 +68,17 @@ make parsingcompetition
6868
To simplify the engineering, we make some assumptions.
6969

7070
- We support UTF-8 (and thus ASCII), nothing else (no Latin, no UTF-16). We do not believe that this is a genuine limitation in the sense that we do not think that there is any serious application that needs to process JSON data without an ASCII or UTF-8 encoding.
71-
- We assume AVX2 support which is available in all recent mainstream x86 processors produced by AMD and Intel. No support for non-x86 processors is included though it can be done.
72-
- We only support GNU GCC and LLVM Clang at this time. There is no support for Microsoft Visual Studio, though it should not be difficult.
73-
- We expect the input memory to be readable up to 32 bytes beyond the end of the JSON document (to support fast vector loads). All bytes beyond the end of the JSON document are ignored (can be garbage) and the JSON document does not need to be NULL terminated. You can allocate a properly overallocated memory region with the provided `allocate_padded_buffer` function or simply by allocating your memory with extra capacity (`malloc(length + SIMDJSON_PADDING)`). (We expect that this limitation can be lifted without performance penalty, but at the cost of a bit of some code complexity.)
71+
- We assume AVX2 support which is available in all recent mainstream x86 processors produced by AMD and Intel. No support for non-x86 processors is included though it can be done. We plan to support ARM processors (help is invited).
72+
- We only support GNU GCC and LLVM Clang at this time. There is no support for Microsoft Visual Studio, though it should not be difficult (help is invited).
7473
- In cases of failure, we just report a failure without any indication as to the nature of the problem. (This can be easily improved without affecting performance.)
7574

7675
## Features
7776

77+
- The input string is unmodified. (Parsers like sajson and RapidJSON use the input string as a buffer.)
7878
- We parse integers and floating-point numbers as separate types which allows us to support large 64-bit integers.
7979
- We do full UTF-8 validation as part of the parsing. (Parsers like fastjson, gason and dropbox json11 do not do UTF-8 validation.)
8080
- We fully validate the numbers. (Parsers like gason and ultranjson will accept `[0e+]` as valid JSON.)
8181
- We validate string content for unescaped characters. (Parsers like fastjson and ultrajson accept unescaped line breaks and tags in strings.)
82-
- The input string is unmodified. (Parsers like sajson and RapidJSON use the input string as a buffer.)
8382

8483
## Architecture
8584

‎include/simdjson/jsonparser.h‎

Lines changed: 50 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -12,46 +12,71 @@
1212

1313
// Parse a document found in buf, need to preallocate ParsedJson.
1414
// Return false in case of a failure. You can also check validity
15-
// by calling pj.isValid(). The same ParsedJson can be reused.
16-
// the input buf should be readable up to buf + len + SIMDJSON_PADDING
17-
// all bytes at and after buf + len are ignored (can be garbage)
15+
// by calling pj.isValid(). The same ParsedJson can be reused for other documents.
16+
//
17+
// If reallocifneeded is true (default) then a temporary buffer is created when needed during processing
18+
// (a copy of the input string is made).
19+
// The input buf should be readable up to buf + len + SIMDJSON_PADDING if reallocifneeded is false,
20+
// all bytes at and after buf + len are ignored (can be garbage).
1821
WARN_UNUSED
19-
bool json_parse(const u8 *buf, size_t len, ParsedJson &pj);
22+
bool json_parse(const u8 *buf, size_t len, ParsedJson &pj, bool reallocifneeded = true);
2023

21-
// the input buf should be readable up to buf + len + SIMDJSON_PADDING
22-
// all bytes at and after buf + len are ignored (can be garbage)
24+
// Parse a document found in buf, need to preallocate ParsedJson.
25+
// Return false in case of a failure. You can also check validity
26+
// by calling pj.isValid(). The same ParsedJson can be reused for other documents.
27+
//
28+
// If reallocifneeded is true (default) then a temporary buffer is created when needed during processing
29+
// (a copy of the input string is made).
30+
// The input buf should be readable up to buf + len + SIMDJSON_PADDING if reallocifneeded is false,
31+
// all bytes at and after buf + len are ignored (can be garbage).
2332
WARN_UNUSED
24-
static inline bool json_parse(const char * buf, size_t len, ParsedJson &pj) {
25-
return json_parse((const u8 *) buf, len, pj);
33+
static inline bool json_parse(const char * buf, size_t len, ParsedJson &pj, bool reallocifneeded = true) {
34+
return json_parse((const u8 *) buf, len, pj, reallocifneeded);
2635
}
2736

28-
// convenience function
29-
// the input s should be readable up to s.data() + s.size() + SIMDJSON_PADDING
30-
// all bytes at and after s.data()+s.size() are ignored (can be garbage)
37+
// Parse a document found in buf, need to preallocate ParsedJson.
38+
// Return false in case of a failure. You can also check validity
39+
// by calling pj.isValid(). The same ParsedJson can be reused for other documents.
40+
//
41+
// If reallocifneeded is true (default) then a temporary buffer is created when needed during processing
42+
// (a copy of the input string is made).
43+
// the input s should be readable up to s.data() + s.size() + SIMDJSON_PADDING if reallocifneeded is false,
44+
// all bytes at and after s.data()+s.size() are ignored (can be garbage).
3145
WARN_UNUSED
32-
static inline bool json_parse(const std::string_view &s, ParsedJson &pj) {
33-
return json_parse(s.data(), s.size(), pj);
46+
static inline bool json_parse(const std::string_view &s, ParsedJson &pj, bool reallocifneeded = true) {
47+
return json_parse(s.data(), s.size(), pj, reallocifneeded);
3448
}
3549

3650

3751
// Build a ParsedJson object. You can check validity
38-
// by calling pj.isValid(). This does memory allocation.
39-
// the input buf should be readable up to buf + len + SIMDJSON_PADDING
40-
// all bytes at and after buf + len are ignored (can be garbage)
52+
// by calling pj.isValid(). This does the memory allocation needed for ParsedJson.
53+
// If reallocifneeded is true (default) then a temporary buffer is created when needed during processing
54+
// (a copy of the input string is made).
55+
//
56+
// the input buf should be readable up to buf + len + SIMDJSON_PADDING if reallocifneeded is false,
57+
// all bytes at and after buf + len are ignored (can be garbage).
4158
WARN_UNUSED
42-
ParsedJson build_parsed_json(const u8 *buf, size_t len);
59+
ParsedJson build_parsed_json(const u8 *buf, size_t len, bool reallocifneeded = true);
4360

4461
WARN_UNUSED
45-
// the input buf should be readable up to buf + len + SIMDJSON_PADDING
46-
// all bytes at and after buf + len are ignored (can be garbage)
47-
static inline ParsedJson build_parsed_json(const char * buf, size_t len) {
48-
return build_parsed_json((const u8 *) buf, len);
62+
// Build a ParsedJson object. You can check validity
63+
// by calling pj.isValid(). This does the memory allocation needed for ParsedJson.
64+
// If reallocifneeded is true (default) then a temporary buffer is created when needed during processing
65+
// (a copy of the input string is made).
66+
// The input buf should be readable up to buf + len + SIMDJSON_PADDING if reallocifneeded is false,
67+
// all bytes at and after buf + len are ignored (can be garbage).
68+
static inline ParsedJson build_parsed_json(const char * buf, size_t len, bool reallocifneeded = true) {
69+
return build_parsed_json((const u8 *) buf, len, reallocifneeded);
4970
}
5071

5172
// convenience function
5273
WARN_UNUSED
53-
// the input s should be readable up to s.data() + s.size() + SIMDJSON_PADDING
54-
// all bytes at and after s.data()+s.size() are ignored (can be garbage)
55-
static inline ParsedJson build_parsed_json(const std::string_view &s) {
56-
return build_parsed_json(s.data(), s.size());
74+
// Build a ParsedJson object. You can check validity
75+
// by calling pj.isValid(). This does the memory allocation needed for ParsedJson.
76+
// If reallocifneeded is true (default) then a temporary buffer is created when needed during processing
77+
// (a copy of the input string is made).
78+
// The input s should be readable up to s.data() + s.size() + SIMDJSON_PADDING if reallocifneeded is false,
79+
// all bytes at and after s.data()+s.size() are ignored (can be garbage).
80+
static inline ParsedJson build_parsed_json(const std::string_view &s, bool reallocifneeded = true) {
81+
return build_parsed_json(s.data(), s.size(), reallocifneeded);
5782
}

‎src/jsonparser.cpp‎

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,49 @@
11
#include "simdjson/jsonparser.h"
2-
2+
#include <unistd.h>
33

44
// parse a document found in buf, need to preallocate ParsedJson.
55
WARN_UNUSED
6-
bool json_parse(const u8 *buf, size_t len, ParsedJson &pj) {
6+
bool json_parse(const u8 *buf, size_t len, ParsedJson &pj, bool reallocifneeded) {
77
if (pj.bytecapacity < len) {
88
std::cerr << "Your ParsedJson cannot support documents that big: " << len
99
<< std::endl;
1010
return false;
1111
}
12+
bool reallocated = false;
13+
if(reallocifneeded) {
14+
// realloc is needed if the end of the memory crosses a page
15+
long pagesize = sysconf (_SC_PAGESIZE); // on windows this should be SYSTEM_INFO sysInfo; GetSystemInfo(&sysInfo); sysInfo.dwPageSize
16+
if ( (reinterpret_cast<uintptr_t>(buf + len - 1) % pagesize ) < SIMDJSON_PADDING ) {
17+
const u8 *tmpbuf = buf;
18+
buf = (u8 *) allocate_padded_buffer(len);
19+
if(buf == NULL) return false;
20+
memcpy((void*)buf,tmpbuf,len);
21+
reallocated = true;
22+
}
23+
}
1224
bool isok = find_structural_bits(buf, len, pj);
1325
if (isok) {
1426
isok = flatten_indexes(len, pj);
1527
} else {
28+
if(reallocated) free((void*)buf);
1629
return false;
17-
}//printf("ok\n");
30+
}
1831
if (isok) {
1932
isok = unified_machine(buf, len, pj);
20-
//printf("ok %d \n",isok);
2133
} else {
34+
if(reallocated) free((void*)buf);
2235
return false;
2336
}
37+
if(reallocated) free((void*)buf);
2438
return isok;
2539
}
2640

2741
WARN_UNUSED
28-
ParsedJson build_parsed_json(const u8 *buf, size_t len) {
42+
ParsedJson build_parsed_json(const u8 *buf, size_t len, bool reallocifneeded) {
2943
ParsedJson pj;
3044
bool ok = pj.allocateCapacity(len);
3145
if(ok) {
32-
ok = json_parse(buf, len, pj);
46+
ok = json_parse(buf, len, pj, reallocifneeded);
3347
assert(ok == pj.isValid());
3448
} else {
3549
std::cerr << "failure during memory allocation " << std::endl;

0 commit comments

Comments
 (0)