Skip to content

Commit 50be762

Browse files
jdatcmdclaude
jdatcmd
and
claude
committed
Map bytea to a binary String instead of hex text
A bytea value now reaches Ruby as a binary (ASCII-8BIT) String holding its exact bytes -- NUL-safe -- rather than as its \x... hex text output, and returning a String into a bytea takes the string's raw bytes verbatim (any encoding), with no hex/escape parsing. This matches PL/Python's bytea <-> bytes mapping. The conversion is intercepted at the datum level (before the type's text I/O) wherever a bytea crosses between SQL and Ruby: - scalar arguments and bytea[] arguments (plruby_func_build_args) - scalar and bytea[] return values (plruby_func_handler return path, which otherwise NUL-truncates via the text-input path) - composite fields, SPI result rows, and trigger $_TD (the shared plruby_hash_from_tuple) - fields, OUT params, and return_next rows via plruby_datum_from_value FROM-SQL uses PG_DETOAST_DATUM_PACKED + VARDATA_ANY/VARSIZE_ANY_EXHDR into an ASCII-8BIT string; TO-SQL builds the varlena directly from the string's raw bytes. bytea[] shares the array datum walk, generalized to convert elements binary-wise when no FromSQL transform applies. BREAKING: bodies that produced or consumed the hex text form must be updated (e.g. build bytes with [..].pack('C*') rather than a \x string). Rewrote sql/bytea for the new semantics (round-trips with NULs, high bytes, composite/SPI/array coverage). Full suite green on PG 12 and 18; jsonb_plruby and hstore_plruby suites still pass (shared array walk). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 0ae8101 commit 50be762

7 files changed

Lines changed: 298 additions & 57 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,18 @@ and the project aims to follow [Semantic Versioning](https://semver.org/).
1515
independent and resets when the function is recompiled; an anonymous `DO` block
1616
gets a fresh, empty `$_SD` each run.
1717

18+
### Changed
19+
20+
- **`bytea` maps to a binary `String`, not hex text (breaking).** A `bytea`
21+
value now reaches Ruby as a binary (`ASCII-8BIT`) `String` of its raw bytes,
22+
NUL-safe, instead of its `\x...` hex text; returning a `String` into a `bytea`
23+
takes the string's raw bytes verbatim, with no hex/escape parsing. This
24+
matches PL/Python's `bytea` <-> `bytes` mapping and applies wherever a `bytea`
25+
crosses between SQL and Ruby (arguments, returns, `bytea[]` elements,
26+
composite fields, and SPI result rows). Function bodies that previously
27+
produced or consumed the hex text form must be updated (e.g. build bytes with
28+
`[..].pack('C*')` rather than assembling a `\x` string).
29+
1830
## [2.4.0] - 2026-07-06
1931

2032
Feature parity with the sibling PL/php 2.4: an `on_init` hook, the

‎doc/plruby.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ function.
9898
| numeric | `String` (lossless) | `Numeric` or `String` |
9999
| boolean | `true` / `false` | `true` / `false` |
100100
| text / varchar / etc. | `String` | `String` |
101+
| bytea | binary `String` (ASCII-8BIT) | `String` (raw bytes) |
101102
| arrays (e.g. `int[]`) | nested `Array` | nested `Array` |
102103
| composite / row / record | `Hash` (string keys) | `Hash` (or positional `Array`) |
103104
| NULL | `nil` | `nil` |
@@ -115,8 +116,16 @@ Strings arrive tagged with the Ruby encoding that matches the database encoding
115116
(`UTF8` to `UTF-8`, `LATIN1` to `ISO-8859-1`, `EUC_JP` to `EUC-JP`, `WIN1251` to
116117
`Windows-1251`, and so on), so `length`, `reverse`, and regexps operate on
117118
characters, not bytes. Encodings Ruby does not recognize fall back to
118-
`ASCII-8BIT` (binary), which is byte-preserving. `bytea` is passed as its
119-
textual `\x...` hex representation (a `String`), not raw bytes.
119+
`ASCII-8BIT` (binary), which is byte-preserving.
120+
121+
`bytea` is the exception: it is a raw byte string, so it arrives as a binary
122+
(`ASCII-8BIT`) `String` holding its exact bytes, including any NULs, rather than
123+
as hex text. Returning a `String` into a `bytea` takes the string's raw bytes
124+
verbatim (whatever its encoding), with no hex or escape parsing. This holds
125+
wherever a `bytea` crosses between SQL and Ruby: plain arguments and returns,
126+
`bytea[]` elements, fields of composite arguments and results, and SPI result
127+
rows. To emit specific bytes, build the `String` directly, e.g.
128+
`[0, 255, 16].pack('C*')`.
120129

121130
## Composite types and records
122131

‎expected/bytea.out‎

Lines changed: 106 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,130 @@
11
--
2-
-- bytea conversion. Like every type, a bytea reaches Ruby through its output
3-
-- function, so it arrives as its *textual* (hex) representation -- a String --
4-
-- not as raw bytes. Returning a bytea feeds the String back through bytea's
5-
-- input function.
2+
-- bytea conversion. A bytea is a raw byte string, so it reaches Ruby as a
3+
-- binary (ASCII-8BIT) String holding its exact bytes -- NUL-safe -- rather than
4+
-- as its hex text. Returning a Ruby String into a bytea takes the string's
5+
-- raw bytes verbatim (any encoding), with no hex/escape parsing.
66
--
77
SET bytea_output = 'hex';
8-
-- A bytea argument is the hex text string (UTF-8 tagged), preserving NUL and
9-
-- high bytes in that representation.
8+
-- A bytea argument is a binary String; its bytes match the input exactly.
109
CREATE FUNCTION b_form(bytea) RETURNS text LANGUAGE plruby AS $$
11-
"#{args[0].class}:#{args[0]}"
10+
"#{args[0].class}:#{args[0].encoding}:#{args[0].bytesize}"
1211
$$;
1312
SELECT b_form('\xdeadbeef'::bytea);
14-
b_form
15-
-------------------
16-
String:\xdeadbeef
13+
b_form
14+
---------------------
15+
String:ASCII-8BIT:4
1716
(1 row)
1817

19-
SELECT b_form('\x00ff01fe'::bytea);
20-
b_form
21-
-------------------
22-
String:\x00ff01fe
18+
SELECT b_form('\x00ff01fe'::bytea); -- NUL and high bytes preserved
19+
b_form
20+
---------------------
21+
String:ASCII-8BIT:4
2322
(1 row)
2423

25-
-- Round-trip: returning the hex string yields the identical bytea.
24+
SELECT b_form('\x'::bytea); -- empty
25+
b_form
26+
---------------------
27+
String:ASCII-8BIT:0
28+
(1 row)
29+
30+
-- The bytes are the raw content, so encode() of the echoed value round-trips.
2631
CREATE FUNCTION b_echo(bytea) RETURNS bytea LANGUAGE plruby AS $$
2732
args[0]
2833
$$;
29-
SELECT b_echo('\x00ff01fe'::bytea);
30-
b_echo
31-
------------
32-
\x00ff01fe
34+
SELECT encode(b_echo('\x00ff01fe'::bytea), 'hex');
35+
encode
36+
----------
37+
00ff01fe
38+
(1 row)
39+
40+
-- The function can inspect and transform the raw bytes.
41+
CREATE FUNCTION b_first_byte(bytea) RETURNS int LANGUAGE plruby AS $$
42+
args[0].bytes[0]
43+
$$;
44+
SELECT b_first_byte('\x41ff00'::bytea); -- 65
45+
b_first_byte
46+
--------------
47+
65
3348
(1 row)
3449

35-
-- Building a bytea from a Ruby-produced hex string.
50+
-- Building a bytea from raw Ruby bytes.
3651
CREATE FUNCTION b_make() RETURNS bytea LANGUAGE plruby AS $$
37-
"\\x" + [0, 255, 16, 32].map { |n| "%02x" % n }.join
52+
[0, 255, 16, 32].pack('C*')
53+
$$;
54+
SELECT encode(b_make(), 'hex');
55+
encode
56+
----------
57+
00ff1020
58+
(1 row)
59+
60+
-- A Ruby String with embedded NULs becomes those exact bytes.
61+
CREATE FUNCTION b_nul() RETURNS bytea LANGUAGE plruby AS $$
62+
"a\x00b".b
3863
$$;
39-
SELECT b_make();
40-
b_make
41-
------------
42-
\x00ff1020
64+
SELECT encode(b_nul(), 'hex');
65+
encode
66+
--------
67+
610062
4368
(1 row)
4469

4570
-- The empty bytea.
4671
CREATE FUNCTION b_empty() RETURNS bytea LANGUAGE plruby AS $$
47-
"\\x"
72+
"".b
73+
$$;
74+
SELECT encode(b_empty(), 'hex');
75+
encode
76+
--------
77+
78+
(1 row)
79+
80+
-- bytea inside a composite: the field is a binary String too.
81+
CREATE TYPE b_rec AS (tag text, blob bytea);
82+
CREATE FUNCTION b_rec_in(b_rec) RETURNS int LANGUAGE plruby AS $$
83+
args[0]['blob'].bytesize
84+
$$;
85+
SELECT b_rec_in(ROW('x', '\x00ff01fe')::b_rec); -- 4
86+
b_rec_in
87+
----------
88+
4
89+
(1 row)
90+
91+
CREATE FUNCTION b_rec_out() RETURNS b_rec LANGUAGE plruby AS $$
92+
{'tag' => 'y', 'blob' => [1, 2, 3].pack('C*')}
93+
$$;
94+
SELECT (b_rec_out()).tag, encode((b_rec_out()).blob, 'hex');
95+
tag | encode
96+
-----+--------
97+
y | 010203
98+
(1 row)
99+
100+
-- bytea read back through SPI is also a binary String.
101+
CREATE FUNCTION b_via_spi() RETURNS int LANGUAGE plruby AS $$
102+
r = spi_exec("SELECT '\\x00ff01fe'::bytea AS b")
103+
spi_fetch_row(r)['b'].bytesize
104+
$$;
105+
SELECT b_via_spi(); -- 4
106+
b_via_spi
107+
-----------
108+
4
109+
(1 row)
110+
111+
-- bytea[] arguments arrive as an Array of binary Strings, and a returned
112+
-- Array of binary Strings becomes a bytea[].
113+
CREATE FUNCTION b_arr_in(bytea[]) RETURNS int LANGUAGE plruby AS $$
114+
args[0].map(&:bytesize).inject(0, :+)
115+
$$;
116+
SELECT b_arr_in(ARRAY['\x0000'::bytea, '\xffffff'::bytea]); -- 5
117+
b_arr_in
118+
----------
119+
5
120+
(1 row)
121+
122+
CREATE FUNCTION b_arr_out() RETURNS bytea[] LANGUAGE plruby AS $$
123+
[[0, 255].pack('C*'), [16].pack('C*')]
48124
$$;
49-
SELECT b_empty();
50-
b_empty
51-
---------
52-
\x
125+
SELECT encode((b_arr_out())[1], 'hex'), encode((b_arr_out())[2], 'hex');
126+
encode | encode
127+
--------+--------
128+
00ff | 10
53129
(1 row)
54130

‎plruby.c‎

Lines changed: 39 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -724,7 +724,26 @@ plruby_func_handler(FunctionCallInfo fcinfo, plruby_proc_desc *desc)
724724
case T_FIXNUM:
725725
case T_BIGNUM:
726726
case T_FLOAT:
727+
retvalbuffer = plruby_value_to_cstring(result, false, false);
728+
break;
727729
case T_STRING:
730+
/*
731+
* bytea takes the String's raw bytes (NUL-safe) via the datum
732+
* path; every other scalar keeps the text-input path below.
733+
*/
734+
if (desc->ret_oid == BYTEAOID)
735+
{
736+
bool isnull;
737+
Datum d = plruby_datum_from_value(result, desc->ret_oid,
738+
(int32) -1, &isnull);
739+
740+
if (isnull)
741+
{
742+
fcinfo->isnull = true;
743+
return (Datum) 0;
744+
}
745+
return d;
746+
}
728747
retvalbuffer = plruby_value_to_cstring(result, false, false);
729748
break;
730749
case T_ARRAY:
@@ -733,14 +752,15 @@ plruby_func_handler(FunctionCallInfo fcinfo, plruby_proc_desc *desc)
733752
Oid elemtype = get_element_type(desc->ret_oid);
734753

735754
/*
736-
* Array of composite (or of a transformed type, e.g.
737-
* jsonb[]): build the array datum directly so each element
738-
* is assembled recursively/through its ToSQL function.
739-
* Scalar arrays keep the (multidimensional-capable) text
740-
* path below.
755+
* Array of composite (or of a transformed type, e.g. jsonb[],
756+
* or of bytea, whose elements are raw bytes): build the array
757+
* datum directly so each element is assembled recursively /
758+
* through its ToSQL function. Scalar arrays keep the
759+
* (multidimensional-capable) text path below.
741760
*/
742761
if (OidIsValid(elemtype) &&
743762
(type_is_rowtype(elemtype) ||
763+
elemtype == BYTEAOID ||
744764
OidIsValid(plruby_transform_tosql(elemtype))))
745765
{
746766
bool isnull;
@@ -1202,6 +1222,20 @@ plruby_func_build_args(plruby_proc_desc *desc, FunctionCallInfo fcinfo)
12021222
plruby_array_from_datum(PLRUBY_ARG_VALUE(fcinfo, j),
12031223
elemtype, etrf));
12041224
}
1225+
else if (argtype == BYTEAOID)
1226+
{
1227+
/* bytea: raw binary String, not hex text */
1228+
rb_ary_push(args,
1229+
plruby_binary_from_datum(PLRUBY_ARG_VALUE(fcinfo, j),
1230+
BYTEAOID));
1231+
}
1232+
else if (elemtype == BYTEAOID)
1233+
{
1234+
/* bytea[]: elements as binary Strings */
1235+
rb_ary_push(args,
1236+
plruby_array_from_datum(PLRUBY_ARG_VALUE(fcinfo, j),
1237+
BYTEAOID, InvalidOid));
1238+
}
12051239
else
12061240
{
12071241
char *tmp = OutputFunctionCall(&(desc->arg_out_func[i]),

‎plruby_io.c‎

Lines changed: 61 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ plruby_transform_fromsql(Oid typid)
6767
static VALUE
6868
plruby_array_from_datum_walk(Datum *elems, bool *nulls, int *idx,
6969
int ndims, const int *dims, int depth,
70-
Oid fromsql_fn)
70+
Oid elemtype, Oid fromsql_fn)
7171
{
7272
VALUE ary = rb_ary_new();
7373
int i;
@@ -78,20 +78,29 @@ plruby_array_from_datum_walk(Datum *elems, bool *nulls, int *idx,
7878
rb_ary_push(ary, plruby_array_from_datum_walk(elems, nulls, idx,
7979
ndims, dims,
8080
depth + 1,
81-
fromsql_fn));
81+
elemtype, fromsql_fn));
8282
else
8383
{
8484
if (nulls[*idx])
8585
rb_ary_push(ary, Qnil);
86-
else
86+
else if (OidIsValid(fromsql_fn))
8787
rb_ary_push(ary,
8888
(VALUE) OidFunctionCall1(fromsql_fn, elems[*idx]));
89+
else
90+
/* no transform: the only element-wise binary type is bytea */
91+
rb_ary_push(ary, plruby_binary_from_datum(elems[*idx], elemtype));
8992
(*idx)++;
9093
}
9194
}
9295
return ary;
9396
}
9497

98+
/*
99+
* An array datum -> nested Ruby Array. When fromsql_fn is valid, each element
100+
* converts through that FromSQL transform; when it is InvalidOid, elements
101+
* convert per plruby_binary_from_datum (used for bytea[], whose elements are
102+
* raw byte strings).
103+
*/
95104
VALUE
96105
plruby_array_from_datum(Datum d, Oid elemtype, Oid fromsql_fn)
97106
{
@@ -112,7 +121,8 @@ plruby_array_from_datum(Datum d, Oid elemtype, Oid fromsql_fn)
112121
deconstruct_array(arr, elemtype, elmlen, elmbyval, elmalign,
113122
&elems, &nulls, &nelems);
114123
return plruby_array_from_datum_walk(elems, nulls, &idx,
115-
ndims, ARR_DIMS(arr), 0, fromsql_fn);
124+
ndims, ARR_DIMS(arr), 0,
125+
elemtype, fromsql_fn);
116126
}
117127

118128
/* ---------------------------------------------------------------------
@@ -224,6 +234,29 @@ plruby_scalar_from_cstring(const char *str, Oid typeoid)
224234
}
225235
}
226236

237+
/*
238+
* bytea -> Ruby. A bytea Datum is a raw varlena, so it maps to a binary
239+
* (ASCII-8BIT) Ruby String holding its exact bytes -- NUL-safe -- rather than
240+
* through bytea's hex text output. Returns Qundef when typeoid is not bytea,
241+
* so callers can fall back to their normal text path.
242+
*/
243+
VALUE
244+
plruby_binary_from_datum(Datum value, Oid typeoid)
245+
{
246+
struct varlena *vl;
247+
VALUE s;
248+
249+
if (typeoid != BYTEAOID)
250+
return Qundef;
251+
252+
vl = PG_DETOAST_DATUM_PACKED(value);
253+
s = rb_enc_str_new(VARDATA_ANY(vl), VARSIZE_ANY_EXHDR(vl),
254+
rb_ascii8bit_encoding());
255+
if ((Pointer) vl != DatumGetPointer(value))
256+
pfree(vl);
257+
return s;
258+
}
259+
227260
/* ---------------------------------------------------------------------
228261
* PostgreSQL array text -> nested Ruby Array
229262
* ------------------------------------------------------------------- */
@@ -480,6 +513,20 @@ plruby_datum_from_value(VALUE val, Oid typeoid, int32 typmod, bool *isnull)
480513
(RB_TYPE_P(val, T_HASH) || RB_TYPE_P(val, T_ARRAY)))
481514
return plruby_composite_datum(val, typeoid, typmod);
482515

516+
/*
517+
* bytea fed a Ruby String: take its raw bytes verbatim (any encoding),
518+
* building the varlena directly rather than parsing hex/escape text.
519+
*/
520+
if (typeoid == BYTEAOID && RB_TYPE_P(val, T_STRING))
521+
{
522+
long len = RSTRING_LEN(val);
523+
bytea *result = (bytea *) palloc(len + VARHDRSZ);
524+
525+
SET_VARSIZE(result, len + VARHDRSZ);
526+
memcpy(VARDATA(result), RSTRING_PTR(val), len);
527+
return PointerGetDatum(result);
528+
}
529+
483530
/* Scalar leaf: convert via the target type's input function. */
484531
{
485532
char *s = plruby_value_to_cstring(val, true, true);
@@ -675,6 +722,16 @@ plruby_hash_from_tuple(HeapTuple tuple, TupleDesc tupdesc)
675722
continue;
676723
}
677724

725+
/* bytea (and bytea[]) map to raw binary Strings, not hex text. */
726+
v = plruby_binary_from_datum(attr, att->atttypid);
727+
if (v == Qundef && get_element_type(att->atttypid) == BYTEAOID)
728+
v = plruby_array_from_datum(attr, BYTEAOID, InvalidOid);
729+
if (v != Qundef)
730+
{
731+
rb_hash_aset(h, rb_str_new_cstr(attname), v);
732+
continue;
733+
}
734+
678735
getTypeOutputInfo(att->atttypid, &typoutput, &typisvarlena);
679736
outputstr = OidOutputFunctionCall(typoutput, attr);
680737

0 commit comments

Comments
 (0)