DoS vector: TXID calculation #10522

When validating that a transaction has some PoW bound to it, either during block sync or PoWER we usually use a transaction's TXID to bind that PoW. And transactions are transmitted over the network as opaque byte blobs. The current way that the TXID is calculated from a transaction blob, is that it is A) deserialized, then B) expanded, then C) the subsections are hashed, and finally, D) the final transaction hash is calculated from the subsection hashes. Ideally, this process should be as quick as possible, so that bad transaction data can be failed as quickly as possible, but in reality, this process is much more expensive than it needs to be.

1. There's one Ed25519 point decompression, variable-base scalar-point multiplication, and point compression in cryptonote::expand_transaction_1() per output here and here. Each decompress + multiply + compress op is ~100x slower (citation needed) than crypto::cn_fast_hash()!!!
2. For a $N$-input RCTTypeBulletproofPlus (v6) RingCT transaction, deserializing the transaction takes $2N +10$ memory allocations:

• monero/src/cryptonote_basic/cryptonote_basic.h

  Line 146 in 4a24ca3

  FIELD(key_offsets)

  ($N$)

• monero/src/cryptonote_basic/cryptonote_basic.h

  Lines 187 to 189 in 4a24ca3

  	FIELD(vin)
  	FIELD(vout)
  	FIELD(extra)

  ($3$)

• monero/src/ringct/rctTypes.h

  Line 363 in 4a24ca3

  PREPARE_CUSTOM_VECTOR_SERIALIZATION(outputs, ecdhInfo);

  ($1$)

• monero/src/ringct/rctTypes.h

  Line 394 in 4a24ca3

  PREPARE_CUSTOM_VECTOR_SERIALIZATION(outputs, outPk);

  ($1$)

• monero/src/ringct/rctTypes.h

  Line 447 in 4a24ca3

  PREPARE_CUSTOM_VECTOR_SERIALIZATION(nbp, bulletproofs_plus);

  ($1$)

• monero/src/ringct/rctTypes.h

  Lines 274 to 275 in 4a24ca3

  	FIELD(L)
  	FIELD(R)

  ($2$)

• monero/src/ringct/rctTypes.h

  Line 500 in 4a24ca3

  PREPARE_CUSTOM_VECTOR_SERIALIZATION(inputs, CLSAGs);

  ($1$)

• monero/src/ringct/rctTypes.h

  Line 511 in 4a24ca3

  PREPARE_CUSTOM_VECTOR_SERIALIZATION(mixin + 1, CLSAGs[i].s);

  ($N$)

• monero/src/ringct/rctTypes.h

  Line 591 in 4a24ca3

  PREPARE_CUSTOM_VECTOR_SERIALIZATION(inputs, pseudoOuts);

  ($1$)

3. Expanding the transaction takes a memory allocation:

   monero/src/cryptonote_basic/cryptonote_format_utils.cpp

   Line 168 in 4a24ca3

   rv.p.bulletproofs_plus[0].V.resize(n_amounts);

4. Serializing the transaction prefix inside the hashing function takes at least one allocation (calculate_transaction_prunable_hash() uses cached unprunable_size field instead of serializing):

• monero/src/cryptonote_basic/cryptonote_format_utils_basic.cpp

  Line 38 in 4a24ca3

  binary_archive<true> a(s);

So all told, for a N-in M-out transaction today, the daemon does $2N + 12$ memory allocations, $M$ Ed25519 point decompressions, $M$ Ed25519 variable-base scalar-point multiplications, $M$ Ed25519 point compressions, and 4 Keccak256 hashes, just to calculate the TXID. This is a DoS vector. Without a fork, performant code could reduce this to just the 4 Keccak256 hashes. With a hard fork, keeping the TXID bound to its proof data, we could reduce this to 2 Keccak256 hash, and still retain pruning capabilities. With a hard fork, moving the transaction proof data to a field in the block, we could reduce this to just 1 Keccak256 hash, and still retain pruning capabilities.