Early Security Stories — Green Shellcode Contest
When I was working in the MSRC and SDL teams, I ran a series of contests. The goals were to encourage learning, foster a team culture around technical excellence, and have some fun. I wanted them to be accessible across program managers, vulnerability researchers, and engineers.
Contest #1
The first one was to calculate a Fibonacci number in assembly. I chose this because it’s a simple problem to learn more about assembly, which was relevant to vulnerability and exploit analysis. The contest part was to do it in the fewest number of clock cycles.
I wish I had known @BruceDawson0xB then because the most challenging aspect was measuring the winner! It seemed so simple at first. Call RDTSC (Read Timestamp Counter) before and after. Do it a few times in a loop to ensure consistency and done. Hardly.
With out-of-order execution, you need to serialize the instruction queue (CPUID trick https://stackoverflow.com/questions/2918113/cpuid-before-rdtsc). Then there’s background noise. I wrote a kernel driver and disabled interrupts. I think I learned more than the contestants! @MattT_Cyber won ( https://gist.github.com/JohnLaTwC/e9a2665ce9ae0b460fc4d2f676390f85)
Contest #2
The next challenge was to represent the state of a chessboard in the fewest number of bits. I wrote a program to generate random piece layouts with varying densities and judged the algorithms on the average bit length of their representations.
This is a compression problem. Entries used Huffman coding, combinatorics, arithmetic coding, and adaptive algorithms. The winner (Greg W.) was the MSRC engineer who worked on vulnerability reports in Microsoft decompression code!
Winning entry: https://gist.github.com/JohnLaTwC/ee7550e74ca23a52f029770f39cf9f7f
Contest #3
The contest was to write pop-calc shellcode. Rewarding the shortest shellcode would have given an advantage to experienced exploit writers so I called this the Green Shellcode challenge. The winner would use the fewest number of unique bytes. Tiebreaker was length.
While people have written shellcode to avoid certain bytes (like NULLs), used AlphaNumeric encoders, or even ASCII Art (@berendjanwever) this was taking it to another level. Could you pop calc using only 6 unique bytes? What about 5 or less?
People teamed up. I only provided the standings to people that had submitted to account for stealth teams that lurked while fine-tuning their algorithms. The contest became about writing a decoder and managing the length of your encoded shellcode.
Early in the contest they got down to 4 unique bytes. Then a few got to 3 unique bytes. There was some drama. The 3rd place algorithm led the contest for a long time, but at 11:36 PM on the last day of the contest, the winning team submitted twice for first and second place.
The winners were Michal C. and Sergiusz F. If you know them, you won’t be surprised. You can find their entry here: https://gist.github.com/JohnLaTwC/d2c3e7f54e256aa2fd5ce4b86a1d6d54. This is it rendered as a bitmap and in #Ghidra.
One brilliant entrant submitted a 2 unique byte shellcode entry that only worked on Windows 2000. I was unable to verify it. The encoded payload used only 0x40 and 0xff and was over half a megabyte.
Many thanks to @fjserna @brucedang @markwo @epakskape @BruceDawson0xB and all the others for the memories and team camaraderie! Thanks to @epakskape for poking me to unearth this little bit of Trustworthy Computing lore.
This story was originally published here: https://twitter.com/JohnLaTwC/status/1107380877762260992
