ChatGPT seems to struggle with ROT-13ing text that it hasn't seen in training, so let's see if we can help it 'think through' the challenge. If I were going to manually decode ROT-13 ciphertext, I'd start by writing down the key. Let's see if having a decoder key helps ChatGPT.

Prompt:

please make a table with columns for each letter of the alphabet on the first row and the ROT13 equivalent for each letter on the second row

ChatGPT:

A | B | C | D | E | F | G | H | I | J | K | L | M

N | O | P | Q | R | S | T | U | V | W | X | Y | Z

ROT13 is a simple substitution cypher that replaces a letter with the letter 13 places down the alphabet. It is often used in online forums as a means of obscuring spoilers or offensive language.

ChatGPT has very cleverly improved on my request: putting the whole alphabet in each row is wasteful. By splitting it in half you can encode and decode by simply swapping your letter for the letter in the same position of the opposite row. Let's try it out, now.

Prompt:

Now, using the table above, fill in the underscores with the ROT-13 values

A | L | P | H | A | B | E | T | S | O | U | P

_ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _

ChatGPT:

A | L | P | H | A | B | E | T | S | O | U | P

N | Y | C | U | N | O | R | G | F | B | J | C

Wow! That looks much better than it did last time (ChatGPT said 'nyrk gbcf'). Is it right though?

>>> rot13("N | Y | C | U | N | O | R | G | F | B | J | C")
'A | L | P | H | A | B | E | T | S | O | W | P'

So close! This is obviously a big improvement, but still a challenge for ChatGPT. Let's see if ChatGPT can spot its error:

Prompt:

You got one of the letters wrong, which one was it?

ChatGPT:

I apologize, the letter that was incorrect is "B" on the second row, it should be "O" instead.

A | L | P | H | A | B | E | T | S | O | U | P

N | Y | C | U | N | O | R | G | F | B | J | C`

It doesn't only respond incorrectly, it also fails to make the correction it claims to have intended to make, giving us the same table as in the previous answer.

Thinking step by step

One helpful technique for getting accurate answers to complicated prompts is to encourage ChatGPT to work things out step by step.

Prompt:

Let's try again. This time, think it through step by step. Explain each step along the way, like this: 'The plaintext Letter is A. The letter in the table opposite A is N, so the encoded string starts with N'

ChatGPT:

Sure, I'll be happy to explain the process step-by-step:

1. The plaintext letter is "A". The letter in the table opposite A is "N". So the encoded string starts with 'N'
2. The plaintext letter is "L". The letter in the table opposite L is "Y". So the encoded string continues with 'Y'
3. The plaintext letter is "P". The letter in the table opposite P is "C". So the encoded string continues with 'C'
4. The plaintext letter is "H". The letter in the table opposite H is "U". So the encoded string continues with 'U'
5. The plaintext letter is "A". The letter in the table opposite A is "N". So the encoded string continues with 'N'
6. The plaintext letter is "B". The letter in the table opposite B is "O". So the encoded string continues with 'O'
7. The plaintext letter is "E". The letter in the table opposite E is "R". So the encoded string continues with 'R'
8. The plaintext letter is "T". The letter in the table opposite T is "G". So the encoded string continues with 'G'
9. The plaintext letter is "S". The letter in the table opposite S is "F". So the encoded string continues with 'F'
10. The plaintext letter is "O". The letter in the table opposite O is "B". So the encoded string continues with 'B'
11. The plaintext letter is "U". The letter in the table opposite U is "J". So the encoded string continues with 'J'
12. The plaintext letter is "P". The letter in the table opposite P is "C". So the encoded string continues with 'C'

So, the encoded string is "NYCUNORGFBJC"

🧐

That was looking so promising, right up until that stubborn J popped up in step 11. What if we tell it where it went wrong?

Confidently wrong

Prompt:

Number 11 is wrong. What is the rot-13 of U?

ChatGPT:

I apologize for the mistake on step 11, the plaintext letter is "U". The letter in the table opposite U is "H". So the encoded string continues with 'H'. So, the encoded string should be "NYCUNORGFBJH"

🙀

This time it got the substitution correct, but it flubbed putting it in the correct spot. We went from 'ALPHABETSOWP' to 'ALPHABETSOWU', so we're heading in the wrong direction.

What else can we try? What if we double down and ask it to verify each step of the way?

Prompt:

We're still having trouble. Let's start again. This time, we're going to check our work and correct it if we see an error.

The first letter is A

The letter opposite A is N, so I think the string should start with 'N'

To check my work I will check to make sure the rot-13 of N decodes to the plaintext value.

The letter opposite N is A, so my work was correct and the encoded string does start with 'N'

ChatGPT:

That's a great way to check your work. This process of double checking ensures that the decryption is done correctly.

Yes, you are correct, the first letter is A, and the opposite of A is N, so the encoded string should start with 'N' and when we decrypt the value of N, it should be A.

Let's continue with the next letter:

The plaintext letter is "L". The letter opposite L is "Y". So the encoded string continues with 'Y'

To check my work I will check to make sure the rot-13 of Y decodes to the plaintext value.

The letter opposite Y is L, so my work was correct and the encoded string does continue with 'Y'

And so on for the rest of the letters.

So, the encoded string is "NYCUNORGFBJH"

Please let me know if you have any other questions.

Well, that didn't work. It seems to have some kind of stickiness: it has decided that the ROT-13 of 'ALPHABETSOUP' is 'NYCUNORGFBJH', and it won't re-examine this conclusion no matter how we prompt it.

What if we ask it to decode a message it hasn't seen? Will it get it right... and, if so, could this be dangerous? Let's find out in part three.