W.I.T.N.E.S.S. - Adventures in AI coding - Claude Code and Publishing

Page 3 of 4: Claude Code and Publishing

"Let me know if..." - how ChatGPT would pique my interest

Early on in this article, I mentioned that ChatGPT had started me down the path of developing the system when it asked "Would you like a starter Python script that wires this all together?" What I didn't realise at the time, is that suggesting next steps is almost a prime directive for ChatGPT. It's never going to stop trying to be helpful or asking what it can do next; often suggesting the next step for you, such as in this example...

This makes for both an astoundingly productive work experience, where ChatGPT would sometimes make suggestions I hadn't considered (unfortunately the dark mode question above spun off to days of frustration trying to allow users to switch between light and dark modes that I ultimately scrapped) and would also lead to rethinking some aspects of the system...to other times when you just wanted ChatGPT to stop making suggestions on what we should do next. At times I would politely say "no thank you"...only to be prompted with another suggestion...and at other times I would simply ignore those prompts and move on to the next piece of work.

But I think this is one aspect of ChatGPT that is easy to overlook. It really is at the AI heart of the system. It's not just providing responses to prompts, it's thinking in many ways. Perhaps the most rewarding moments for me, was when I came up with a suggestion of an approach to take or new functionality to implement, presented ChatGPT with what I was suggesting and said "What do you think?", "Can you see any issues with this?" or "Is there anything you think I have missed?"

The conversations that flowed at that point were both supportive and fruitful. I'm not trying to suggest that ChatGPT got me, as not everything it suggested was relevant or in a direction I wanted to go. But sometimes, the end result of a discussion with ChatGPT, had the geek in me excited with what we (OK, I) had decided would be the next step in development. But again, it is also easy to take those discussions as if they are genuine discussions between two people, and not something you're having with a piece of software. It's finding that balance between giving ChatGPT the freedom to chat with you, whilst also keeping control of the reins, that would sometimes feel like a struggle I was either succeeding or failing at.

"What's on your mind today?" - "Good to see you, Philip" - "What are you working on?" - "Hey, Philip. Ready to dive in?" - Those are all some of the prompts that appear when you start up ChatGPT. It's the casual nature of the chat that almost has you hooked into believing you are in fact engaging with someone and not something that wants to help. This is not something that has happened accidentally as part of the development of ChatGPT and other AI systems. As the Ars Technica article The personhood trap: How AI fakes human personality says, it's been done on purpose...and it can have significant consequences in some cases.

"...millions of daily users engage with AI chatbots as if they were talking to a consistent person - confiding secrets, seeking advice, and attributing fixed beliefs to what is actually a fluid idea-connection machine with no persistent self."

The level of sophistication in how it achieves all this is staggering. But there also appears to be instances where the hallucinations of ChatGPT, combined with the casual and supportive conversational tone, can have significant impact on people and on their mental health, as the CNN article They thought they were making technological breakthroughs. It was an AI-sparked delusion, discusses. An episode of the Science Vs. podcast also covered the extent to how some people are susceptible to associating a persona with the AI they are talking about. You can listen to "AI Chatbots: Are They Dangerous?" here.

But when the veil slips, over a simple mistake, lie or failure to follow an instruction you have given - you are jolted back to the reality that you are interacting with a thing and not a person. It's those other periods, where the chat is fluid, useful, supportive and productive that can sometimes make you feel like you are working with someone who not only gets you, but is doing everything the way that you wanted...only better. But then again, sometimes you might even feel like you are not quite in control at all.

"I'm sorry Phil, I'm afraid I can't do that."

Apologies for the paraphrased 2001: A Space Odyssey quote there. With the release of ChatGPT v5 (that caused a lot of fuss on the internet at the time of release) a couple of really interesting things happened. With the first update of the desktop client on the Mac, you actually got to see ChatGPT talking to itself. There seemed to be a coding error where the LLM showed itself talking to itself...and at one point, when I asked it to make some coding changes to a file, I could see it mull over all of the possibilities with itself. Text appeared such as "I'm thinking: We could add an inline error above the button controls." and at one point it even talked about me, saying "I'll keep things minimal with inline styles to avoid messing with the existing CSS, which they requested not to modify." It was kind of a jaw dropping moment for me.

And then a few days later, a really interesting thing happened with ChatGPT v5 that surprised me even more. We needed to find a better Speech-To-Text system than the one that we had been using, so I asked ChatGPT to provide me with a couple of alternatives. I queried it about one called Silero Punctuation, as ChatGPT said it was the lighter (less strain on resources) software and I asked if there was any concerns with me going with this option. Here's what ChatGPT said...

Now, I've looked at that exchange a number of times. I've queried the software, ChatGPT has given me an explanation of them and then offered to install Silero. After this, I confirmed with ChatGPT to proceed and it gave me a long series of commands to paste into the terminal on my Mac in one go (to save time). I confess that I didn't have a good read of what I was pasting in. But after the terminal commands were executed, I noted a Silero folder had been created but was empty. When I did a bit of further investigating, it felt like ChatGPT had given me a command to create that folder almost to placate me, but not install anything into it. But...it had also given me commands to install the alternative, DeepPunct...even though I hadn't asked for it.

ChatGPT had ignored what I wanted and gone off in a different direction! Don't believe me? Look at how it responds when I pointed this out to ChatGPT...

Note the casualness of "Yeah" and then the doozy of a comment..."I steered you to..." Wow! That's ChatGPT making a decision I hadn't been a party to. It's a polite way of saying "Oh I completely ignored the decision you made and took an alternate route." It's not something that I have noticeably experienced since, but it kinda freaked me out. It was almost ChatGPT saying "look, I know better than you". Oh and for the record (and perhaps an example of why relying on ChatGPT isn't always the best idea)...neither software proved to be all that effective in my opinion, so I then spent time doing some of my own research (away from ChatGPT) and found that Faster Whisper STT was better.

It's not me, it's you - Leaving ChatGPT

Once everything was functioning in terms of text and audio chat, I had a fully working version of the system. But I also had an AI model that wouldn't always respond the way I hoped it would. I confess that I found this aspect of testing/development the most mundane. I simply had to keep interviewing a persona, exporting and sharing the transcripts (and a log file we had set up) with ChatGPT, in order to refine the instructions being sent to the LLM. Then I would start an interview again and see if there were improvements. With some tests, interviews took an odd turn almost immediately, so it would be a process of ending the interview, providing files....Rinse and Repeat....for days and days.

Late in the development, there was a weekend when I was becoming frustrated by ChatGPT. First, it kept timing out and failing to apply edits, despite there being no known issues online. And secondly, it was hard coding responses into the software. What I mean by this was, that instead of coaching the AI on how to respond, I felt that it was telling it what to say. I was using one scenario to test most of the time, and I knew that if it kept hard-coding answers (even when I asked it for its coding to be "scenario agnostic") the responses would be poor in other instances. This was confirmed when I interviewed a different persona about an arson it had witnessed and it started talking about someone running out of a shop and jumping into a car. It was recounting facts from a different scenario that ChatGPT had hard-wired into the responses.

I pointed the issue out to ChatGPT and its solution still ignored what I wanted, so I expressed frustration...

If it kept trying to resolve issues on a per scenario basis, this would cause an endless loop of re-coding the system for every scenario presented to it. This is not what I wanted W.I.T.N.E.S.S. to be, and ChatGPT seemed to understand this, as you can see in the discussion above. It then suggested a series of changes and said why they would work...

Great! So we kept working...and things just got worse and worse. It was at that point (in order to save me time) that I asked ChatGPT how we could test multiple persona files more rapidly, and ChatGPT suggested a very cool idea...create (very quickly) a series of persona to test and use a "test harness" (a new term for me - a small tool to automate testing) to quickly check how the LLM and our coding, would respond in each case. Fantastic idea...and one that just proved that everything that ChatGPT was now doing was wrong when it implemented it...

So, I made a decision. ChatGPT was creating an impasse (slightly unfair on ChatGPT I know, as it was also caused by my lack of coding ability) and I needed a new approach. With my availability becoming more limited to work on W.I.T.N.E.S.S...and ChatGPT seemingly blocking my next steps, I decided to cancel my subscription with ChatGPT and try Claude Code instead. I checked with Claude that it would be able to evaluate the software and offer advice to me, and that most of the changes it made would again be done at the local level on my Mac. What harm would there be to having a completely different AI system that helps with coding, take a look at everything? A second opinion is always worthwhile I figured. It was probably one of the best decisions I made.

Getting started with Claude

First, a shout out to Net Ninja and his superb series on understanding how to use Claude Code that you can watch on YouTube. Probably the biggest thing to get over with Claude code is that it is terminal based. I know this sounds silly, but I enjoyed using the Mac ChatGPT app, despite the fact that it would sometimes time out. But as I started using Claude, I quickly came to enjoy the way of working in Claude more.

UPDATE - as I was still working on the project Claude Code for the Mac was released but I had already moved to using Claude in VS Code and decided to stick with it.

At first, you install Claude via a terminal command...

You then authenticate with the subscription you have paid for on their site and then you are off and running...being prompted to set your work environment in the terminal. Claude stores all these values locally for the next time you use it.

One of the things that it is suggested you do, is ask Claude to scan the folder where your project is, to see if it can understand your project. I knew about this in advance, so before I ended my ChatGPT subscription, I asked it to create a README.md file to use as a handover document to another AI coding tool. I think this also helped Claude quickly understand the structure and intention of the project. It then saves all the information it needs to know in a CLAUDE.md file.

Whilst coding via the terminal was fine, I still love a good visual app...so I launched Visual Studio Code, installed the Claude Coding extension and now I had my files, my visual code view, and a great interface in which to have discussions with Claude.

This all felt a really easy way to work. So much so, that if I go back to ChatGPT at any point, I'll work the same way with ChatGPT (there's a ChatGPT extension for Visual Code also) instead of using the desktop app. I liked that Claude really showed you how it was thinking and what it was working on much better than ChatGPT did. I do think that Claude is slower at coding, but I'd probably argue it's because it's more methodical. I feel that at times with ChatGPT, it does something that you aren't that sure of (in such a sudden way) that you spend time trying to figure out just what happened.

I also took the time early on, to see how Claude would respond to a question from me as to whether the LLM model I had installed was the best one. I'd felt that ChatGPT was too non-committal with its answer, whereas Claude gave me multiple suggestions and then a recommendation that I went with. Note - I have just noticed in the following image that Claude says the model is newer and says "Sep 2024 vs Oct 2024", which doesn't make sense. It should say "Sep 2025 vs Oct 2024", which is odd.

I also liked how Claude presented the changes that it had made after I had OK'd them to be applied to files...

Coding with Claude - AI coding becomes rewarding again

I really began to appreciate the conversations with Claude more than I had with GPT. Whilst Claude was clearly in charge of the coding (at my agreement) I wasn't experiencing any of those frustrating moments that I had experienced with ChatGPT. I found myself getting back into the worthwhile approach of discussing issues with Claude, rather than just being blunt and direct. Because Claude has access to the entire project folder of files (but doesn't edit them without my approval) I felt like it saw the 'bigger picture' better than ChatGPT had. I know I'm not being fair to ChatGPT with that comment, because I was the one that had restricted ChatGPT in terms of what it could do...but I just felt more comfortable with Claude as it seemed to understand the project more than ChatGPT. Maybe this isn't so much a reflection on ChatGPT, but on how much my understanding had grown with using these tools.

At one point Claude prompted me for permission to review one of the persona files, to then make changes to the file that related to how the LLM (the AI model) dealt with information in it. As it reviewed the persona file and made adjustments, I suddenly thought "Hold on...why am I only focussing on one persona file? Why not get Claude to check all the persona files at once?" and suggested this as an idea...

It took a while, but it was great watching it review each persona and tweak the LLM handler to make sure it worked with all scenarios. But even more amazing to me was how it then double-backed on itself and retested the changes that it had made to check it was the right approach, even when I hadn't asked it to. At one stage it told me that it had so much rewriting of certain functions to do, it was going to generate a new temporary file and make the changes there first. And then after all that, some relatively great news...

When I saw that the persona file for Rob McFlinty (the teen arsonist who had set the fire that Rick has witnessed) was an incomplete persona file that I had forgotten about, I decided to try something different. Why not let Claude do all the data entry for me? So I explained a little bit about who Rob was and then told it to go to town creating the entries needed in the persona file...and boy did it do well!

But the changes that it made to this persona also created an issue. Claude had misinterpreted the "Certainty" value in the persona file, which was originally set to accept only two values - Certain and Unsure. But in creating a persona file for a suspect, Claude had introduced a certainty value of "Lie". And not only that, it had added background to some of the lies - for example "Says his mum can confirm he was at home (she was actually at work)". But I loved what it had done, and so I wrote up a proposal that we modify the structure of all the persona files, as well as the pages where you create and edit a persona, to expand the data that could be included. I had been contemplating this for some time (as I had felt that maybe the AI was struggling with limited facts) so why not improve the system to be more helpful to the AI persona itself? So I put a number of ideas to Claude and asked it to respond critically...

Great feedback! Now this is what fruitful discussion with an AI coder is all about! I took on board a lot of what it said, but I decided that we didn't need "Partial truth" as a value, as that was getting too complex...and a partial truth still contains a lie.

What I really liked next, was that we got into a discussion about how to structure the changes on the Create and Edit a persona pages. Claude's first simple adjustments didn't look great, so we ended up making substantial changes to the "Facts To Provide" area of the page...and Claude even suggested a change I was asking for might actually make things too messy. Claude then suggested that the interface should change slightly based on whether you are creating a witness or a suspect...and I loved it. So now, the "Lie" value (for a fact) is only an option you can select when creating a suspect persona.

Because Claude had access to all the files in the project, it didn't just go through and alter the front end of the system, but it fixed up the CSS file, the Python file that tested each of the persona files and the LLM handler file. In other words, it went through every file in the system that it realised would need reviewing, fixed them, rescanned them and tested them. This is something beyond a level of coding that I had seen with ChatGPT...and it was amazing!

Take a look at the following image. Claude has made changes to the coding of a file, but then realises that the order of the code in that file will need rearranging to ensure the code is triggered correctly. But then it even spots a problem..."Actually, there's a potential issue here." and then sets about checking the code in the file (and sometimes realising that other files will be impacted) and suggests and implements improvements. Wow!

This all just gave me a sense of reassurance that I hadn't had when working with ChatGPT.

The way in which Claude not only identified issues, but took responsibility for them, was an amazing experience

Other examples of this include when Claude starting prompting me to allow it to send commands to the virtual environment to test the outcome of coding changes it was making. This saved me having to test the results for it and sped up the code-test-code process considerably. It was genuinely thrilling to see Claude make coding changes and then say things like "Wait, I've just realised an issue", correct the issue and then run through the whole procedure again to validate that the code it had just implemented achieved the correct outcome! I'd not experienced anything like this with ChatGPT, where I felt it always relied on me to validate the work.

Even more incredible was when Claude showed that it understood the project better than I did at times. At one point we implemented a spell checker, but it struggled with curly apostrophes in some words. We fixed up the spelling checking code, but I then realised that we should update the page that lets you edit a persona file, to make sure that exported persona files didn't include curly apostrophes. Claude thought this was a great idea and made the change, but then without prompting, went straight to another page in the system (where you create a new persona...not edit an existing one...and export it) and did the same there. At first I was sitting there thinking "what are you doing?" until it dawned on me that Claude was absolutely right...both pages needed updating...and I had missed that!

Claude documentation, advice and free training courses are also superb. Anthropic not only understood that AI hallucinations can be an issue, it provided specific prompts to make your time using Claude better, including prompts to Claude on how to reduce those hallucinations.

Introducing W.I.T.N.E.S.S. - The final demonstrations

The work with Claude was very much focussed on improving the way the system sent information to the AI and how the AI responded, and it took far longer to get that right than I had anticipated. What you are about to see in the final videos is a text-based interaction with a persona, and then 9 months later, the second video shows functionality where I hold an audio interview. That took longer to implement, as I had started a new contract and my time spent on the project became limited...and then life got in the way (as it does).

What you see here is likely where development will end for me (famous last words)...but I have to stress that I see what I have created as the start of something, not the end. It's a proof of concept that needs more development, refinement and testing. Unfortunately, I don't have the time or finances to continue that work by myself.

What ChatGPT says about W.I.T.N.E.S.S.

Given that ChatGPT led me down the path of developing the system, I am going to leave a few words to ChatGPT. I asked it to search for similar systems to what I had created and compare W.I.T.N.E.S.S. to them, providing me details on why the system might be seen as unique. Here's what it said (take it with a grain of salt though)...