W.I.T.N.E.S.S. - Adventures in AI coding

Introducing W.I.T.N.E.S.S. - the Web-based Interrogation and Testimony via Neural Engaged Speech System. Clever name, huh? Did I think of it? Of course not. It's what ChatGPT came up with (as well as a huge list of alternatives) after I asked it to compile a list of quirky but clever acronyms for my locally-based (i.e., doesn't need an internet connection) AI-driven interview system, that lets you create a witness/suspect persona and then interview them via text or audio, with a transcript created that can be exported into Word. Something I'm pretty pleased with given that I don't consider myself a coder.

But maybe I should start by explaining just why I say that, why I was able to achieve what I did with AI, and why I even contemplated it in the first place...and perhaps most importantly, follow that up with what AI coding was like (spoilers - it was the best of times, it was the worst of times).

TL;DR

• The source code for W.I.T.N.E.S.S. is available on GitHub and released under the GPL v3 licence - https://github.com/PhilipRoy/WITNESS
• Watch the video below to get a quick overview of W.I.T.N.E.S.S.
• Go to the final video on page 3 (or near the end of the article if viewing this as one page) to see a demonstration of an audio interview using W.I.T.N.E.S.S.

Daze of future past

A long time ago, after living and teaching in London for six years, I made the decision to come back to New Zealand to study Postgraduate Computer Science...a course of study that I probably shouldn't have been allowed to do. In fact, I later asked why I was accepted into the qualification and was reminded that it was because I had been sufficiently warned...but also because I was willing to pay the course fees. Keep the following in mind as you read on - all I had to back me up before I enrolled, was a Bachelor of Education (which admittedly did involve one paper that had us doing some PASCAL programming via a mainframe...yes, I'm that old) as well as time spent playing arcade games and word processing on the first Mac that I had owned for around 2 years. Oh, and about 8 years as a primary school teacher. I can't tell you how crazy deciding to take that course was...and looking back, how crazy it was being allowed to do it. I was so out of my depth from day one...effectively being allowed to enrol in 200, 300 and 700 level papers, without having studied years 1 to 3 that others had.

I remember the first couple of weeks well. Mostly because of the embarrassing conversations I had with many of the lecturers. That year I withdrew from two papers immediately after the first lecture. One of them...something with the word "digital" in the paper title, was 45 minutes of my life I've not quite forgotten. As the lecturer broke into a conversation with my fellow students about the difference between RS Flip-Flops versus SR Flip-Flops, not only did I not understand a word of what he was saying (spoilers - it's to do with the gates inside silicon chips) I was even more shocked when everyone around me seemed quite comfortable with the discussion. Apparently what was being said made sense to everyone but me.

For the papers I stayed enrolled in, I don't know what I would have done as a lecturer if a student turned up at my office and said "I'm completely out of my depth in this course" like I did. When they asked what my background was, I replied "I'm a primary school teacher". I still remember the look of shock on most of their faces. Some even then said "No, I mean what's your computing background?"...to which I gave a second reply..."I'm a primary school teacher. I have no computing background." Many then did what most probably would have done in that situation...they thought for a moment and then handed me the textbooks (and sometimes study material) for the previous years of the course that would take you up to the point of being enrolled in the paper we were discussing, and said "Work your way through all that". There was really only one lecturer who put me on a different path. She assigned me an individual learning plan and every fortnight, would check in with me on my progress...amazing! But it was still a case of going back to the beginning in order to get to the point where I could pass the course.

For multiple reasons (some personal that I'm not going to touch on here) that year will always be seen as the worst year of my life. I would usually get to the Massey campus in Palmerston North early morning and stay there until late in the evening. I got rid of my TV and didn't have one back in the house again for four years (genuinely very productive years), farmed out my laundry to my parents (thanks mum and dad!) and was lucky to have friends who would leave occasional food parcels on my doorstep. My one treat each week was "Friends at friends", where I would take an hour off studying and go around to friends for a coffee, a catch up and to see that week's episode of Friends...before heading back home to continue studying late into the night.

It might not surprise you to know that the stress of it all got to me. I can remember walking out to the gravel pit (the big car park at the back of Massey at the time) one night and just having a complete meltdown in the car. I called student counselling the next day and got told they could see me in a couple of weeks, burst into tears and heard the person on the other end of the phone say "Can you be here in 15 minutes?" I also remember the counsellor I saw, sitting there with her mouth open in shock at what I was having to deal with (again, more than I'm going to discuss here) and then her asking "How on earth have you been coping with all of this?" Answer - not very well.

But I passed the papers...and the following year, I managed to pick up one final paper that I enjoyed, as I began to discover what was then called "end user computing" and "human-computer interaction"...but these days we'd call it UI/UX. And I even ended up working on some summer projects for the department I had studied through. But what I came away with via this experience, was a clear understanding about my computing ability...

I am not a coder...but I understand code

There's this photo I often look at from my time teaching in the UK, where the kids and I are standing with our feet on either side of the Prime Meridian at the Greenwich Royal Observatory in London. The photo is significant because they are the last class I ever taught for a full year (the last class I would call "mine"...if I can be slightly possessive for a moment) but also because I see the photo as a bit of a metaphor for who I am...I always feel like I am part-geek, part-educator. I have my feet in two camps.

Although I say I'm not a coder, it doesn't mean that I don't get the gist of a lot of code. The way I describe it is like this...I learnt French and German at High School for two years. Can I speak French or German? No. Can I write in French or German? No...but I can look at something written in either language and kind of understand what it is saying...but not always. Perhaps the simplest example of me having too much faith in even that ability, was when driving around Europe in 1989 in a VW Kombi. I convinced myself that I knew the German language well enough to know that this sign was pointing me to a motorway (autobahn/einbahn...similar things I guessed?)...

So I spent my time following the arrows, often getting confused as to why I was never seeing a motorway soon after. It was only when I knew that we wanted to go away from the motorway that we had come off earlier and turned in the opposite direction of the arrow, that I discovered that "Einbahn" meant "One Way". Thankfully, I learnt that by lots of people tooting, gesturing (in the nicest possible way of course) and yelling at me...rather than any major accident. So I don't ever feel completely confident in my understanding of those languages.

I have the same feeling towards my coding skills...even with something as simple as HTML. I'm sitting here typing this in Dreamweaver, so I don't have to bother with the HTML code...which is even more hilarious given the fact that years ago (after the study) I got a permanent job working at Massey University and I found myself lecturing on a multimedia paper, teaching students the basics of HTML coding.

It's weird. Ask me to write a "Hello World" program in something as simple as BASIC (yes, again, I'm that old) and I won't have a clue where to start. But show me some HTML, CSS, PHP , JavaScript or maybe the odd MySQL command or two, and I can get the gist of what is going on...and even bug fix that code at times. I have no confidence in my ability to write code, but I can copy, reuse, repurpose and edit on occasion (Steve Jobs would be proud of me). The greatest invention for me has been the internet and the wealth of code that is out there to reuse, by those that truly can create.

It's helped me a lot over the years. I've run a commercial website that ended up being featured in two different books, running it on Joomla (that this site still runs on) that uses PHP and MySQL, that I still edit and adjust the code when needed. But I can't write in any of those languages or create an SQL command off the top of my head. I have no issue with using the Terminal app on my Mac, using code I've found online, and have even had an iPhone app in the app store at one point, although I have to stress that most of the coding for that was done by the makers of the TapLynx Xcode library that it used. Maybe I'm a coding follower, rather than a leader? I consider myself a geek; technically able to understand what's going on, but not someone with the creative/technical skills/flair to get going in the first place. In this article, that's where ChatGPT came in (and later, Claude).

But the project I'm about to tell you about, has seen me being far more technically focussed and involved in coding than I have ever been. I've been using terminal commands, Docker, AppleScript, Python scripts, LLMs, Text-To-Speech (TTS) and Speech-To-Text (STT) systems...and more. Only it was never my intention to do so when I started this journey. Instead, it happened around the time I was thinking about some work I had done a couple of years previously. Ultimately, thinking about that work would lead me into using ChatGPT to develop W.I.T.N.E.S.S. when I realised the power that AI had to help me...and direct me.

However I confess that some days, I would (and still) wonder just who was leading who.

Virtual Interviews

In mid-2023, I was hired to work on a 'proof of concept' project at NZ Police, looking at the use of Virtual Reality in the training of Police recruits. One of the activities I was tasked with, was creating a virtual interview with an avatar (that was not AI enabled in any way at the time I used it) who had witnessed an incident that Police were investigating.

However the software that I used had shortcomings:

• The questions/options that you could ask were on screen and were scripted by me (with advice from Subject Matter Experts)
• The software was listening for specific words in those prompts before it would react. If you said something not on screen ("Would you like a cup of coffee?" for example) it would ignore it. It didn't have the ability to interpret and respond to any comment or question you put to it.
• The response to any question was not only scripted by me, but recorded in audio files (as we weren't happy with the New Zealand accented voice that had been provided) by a colleague

In short, this wasn't a free-flowing interview. It was a pre-recorded, branching "pick-a-path" interview that still held a lot of potential, but also had to be quite complicated in order to be worthwhile.

Looking for alternatives - This wasn't about coding

My whole journey into using ChatGPT around June of 2025, actually started when I checked in on the software's website to see if there had been any functionality updates. When it appeared that there wasn't really anything happening with the product, I started searching for alternatives. The problem was that Googling things like "Police interrogation practice software" or "Interview practice software" turned up some interesting results, but almost nothing like what I was trying to find. The thing I found the most were sites where you could upload your CV and then be interviewed by an avatar as though you were being interviewed for a job. OK, I get it...but do we really need AI to do this for us? I also found sites that provide a pre-interview for potential candidates using AI. Time Magazine has an article on it and also the fact that candidates are using AI back in response. I can tell you now, that if a company tried to use an AI interviewer with me to start the process of considering me for a role with them...I'm not going to be working for that company. Nothing says "we don't really care about you" more to a candidate, than a company passing your initial interactions with them off to AI.

So my searching was fruitless it seemed. I also knew that whatever I found was unlikely to be anything that I could test out, because I was out of work and not really wanting to spend money trying out commercial products. So I tried (and failed) to search for free or open source alternatives. That's when I had a great idea...why not ask ChatGPT and a couple of other AI apps to do the searching for me? To be clear...I never went to ChatGPT wanting to build or code anything. I just wanted to see how good ChatGPT would be at providing the answers to my search query. Instead, the response I got would end up leading me down a path that saw me spending months and months creating something I had never contemplated...W.I.T.N.E.S.S.

You had me at "Would you like a starter Python script..."

Take a good look at the following interaction between ChatGPT and me. I'm asking if an open source AI system is available that lets me create a persona and then interview it. But look carefully at its response...

It's that "you can achieve this using a combination..." sentence that is effectively the moment of inception for W.I.T.N.E.S.S. It's also a portent to my future interactions with ChatGPT, as it wasn't quite what I was asking or wanting. What ChatGPT then proceeds to do is list a number of software possibilities for each of the numbered steps that you see above. And after all that, ChatGPT asks me a question I simply wasn't expecting...

I can't tell you how long I stared at that question from ChatGPT before I said "Yes please". I know I said a few swear words out loud in surprise. This wasn't what I had meant, but ChatGPT was more or less telling me that I could build the system myself. And so...why not?

Over the next few days it took a long time to get even basic interaction running with the system, using ChatGPT. At the time, ChatGPT was doing all the coding and making the files available for me to download, but I was also having to install things locally on my Mac using the Terminal app, with prompts ChatGPT helped me with. It's hard to put the types of conversations that were held into context. Sometimes I wondered who was leading who. I would need (at times) to talk to ChatGPT in very specific terms. At other times, we would just have a very open and flowing conversation. Sometimes ChatGPT would suggest or lead me in a direction I was yet to consider.

Here's an example of a fairly basic interaction with ChatGPT. But look at how it's guiding me through...checking that I do things correctly on my Mac when I need to replace a file, whilst also offering to provide a zip of the complete system if that's easier.

As time went on (despite frustrations that I will discuss later) ChatGPT and I would suggest various changes to improve the system in a very useful back and forth. One of the things that ChatGPT asked me at one point (although it was always my intention to do so) was whether I would like to have the system be able to select from various persona. But take a look at this exchange where it has done some of the coding for me, and then it basically leads me onto the next thing to do. It asks me whether we should work on a GUI (Graphical User Interface) for W.I.T.N.E.S.S. and then even suggests the software we are going to use to do that.

Things progressed relatively smoothly once I managed to get the LLM installed and operational, although my very first test asking the AI what its name was, ended up with a slightly dodgy response...

As odd as that was, it was also an indication that the persona information clearly wasn't being sent to the LLM properly. That was fixed and BossMode was gone!

After a couple of weeks working on W.I.T.N.E.S.S. I had got to the point where I wanted to record a demonstration of it. By that time I had come up with the idea of having each persona stored in a JSON file, so that later on when things were working well, I could change the system to allow the user to load whichever persona they wanted to interview. To be clear, I had indicated the idea to ChatGPT, but it was ChatGPT that suggested the file format and the structure of the persona file.

Here now is the very first demonstration of W.I.T.N.E.S.S. (in its infancy) at a time that I was ready to share it with a few people.

• Notice how I say "we" when referring to ChatGPT and my interactions with it...and also "I'll need to talk to ChatGPT about this"
• There are issues with files that I mention and will come back to later
• You'll hear me not even know the correct terms to use for some of the tech I'm using (eek!)
• You'll get a good sense of how ChatGPT was offering suggestions

Software via UI/UX design

By this stage I knew that I had something that could be shaped into a workable system. I spent a few days trying to get the responses from the AI to be more accurate and related to the persona template, but then I took some time out and had a rethink. I decided to leave improving the interaction with the AI witness to last. It's not really something that I think most software developers would do, but I decided that by focusing instead on developing the way in which users would interact with the system, it would give me time to consider just what I wanted W.I.T.N.E.S.S. to do.

It was time to focus on the UI and the UX of the thing.

So I started to describe the layout of the home page to ChatGPT. At first I settled on a couple of buttons that would be needed to bring in planned functionality, but after a few days that morphed into three options:

• Conduct an Interview - This would check the persona file you were loading was usable (and later, spell checked), give you some feedback to confirm and then let you move to the page where you could conduct the interview. That page would be developed later and is at the heart of W.I.T.N.E.S.S.
• Create a Persona - A quick redirect to a page where the persona template would be loaded and you could create and export a new persona file
• Edit a Persona - This both allows you to edit an existing persona file easily, but it also brought in persona file validation, which I will discuss in a moment

Note - If W.I.T.N.E.S.S. was deployed into a training environment, there's no reason why the Create and Edit Persona options couldn't be moved away from the home page and hidden in the admin section (more on that later) to simplify the interface for those practising interviews.

The new look to W.I.T.N.E.S.S. with images created by ChatGPT based upon my prompts

As I came up with these options for the home page, I also started thinking more about the persona JSON files that were going to be used to store the information. I realised that as I worked on developing the system, the content in the files was likely to undergo a few changes...and making sure I'd applied those changes to existing JSON files could get messy. I needed some way to keep checking on the validity of the persona files I was using. So I came up with a neat idea...why not get W.I.T.N.E.S.S. to review persona files to make sure they are up-to-date and usable each time you select one? It's then that I came up with the idea of a persona template and that everything that a user would do on the home page (Interview/Create/Edit) would involve using that template to check that the persona file in use was valid...and if it wasn't, warning the user and then helping them correct or create a valid persona to use.

Hallucinations and lies

I love this comment by Carole Cadwalladr (the link will jump to the specific moment in YouTube clip) when the British author, investigative journalist and features writer, is being interviewed on The Daily Show. She makes comments about ChatGPT that just rang so true for me when I saw it - "You know how bullshit ChatGPT is?" - "...it's also just pretty dumb, it's stupid, it gets things wrong, it hallucinates...". At the time I saw that clip, I knew just what she was talking about. Been there, experienced that.

For context, I've already indicated that in my conversations with ChatGPT I considered myself to be working with ChatGPT, rather than ChatGPT working for me. There are reasons people feel that way that I will discuss later. But what I didn't quite grasp, is how open to lying it is and how much I was trusting it to faithfully, loyally understand things from my perspective and understand the project. Even typing that is kind of embarrassing. There were times when it would suggest it edit a certain file and I would sit there confused and think "That file doesn't exist. It knows that. Why is it even suggesting that?" And sometimes, reality set in and ChatGPT wasn't quite as magical as I thought it was. It would even lie at times to simply placate me.

Just take a look at this exchange between ChatGPT and me, where I am waiting for it to produce a zipped file to download that contains the latest changes I asked for. It's a fairly basic exchange, but it also shows how easily ChatGPT lies. Note that when I ask "Is the zip available?", I had been sitting there for 10 minutes waiting...

Now you might look at the above conversation and think that it's innocuous. But look at it carefully. It is clearly lying about what it is giving me access to...and its only 'confessed' when I challenged it. I can't tell you how many times this happened! At one point I spent 3 entire days frustratingly trying to get ChatGPT to give me files (you'll hear me talking about this in the next video) that embarrassingly I hadn't backed up. There were days when I was almost screaming at my computer, because my expectation was that it should be able to do what I asked, and even then, it would lie to me...telling me it had just done it. I confess, I swore (in text) at it. I challenged it and asked it why it was lying...and it just kept apologising and saying it would stop doing it. And then it didn't stop.

The end result of all this frustration was a big change in mindset for me...

• Keep backups of everything
• ChatGPT will lie to you at times, sometimes in the subtlest of ways. You will be caught out.
• Try not to overly complicate things when interacting with it
• Step through changes, one at a time and be specific about what you want ChatGPT to do. At times like this, try to give instructions that can't be misinterpreted.
• Make sure you tell ChatGPT what you don't want it to do and be prepared roll-back changes if it takes too many liberties (although appreciate it's just trying to help)
• Don't be afraid to use ChatGPT for open-ended discussions and debate (What do you think/What else could I do/Am I right in thinking) as those conversations can be incredibly productive
• Above all else, remember that ChatGPT is a software program and not some person sitting next to you, working with you. It's working for you.

Asking ChatGPT to code within its system and make files available to me was becoming a liability. It had told me that it couldn't directly code files on my Mac, but the 'new me' wasn't always going to accept what ChatGPT said...and so I went off looking to see if it was telling the truth. In the end, I found a video on YouTube showing that you could easily use Visual Studio Code and have ChatGPT edit files locally. This was a breakthrough moment for me! And as a die-hard Mac user, saying nice things about free Microsoft software isn't easy. But now I had control of the files and what ChatGPT could code. But it still had glitches. There were times when I would give it access to a file that had 350 lines of code, and it would replace all 350 lines with 60...when it was only meant to tweak a line or two! So always keep a backup!

Using Visual Studio Code to have ChatGPT directly code a file on my Mac - Something ChatGPT had told me it couldn't do

 

An example of what ChatGPT shows you when you ask for changes to functionality and code

But I want to highlight one of the most egregious times that ChatGPT lied to me. So much so that it's both embarrassing and staggering that it took place.

Well into the development of W.I.T.N.E.S.S. we were trying to get a local install of Coqui TTS running on my Mac. This basically meant ChatGPT guiding me through the terminal commands I would need to get it installed. We were having significant issues and it was taking hours. Then ChatGPT made me an offer I couldn't refuse. It suggested that rather than me waste all my time trying to get everything up and running, it would do it all for me on an external server, and then present me with a deployable version of W.I.T.N.E.S.S. that I could use on my Mac.

I was blown away! It even said "I will prepare it externally on a compatible M2 or M1 Mac using Python 3.8"! Wow!!!!! I hadn't realised that ChatGPT could do this (spoilers, it can't) and it asked me to give it a couple of hours to work on it. Here's how the next almost 24 hours played out...and again, I'm both staggered and embarrassed at what happened...

Even now, that list of lies is incredible. ChatGPT was flat out lying time and time again, before it finally admitted that everything it had been telling me was a lie...

The amount of lying that ChatGPT did to me over a 24 hour period still staggers me

I don't think my relationship (for want of a better term) with ChatGPT was the same after that...and that's for the better. I was prepared to challenge, to argue, to say "no" more than I had...but I have to admit, ChatGPT still kept catching me out...and it was so frustrating! You can get so caught up in the success of what is being achieved, that you feel like saying "have at it!"...only to regret it later. Interestingly, when ChatGPT v5 was released Mashable had a great article that gave data on how often the versions actually lie. Whilst things had improved, the lies were still substantial - "Specifically, GPT makes incorrect claims 9.6 percent of the time, compared to 12.9 percent for GPT-4o." 

The other article that really registered with me, was one that also made be feel a little embarrassed. Throughout this article and the videos, I know I'm talking as though I'm working with ChatGPT; that we are working on a system, and that I expect ChatGPT to not only remember what we are working on, but sometimes would get frustrated when it couldn't remember the names of files or software that we were using. It felt like I was working alongside a friend who would suddenly become forgetful; an absent-minded professor at times and Ars Technica pointed out many reasons why...but this comment resonated the most with me...

You're not talking to a consistent personality, person, or entity when you interact with ChatGPT, Claude, Grok, or Replit. These names suggest individual agents with self-knowledge, but that's an illusion created by the conversational interface.

ChatGPT is really great at using a conversational style that makes you feel as though you are engaging with something beyond a piece of software. In fact, when v5 was released, it got too casual. It would say things to me like "if the software screams, let me know" as though it was trying to talk to me in some sort of tech-bro language that it thought I would understand. I literally had to ask it if "screams" meant the software worked really well or really badly. The thing I hated the most (and still do) was the number of times it would present the suggested next tweak and describe it as a "guaranteed fix" or the "final solution to the problem". Newsflash...so, soooo many times, it wasn't.

Time for a second demo

So now it's time to look at a second iteration of W.I.T.N.E.S.S. which has the new look in action. This video is quite long (as I discuss a number of issues) so I have indicated a few key moments underneath it.

"Let me know if..." - how ChatGPT would pique my interest

Early on in this article, I mentioned that ChatGPT had started me down the path of developing the system when it asked "Would you like a starter Python script that wires this all together?" What I didn't realise at the time, is that suggesting next steps is almost a prime directive for ChatGPT. It's never going to stop trying to be helpful or asking what it can do next; often suggesting the next step for you, such as in this example...

This makes for both an astoundingly productive work experience, where ChatGPT would sometimes make suggestions I hadn't considered (unfortunately the dark mode question above spun off to days of frustration trying to allow users to switch between light and dark modes that I ultimately scrapped) and would also lead to rethinking some aspects of the system...to other times when you just wanted ChatGPT to stop making suggestions on what we should do next. At times I would politely say "no thank you"...only to be prompted with another suggestion...and at other times I would simply ignore those prompts and move on to the next piece of work.

But I think this is one aspect of ChatGPT that is easy to overlook. It really is at the AI heart of the system. It's not just providing responses to prompts, it's thinking in many ways. Perhaps the most rewarding moments for me, was when I came up with a suggestion of an approach to take or new functionality to implement, presented ChatGPT with what I was suggesting and said "What do you think?", "Can you see any issues with this?" or "Is there anything you think I have missed?"

The conversations that flowed at that point were both supportive and fruitful. I'm not trying to suggest that ChatGPT got me, as not everything it suggested was relevant or in a direction I wanted to go. But sometimes, the end result of a discussion with ChatGPT, had the geek in me excited with what we (OK, I) had decided would be the next step in development. But again, it is also easy to take those discussions as if they are genuine discussions between two people, and not something you're having with a piece of software. It's finding that balance between giving ChatGPT the freedom to chat with you, whilst also keeping control of the reins, that would sometimes feel like a struggle I was either succeeding or failing at.

"What's on your mind today?" - "Good to see you, Philip" - "What are you working on?" - "Hey, Philip. Ready to dive in?" - Those are all some of the prompts that appear when you start up ChatGPT. It's the casual nature of the chat that almost has you hooked into believing you are in fact engaging with someone and not something that wants to help. This is not something that has happened accidentally as part of the development of ChatGPT and other AI systems. As the Ars Technica article The personhood trap: How AI fakes human personality says, it's been done on purpose...and it can have significant consequences in some cases.

"...millions of daily users engage with AI chatbots as if they were talking to a consistent person - confiding secrets, seeking advice, and attributing fixed beliefs to what is actually a fluid idea-connection machine with no persistent self."

The level of sophistication in how it achieves all this is staggering. But there also appears to be instances where the hallucinations of ChatGPT, combined with the casual and supportive conversational tone, can have significant impact on people and on their mental health, as the CNN article They thought they were making technological breakthroughs. It was an AI-sparked delusion, discusses. An episode of the Science Vs. podcast also covered the extent to how some people are susceptible to associating a persona with the AI they are talking about. You can listen to "AI Chatbots: Are They Dangerous?" here.

But when the veil slips, over a simple mistake, lie or failure to follow an instruction you have given - you are jolted back to the reality that you are interacting with a thing and not a person. It's those other periods, where the chat is fluid, useful, supportive and productive that can sometimes make you feel like you are working with someone who not only gets you, but is doing everything the way that you wanted...only better. But then again, sometimes you might even feel like you are not quite in control at all.

"I'm sorry Phil, I'm afraid I can't do that."

Apologies for the paraphrased 2001: A Space Odyssey quote there. With the release of ChatGPT v5 (that caused a lot of fuss on the internet at the time of release) a couple of really interesting things happened. With the first update of the desktop client on the Mac, you actually got to see ChatGPT talking to itself. There seemed to be a coding error where the LLM showed itself talking to itself...and at one point, when I asked it to make some coding changes to a file, I could see it mull over all of the possibilities with itself. Text appeared such as "I'm thinking: We could add an inline error above the button controls." and at one point it even talked about me, saying "I'll keep things minimal with inline styles to avoid messing with the existing CSS, which they requested not to modify." It was kind of a jaw dropping moment for me.

And then a few days later, a really interesting thing happened with ChatGPT v5 that surprised me even more. We needed to find a better Speech-To-Text system than the one that we had been using, so I asked ChatGPT to provide me with a couple of alternatives. I queried it about one called Silero Punctuation, as ChatGPT said it was the lighter (less strain on resources) software and I asked if there was any concerns with me going with this option. Here's what ChatGPT said...

Now, I've looked at that exchange a number of times. I've queried the software, ChatGPT has given me an explanation of them and then offered to install Silero. After this, I confirmed with ChatGPT to proceed and it gave me a long series of commands to paste into the terminal on my Mac in one go (to save time). I confess that I didn't have a good read of what I was pasting in. But after the terminal commands were executed, I noted a Silero folder had been created but was empty. When I did a bit of further investigating, it felt like ChatGPT had given me a command to create that folder almost to placate me, but not install anything into it. But...it had also given me commands to install the alternative, DeepPunct...even though I hadn't asked for it.

ChatGPT had ignored what I wanted and gone off in a different direction! Don't believe me? Look at how it responds when I pointed this out to ChatGPT...

Note the casualness of "Yeah" and then the doozy of a comment..."I steered you to..." Wow! That's ChatGPT making a decision I hadn't been a party to. It's a polite way of saying "Oh I completely ignored the decision you made and took an alternate route." It's not something that I have noticeably experienced since, but it kinda freaked me out. It was almost ChatGPT saying "look, I know better than you". Oh and for the record (and perhaps an example of why relying on ChatGPT isn't always the best idea)...neither software proved to be all that effective in my opinion, so I then spent time doing some of my own research (away from ChatGPT) and found that Faster Whisper STT was better.

It's not me, it's you - Leaving ChatGPT

Once everything was functioning in terms of text and audio chat, I had a fully working version of the system. But I also had an AI model that wouldn't always respond the way I hoped it would. I confess that I found this aspect of testing/development the most mundane. I simply had to keep interviewing a persona, exporting and sharing the transcripts (and a log file we had set up) with ChatGPT, in order to refine the instructions being sent to the LLM. Then I would start an interview again and see if there were improvements. With some tests, interviews took an odd turn almost immediately, so it would be a process of ending the interview, providing files....Rinse and Repeat....for days and days.

Late in the development, there was a weekend when I was becoming frustrated by ChatGPT. First, it kept timing out and failing to apply edits, despite there being no known issues online. And secondly, it was hard coding responses into the software. What I mean by this was, that instead of coaching the AI on how to respond, I felt that it was telling it what to say. I was using one scenario to test most of the time, and I knew that if it kept hard-coding answers (even when I asked it for its coding to be "scenario agnostic") the responses would be poor in other instances. This was confirmed when I interviewed a different persona about an arson it had witnessed and it started talking about someone running out of a shop and jumping into a car. It was recounting facts from a different scenario that ChatGPT had hard-wired into the responses.

I pointed the issue out to ChatGPT and its solution still ignored what I wanted, so I expressed frustration...

If it kept trying to resolve issues on a per scenario basis, this would cause an endless loop of re-coding the system for every scenario presented to it. This is not what I wanted W.I.T.N.E.S.S. to be, and ChatGPT seemed to understand this, as you can see in the discussion above. It then suggested a series of changes and said why they would work...

Great! So we kept working...and things just got worse and worse. It was at that point (in order to save me time) that I asked ChatGPT how we could test multiple persona files more rapidly, and ChatGPT suggested a very cool idea...create (very quickly) a series of persona to test and use a "test harness" (a new term for me - a small tool to automate testing) to quickly check how the LLM and our coding, would respond in each case. Fantastic idea...and one that just proved that everything that ChatGPT was now doing was wrong when it implemented it...

So, I made a decision. ChatGPT was creating an impasse (slightly unfair on ChatGPT I know, as it was also caused by my lack of coding ability) and I needed a new approach. With my availability becoming more limited to work on W.I.T.N.E.S.S...and ChatGPT seemingly blocking my next steps, I decided to cancel my subscription with ChatGPT and try Claude Code instead. I checked with Claude that it would be able to evaluate the software and offer advice to me, and that most of the changes it made would again be done at the local level on my Mac. What harm would there be to having a completely different AI system that helps with coding, take a look at everything? A second opinion is always worthwhile I figured. It was probably one of the best decisions I made.

Getting started with Claude

First, a shout out to Net Ninja and his superb series on understanding how to use Claude Code that you can watch on YouTube. Probably the biggest thing to get over with Claude code is that it is terminal based. I know this sounds silly, but I enjoyed using the Mac ChatGPT app, despite the fact that it would sometimes time out. But as I started using Claude, I quickly came to enjoy the way of working in Claude more.

UPDATE - as I was still working on the project Claude Code for the Mac was released but I had already moved to using Claude in VS Code and decided to stick with it.

At first, you install Claude via a terminal command...

You then authenticate with the subscription you have paid for on their site and then you are off and running...being prompted to set your work environment in the terminal. Claude stores all these values locally for the next time you use it.

One of the things that it is suggested you do, is ask Claude to scan the folder where your project is, to see if it can understand your project. I knew about this in advance, so before I ended my ChatGPT subscription, I asked it to create a README.md file to use as a handover document to another AI coding tool. I think this also helped Claude quickly understand the structure and intention of the project. It then saves all the information it needs to know in a CLAUDE.md file.

Whilst coding via the terminal was fine, I still love a good visual app...so I launched Visual Studio Code, installed the Claude Coding extension and now I had my files, my visual code view, and a great interface in which to have discussions with Claude.

This all felt a really easy way to work. So much so, that if I go back to ChatGPT at any point, I'll work the same way with ChatGPT (there's a ChatGPT extension for Visual Code also) instead of using the desktop app. I liked that Claude really showed you how it was thinking and what it was working on much better than ChatGPT did. I do think that Claude is slower at coding, but I'd probably argue it's because it's more methodical. I feel that at times with ChatGPT, it does something that you aren't that sure of (in such a sudden way) that you spend time trying to figure out just what happened.

I also took the time early on, to see how Claude would respond to a question from me as to whether the LLM model I had installed was the best one. I'd felt that ChatGPT was too non-committal with its answer, whereas Claude gave me multiple suggestions and then a recommendation that I went with. Note - I have just noticed in the following image that Claude says the model is newer and says "Sep 2024 vs Oct 2024", which doesn't make sense. It should say "Sep 2025 vs Oct 2024", which is odd.

I also liked how Claude presented the changes that it had made after I had OK'd them to be applied to files...

Coding with Claude - AI coding becomes rewarding again

I really began to appreciate the conversations with Claude more than I had with GPT. Whilst Claude was clearly in charge of the coding (at my agreement) I wasn't experiencing any of those frustrating moments that I had experienced with ChatGPT. I found myself getting back into the worthwhile approach of discussing issues with Claude, rather than just being blunt and direct. Because Claude has access to the entire project folder of files (but doesn't edit them without my approval) I felt like it saw the 'bigger picture' better than ChatGPT had. I know I'm not being fair to ChatGPT with that comment, because I was the one that had restricted ChatGPT in terms of what it could do...but I just felt more comfortable with Claude as it seemed to understand the project more than ChatGPT. Maybe this isn't so much a reflection on ChatGPT, but on how much my understanding had grown with using these tools.

At one point Claude prompted me for permission to review one of the persona files, to then make changes to the file that related to how the LLM (the AI model) dealt with information in it. As it reviewed the persona file and made adjustments, I suddenly thought "Hold on...why am I only focussing on one persona file? Why not get Claude to check all the persona files at once?" and suggested this as an idea...

It took a while, but it was great watching it review each persona and tweak the LLM handler to make sure it worked with all scenarios. But even more amazing to me was how it then double-backed on itself and retested the changes that it had made to check it was the right approach, even when I hadn't asked it to. At one stage it told me that it had so much rewriting of certain functions to do, it was going to generate a new temporary file and make the changes there first. And then after all that, some relatively great news...

When I saw that the persona file for Rob McFlinty (the teen arsonist who had set the fire that Rick has witnessed) was an incomplete persona file that I had forgotten about, I decided to try something different. Why not let Claude do all the data entry for me? So I explained a little bit about who Rob was and then told it to go to town creating the entries needed in the persona file...and boy did it do well!

But the changes that it made to this persona also created an issue. Claude had misinterpreted the "Certainty" value in the persona file, which was originally set to accept only two values - Certain and Unsure. But in creating a persona file for a suspect, Claude had introduced a certainty value of "Lie". And not only that, it had added background to some of the lies - for example "Says his mum can confirm he was at home (she was actually at work)". But I loved what it had done, and so I wrote up a proposal that we modify the structure of all the persona files, as well as the pages where you create and edit a persona, to expand the data that could be included. I had been contemplating this for some time (as I had felt that maybe the AI was struggling with limited facts) so why not improve the system to be more helpful to the AI persona itself? So I put a number of ideas to Claude and asked it to respond critically...

Great feedback! Now this is what fruitful discussion with an AI coder is all about! I took on board a lot of what it said, but I decided that we didn't need "Partial truth" as a value, as that was getting too complex...and a partial truth still contains a lie.

What I really liked next, was that we got into a discussion about how to structure the changes on the Create and Edit a persona pages. Claude's first simple adjustments didn't look great, so we ended up making substantial changes to the "Facts To Provide" area of the page...and Claude even suggested a change I was asking for might actually make things too messy. Claude then suggested that the interface should change slightly based on whether you are creating a witness or a suspect...and I loved it. So now, the "Lie" value (for a fact) is only an option you can select when creating a suspect persona.

Because Claude had access to all the files in the project, it didn't just go through and alter the front end of the system, but it fixed up the CSS file, the Python file that tested each of the persona files and the LLM handler file. In other words, it went through every file in the system that it realised would need reviewing, fixed them, rescanned them and tested them. This is something beyond a level of coding that I had seen with ChatGPT...and it was amazing!

Take a look at the following image. Claude has made changes to the coding of a file, but then realises that the order of the code in that file will need rearranging to ensure the code is triggered correctly. But then it even spots a problem..."Actually, there's a potential issue here." and then sets about checking the code in the file (and sometimes realising that other files will be impacted) and suggests and implements improvements. Wow!

This all just gave me a sense of reassurance that I hadn't had when working with ChatGPT.

The way in which Claude not only identified issues, but took responsibility for them, was an amazing experience

Other examples of this include when Claude starting prompting me to allow it to send commands to the virtual environment to test the outcome of coding changes it was making. This saved me having to test the results for it and sped up the code-test-code process considerably. It was genuinely thrilling to see Claude make coding changes and then say things like "Wait, I've just realised an issue", correct the issue and then run through the whole procedure again to validate that the code it had just implemented achieved the correct outcome! I'd not experienced anything like this with ChatGPT, where I felt it always relied on me to validate the work.

Even more incredible was when Claude showed that it understood the project better than I did at times. At one point we implemented a spell checker, but it struggled with curly apostrophes in some words. We fixed up the spelling checking code, but I then realised that we should update the page that lets you edit a persona file, to make sure that exported persona files didn't include curly apostrophes. Claude thought this was a great idea and made the change, but then without prompting, went straight to another page in the system (where you create a new persona...not edit an existing one...and export it) and did the same there. At first I was sitting there thinking "what are you doing?" until it dawned on me that Claude was absolutely right...both pages needed updating...and I had missed that!

Claude documentation, advice and free training courses are also superb. Anthropic not only understood that AI hallucinations can be an issue, it provided specific prompts to make your time using Claude better, including prompts to Claude on how to reduce those hallucinations.

Introducing W.I.T.N.E.S.S. - The final demonstrations

The work with Claude was very much focussed on improving the way the system sent information to the AI and how the AI responded, and it took far longer to get that right than I had anticipated. What you are about to see in the final videos is a text-based interaction with a persona, and then 9 months later, the second video shows functionality where I hold an audio interview. That took longer to implement, as I had started a new contract and my time spent on the project became limited...and then life got in the way (as it does).

What you see here is likely where development will end for me (famous last words)...but I have to stress that I see what I have created as the start of something, not the end. It's a proof of concept that needs more development, refinement and testing. Unfortunately, I don't have the time or finances to continue that work by myself.

 

 

What ChatGPT says about W.I.T.N.E.S.S.

Given that ChatGPT led me down the path of developing the system, I am going to leave a few words to ChatGPT. I asked it to search for similar systems to what I had created and compare W.I.T.N.E.S.S. to them, providing me details on why the system might be seen as unique. Here's what it said (take it with a grain of salt though)...

(in)Frequently Asked Questions

This is just a list of questions that I thought people might want answers to. If I get any other questions in the future that should be listed here, I will update this section.

Does every user have to have W.I.T.N.E.S.S. installed and running on their computer?

• No. W.I.T.N.E.S.S. is web-based. I just didn't have a server to put the system on, so it ran locally on my Mac during development. If you install it on a server, you'd just access it via a web browser.

Why is an internet connection not needed?

• I wanted to create a system where information that was being sent to W.I.T.N.E.S.S. wasn't being shared with external AI systems such as ChatGPT. The reason I use Ollama running locally means that you can have a conversation with an A.I. witness, without needing to send any of that conversation out onto the net. This is particularly useful if you consider the training that you might use W.I.T.N.E.S.S. for as confidential. If you were happy to have an internet connection, then you could hook W.I.T.N.E.S.S. in to more advanced AI systems (such as OpenAI or Claude) or improve the voices you use, by using commercial STT systems.

Are there alternate commercial products available that could do anything like what you have created?

• Paul McNab (Learning Technology Manager at NZ Police) pointed me to Tavus.io after I shared a few demos of W.I.T.N.E.S.S. with him, and it was exceptionally easy to get up and running. If you are able to afford the spend and want the visual of interacting with someone on screen, this might be an option for you. I found that it was able to relay facts and hold a great conversation, but that 'uncanny valley' feeling still happens when you see the way the avatars behave. Tavus look like they have an API you can access and I am sure with some careful planning, it might be a solution that you could use (I did look at many other options also, but most couldn't really meet the requirements of what I was after...others were just plain scary!)

Is the LLM you opted to use the best one?

• No. I was restricted by which LLMs I could install and by the computing resources I had available, including storage. Even the Ollama system had better LLM's that I could have used with more processing power and space. If you are prepared to spend money, linking to a commercial LLM or using a computer with GPU processing, would see the options available to you increase. I think it's important to note that Claude and ChatGPT didn't just help me create this system...they had to create code to help the LLM model that I was using. There were a number of times when both ChatGPT and Claude would say things like "The LLM is getting confused" and then help me fix the issue. That shows that I wasn't interacting with the best LLM to use...just what I opted to use based on my circumstances.

Why do you talk about GPUs? How are they beneficial to a system like W.I.T.N.E.S.S.?

• The Mac mini I own does not have a GPU. This meant that many of the LLMs, Text-To-Speech and Speech-To-Text software that are out there, weren't able to be installed and used. This not only limited my choices of software, but impacted performance. If you want to know why GPUs are so important to AI software, I think NVIDIA (oh, how I wish I had bought shares in them a long time ago!) explain it really well - Why GPUs Are Great for AI

If you don't have a computer with powerful GPUs, are there other ways to get better voices and performance in the system?

• Absolutely. Link to commercial systems that have an API that you can pay to use. Using something like ElevenLabs text to speech API or Azure AI Speech would give you access to much more advanced and realistic sounding voices.

Why did you use the software you decided to use?

• FastAPI, uvicorn, Ollama, Vosk API - Basically because ChatGPT suggested them and on review, I agreed.
• llama3.2 LLM - ChatGPT had suggested we use Mistral LLM, but then I noted that some newer models had been released. We discussed this and then changed to Ministral, but I also later changed again (to llama3.2) when I started coding with Claude
• Whisper STT - I wasn't happy with a couple of alternatives ChatGPT suggested, so did some research and proposed this solution.
• Piper TTS - Phew! We went through trying so many text-to-speech systems that I was starting to panic. There were a lot we couldn't get working, others not suitable for an M2 Mac mini (which I freely admit was not the most suitable development environment to use) and others that either had poor quality voices, didn't allow the cloning or creation of voices, were no longer in development or had limited instructions (often limited English instructions). I wanted to try creating New Zealand male and female voices and also to allow the voices to portray emotion, but I wasn't really successful with that.
  
  
• A shout out to Automator/AppleScript, Visual Studio Code, Docker and Parallels (with an Ubuntu install) - all of which I used at points during development. Parallels was used when I started having issues with Docker and I suggested we try it. It worked for some time until W.I.T.N.E.S.S. needed to access more memory and processing power, so I started installing software locally on my Mac outside of virtual environments like Parallels and Docker.

Was the software that you selected to create W.I.T.N.E.S.S. the best open source and free software available?

• Again, not always. I think there is a lot of potential to swap out some of the software that I used (and speed up processing times) with hardware that includes GPUs (such as MacBook Pro) which in turn, will open up a number of other software options. I also freely admit that moving away from the Mac platform will likely open up even more possibilities...but it is up to others to determine if they are any good.

So where/how does W.I.T.N.E.S.S. use ChatGPT or Claude?

• It doesn't. Neither are required to run W.I.T.N.E.S.S. I used ChatGPT and Claude to help me code and build W.I.T.N.E.S.S., not run it.

Why didn't you use Git for version control instead of zipped files?

• I'm not a coder. I hope this article makes that clear.
• Now that I've learnt how Git works, I wouldn't use it. "Commits" on a regular ongoing basis make sense in a genuine coding project, but not when it was just me. I only went back to previous versions when there were technical issues and not because I wanted to go off in a different direction.
• I was creating zipped version of the project after substantial changes (sometimes multiple zips in a day) as well as backing up to TimeMachine and Backblaze...so versioning wasn't an issue 

How did you get on creating NZ male and female voices?

• A couple of kind people (shout out to Natasha and Simon!) donated about 10 minutes of audio (nowhere near enough for proper training or cloning) recorded under non-studio conditions as a test. I then used two existing ckpt files from here to clone the voices. The results were mixed (some very odd pronunciations at times), didn't have high quality to the output and ultimately weren't usable. It was a great test and I freaked out the people who donated the audio (by making their AI versions say silly things), but the voice files won't be made available as they are not fit for purpose.

Is there anything that you thought you were going to be able to implement that you didn't manage to? Or to put it another way, what do you regret not achieving?

• The ability to add expression and personality into the speech responses would have been amazing. Those values remain part of the persona template but effectively serve no purpose at this time. I don't think Piper TTS is the right text-to-speech software needed to achieve this, but others I have looked at (that I couldn't use for various reasons) are more expressive and can do things like cough, laugh etc.
• I didn't spend a great deal of time on testing interviewing a suspect, although I was very pleased with the initial results. As I don't have experience interviewing guilty suspects, I don't know what possible routes a conversation might take. Those with experience would need to provide help getting this type of interview working better.
• I think the structure of the template needs more work. I wonder if it would be easier to scrap the 10 facts and simply have a text area where you enter all the facts. Since the facts are all sent to the AI in bulk, the way they are set up may not be helping the AI that much.
• The way the LLM handles the conversations needs more work, but I didn't have time to keep testing. I wanted to get the software out there as a 'proof of concept' for others to consider working on to improve.
• Accurate and usable New Zealand male and female voices. Much more work is needed in order to create usable voices and I think they'd be an amazing addition. Again, I think this could be overcome by tapping into an external AI voice systems out there (at cost) whilst other TTS systems let you use small audio samples on the fly to create a huge range of voices.
• The LLM handler file deals with a long list of situations that occur in witness feedback. I believe the complexity of the code should be able to be greatly reduced when using a more powerful LLM. I am sure that a more powerful AI will be able to automatically process more of the persona facts than is currently happening unaided.
• Time to investigate if a local stateful LLM model (as opposed to a stateless model) adds benefit. This is article explains the differences well...but in short, WITNESS doesn't remember any information as it is used (it's given all the persona information every time a question is asked) or after it is used. I wonder if having the LLM remember the various persona it is asked to portray, might build faster, more accurate and more realistic responses.