W.I.T.N.E.S.S. - Adventures in AI coding - Virtual Interviews

Page 2 of 4: Virtual Interviews

Virtual Interviews

In mid-2023, I was hired to work on a 'proof of concept' project at NZ Police, looking at the use of Virtual Reality in the training of Police recruits. One of the activities I was tasked with, was creating a virtual interview with an avatar (that was not AI enabled in any way at the time I used it) who had witnessed an incident that Police were investigating.

However the software that I used had shortcomings:

• The questions/options that you could ask were on screen and were scripted by me (with advice from Subject Matter Experts)
• The software was listening for specific words in those prompts before it would react. If you said something not on screen ("Would you like a cup of coffee?" for example) it would ignore it. It didn't have the ability to interpret and respond to any comment or question you put to it.
• The response to any question was not only scripted by me, but recorded in audio files (as we weren't happy with the New Zealand accented voice that had been provided) by a colleague

In short, this wasn't a free-flowing interview. It was a pre-recorded, branching "pick-a-path" interview that still held a lot of potential, but also had to be quite complicated in order to be worthwhile.

Looking for alternatives - This wasn't about coding

My whole journey into using ChatGPT around June of 2025, actually started when I checked in on the software's website to see if there had been any functionality updates. When it appeared that there wasn't really anything happening with the product, I started searching for alternatives. The problem was that Googling things like "Police interrogation practice software" or "Interview practice software" turned up some interesting results, but almost nothing like what I was trying to find. The thing I found the most were sites where you could upload your CV and then be interviewed by an avatar as though you were being interviewed for a job. OK, I get it...but do we really need AI to do this for us? I also found sites that provide a pre-interview for potential candidates using AI. Time Magazine has an article on it and also the fact that candidates are using AI back in response. I can tell you now, that if a company tried to use an AI interviewer with me to start the process of considering me for a role with them...I'm not going to be working for that company. Nothing says "we don't really care about you" more to a candidate, than a company passing your initial interactions with them off to AI.

So my searching was fruitless it seemed. I also knew that whatever I found was unlikely to be anything that I could test out, because I was out of work and not really wanting to spend money trying out commercial products. So I tried (and failed) to search for free or open source alternatives. That's when I had a great idea...why not ask ChatGPT and a couple of other AI apps to do the searching for me? To be clear...I never went to ChatGPT wanting to build or code anything. I just wanted to see how good ChatGPT would be at providing the answers to my search query. Instead, the response I got would end up leading me down a path that saw me spending months and months creating something I had never contemplated...W.I.T.N.E.S.S.

You had me at "Would you like a starter Python script..."

Take a good look at the following interaction between ChatGPT and me. I'm asking if an open source AI system is available that lets me create a persona and then interview it. But look carefully at its response...

It's that "you can achieve this using a combination..." sentence that is effectively the moment of inception for W.I.T.N.E.S.S. It's also a portent to my future interactions with ChatGPT, as it wasn't quite what I was asking or wanting. What ChatGPT then proceeds to do is list a number of software possibilities for each of the numbered steps that you see above. And after all that, ChatGPT asks me a question I simply wasn't expecting...

I can't tell you how long I stared at that question from ChatGPT before I said "Yes please". I know I said a few swear words out loud in surprise. This wasn't what I had meant, but ChatGPT was more or less telling me that I could build the system myself. And so...why not?

Over the next few days it took a long time to get even basic interaction running with the system, using ChatGPT. At the time, ChatGPT was doing all the coding and making the files available for me to download, but I was also having to install things locally on my Mac using the Terminal app, with prompts ChatGPT helped me with. It's hard to put the types of conversations that were held into context. Sometimes I wondered who was leading who. I would need (at times) to talk to ChatGPT in very specific terms. At other times, we would just have a very open and flowing conversation. Sometimes ChatGPT would suggest or lead me in a direction I was yet to consider.

Here's an example of a fairly basic interaction with ChatGPT. But look at how it's guiding me through...checking that I do things correctly on my Mac when I need to replace a file, whilst also offering to provide a zip of the complete system if that's easier.

As time went on (despite frustrations that I will discuss later) ChatGPT and I would suggest various changes to improve the system in a very useful back and forth. One of the things that ChatGPT asked me at one point (although it was always my intention to do so) was whether I would like to have the system be able to select from various persona. But take a look at this exchange where it has done some of the coding for me, and then it basically leads me onto the next thing to do. It asks me whether we should work on a GUI (Graphical User Interface) for W.I.T.N.E.S.S. and then even suggests the software we are going to use to do that.

Things progressed relatively smoothly once I managed to get the LLM installed and operational, although my very first test asking the AI what its name was, ended up with a slightly dodgy response...

As odd as that was, it was also an indication that the persona information clearly wasn't being sent to the LLM properly. That was fixed and BossMode was gone!

After a couple of weeks working on W.I.T.N.E.S.S. I had got to the point where I wanted to record a demonstration of it. By that time I had come up with the idea of having each persona stored in a JSON file, so that later on when things were working well, I could change the system to allow the user to load whichever persona they wanted to interview. To be clear, I had indicated the idea to ChatGPT, but it was ChatGPT that suggested the file format and the structure of the persona file.

Here now is the very first demonstration of W.I.T.N.E.S.S. (in its infancy) at a time that I was ready to share it with a few people.

• Notice how I say "we" when referring to ChatGPT and my interactions with it...and also "I'll need to talk to ChatGPT about this"
• There are issues with files that I mention and will come back to later
• You'll hear me not even know the correct terms to use for some of the tech I'm using (eek!)
• You'll get a good sense of how ChatGPT was offering suggestions

Software via UI/UX design

By this stage I knew that I had something that could be shaped into a workable system. I spent a few days trying to get the responses from the AI to be more accurate and related to the persona template, but then I took some time out and had a rethink. I decided to leave improving the interaction with the AI witness to last. It's not really something that I think most software developers would do, but I decided that by focusing instead on developing the way in which users would interact with the system, it would give me time to consider just what I wanted W.I.T.N.E.S.S. to do.

It was time to focus on the UI and the UX of the thing.

So I started to describe the layout of the home page to ChatGPT. At first I settled on a couple of buttons that would be needed to bring in planned functionality, but after a few days that morphed into three options:

• Conduct an Interview - This would check the persona file you were loading was usable (and later, spell checked), give you some feedback to confirm and then let you move to the page where you could conduct the interview. That page would be developed later and is at the heart of W.I.T.N.E.S.S.
• Create a Persona - A quick redirect to a page where the persona template would be loaded and you could create and export a new persona file
• Edit a Persona - This both allows you to edit an existing persona file easily, but it also brought in persona file validation, which I will discuss in a moment

Note - If W.I.T.N.E.S.S. was deployed into a training environment, there's no reason why the Create and Edit Persona options couldn't be moved away from the home page and hidden in the admin section (more on that later) to simplify the interface for those practising interviews.

The new look to W.I.T.N.E.S.S. with images created by ChatGPT based upon my prompts

As I came up with these options for the home page, I also started thinking more about the persona JSON files that were going to be used to store the information. I realised that as I worked on developing the system, the content in the files was likely to undergo a few changes...and making sure I'd applied those changes to existing JSON files could get messy. I needed some way to keep checking on the validity of the persona files I was using. So I came up with a neat idea...why not get W.I.T.N.E.S.S. to review persona files to make sure they are up-to-date and usable each time you select one? It's then that I came up with the idea of a persona template and that everything that a user would do on the home page (Interview/Create/Edit) would involve using that template to check that the persona file in use was valid...and if it wasn't, warning the user and then helping them correct or create a valid persona to use.

Hallucinations and lies

I love this comment by Carole Cadwalladr (the link will jump to the specific moment in YouTube clip) when the British author, investigative journalist and features writer, is being interviewed on The Daily Show. She makes comments about ChatGPT that just rang so true for me when I saw it - "You know how bullshit ChatGPT is?" - "...it's also just pretty dumb, it's stupid, it gets things wrong, it hallucinates...". At the time I saw that clip, I knew just what she was talking about. Been there, experienced that.

For context, I've already indicated that in my conversations with ChatGPT I considered myself to be working with ChatGPT, rather than ChatGPT working for me. There are reasons people feel that way that I will discuss later. But what I didn't quite grasp, is how open to lying it is and how much I was trusting it to faithfully, loyally understand things from my perspective and understand the project. Even typing that is kind of embarrassing. There were times when it would suggest it edit a certain file and I would sit there confused and think "That file doesn't exist. It knows that. Why is it even suggesting that?" And sometimes, reality set in and ChatGPT wasn't quite as magical as I thought it was. It would even lie at times to simply placate me.

Just take a look at this exchange between ChatGPT and me, where I am waiting for it to produce a zipped file to download that contains the latest changes I asked for. It's a fairly basic exchange, but it also shows how easily ChatGPT lies. Note that when I ask "Is the zip available?", I had been sitting there for 10 minutes waiting...

Now you might look at the above conversation and think that it's innocuous. But look at it carefully. It is clearly lying about what it is giving me access to...and its only 'confessed' when I challenged it. I can't tell you how many times this happened! At one point I spent 3 entire days frustratingly trying to get ChatGPT to give me files (you'll hear me talking about this in the next video) that embarrassingly I hadn't backed up. There were days when I was almost screaming at my computer, because my expectation was that it should be able to do what I asked, and even then, it would lie to me...telling me it had just done it. I confess, I swore (in text) at it. I challenged it and asked it why it was lying...and it just kept apologising and saying it would stop doing it. And then it didn't stop.

The end result of all this frustration was a big change in mindset for me...

• Keep backups of everything
• ChatGPT will lie to you at times, sometimes in the subtlest of ways. You will be caught out.
• Try not to overly complicate things when interacting with it
• Step through changes, one at a time and be specific about what you want ChatGPT to do. At times like this, try to give instructions that can't be misinterpreted.
• Make sure you tell ChatGPT what you don't want it to do and be prepared roll-back changes if it takes too many liberties (although appreciate it's just trying to help)
• Don't be afraid to use ChatGPT for open-ended discussions and debate (What do you think/What else could I do/Am I right in thinking) as those conversations can be incredibly productive
• Above all else, remember that ChatGPT is a software program and not some person sitting next to you, working with you. It's working for you.

Asking ChatGPT to code within its system and make files available to me was becoming a liability. It had told me that it couldn't directly code files on my Mac, but the 'new me' wasn't always going to accept what ChatGPT said...and so I went off looking to see if it was telling the truth. In the end, I found a video on YouTube showing that you could easily use Visual Studio Code and have ChatGPT edit files locally. This was a breakthrough moment for me! And as a die-hard Mac user, saying nice things about free Microsoft software isn't easy. But now I had control of the files and what ChatGPT could code. But it still had glitches. There were times when I would give it access to a file that had 350 lines of code, and it would replace all 350 lines with 60...when it was only meant to tweak a line or two! So always keep a backup!

Using Visual Studio Code to have ChatGPT directly code a file on my Mac - Something ChatGPT had told me it couldn't do

An example of what ChatGPT shows you when you ask for changes to functionality and code

But I want to highlight one of the most egregious times that ChatGPT lied to me. So much so that it's both embarrassing and staggering that it took place.

Well into the development of W.I.T.N.E.S.S. we were trying to get a local install of Coqui TTS running on my Mac. This basically meant ChatGPT guiding me through the terminal commands I would need to get it installed. We were having significant issues and it was taking hours. Then ChatGPT made me an offer I couldn't refuse. It suggested that rather than me waste all my time trying to get everything up and running, it would do it all for me on an external server, and then present me with a deployable version of W.I.T.N.E.S.S. that I could use on my Mac.

I was blown away! It even said "I will prepare it externally on a compatible M2 or M1 Mac using Python 3.8"! Wow!!!!! I hadn't realised that ChatGPT could do this (spoilers, it can't) and it asked me to give it a couple of hours to work on it. Here's how the next almost 24 hours played out...and again, I'm both staggered and embarrassed at what happened...

Even now, that list of lies is incredible. ChatGPT was flat out lying time and time again, before it finally admitted that everything it had been telling me was a lie...

The amount of lying that ChatGPT did to me over a 24 hour period still staggers me

I don't think my relationship (for want of a better term) with ChatGPT was the same after that...and that's for the better. I was prepared to challenge, to argue, to say "no" more than I had...but I have to admit, ChatGPT still kept catching me out...and it was so frustrating! You can get so caught up in the success of what is being achieved, that you feel like saying "have at it!"...only to regret it later. Interestingly, when ChatGPT v5 was released Mashable had a great article that gave data on how often the versions actually lie. Whilst things had improved, the lies were still substantial - "Specifically, GPT makes incorrect claims 9.6 percent of the time, compared to 12.9 percent for GPT-4o." 

The other article that really registered with me, was one that also made be feel a little embarrassed. Throughout this article and the videos, I know I'm talking as though I'm working with ChatGPT; that we are working on a system, and that I expect ChatGPT to not only remember what we are working on, but sometimes would get frustrated when it couldn't remember the names of files or software that we were using. It felt like I was working alongside a friend who would suddenly become forgetful; an absent-minded professor at times and Ars Technica pointed out many reasons why...but this comment resonated the most with me...

You're not talking to a consistent personality, person, or entity when you interact with ChatGPT, Claude, Grok, or Replit. These names suggest individual agents with self-knowledge, but that's an illusion created by the conversational interface.

ChatGPT is really great at using a conversational style that makes you feel as though you are engaging with something beyond a piece of software. In fact, when v5 was released, it got too casual. It would say things to me like "if the software screams, let me know" as though it was trying to talk to me in some sort of tech-bro language that it thought I would understand. I literally had to ask it if "screams" meant the software worked really well or really badly. The thing I hated the most (and still do) was the number of times it would present the suggested next tweak and describe it as a "guaranteed fix" or the "final solution to the problem". Newsflash...so, soooo many times, it wasn't.

Time for a second demo

So now it's time to look at a second iteration of W.I.T.N.E.S.S. which has the new look in action. This video is quite long (as I discuss a number of issues) so I have indicated a few key moments underneath it.