W.I.T.N.E.S.S. - Adventures in AI coding - (in)Frequently Asked Questions

Page 4 of 4: (in)Frequently Asked Questions

(in)Frequently Asked Questions

This is just a list of questions that I thought people might want answers to. If I get any other questions in the future that should be listed here, I will update this section.

Does every user have to have W.I.T.N.E.S.S. installed and running on their computer?

• No. W.I.T.N.E.S.S. is web-based. I just didn't have a server to put the system on, so it ran locally on my Mac during development. If you install it on a server, you'd just access it via a web browser.

Why is an internet connection not needed?

• I wanted to create a system where information that was being sent to W.I.T.N.E.S.S. wasn't being shared with external AI systems such as ChatGPT. The reason I use Ollama running locally means that you can have a conversation with an A.I. witness, without needing to send any of that conversation out onto the net. This is particularly useful if you consider the training that you might use W.I.T.N.E.S.S. for as confidential. If you were happy to have an internet connection, then you could hook W.I.T.N.E.S.S. in to more advanced AI systems (such as OpenAI or Claude) or improve the voices you use, by using commercial STT systems.

Are there alternate commercial products available that could do anything like what you have created?

• Paul McNab (Learning Technology Manager at NZ Police) pointed me to Tavus.io after I shared a few demos of W.I.T.N.E.S.S. with him, and it was exceptionally easy to get up and running. If you are able to afford the spend and want the visual of interacting with someone on screen, this might be an option for you. I found that it was able to relay facts and hold a great conversation, but that 'uncanny valley' feeling still happens when you see the way the avatars behave. Tavus look like they have an API you can access and I am sure with some careful planning, it might be a solution that you could use (I did look at many other options also, but most couldn't really meet the requirements of what I was after...others were just plain scary!)

Is the LLM you opted to use the best one?

• No. I was restricted by which LLMs I could install and by the computing resources I had available, including storage. Even the Ollama system had better LLM's that I could have used with more processing power and space. If you are prepared to spend money, linking to a commercial LLM or using a computer with GPU processing, would see the options available to you increase. I think it's important to note that Claude and ChatGPT didn't just help me create this system...they had to create code to help the LLM model that I was using. There were a number of times when both ChatGPT and Claude would say things like "The LLM is getting confused" and then help me fix the issue. That shows that I wasn't interacting with the best LLM to use...just what I opted to use based on my circumstances.

Why do you talk about GPUs? How are they beneficial to a system like W.I.T.N.E.S.S.?

• The Mac mini I own does not have a GPU. This meant that many of the LLMs, Text-To-Speech and Speech-To-Text software that are out there, weren't able to be installed and used. This not only limited my choices of software, but impacted performance. If you want to know why GPUs are so important to AI software, I think NVIDIA (oh, how I wish I had bought shares in them a long time ago!) explain it really well - Why GPUs Are Great for AI

If you don't have a computer with powerful GPUs, are there other ways to get better voices and performance in the system?

• Absolutely. Link to commercial systems that have an API that you can pay to use. Using something like ElevenLabs text to speech API or Azure AI Speech would give you access to much more advanced and realistic sounding voices.

Why did you use the software you decided to use?

• FastAPI, uvicorn, Ollama, Vosk API - Basically because ChatGPT suggested them and on review, I agreed.
• llama3.2 LLM - ChatGPT had suggested we use Mistral LLM, but then I noted that some newer models had been released. We discussed this and then changed to Ministral, but I also later changed again (to llama3.2) when I started coding with Claude
• Whisper STT - I wasn't happy with a couple of alternatives ChatGPT suggested, so did some research and proposed this solution.
• Piper TTS - Phew! We went through trying so many text-to-speech systems that I was starting to panic. There were a lot we couldn't get working, others not suitable for an M2 Mac mini (which I freely admit was not the most suitable development environment to use) and others that either had poor quality voices, didn't allow the cloning or creation of voices, were no longer in development or had limited instructions (often limited English instructions). I wanted to try creating New Zealand male and female voices and also to allow the voices to portray emotion, but I wasn't really successful with that.
  
  
• A shout out to Automator/AppleScript, Visual Studio Code, Docker and Parallels (with an Ubuntu install) - all of which I used at points during development. Parallels was used when I started having issues with Docker and I suggested we try it. It worked for some time until W.I.T.N.E.S.S. needed to access more memory and processing power, so I started installing software locally on my Mac outside of virtual environments like Parallels and Docker.

Was the software that you selected to create W.I.T.N.E.S.S. the best open source and free software available?

• Again, not always. I think there is a lot of potential to swap out some of the software that I used (and speed up processing times) with hardware that includes GPUs (such as MacBook Pro) which in turn, will open up a number of other software options. I also freely admit that moving away from the Mac platform will likely open up even more possibilities...but it is up to others to determine if they are any good.

So where/how does W.I.T.N.E.S.S. use ChatGPT or Claude?

• It doesn't. Neither are required to run W.I.T.N.E.S.S. I used ChatGPT and Claude to help me code and build W.I.T.N.E.S.S., not run it.

Why didn't you use Git for version control instead of zipped files?

• I'm not a coder. I hope this article makes that clear.
• Now that I've learnt how Git works, I wouldn't use it. "Commits" on a regular ongoing basis make sense in a genuine coding project, but not when it was just me. I only went back to previous versions when there were technical issues and not because I wanted to go off in a different direction.
• I was creating zipped version of the project after substantial changes (sometimes multiple zips in a day) as well as backing up to TimeMachine and Backblaze...so versioning wasn't an issue 

How did you get on creating NZ male and female voices?

• A couple of kind people (shout out to Natasha and Simon!) donated about 10 minutes of audio (nowhere near enough for proper training or cloning) recorded under non-studio conditions as a test. I then used two existing ckpt files from here to clone the voices. The results were mixed (some very odd pronunciations at times), didn't have high quality to the output and ultimately weren't usable. It was a great test and I freaked out the people who donated the audio (by making their AI versions say silly things), but the voice files won't be made available as they are not fit for purpose.

Is there anything that you thought you were going to be able to implement that you didn't manage to? Or to put it another way, what do you regret not achieving?

• The ability to add expression and personality into the speech responses would have been amazing. Those values remain part of the persona template but effectively serve no purpose at this time. I don't think Piper TTS is the right text-to-speech software needed to achieve this, but others I have looked at (that I couldn't use for various reasons) are more expressive and can do things like cough, laugh etc.
• I didn't spend a great deal of time on testing interviewing a suspect, although I was very pleased with the initial results. As I don't have experience interviewing guilty suspects, I don't know what possible routes a conversation might take. Those with experience would need to provide help getting this type of interview working better.
• I think the structure of the template needs more work. I wonder if it would be easier to scrap the 10 facts and simply have a text area where you enter all the facts. Since the facts are all sent to the AI in bulk, the way they are set up may not be helping the AI that much.
• The way the LLM handles the conversations needs more work, but I didn't have time to keep testing. I wanted to get the software out there as a 'proof of concept' for others to consider working on to improve.
• Accurate and usable New Zealand male and female voices. Much more work is needed in order to create usable voices and I think they'd be an amazing addition. Again, I think this could be overcome by tapping into an external AI voice systems out there (at cost) whilst other TTS systems let you use small audio samples on the fly to create a huge range of voices.
• The LLM handler file deals with a long list of situations that occur in witness feedback. I believe the complexity of the code should be able to be greatly reduced when using a more powerful LLM. I am sure that a more powerful AI will be able to automatically process more of the persona facts than is currently happening unaided.
• Time to investigate if a local stateful LLM model (as opposed to a stateless model) adds benefit. This is article explains the differences well...but in short, WITNESS doesn't remember any information as it is used (it's given all the persona information every time a question is asked) or after it is used. I wonder if having the LLM remember the various persona it is asked to portray, might build faster, more accurate and more realistic responses.