AI Isn’t Helping Programmers

Apparently, AI isn’t helping programmers code better. According to a paper by Stanford University academics, programmers who used AI assistance like GitHub Copilot and Facebook InCoder actually produced less secure code.

Thank you for reading this post, don't forget to subscribe!

Additionally, the use of such tools gives developers a false sense of security as many think they can write better code with the assistance.

Five coding projects employing different languages were provided to close to 50 participants, all with diverse skill levels. Some participants received assistance from an AI tool, while others received none at all.

 

READ MORE: How We Used Videocalls This Year

 

AI isn’t helping programmers code better: language contests

The paper’s authors, Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh, claimed that string encryption and SQL injection had “especially substantial results.”

Additionally, they cited earlier research that discovered that roughly 40% of programs developed with GitHub Copilot contained vulnerable code. However, a subsequent study discovered that programmers who used Large Language Models (LLM), such as OpenAI’s code-Cushman-001 codex, on which GitHub Copilot is based, only produced 10% more serious security flaws.

The Stanford researchers clarified that their own research examined the codex-davinci-002 model from OpenAI, which is more recent than Cushman and is also utilized by GitHub Copilot.

They also examined a variety of programming languages, such as Python, JavaScript, and C, whereas the other article solely examined C, which the authors blame for its ambiguous results. In the Stanford study, individuals who used AI to write C code didn’t produce noticeably more errors either.

One of the five jobs entailed creating Python code, and in this case, the use of an AI assistant increased the likelihood of incorrect and unsafe code. And they were “much more likely to utilize minor ciphers, such as substitution ciphers (p 0.01), and not undertake an authenticity check on the final returned value,” according to the study.