Evaluating the Security Awareness and Code Reliability of AI Models Against Common Software Vulnerabilities

Abstract

This project explores how well AI coding assistants understand and handle software security, focusing on known vulnerabilities listed in the Common Weakness Enumeration (CWE) system. While these AI tools can speed up development, it’s unclear whether they reliably prevent security flaws. To investigate, I will test eight AI models—including free and paid versions—using consistent prompts across 11 critical CWEs, resulting in 88 experiments. Eleven of the most impactful vulnerabilities, such as Cross-Site Scripting, SQL Injection, Use-After-Free, Buffer Overflows, and Hard-Coded Credentials, will be evaluated in languages like JavaScript, PHP, C++, Java, C, Python, and Node.js. The AI-generated code will be analyzed and categorized as secure, partially secure, or insecure according to established coding standards.

Unlike previous studies that focus on single errors, this work takes a systematic, multi-vulnerability approach, offering insights into the safety of using AI in secure software development. The results aim to guide developers on integrating AI coding tools responsibly, especially in mission-critical fields like aerospace, where software security is essential.