GitHub Copilot vs. ChatGPT: How do they compare?

What is GitHub Copilot and how does it work?

GitHub Copilot, one of the most popular coding assistants, focuses specifically on code completion and provides suggestions for code lines or entire functions directly within integrated development environments (IDEs). It was developed by GitHub and OpenAI, and is built on OpenAI's language models. Other coding assistants are available from vendors such as Google, IBM and Tabnine.

Copilot users can get help directly within popular tools such as Visual Studio, VS Code and Neovim, and IDEs from JetBrains. This also lets it analyze a large context of code without having to cut and paste short snippets into ChatGPT.

GitHub Copilot uses large language models to generate responses and suggestions. GitHub Copilot's LLM, called Codex, is fine-tuned on a massive data set of source code and natural language text, explained Nikita Povarov, data analytics and machine learning (ML) teams lead at JetBrains.

"This fine-tuning process allows Codex to understand the syntax and structure of code," Povarov said.

Here is the response from OpenAI's Codex to a request by the author.

GitHub also offers Copilot Chat, a tool that combines chat and terminal interfaces directly into the IDE. It can automate more aspects of the development experience. For example, it can detect code changes and automatically suggest descriptions, called pull requests, to accompany software updates.

The tool provides context-aware conversations to explain code, find bugs and suggest fixes. It also personalizes answers that are linked to official documentation. In addition, it can write software tests and identify missing tests that might be required.

What is ChatGPT and how does it work?

ChatGPT, developed by OpenAI and launched in November 2022, is accessed as a separate tool and is better suited to summarizing complex code or generating a starting template for a specific coding task. Numerous other code-generation AI chat tools have emerged besides ChatGPT. Many of these can be accessed on their own, integrated into various code development tools or offered as a feature in some IDEs.

ChatGPT's LLM is trained on human language data. Because it is a general-purpose conversational AI platform that uses natural language processing to respond to user input, ChatGPT can provide answers to a broader range of questions outside of typical programming workflows.

The most common application development use case for ChatGPT is understanding and writing code based on the task description, said Agur Jõgi, CTO at CRM platform vendor Pipedrive.

Here is an example of ChatGPT explaining code submitted by the author.

Strengths and weaknesses of GitHub Copilot and ChatGPT

Here is a rundown of the respective strengths and weaknesses of each tool.

GitHub Copilot

Strengths. GitHub Copilot excels at generating code snippets and suggestions based on the context of the code being written, said Jorge Torres, co-founder and CEO at MindsDB, a platform designed to democratize ML.

Copilot can suggest lines of code, variables and function names relevant to the context of the code, and it can assist with granular snippets. GitHub Copilot also acts as a code completion agent that can complete tasks that the usual code completion tools built into the IDE cannot.

As such, it can save developers time and help them code more efficiently. In addition, GitHub Copilot can learn from the code that developers write, thereby improving its suggestions and accuracy over time.

Another code-writing strength of GitHub Copilot is its integration with the IDE, making it more efficient for real-time coding than ChatGPT and providing a better user experience overall for programmers, according to Jõgi. "Since it's embedded into the GitHub ecosystem, developers can use it without switching contexts or opening additional tools," he said.

Weaknesses. GitHub Copilot's weaknesses include generating incorrect or inefficient code suggestions, Torres said. It also might not be suitable for complex programming tasks that require extensive knowledge and expertise.

ChatGPT

Strengths. ChatGPT's strength is automating customer service interactions and providing virtual assistance for various tasks, Torres said. It can also generate text-based content, such as articles, stories and summaries, which makes it useful for content creation.

Povarov finds ChatGPT helpful for generating a wide range of code responses and explaining code concepts. It is also more suitable for nontechnical stakeholders and more flexible: Users communicate through a chatlike interface with a helper that can answer virtually any question. This flexibility lets users clarify and reformulate questions, leading to more refined answers and distinctions.

Jõgi also uses ChatGPT to generate function and unit tests to validate outcomes. This can streamline test-driven development practices.

Weaknesses. In terms of shortcomings, Torres finds that ChatGPT might struggle with complex or technical language, and not always generate accurate or relevant responses. It also might not be suitable for applications that require real-time interactions, such as gaming or trading.

Some researchers found that ChatGPT struggled to achieve the same level of measures for correctness, consistency, comprehensiveness and conciseness as answers written by humans on Stack Overflow. In addition, they observed that 52% of ChatGPT's answers contained inaccuracies, 62% were less concise than human answers and 78% of answers suffered from different degrees of inconsistency with human answers.

However, users still preferred ChatGPT responses approximately 39% of the time due to their comprehensiveness and more articulate language style. The danger, researchers wrote, is that users make occasional mistakes by preferring incorrect ChatGPT responses owing to better-articulated results and seemingly correct logic presented with positive assertions.

Similarities and differences between Copilot and ChatGPT

Both tools take advantage of OpenAI's GPT LLMs to generate results in different ways. They also suggest different paths for improving all kinds of tasks besides just programming. For example, Microsoft has rolled out a range of Copilots for various Office applications and is also integrating ChatGPT capabilities into its Bing search engine.

"They both represent the future of generative AI experience design, where AI-powered assistants can help us accomplish tasks more efficiently and effectively," Torres said. As AI technology continues to improve, he said, we will see ever more sophisticated AI-powered assistants that can understand human language and context more accurately and generate more complex and sophisticated outputs.

The tools also promise to make developers more productive by automating the writing of mundane, boilerplate code -- and could potentially make developers more effective by letting them spend more time on higher-level tasks. For example, GitHub's Copilot Chat interface adds a ChatGPT-like experience to explain code, suggest best practices, generate tests and fix errors. GitHub has also added new filtering to reduce the risk of generating existing code snippets that might impose licensing restrictions, as well as an AI-based vulnerability prevention system to help reduce instances of insecure code.

"In the future, we may see the integration of an AI assistant similar to ChatGPT directly into IDEs, enabling developers to communicate with it and request it to perform routine tasks, such as checking out code from repositories, running tests or building code," Povarov said.

If something goes wrong, such as a test failure or a build crash, the developer could ask the assistant to identify the error and suggest how to fix it. The AI assistant could do so, thereby reducing the time and effort required to maintain the codebase.

"The main difference [between the two] is the tight integration of Copilot with Visual Studio Code and the fact that Copilot is trained on vast amounts of code from GitHub and other places," explained Guido Hoffmann, technical fellow at Tech Soft 3D.

"Overall, Copilot is a great tool for quick, tactical tasks, and ChatGPT is better suited for broader tasks. Like any early technology, they both have areas of improvement and will get better and more advanced with time," Jõgi said.

Both ChatGPT and Copilot are free to try. A ChatGPT subscription with access to the latest language model is $20 per month. Copilot pricing starts at $10 per month for individuals and $19 per month for organizations.

Coding assistant alternatives to GitHub Copilot and ChatGPT

GitHub Copilot and ChatGPT are no longer the only games in town. Some coding assistants, such as Tabnine, actually preceded the recent buzz of using LLMs to generate code. In addition, other coding-specific LLMs have been developed that promise an improved ability to securely fine-tune the models on existing or proprietary codebases and enterprise coding practices. In no particular order, here is a list of promising alternatives.

IBM Watsonx Code Assistant uses pre-trained models based on specific programming languages to improve trust and efficiency. It can help enhance transparency by providing visibility into the potential origin of generated code, translate code from one language to another or modernize legacy code. Watsonx Code Assistant for IBM Z helps organizations refactor COBOL into Java.

JetBrains has been weaving AI coding assistant and chat capabilities across its portfolio of IDE tools. The features are powered by the JetBrains AI service that connects users to various LLMs. The assistant needs to be installed separately from the JetBrains marketplace. It includes a prompt library, AI chat, documentation generation, code explanation and commit message generation. Ask AI is a new feature in JetBrains' Datalore collaborative data science platform that is said to simplify coding in Python, R and SQL by supporting code generation and modification through natural language commands.

Google offers several coding assistance tools. Google Gemini now supports code analysis and development. Google Studio Bot provides a conversational experience like ChatGPT for Android developers. Google Colab, a data science development platform, uses Google's Codey LLMs optimized for code to generate larger blocks of code and write whole functions from prompts.

CodeGPT is an open source coding assistant extension that runs within the Microsoft Visual Studio IDE. It can connect to a wide range of LLMs optimized for code on the back end from OpenAI, Cohere, AI21 Labs, Meta and Anthropic. It can explain, improve and create new code directly within the IDE. A professional service starts at $9.99 plus taxes per user, per month, and supports context management, AI agent creation, GitHub repository synchronization and other features.

Cursor from Anysphere is a newer AI-powered code editor built around generative AI capabilities that combines elements of an AI copilot and chat interface to speed development workflows. It includes a tool for generating code with a particular dependency, and it can answer questions about a codebase, analyze third-party libraries and automatically debug code. There is a free tier for basic access; there are also plans priced at $20 per month and $40 per user, per month, based on access, features and control.

Coding assistant adoption: Proceed with caution

Developers need to proceed cautiously in adopting both coding assistants and chatbots, said Kevin Smith, CTO at systems integration consultancy Dootrix.

The first big challenge relates to data security. "If you're typing sensitive data into ChatGPT, you may be in violation of the law or in breach of nondisclosure," he said. It's also important to be clear regarding the terms under which you are using the tools and to invest in alternative plans or enterprise agreements where necessary.

The new tools also increase the need for accountability. These tools can help developers be more productive, but they shouldn't replace experts. "You should treat ChatGPT like you'd treat a junior developer; you still need to check its work, the results may still need some significant reworking, and in some cases, it might just be wrong," Smith said.

Future of coding assistants

Smith expects coding assistant capabilities to become more pervasive, much like how every word processor ended up with spelling and grammar checking. Every development tool will end up making use of AI to improve developer throughput.

The tools will likely become more advanced as well through innovations in developer experience and coding LLMs. Smith also expects future gains from more specific models trained on specific data sets and problem domains.

In the long run, coding assistants and chat interfaces could also converge. "Just as we all gravitated toward the search engine and started seeing little search boxes in all our apps, maybe the chat interface will end up dominating and be the primary way we interact with AI," Smith said.

Editor's note: This article was updated in July 2024 to improve the reader experience and update some tool descriptions.

This list was compiled based on extensive long-term research into the field of code completion tools and analysis of some of the leading players, as well as newer entrants into this field. It should be noted that some consideration was also given to a new crop of LLMs being developed to support code assistant and chat capabilities on the back end, but not the front-end developer experience directly. For brevity and clarity, we decided to focus this article only on those tools that currently implement code completion and code chat capabilities on the front end and that work in conjunction with one or more code completion LLMs on the back end.

George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.