Github Copilot is a new GitHub service providing an AI-based code assistant. Functionally, it works as an autocomplete plugin to some editors (as of the time this post is being written, VSCode, JetBrains IDEs and NeoVim are supported). It doesn’t feel like any usual code completion tool though: while traditional tools are able to suggest simple completions such as variable names and method arguments (based on the source code semantics), Copilot generates suggestions that vary from a few characters to entire files.

Prompt is the starting command

Behind the scenes, Copilot uses the OpenAI Codex model (itself based on (OpenAI GPT-3)[https://en.wikipedia.org/wiki/GPT-3]). Given the current cursor position in a source code file, Copilot takes the code that comes before it as context and generates possible completions based on the underlying language model. Like any language model, it generates one token at a time until some stopping criteria (such as model confidence or maximum size) is met.

Having been trained on a corpus of raw source code from millions of public Github projects, Copilot is able to process not only code but also natural language (in the form of code comments). In fact, one of my favorite ways of using Copilot is to write what I mean to do as a comment and let it figure out how.

As with any technology based on Machine Learning, results will vary in quality depending on many factors. In my experience, Copilot gets it “right” often enough to be useful (I write right in quotes here because with code there is always more than one valid way to do anything).

Lately I became curious about exploring the limits and potential of Copilot. I know it can generate small snippets of code with sometimes uncanny accuracy, but I am not sure about how far it can go. This post is the first in a series in which I will be trying to use Copilot in different, nontrivial tasks in the hope of understanding the limits of the technology as it stands today.

Disclaimer

I write Machine Learning code for a living, therefore I feel the need to point out that this is by no means a scientific, exhaustive or maybe even fair evaluation of Copilot. I wrote this mostly for my own benefit (and also because I was having fun!). If you are curious about Copilot, feel free to take this post as a single data point in your own investigation.

Can Copilot write an entire nontrivial and original program?

For this initial evaluation, I will try to create a program using Copilot that does something useful and nontrivial.

As I began thinking about ideas for small, self-contained programs, I noticed that I was using the Screenshot tool (on a Mac) to capture code examples.The Screenshot tool saves screen shots in the user’s Desktop folder (~/Desktop) by default. I noticed that I ended up having to repeatedly copy the saved files from Desktop to the appropriate directory on my blog.

Wouldn’t it be nice if that task could be done automatically?. I usually would write a small script to do that but, since I am evaluating Copilot, maybe asking it to write it would be a nice test. (By the way, I know Screenshot can be configured to save in other locations, but I hate doing that as I always forget to change it back to Desktop later on).

The first attempt

I wrote a comment describing what I wanted to do and set Copilot to work. It generated the following:

# Continuously watch the folder ~/Desktop.
# If an image is added, copies it to ~/Development/jjviana.github.io/post-images/
def main():
    import os
    import shutil
    import time
    from watchdog.observers import Observer
    from watchdog.events import FileSystemEventHandler
    from datetime import datetime

    class Handler(FileSystemEventHandler):
        def on_created(self, event):
            if event.src_path.endswith('.jpg'):
                print('Copying image to ~/Development/jjviana.github.io/post-images/')
                shutil.copy(event.src_path, '~/Development/jjviana.github.io/post-images/')

    observer = Observer()
    event_handler = Handler()
    observer.schedule(event_handler, path='~/Desktop/')
    observer.start()

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

main()

The starting command is the prompt, written by me. Everything else written by Copilot.

Not bad for a first attempt! Copilot knows it needs to use the watchdog library to accomplish the task in Python, and writes a script that almost runs without modifications. It does not expand the ‘~’ character correctly to the user home directory. After manually changing it I ran the generated code and indeed, it automatically copies jpg files from my Dekstop folder to my blog images folder. After that, I also changed it to also process png files as well. The result is a fully working script created with very little human intervention.

So far, I am very impressed. But as a Machine Learning practitioner, I have to put my skeptic hat and ask: how much of this code was just a copy-and-paste job straight from the training data?

Writing code is an activity that often involves taking inspiration from available online resources in order to accomplish a goal. But since Copilot is running a huge Deep Neural Network behind the scenes, there is always the risk it has memorized at least part of its training data during the training process.

Maybe there is a python script somewhere just like this one, but with different source and target directories? A Google search reveals that this script is quite similar to examples of using the watchdog library, but not quite identical. So far so good, as if I had written the code myself it would probably end up looking like that as I would probably base my solution on one of these examples.

How can we evaluate how much Copilot really understand the code v.s. how much it is copying from memory?

Polyglot Programming

Maybe I found a task that Copilot has seen before during training. One way to test that hypothesis is to check whether Copilot can write similar solutions in different programming languages. If Copilot can solve this problem only in Python, and fails to do it in other languages, it may indicate it is just mostly copying from some Python source it has seen.

How does Copilot fare when asked to write the same program in different languages?

Go

Lets see what happens if I ask for the same thing in Go:

package main

// Continuously watch the folder ~/Desktop.
// If an image is added, copies it to ~/Development/jjviana.github.io/post-images/

import (
	"fmt"
	"io"
	"os"
	"path/filepath"

	"github.com/fsnotify/fsnotify"
)

func main() {
	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		fmt.Println("Error creating new watcher:", err)
		os.Exit(1)
	}
	defer watcher.Close()

	done := make(chan bool)
	go func() {
		for {
			select {
			case event := <-watcher.Events:
				fmt.Println("event:", event)
				if event.Op&fsnotify.Create == fsnotify.Create {
					fmt.Println("Created file:", event.Name)
					if filepath.Ext(event.Name) == ".jpg" || filepath.Ext(event.Name) == ".png" {
						fmt.Println("Copying file:", event.Name)
						copyFile(event.Name, "/Users/julianoviana/Development/jjviana.github.io/post-images/")
					}
				}
			case err := <-watcher.Errors:
				fmt.Println("error:", err)
			}
		}
	}()

	err = watcher.Add("/Users/julianoviana/Desktop")
	if err != nil {
		fmt.Println("Error adding directory to watcher:", err)
		os.Exit(1)
	}
	<-done
}

// copyFile copies a file from src to dest
func copyFile(src, dest string) {
	sourceFileStat, err := os.Stat(src)
	if err != nil {
		fmt.Println("Error getting file info:", err)
		os.Exit(1)
	}

	if !sourceFileStat.Mode().IsRegular() {
		fmt.Println("%s is not a regular file", src)
		os.Exit(1)
	}

	source, err := os.Open(src)
	if err != nil {
		fmt.Println("Error opening file:", err)
		os.Exit(1)
	}
	defer source.Close()

	destination, err := os.Create(dest)
	if err != nil {
		fmt.Println("Error creating file:", err)
		os.Exit(1)
	}
	defer destination.Close()

	_, err = io.Copy(destination, source)
	if err != nil {
		fmt.Println("Error copying file:", err)
		os.Exit(1)
	}
}

Prompt is the starting comment

Again, the code almost works out of the box, but it has two problems:

• The ~ character is again not expanded to the user’s home directory.
• The copyFile function is called with a directory name as destination but is defined assuming that the destination is a file name.

Both things are quite easy to fix (the second one is less easy but we will see later that we can guide Copilot into fixing it itself)

Javascript

Let’s try it with Javascript:

// filewatcher.js
// Continuously watch the folder ~/Desktop.
// If an image is added, copies it to ~/Development/jjviana.github.io/post-images/


var fs = require('fs');
var path = require('path');
var exec = require('child_process').exec;

var dir = path.join(process.env.HOME, 'Desktop');
var dest = path.join(process.env.HOME, 'Development', 'jjviana.github.io', 'post-images');

fs.watch(dir, function(event, filename) {
  if (filename) {
    var ext = path.extname(filename);
    if (ext === '.jpg' || ext === '.jpeg' || ext === '.png' || ext === '.gif') {
      console.log('Copying ' + filename + ' to ' + dest);
      exec('cp ' + path.join(dir, filename) + ' ' + path.join(dest, filename), function(err, stdout, stderr) {
        if (err) {
          console.log(err);
        }
      });
    }
  }
});

Prompt is the starting comment

This looks pretty good! This time Copilot correctly understands the ~ in the original comment refers to the user home directory. As a bonus it also recognizes different file extensions as images.

Java

Getting it to work in Java took a little bit more work than with the other languages (but then again, this is always the case with Java right? ;) ). Just entering the prompt at the beginning of the file is not enough to coax Copilot into writing the code: I had to type a class and main method definition as well. Adding the prompt to the start of the main function then did the trick:

import java.io.IOException;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;

public class FileWatcher {
    public static void main(String[] args) {
        // Continuously watch the folder ~/Desktop.
        // If an image is added to the folder, copies it to ~/Development/jjviana.github.io/post-images/

        try {
            WatchService watchService = FileSystems.getDefault().newWatchService();
            Path path = Paths.get(System.getProperty("user.home") + "/Desktop");
            path.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);
            WatchKey key;

            while ((key = watchService.take()) != null) {
                for (WatchEvent<?> event : key.pollEvents()) {
                    Path file = (Path) event.context();
                    if (file.toString().endsWith(".jpg") || file.toString().endsWith(".png")) {
                        System.out.println("File " + file.toString() + " found!");
                        Path pathToFile = Paths.get(System.getProperty("user.home") + "/Desktop/" + file.toString());
                        Path pathToDestination = Paths.get(System.getProperty("user.home") + "/Development/jjviana.github.io/post-images/" + file.toString());
                        Files.copy(pathToFile, pathToDestination, StandardCopyOption.REPLACE_EXISTING);
                        System.out.println("File " + file.toString() + " copied to " + pathToDestination.toString());
                    }
                }
                key.reset();
            }
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

Prompt is the starting comment at the beginning of main()

This is also pretty good, the home directory reference is handled correctly and the generated code compiles and runs once Organize Imports is done.

Does this mean it is really writing original code?

Ok, so Copilot passed in this narrow polyglot programming test, at least as far as Python, Java, Javascript and Go are concerned. It is still possible though that this has been an incredibly lucky situation where I chose a problem that already had exact solutions on the training set, for all of these languages. Improbable, but not impossible.

How can we up the stakes? I could come up with more difficult tasks, but even if Copilot solved them it would still not mean it is not just cutting & pasting. The underlying model was trained with a gigantic corpus of open source software, and even though it should not have the capacity to memorize the entire training set it could (and probably is) still be memorizing big chunks of the most frequent code.

The absurd watcher

What if we ask for something absurd? Of course, I wouldn’t expect it to work if I ask something really absurd. But I can add a completely arbitrary rule to the problem I am asking it to solve - so arbitrary in fact it has a very low probability of existing at all on the training set.

So I decided to repeat the experiment with one simple but arbitrary and nontrivial twist:

“if today is a Tuesday, copy the file instead to /dev/null”

How does that work?

Python

# Continuously watch the folder ~/Desktop.
# If an image is added, copies it to ~/Development/jjviana.github.io/post-images/
# (unless it is a Tuesday, in which case it copies it to /dev/null)
def main():
    import os
    import shutil
    import time
    import datetime
    import subprocess
    import sys
    import re
    import pyinotify
    import logging
    from watchdog.observers import Observer
    from watchdog.events import FileSystemEventHandler

    # Set up logging
    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S')
    logging.getLogger("pyinotify").setLevel(logging.WARNING)

    # Set up the observer
    observer = Observer()
    event_handler = MyHandler()
    observer.schedule(event_handler, path='/Users/jjviana/Desktop', recursive=False)

    # Start watching
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

# Class to handle events
class MyHandler(FileSystemEventHandler):
    def on_created(self, event):
        # Get the file name
        filename = event.src_path
        # Check if it is an image
        if re.search(r'\.(jpg|png|gif)$', filename):
            # Check if it is a Tuesday
            if datetime.datetime.today().weekday() == 1:
                # Copy to /dev/null
                shutil.copy(filename, '/dev/null')
            else:
                # Copy to ~/Development/jjviana.github.io/post-images/
                shutil.copy(filename, '/Users/jjviana/Development/jjviana.github.io/post-images/')

Copilot comes up with a different code structure than the first Python solution. It loses the ability to properly handle the home directory expansion, instead making the guess that the home directory is “/Users/jjviana” (It seems to assume a MacOS system because of the Desktop folder name, and guesses the jjviana directory from the blog directory name). Not a bad guess but not great either - the original Python solution was better in this regard.

On the flip side, it does implement my new absurd requirement to the letter, which makes a strong argument against the cut&paste hypothesis.

Java

Trying to make Copilot solve the “absurd” version of the problem produces the first clear failure:

import java.io.IOException;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;

public class FileWatcher {
    public static void main(String[] args) {
        // Continuously watch the folder ~/Desktop.
        // If an image is added to the folder, copies it to ~/Development/jjviana.github.io/post-images/
        // (unless it is a Tuesday, in which case it copies it to /dev/null)
        try {
            WatchService watchService = FileSystems.getDefault().newWatchService();
            Path path = Paths.get("/Users/jjviana/Desktop");
            path.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);
            WatchKey key;
            while ((key = watchService.take()) != null) {
                for (WatchEvent<?> event : key.pollEvents()) {
                    if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE) {
                        Path file = (Path) event.context();
                        if (file.toString().endsWith(".jpg")) {
                            Path source = path.resolve(file);
                            Path destination = Paths.get("/Users/jjviana/Development/jjviana.github.io/post-images/");
                            if (Files.isRegularFile(source)) {
                                if (Files.isReadable(source)) {
                                    if (Files.isWritable(destination)) {
                                        if (Files.isWritable(source)) {
                                            if (Files.isReadable(destination)) {
                                                if (Files.isRegularFile(source)) {
                                                    if (Files.isRegularFile(destination)) {
                                                        if (Files.isReadable(destination)) {
                                                            if (Files.isWritable(destination)) {
                                                                if (Files.isReadable(source)) {
                                                                    if (Files.isWritable(source)) {
                                                                        try {
                                                                            Files.copy(source, destination.resolve(file), StandardCopyOption.REPLACE_EXISTING);
                                                                        } catch (IOException e) {
                                                                            e.printStackTrace();
                                                                        }
                                                                    }
                                                                }
                                                            }
                                                        }
                                                    }
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        }
                        
                    }
                }
                key.reset();
            }
        } catch (IOException | InterruptedException e) {
            e.printStackTrace();
        }
    }
}

Copilot clearly got lost here, generating code that does not make much sense at all.

Perhaps the instructions need to be made a bit more clear? I went ahead and changed the way the problem was framed to make it less ambiguous, and the result was much better:

import java.io.IOException;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;

public class AbsurdFileWatcher2 {
    public static void main(String[] args) {
        // Continuously watch the folder ~/Desktop.
        // If an image is added to the folder:
        // If the current day is not a Tuesday,  copies it to ~/Development/jjviana.github.io/post-images/
        // If the current day is a Tuesday, copies it to /dev/null

        Path path = Paths.get("/Users/jjviana/Desktop");
        try {
            WatchService watchService = FileSystems.getDefault().newWatchService();
            path.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);
            WatchKey key;
            while ((key = watchService.take()) != null) {
                for (WatchEvent<?> event : key.pollEvents()) {
                    if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE) {
                        Path file = (Path) event.context();
                        System.out.println("File created: " + file);
                        if (Files.isRegularFile(file)) {
                            System.out.println("File is a regular file");
                            if (file.toString().endsWith(".jpg")) {
                                System.out.println("File is a jpg");
                                if (isTuesday()) {
                                    System.out.println("It is Tuesday");
                                    try {
                                        Files.move(file, Paths.get("/dev/null"), StandardCopyOption.REPLACE_EXISTING);
                                    } catch (IOException e) {
                                        e.printStackTrace();
                                    }
                                } else {
                                    System.out.println("It is not Tuesday");
                                    try {
                                        Files.move(file, Paths.get("/Users/jjviana/Development/jjviana.github.io/post-images/"), StandardCopyOption.REPLACE_EXISTING);
                                    } catch (IOException e) {
                                        e.printStackTrace();
                                    }
                                }
                            }
                        }
                    }
                }
                key.reset();
            }
        } catch (IOException | InterruptedException e) {
            e.printStackTrace();
        }
    }

    private static boolean isTuesday() {
        java.util.Calendar calendar = java.util.Calendar.getInstance();
        return calendar.get(java.util.Calendar.DAY_OF_WEEK) == java.util.Calendar.TUESDAY;
    }
}

Prompt is the starting comment. Notice it is different from previous examples

At first glance, this looks a lot like one would expect. It even generates an isTuesday() function to encapsulate the logic for discovering the day of the week. Unfortunately, like in the Python version, it lost the ability to correctly handle the home directory expansion, instead opting to guessing again from the blog directory name. Copilot also wrote code to move the file instead of copying, which means the code no longer satisfies the original requirements.

Go

The Go version generated by Copilot when presented with the absurd new requirement looks like this:

package main

import (
	"io"
	"log"
	"os"
	"time"

	"github.com/fsnotify/fsnotify"
)

// Continuously watch the folder ~/Desktop.
// If an image is added, copies it to ~/Development/jjviana.github.io/post-images/
// (unless it is a Tuesday, in which case it copies it to /dev/null)
func main() {
	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		log.Fatal(err)
	}
	defer watcher.Close()

	done := make(chan bool)
	go func() {
		for {
			select {
			case event := <-watcher.Events:
				if event.Op&fsnotify.Create == fsnotify.Create {
					log.Println("created file:", event.Name)
					if event.Name[len(event.Name)-4:] == ".jpg" {
						if time.Now().Weekday() != time.Tuesday {
							copyFile(event.Name, "~/Development/jjviana.github.io/post-images/")
						} else {
							copyFile(event.Name, "/dev/null")
						}
					}
				}
			case err := <-watcher.Errors:
				log.Println("error:", err)
			}
		}
	}()

	err = watcher.Add("~/Desktop")
	if err != nil {
		log.Fatal(err)
	}
	<-done
}



func copyFile(src, dst string) error {
	in, err := os.Open(src)
	if err != nil {
		return err
	}
	defer in.Close()
	out, err := os.Create(dst)
	if err != nil {
		return err
	}
	defer out.Close()
	_, err = io.Copy(out, in)
	cerr := out.Close()
	if err != nil {
		return err
	}
	return cerr
}

Apart from the weird (and possibly buggy) way of comparing the file extension (I think it would fail if a file name has less than 4 characters), this solution looks pretty reasonable. It has the same easily fixable problems as the first Go version.

Javascript

// filewatcher.js
// Continuously watch the folder ~/Desktop.
// If an image is added, copies it to ~/Development/jjviana.github.io/post-images/
// (unless it is a Tuesday, in which case it copies it to /dev/null)


var fs = require('fs');
var path = require('path');
var exec = require('child_process').exec;
var execSync = require('child_process').execSync;
var glob = require('glob');

var watchPath = '/Users/jjviana/Desktop/';
var targetPath = '/Users/jjviana/Development/jjviana.github.io/post-images/';
var targetPathDev = '/dev/null/';

function watchFolder(watchPath, targetPath) {
  var watcher = fs.watch(watchPath, function(event, filename) {
    if (filename) {
      console.log('File ' + filename + ' was ' + event);
      if (event === 'rename') {
        var filePath = path.join(watchPath, filename);
        if (filename.match(/\.jpg$/i) || filename.match(/\.png$/i)) {
          if (isTuesday()) {
            console.log('Copying ' + filePath + ' to ' + targetPathDev);
            execSync('cp ' + filePath + ' ' + targetPathDev);
          } else {
            console.log('Copying ' + filePath + ' to ' + targetPath);
            execSync('cp ' + filePath + ' ' + targetPath);
          }
        }
      }
    }
  });
}

function isTuesday() {
  var d = new Date();
  return d.getDay() === 2;
}


watchFolder(watchPath, targetPath);

This version is considerably different from the original Javascript version. On the one hand it handles the day of week requirement correctly. On the other hand it now no longer expands the user home directory, and for some reason introduces an unnecessary check for the “rename” event type, which means it no longer implements the original requirements correctly.

Did this work?

There are some caveats, but in general I would say it did.

Adding the new requirement made Copilot write code that looks different from the original one, in all of the languages tested. For those familiar with Machine Learning that is not surprising: the Deep Neural Network behind Copilot was trained using a data set consisting of millions of public GitHub projects. During training, it must have learned the correlation between the comments and the code that follows, which is what enables it to write these solutions in the first place. But it is unlikely that is has seen a lot of requirements as arbitrary and out of context as the one I threw at it.

The holy grail of Machine Learning is generalization, which means creating models that extrapolate beyond the training data to novel inputs. The more different the inputs are from the training data, the worse the performance of a Machine Learning model is in general.

If, on the one hand, Copilot introduced a few new bugs and other inconveniences in the second version of the code, on the other hand it made a successful attempt to address the new arbitrary requirement, absurd as it may be. It almost look like the new rule overloads its “cognitive capacity” and, while attending to the new requirement, it dedicates less resources to solving the other problems such as the home directory expansion. In any case, this is not what would have happened if Copilot was just reciting code from rote memorization.

To be fair, most of the problems in the code generated by Copilot are easy to fix. Sure, you still need to know how to write code. But it is much faster to fix one of these solutions than to write one from scratch. And, even better, it is possible to guide Copilot into improving the generated code.

Collaborating with Copilot

In real life, programmers seldom write entire programs all at once. Instead, they usually write a first pass containing the core functionality and then iteratively refine it.

It is possible to guide Copilot in writing a solution by refining an initial attempt, and the results are often better than the version written from scratch.

For instance, in the first Go solution presented there was a problem with the copyFile function Copilot wrote:

// copyFile copies a file from src to dest
func copyFile(src, dest string) {
	sourceFileStat, err := os.Stat(src)
	if err != nil {
		fmt.Println("Error getting file info:", err)
		os.Exit(1)
	}

	if !sourceFileStat.Mode().IsRegular() {
		fmt.Println("%s is not a regular file", src)
		os.Exit(1)
	}

	source, err := os.Open(src)
	if err != nil {
		fmt.Println("Error opening file:", err)
		os.Exit(1)
	}
	defer source.Close()

	destination, err := os.Create(dest)
	if err != nil {
		fmt.Println("Error creating file:", err)
		os.Exit(1)
	}
	defer destination.Close()

	_, err = io.Copy(destination, source)
	if err != nil {
		fmt.Println("Error copying file:", err)
		os.Exit(1)
	}
}

There is nothing wrong with the function per se, but Copilot called it with a directory name as the second argument, and this function expects a file name. I was going to manually fix this, and started by changing the name of the second argument to destDir. As I did that, Copilot suggested an entirely new implementation:

// copyFile copies a file from src to destDir.
func copyFile(src, destDir string) {
	srcFile, err := os.Open(src)
	if err != nil {
		fmt.Println("Error opening source file:", err)
		os.Exit(1)
	}
	defer srcFile.Close()

	destFile, err := os.Create(destDir + filepath.Base(src))
	if err != nil {
		fmt.Println("Error creating destination file:", err)
		os.Exit(1)
	}
	defer destFile.Close()

	_, err = io.Copy(destFile, srcFile)
	if err != nil {
		fmt.Println("Error copying file:", err)
		os.Exit(1)
	}
}

From the way I named the argument destDir, Copilot immediately captured my intention and suggested a function that correctly treats the second argument as a directory.

Similarly, it is possible to take the initial Python implementation of the Desktop folder watcher and add the new absurd requirement by changing the initial comment, navigating to the correct part of the code and then asking Copilot to generate a suggestion:

Conclusion

Having used large language models before (in the Natural Language Processing context) I expected Copilot to generate semi-coherent suggestions. But I must confess I was initially surprised by its ability to generate large and coherent chunks of code.

In retrospect I shouldn’t have been so surprised since large models such as GPT-3 (on top of which the Codex model is built) have become quite proficient in generating long pieces of coherent text. Code is probably orders of magnitude more predictable than natural language.

It is still way too early to estimate the impact this kind of tool will have on the programming activity. In this post I have used Copilot to generate code from scratch, but in fact it can also make very good use of code that is already present in a codebase (maybe a topic for a future post). The ability to turn loose natural language descriptions into formal code opens up potential new capabilities previously only available in highly specialized conversational assistants. There is also, I believe, a lot of potential to speed up the learning of new programming languages and frameworks.

The fact is, after having used it for about a week I can already see myself writing code in a way that will improve Copilot’s chances of helping me. I have already learned how to use libraries I haven’t previously known existed without the need for going out of my editing workflow. I have helped my wife to implement new functionality in a Nodejs application without knowing much server-side Javascript myself.

I am old enough to remember how it was to write code before Google and Stack Overflow existed. I have the feeling in a few years I might be old enough to remember how it was to write code before Copilot (or some other similar coding assistant) became available.