Background Research

There are a variety of tasks that researchers perform in order to conduct background research and learn about both existing and new work in their research area. These include summarizing text, searching relevant literature, seeking to understand new concepts, and translating text from other languages.

Summarizing Text

First, LLMs are very capable at summarizing. They are adept at extracting the most relevant content from a passage of text, as illustrated here:

Paper Summaries

By contrast, the LLM Claude 2 can process texts with up to 100,000 tokens or about 75,000 words at the time of writing, which covers the length of almost all academic papers as well as many books. To provide an example, I used the "Upload'' button to feed the original Feb. 2023 NBER working paper version of this paper (Korinek, 2023)Korinek, A. (2023). Language models and
cognitive automation for economic research. NBER Working
Paper 30957
into Claude 2 and asked for a short summary — "short'' in order to not take up too much space here:

The model can respond to follow-up questions on the content, going into significant detail on the text provided. Additional examples of useful prompts include:
  • What is the evidence supporting these points [or a specific point from the summary]?
  • What are the author's main conclusions?
  • What are the most interesting (or most novel) ideas in this paper?
  • Can you explain the main findings of section [enter number]?
Claude allows the user to upload up to five documents and to ask the LLM to perform tasks such as comparing them or identifying connections between them.

Literature Research

At the time of writing, LLMs are of limited usefulness for searching and referencing the academic literature. When asked for citations or references, they frequently hallucinate and generate authoritative-sounding papers that do not actually exist. Although this has recently improved for cutting-edge LLMs — GPT-4 is significantly better in this respect than GPT-3.5 and is often able to provide standard references that are frequently cited in the literature — all references that LLMs generate should be carefully double-checked. The following is an example:

The reason why LLMs struggle with academic references is that they do not retain their training data verbatim, but instead retain an abstract representation of it that involves some loss of information. While this approach works for many general concepts, it falls short for academic references where precision is crucial.

However, there are alternative tools available that employ LLMs to help researchers find academic references. For example, LLMs with web connections such as Bing or Bard can be prompted to look up specific papers on the internet and generate reference information from their findings. Moreover, the website allows searches of the academic literature and displays a list of papers (that really exist) based on simple user prompts. It also includes a brief summary generated by an LLM.

Formatting References

LLMs can convert a whole bibliography in batch format, for example from APA to Chicago format or into bibtex format. For very highly-cited works that the LLM has frequently encountered in its training data, a prompt like "bibtex reference for stiglitz weiss'' also works well and saves the user from having to copy or type the detailed citation information of the work in question. However, it is advisable to double-check the information for all but the most highly cited papers.

Translating Text

During pre-training, LLMs develop representations of concepts that abstract beyond any single language, and this makes them able to easily map text from one language to the other. Performance on lower-resource languages for which less digitized text and fewer digitized translations are available is worse.

Explaining Concepts

LLMs can act as a tutor and explain many common economic concepts — at a level that is useful for students who are trying to learn new materials, and even for more advanced researchers who are branching out beyond their main area of expertise. For good explanations, it is critical to employ the most cutting-edge LLMs.

When employing LLMs in this way, follow-up questions that go into further detail can be very useful and allow the user to obtain personalized tutoring.

Other concept questions that the LLM answered well included the following:

  • What is the invisible hand theorem and why does it hold?
  • What is Monte Carlo simulation?
  • How does Cournot equilibrium differ from Bertrand equilibrium?
  • What are the leading theories of exchange rate determination?
But sometimes LLMs, especially earlier versions, hallucinate when providing answers. The following is an example using the older GPT-3.5 model text-davinci-003, which powered the original free version of ChatGPT in late 2022:

From: Generative AI for Economic Research: Use Cases and Implications for Economists
by Anton Korinek, Journal of Economic Literature, Vol. 61, No. 4, December 2023.
Copyright (c) by American Economic Association. Reproduced with permission.