Steve Tingiris - Exploring GPT-3: An unofficial first look at the general-purpose language processing API from OpenAI

An unofficial first look at the general-purpose language processing API from OpenAI

Steve Tingiris

BIRMINGHAMMUMBAI

Copyright 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Kunal Parikh

Publishing Product Manager: Sunith Shetty

Senior Editor: David Sugarman

Content Development Editor: Nathanya Dias

Technical Editor: Devanshi Ayare

Copy Editor: Safis Editing

Project Coordinator: Aparna Ravikumar Nair

Proofreader: Safis Editing

Indexer: Rekha Nair

Production Designer: Alishon Mendonca

First published: July 2021

Production reference: 1100621

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80056-319-3

www.packt.com

To my wife, Brigid, for putting up with my constant dabbling for all these years. To my son, Alex, for motivating me to push a little harder. There is no way I would have finished this book without you guys. Thank you!

Steve

What's all the fuss about? Maybe it's the demos. But that would be missing the point. GPT-3 burst into public consciousness in July 2020 not too long after OpenAI first introduced the technical breakthrough in a soberly presented blog post. Introduced in the middle of a global pandemic, rising social unrest, and a US presidential campaign, the message could have been lost entirely. The demos started trickling out in earnest a month later. They ensured that GPT-3 wouldn't be overlooked.

Technology Review claimed the GPT-3 was shockingly good. Many long-time AI researchers expressed both enthusiasm and surprise at its capabilities. But it was the demos that really captured everyone's attention. With a few lines of sample content and a request, the technology was generating poetry, website programming code, analogies, and answers to math questions, to name just a few examples. No one had ever seen a computer create and respond creatively to such a wide range of queries.

GPT-3 seemed to possess magical abilities. In many ways, GPT-3 is very simple. It predicts what words are the most likely to follow in a sequence. However, it is also the finest current example of the potential of Generative Adversarial Networks (GANs). And, it has shown that transformers can be applied to language models at an extremely large scale. GPT-3 provides an intriguing new technical capability while simultaneously resetting expectations about what is possible. As just one example, maybe chatbots don't have to choose between a set number of deterministic responses.

Natural Language Processing (NLP) has improved quickly in recent years. That has led to the more accurate recognition and understanding of speech. At the same time, synthetic speech engines have improved immensely and sound more humanlike each year. What has changed very little is how the systems respond to requests. They are all picking from a predetermined set of responses. GPT-3 offers a capability that enables developers to rethink that approach. But that requires developers to understand the new technology and how to use it.

Steve Tingiris is the first to take that task on with this book. With clear and precise presentation, Tingiris expertly walks new users through the journey from idea through production of a GPT-3 application. He lays out the principles and steps so developers can turn their ideas into (a new) reality. I look forward to seeing what you build.

Bret Kinsella

Founder and CEO of Voicebot.ai

Steve Tingiris is the founder and managing director of Dabble Lab, a technology research and services company that helps businesses accelerate learning and adoption of natural language processing, conversational AI, and other emerging technologies. He has been designing and building automation solutions for over 20 years and has consulted on conversational AI projects for companies including Amazon, Google, and Twilio. He also publishes technical tutorials on Dabble Lab's YouTube channel one of the most popular educational resources for conversational AI developersand manages several open-source projects, including the Autopilot CLI, Twilio's recommended tool for building Autopilot bots. To connect with Steve, you can find him on GitHub@tingiris, via email to steve@dabblelab.com, or on Twitter @tingiris.

This book is the result of contributions from friends, colleagues, and many members of the OpenAI community. There are far too many people to mention everyone by name but for those who contributed directly or reviewed early drafts, I want to extend a special thank you.

First, thank you to the entire Packt team. If Sunith had not reached out with the idea of creating a book on GPT-3, this project would never have started. If David, Gebin, Nathanya, Aishwarya, Roshan, and Devanshi weren't involved the book might never have gotten finished. Thanks to each of you and everyone else on the Packt team who made this possible.

Thanks to Russell, Bakz, Ryan, and Bram for agreeing to be technical reviewers. You guys were so helpful to me in the OpenAI Slack channel when I first got started with GPT-3, I feel so fortunate to have had the opportunity to collaborate together on this book. Again, thank you!

Next, thank you to my work family at Dabble Lab. Kirk Owen for providing early feedback that helped refine the direction, and Mohamad Khalid, Manuel Fernandez, Mark Hovsepyan, Sohini Pattanayak, Shubham Prakash, and Daniela Ramirez, for picking up the slack while I spent much more time than I'd anticipated working on the book.

Finally, I'd like to thank the OpenAI team and the OpenAI community for all of the support and feedback throughout the project, including: Ashley Pilipiszyn, Mark Clintsman, Minal Chhatbar, Rene Diaz, Dan Shaw, Chris Fong, Dariusz Gross, Cristi Vlad, Jonathon Sauceda, Marc-Andre Schenk, Steven Kuo, Narendran Thillaisthanam, Matthew Benites, Manav Goel, Shubham Amraotkar, Mystici Mentis, Fred Zimmerman, Dmitry Kearo, CL Kim, Sudip Lingthep, Joakim Flink, Shubham Saboo, Pedro Ribeiro, Richard Klein, Steve Hoyt, Nicolas Garrel, Sebastian Derewicki, Vikram Pandya, Geoff Davis, Nelson Pereira, Heng Gu, Joey Bertschler, Surendra Reddy, James Morgan, Jon Oakes, Jeetendra K Sharma, Jim Taylor, Rebecca Johnson, Travis Barton, Herber Scrap, Pablo del Ser, Devin Bean, Nik K, Jason Boog, Mohak Agarwal, Sebastian Elliott, and Bjarne Carstensen.