Overview
tty prints the file name of the terminal your input comes from, such as /dev/ttys003 or /dev/pts/0. When input is not a terminal, for example from a pipe or a file, it prints not a tty and exits with status 1. Scripts use it to decide whether they are running interactively.
Background
A tty command appeared in Version 1 AT&T UNIX, according to the macOS manual, which also says it is POSIX.2 compatible. Today the name covers GNU tty, the tty that macOS ships, and the tty applet in BusyBox. They agree on printing the terminal name or not a tty, and on the exit status.
Examples
Ask when input is not a terminal; /dev/null is not one, so the answer is not a tty and the status is 1:
tty < /dev/null; echo $?
Use the exit status in a test:
tty < /dev/null || echo not a terminal
Branch on it in a script:
if tty > /dev/null < /dev/null; then echo interactive; else echo batch; fi
Common mistakes
tty looks at standard input, not output, so tty | cat in a terminal still reports the terminal, while cat file | tty reports not a tty. To test standard output use test -t 1, and to test input in a script test -t 0, which needs no process. Inside a program run from cron, a container or an editor, there may be no terminal at all.
Portability
Only plain tty and its exit status are in the Common examples, run with input from /dev/null so the answer is the same everywhere. The silent option -s is described in the sections below. The Common examples were executed on GNU coreutils 9.11, macOS system tools, and BusyBox 1.37.0 (see the development guide for the exact check).
About GNU tty
GNU tty is part of GNU coreutils, and this page's checks ran coreutils 9.11. -s prints nothing and only sets the exit status.
GNU examples
Use the silent form -s in a test:
tty -s < /dev/null || echo not a terminal
Options
-s
Print nothing and exit with status 0 only when input is a terminal.
References
Original explanations, with upstream documentation for further reading. FreeBSD references are comparison material, not a claim of Apple version equivalence.