"Neurons" in a neural network aren't very neuron-like. The historical inspiration is there, but in practice each of them is closer in complexity to a mechanical gear than to a cell (if this seems hyperbolic, see for example Ken Shirriff's fascinating look at what an electromecanical Central Air Data Computer could do using gears for most of the actual computation - and this as late as the mid-50s).

So it's conceptually straightforward, if perhaps very much impractical, to think of precisely implementing, say, ChatGPT as something like a city-sized mesh of gears and cams. Think of it as the opposite of the Turing test setup: instead of putting a machine in a context where you can't immediately tell whether or not it's a human, build it in a way that's inescapably, viscerally machine-looking.

Imagine yourself at the border of this city, a keyboard in front of you. You type a question, the same one you'd ask ChatGPT. You pull a lever. Gears start moving from where you are, movement rippling across the heretofore still city, a clanking wave that hours or perhaps days later circles back across a different path to a mechanical printer that outputs an answer as distressingly plausible as any you'd get from ChatGPT because it's exactly the same program, just on a different computer.

I suspect the psychological and social response to this would be very different. Somebody familiar with, for example, Stanislaw Lem's stories would look at the mechanical sea and wonder about frozen libraries encoded as topologies, a necromancy of dead thoughts, while others with a more engineering bent of mind would talk about "mechanical librarians" — what nobody would do would be write articles about whether it's self-aware or how close to human it is.

It wouldn't be less powerful, it wouldn't be less sophisticated, and it would look stranger, but it would lack the sense of uncanniness. I believe a lot of our societal responses to interactive generative text are due to the fact that we interact with the software through the same interfaces and patterns with which we interact with other humans, by exchanging text messages across a computer interface. Because the obviously non-human aspects of it — basically that's a very large but structurally relatively homogeneous set of basic operations with data-tuned parameters — are hidden behind the same software structure that often mediates our contact with other people, it feels to us more human-like that it warrants. The features, good and bad, of the text it generates are both undeniable, but we would think about them in a much different way if it were viscerally clear that they come from a computer in a different sense than these words you are reading also come from a computer.

City-sized Steampunk text generation machines would be impractical (but cool-looking) and not really useful (but cool-looking); the point isn't that we should build them. But if when interacting with or thinking about ChatGPT and its ilk you hear behind the faint clicking of a landscape of gears you'll have a better intuitive sense of what's happening, in all its beautiful strangeness.