So the CIA has a program to train a chatbot to answer questions based on public data. This was to be expected — the US intelligence throws money at shiny things in ways that are not coincidentally very similar to the tech industry, although that is another post — but also it’s a horrible idea.
Not in the sci-fi scenario where people build a superintelligent spy and then act surprised when it turns against them, but in the usual scenario in which the wrong technology is incompetently implemented to achieve an undesirable goal, disingenuously used to justify indefensible actions, and then regretfully replaced for the next silver bullet. “Big data”-driven solutions to the problem of having too much intelligence data have, by themselves, a long, dishonorable, and very expensive history .
About this particular nonsense: Large language models don’t think in anything close to what you’d call a Aristoteles/Boole/von Neumann/Bayes model. They are stochastic parrots: they don’t operate on the domain of what words mean, they just generate statistically plausible text conditional on prompts. They are, as it has often been said, the world’s cheapest hard-working interns with eidetic memories, glib pens, and absolutely no knowledge of the world. You see the problem. Even if it’s only used to create “summaries” of public data “with links for validation”, this at best creates extra work for the analyst and, at worst (and most likely) it tempts everybody to treat these as proper summaries of standard knowledge and, on a pinch, “what the AI says.”
The whole point of having an intelligence analyst (instead of a dozen interns) is that this is not only insufficient but dangerous. Intelligence analysts start with the already questionable public data and build better knowledge out of them not by summarization but through analysis, which is an entirely different cognitive operation.
In fact, you can determine — perhaps even define — the degree to which something is a bureaucracy versus a knowledge organization by how well a large language model can do in there. In the pure Platonic form of a bureaucracy, textual patterns are everything and semantics are nothing, and stochastic parrots can and often do achieve power (specially if they also have true domain skills in internal politics and related areas). On the other extreme, a paper summarization program is not a research scientist, because doing research is finding out things that are not already summaries (in the textual sense) of what we already knew.