Computing Machinery and the Individual: the Personal Turing Test - an Addendum
Rollo Carpenter and Dr Jonathan Freeman, 21st November 2005
1. A controlled experiment
In our paper of June 2005 we proposed a variant on Alan Turing's Imitation Game, also known as the Turing Test. For the new Impersonation Game a machine must adopt the persona of a human individual, and converse with a judge via text, or optionally with a fuller set of social presence cues. To win the game the machine must convince the judge either that it 'is' that person, or at least create sufficient uncertainty that the judge says 'I don't know'.
The judge must know the person being impersonated, yet the strength of personal connection cannot be determined. We proposed that 100 Impersonation Games be performed 'as' 100 different people in order to ensure a full and statistically significant range of individual circumstances, and that a machine or technology will pass the Personal Turing Test if 50% or more of the games are won.
Though it is statistically relevant pass mark, 50% is essentially an arbitrary figure. We now propose a modification to the Personal Turing Test, one that is perhaps more scientifically valid. The machine or technology should achieve the same or better percentage pass rate as humans do when impersonating one another. A set of Impersonation Games will be performed entirely amongst humans, to act as a control. Human 'actors', who might even be professionally-trained actors, will be armed with facts about and the characteristics of the person each is to impersonate. Any form of preparation is allowed, including written material, audio, video and personal interaction with the individual, with one the proviso for practical purposes that the actor's preparatory time should be limited to 12 hours. During the Game itself, communication is of course only allowed with the judge.
There should be 100 'control' Impersonation Games, which may take place in advance of the 'real' games, or simultaneously. In the latter case each Impersonation Game may be 3-way, the judge conversing in a random order with machine, human impersonator, and the actual human individual, scoring each as previously defined.
Each time a judge awards a human impersonator an 'I don't know' score, or one closer to 'that is the person I know' on the numeric scale outlined in the paper, it will count 1 towards the control percentage - that of successful human-to-human impersonation. This percentage, whatever it may be, will be the one that a machine or technology will need to equal or exceed to pass the Personal Turing Test.
In our first paper, we suggested that the degree of knowingness between any two individuals - the extent of knowledge about one another, and the nature of psychological bonds - is itself unknowable. Yet we can begin to identify some of its individual components - a task that may assist and inform the development of a Personal Turing Test machine.
A new project plans to investigate aspects relating to this domain. The Pasion project (Psychologically Augmented Social Interaction Over Networks) is currently in the final stages of its funding negotiations with the European Commission (Future and Emerging Technologies - Presence 2 call). It has a key goal of identifying potentially valuable implicit social communication cues that are not usually explicitly shared by people communicating with one and other, and understanding the potential benefits of sharing this information in a group context. Pasion researchers plan to continuously monitor users' physiological signals (such as heart rate, skin conductance, muscle activity) and behavioural indicators (such as variation in voice pitch, gaze direction, posture) in order to infer users' arousal levels and intentional states. The aggregation of data about multiple collaborating users' states has the potential to improve group awareness of the context of communication. An important question that Pasion will be investigating is how to meaningfully present such aggregated data to a collaborating (distributed) group.
Whilst Pasion has its own applications of knowledge work and social gaming as its focus, we see strong potential value in applying outputs of Pasion to an extension of a Personal Turing Test machine. In essence, future machines capable of passing the Personal Turing Test may benefit from contextual information relating to individuals and to groups as envisaged in Pasion, rather than inferring context solely from verbal (text) input. As we noted in our original paper, in addition to learning how likely people are to respond verbally to specific contexts, a future implementation of Jabberwacky may learn "how individuals 'are'; verbally and non-verbally, implicitly and explicitly."
3. Factual updates
The Personal Turing Test paper was published on the web in June 2005. In September 2005 the learning AI at Jabberwacky.com, previously discussed at some length, won the Loebner Prize Bronze Medal, which is awarded yearly to the 'most human' program. This was a first for a learning AI. Also during those five months, the size of the database employed by Jabberwacky has increased from 5 to 6.7 million entries. At the current, increasing rate of growth the 10 million figure, predicted in the paper to be the level at which the AI will appear human to most people in ordinary chat, will be reached in just over a year. We intend to evaluate and report at that time.