User Psychology and Biometric Systems Performance

This paper will address the broader biometric systems performance issue and introduce the subject of user psychology in relation to actual realised performance of biometric systems. The suggestion is that the traditional biometric performance metrics are not sufficient to predict actual systems performance across a broad variety of operational conditions. Furthermore, it is proposed that elements of user psychology undoubtedly affect realised biometric systems performance and as such should be considered a performance parameter when planning or designing a biometric system. Any discussion around FAR, FRR, EER and other accepted metrics should thus be qualified with an understanding of the user base and associated psychology.

At the end of the paper, the original User Psychology Index document is included for reference.

1. Traditional biometric performance metrics

As biometric technology has evolved, a de facto set of measures have served as basic performance benchmarks. These typically include, FAR (false acceptance rate), FRR (false rejection rate), EER (equal error rate), and verification time. However, there are various caveats around the provision and interpretation of such figures.

Firstly, without knowing precisely how the figures were arrived at, the prospective user or systems integrator is relatively little the wiser as to expected performance of a given device within a particular application. Device manufacturers performance figures may have been arrived at via internal laboratory tests, or perhaps via tests in a particular representative environment. How many samples were taken to arrive at these figures? Were results from a small user population extrapolated to provide the quoted performance figures? What was the profile of the test population? Or indeed, are the quoted figures based upon calculation rather than evaluation and observation? Secondly, one or more of these figures may be quoted out of context. For example, an FAR figure on its own has limited merit in helping to understand how a given device might perform overall. FAR, FRR and EER should ideally be quoted together with an indication of threshold level settings where appropriate. Similarly, verification times should ideally be quoted in an operational context, not a figure based upon the theoretical processing power of the device in question, or the software algorithm running on a high specification work station.

From the above comments we might conclude that the traditional performance metrics quoted for biometric devices are certainly useful as an initial guide to the systems integrator, but without detailed qualification they remain simply an indication of theoretically possible performance, rather than an implied guarantee of expected performance within a real system, under real operational conditions.

2. Systems performance

Another issue around traditional biometric performance metrics is that they tend to be based upon the device in isolation. If the device is capable of operating in a true stand alone configuration, and if this is how the user wishes to use the device, then there may be some relevance in this respect. However, more typically the device will be required to work within a larger overall system or process. A network of devices may be envisaged, with processing being undertaken centrally via a template matching ‘engine’. In such a configuration, the performance of the network overall and the time taken to pass template data and deliver the matching result across the network must obviously be taken into consideration. Perhaps the biometric application is deployed across an existing network and thus shares the network with other applications and services. In such a case, there may be greater or lesser amounts of network traffic at certain times and greater or lessor demands upon available central processing power according to other requirements such as batch processes and so forth. How will this affect the realised performance of a biometric system deployed in this manner? In addition, there is the issue of concurrent usage of the biometric devices themselves to consider. The user experience and perception of system performance could thus be quite different at different times of the day, under different load conditions.

So far, we have spoken in rather general terms about systems and provided a flavour of a concept which the author describes as TSP - Total Systems Performance. Perhaps TSP should be regarded as a yardstick for describing biometric systems performance in the real world. But let’s be more specific. What sort of database technology is your biometric system using? What is the specification of the server? How much spare capacity is provided? How much memory is available? What is the optimal client specification? There is much corporate interest currently in web based technology for distributing information, both internally and externally. In such an architecture, what is the relationship between back end database and web server? What are the browser requirements at the client end? There are a host of variables which need to be considered within any medium or large scale system, before we even think about overlaying the biometric functionality. In such a situation, one must concede that the device manufacturers quoted performance figures require a little qualification and interpretation in order to be meaningful. Even within a smaller closed loop system, we cannot ignore the inherent performance characteristics of other systems components and the cumulative effects that they may have on perceived performance at the point of verification.

3. User psychology

We have now expanded our thinking from individual device performance through to considering the total systems performance. However, this is still not the complete picture with regard to biometric systems performance as there is the interaction between man and machine to consider and the user psychology inherent in every biometric transaction.

What do we mean by user psychology and why should biometric systems be any different from other systems in this respect? After all, most systems involve some sort of user interaction at some point. This is undoubtedly true, but there are some important distinctions to be made. Firstly, there is the distinction between a ‘professional’ user such as a systems administrator or somebody that uses a specialist application as part of their job, and a ‘general’ user who is required to use the system, either in relation to their working activity or in order to receive some benefit within a public system. The professional user will typically have a certain interest and understanding of technology issues and will be keen to understand the operation of the system and how to use it properly in order to realise the associated benefits. The general user may or may not have an interest in technology and may feel slightly intimidated by what they see as increasingly complex procedures and operations which they are required to learn. The training requirements will therefore be a little different for the two groups.

The above points may be considered to apply to any IT related system, whether or not biometrics are employed. In a biometric system however, we have a further complication in that not only is the user required to learn the operation of a new high technology system, but they are also required to interact with it in a particularly personal manner (supplying their biometric trait) which they have not previously had to do. Furthermore, if they do not use the system absolutely correctly, they stand a real chance of being rejected by it and not receiving the associated benefit. For most users, this is going several steps beyond that which they are familiar with and may initially seem a little intimidating.

In the above paragraph, I referred to rejection. Let’s consider this from a psychology perspective. People do not like to be rejected. They do not like social or professional rejection. They do not like to be rejected when they have applied for some benefit. They especially do not like being rejected in public. Indeed, many people fear rejection in any shape or form. This is natural, as rejection is not pleasant in the majority of cases. Some people’s fear of rejection causes them to act in an unnatural manner, either saying things which are not quite correct or perhaps behaving in a manner which is slightly out of character. In the context of a biometric transaction in a public place, this might manifest itself in a number of ways, with the user reacting against what they perceive as rejection either with embarrassment or possibly anger. Their subsequent attempts at verification are likely to be affected by this heightened emotion.

We can witness a related ‘fear of rejection’ syndrome sometimes in group training sessions where an individual may not wish to admit that they do not understand a particular point, for fear of embarrassment, or somehow being ‘rejected’ by their peers as not up to the mark. They therefore go along with the flow, hoping that the point they do not understand will prove to be unimportant. This might happen for example if a group of individuals are being instructed as to the correct use of a biometric system. If the new user did not understand the importance of correct and consistent placement of their finger on a fingerprint reader for example, they are likely to struggle with their early attempts at using the device. When they are rejected (through no fault of the biometric technology), they will wonder whether it is because they are doing something wrong. This will itself have a psychological effect, increasing pressure upon them and possibly causing embarrassment into the bargain. This will not endear them to the biometric system, but importantly, what affect is this having on realised performance of the system?

Let’s imagine that we have set up a biometric ATM machine in a local bank and have issued 1000 special bank cards which contain a biometric template for matching against the live sample via the ATM process. It is Saturday morning and there are five people queuing to use the machine. The person currently using it is having difficulty with the biometric verification. They supplied their biometric and were rejected. They repeated the process and were rejected again. Perhaps there is a good reason for this. Maybe they have not inserted the card properly, or did not wait for the proper signal to supply their biometric sample. In any event, it isn’t working and they are conscious that there are five people waiting behind them, one or two of which are getting restless and showing signs of impatience. They try again. The probability that this person will correctly undertake the biometric verification process according to how they were instructed and be accepted accordingly is diminishing with each failed attempt. Again, how is this affecting the realised performance of the system? The next user may have clearly understood how best to use the system and be accepted straight away without any difficulty. Let’s consider this in the context of a single individual. Suppose that I have understood my introductory training and have successfully used the system many times, but this morning I am rushing as I have an important meeting to attend which is currently dominating my thoughts. I supply my biometric in a slightly awkward manner and the system rejects me. Fortunately, I understand the likely reason why and supply me biometric again, this time more attentively and am accepted. For me this inconsistency is perhaps a rare occasion. Now let us consider another individual for whom such inconsistency is more prevalent. Let us also assume that this individual is more impatient by nature and less technically understanding. Given the same configuration, what are the relevant performance figures going to be over the course of 1000 transactions for each of these individuals? Then there is the individual who is fundamentally opposed to the deployment of such systems, even though he or she is required to use them. For this individual, a rejection may simply reinforce their antipathy towards the process. Another type may be the individual who has a certain technical understanding but is highly sceptical as to the capabilities of biometrics and sees each rejection as vindication of their stance. If we place either of these individuals in a busy situation where they are rejected by the system in a public setting, their reactions are going to be interesting and will almost certainly have an affect on the performance of the system. With a concurrent userbase of perhaps 200 individuals, how many will have problems of this nature and what affect will this be having on the false reject performance of the system overall? It may be possible, with time and observation, to categorise users who are likely to experience operational problems and be prepared to adjust the system configuration selectively for those individuals - assuming that the chosen system allows for this. This is an area for the systems administrator to consider.

From the above points we can deduce that training and communication are key elements in the implementation of any biometric system. We can perhaps also acknowledge that human beings are both complex and individual and that it is vitally important to understand your user population and to tune the system accordingly. What is the percentage of elderly people for whom the physical challenge of using such a system may be greater? What is the balance between male and female, technically aware or otherwise and how does this affect the training requirements and the manner in which the system is presented? Are there any noticeable trends according to ethnicity with your chosen biometric, and do these have an affect on performance? There are a number of parameters and variables to understand in this context and if we are planning a large scale implementation, we might usefully run some sort of pilot scheme in order to understand these issues and their implications.

In conclusion, the fundamental point being raised in this paper is that quoting biometric systems performance based upon the manufacturers specifications for an individual device is never going to accurately represent the realised performance within a given environment and with a particular user base. Indeed, the realised performance may even become better than that suggested by the manufacturer in certain cases, although typically the reverse will be true. It is important to understand user perception and reaction to the idea of implementing biometrics before you finalise the detailed system design. It is equally important to configure an appropriate training and communications programme in order to cover any negative perceptions and explain exactly how the system will be operating (including the use and storage of biometric templates) and how to use it effectively. Only when all the variables have been understood and the user base has been properly educated can we realistically configure the system for the best performance and understand what that performance is likely to be. We may be pleasantly surprised. The system may deliver a level of performance which is more than adequate for our requirements and we may be able to configure it accordingly to ensure an absolute minimal occurrence of false rejection whilst still offering a high level of defence against the potential impostor. Of course, psychology also comes into play in this respect as there exists a strong deterrent factor when the criminally inclined realise that a biometric system is in operation.

In any event, when we are discussing the implementation of biometric technology, we simply cannot ignore the users and the affect that they can have upon the realised performance of the overall system. For a given systems specification, a well trained and enthusiastic user base will realise a much higher intrinsic level of performance than a disinterested user base who have not had the background and benefits of the system adequately communicated and have not received quality training. In the latter case, this not only affects their inherent capability to use the system correctly, but also their attitude towards the system and how they interact with it under real operational conditions. It is suggested therefore that user psychology represents as much a performance parameter as any of the established measures and should be taken into consideration accordingly. Like other parameters, it may be intelligently manipulated in order to improve the overall performance of a given system. However, this requires a certain understanding and commitment on behalf of the systems administration function.

4. Moving forward

There are various steps that might be taken along the path of optimising the performance of a given biometric system in this context. The first step is perhaps to understand your user base and the operational environment. This knowledge will enable you to place an interpretation on the quoted performance of the devices under consideration from a systems design perspective. In this respect, the author has previously proposed the User Psychology Index (UPI) as a methodology for consistently interpreting the performance figures quoted by the device manufacturer according to variable user and environment parameters. This is achieved by applying a weighting to the quoted equal error rate in order to derive a much more representative figure which may be used for subsequent system design purposes. A freely available software module automates this process and provides the user with a suggested equal error rate figure for the system under consideration. This concept will be developed further in the future.

Having gone through the above process and defined a robust understanding of the user base, the environment, the requirement, and the expected systems performance, we may now proceed to design the system accordingly, giving full consideration to the generic systems architecture points referred to earlier in this document. As a rule of thumb, the competent systems designer will no doubt allow a margin for network capacity and performance issues within his or her calculations. The physical systems component compliment may then be decided upon. However, this is not the end of the story as the system should be end to end tested in its entirety in order to understand the real performance (and reliability) potential.

Assuming that we now have a system which has been designed with our particular real world operating conditions in mind and that it has been properly tested as an entity, we also have to consider how it will perform with typical users from within our user base and therefore how it should initially be configured for optimal performance. In this context it is probably wise to run a pilot scheme with a subset of users in order to identify and understand the issues which will undoubtedly arise. This will provide the opportunity to build in the appropriate systems responses and compensations before the live system is rolled out. It will also provide the opportunity to start to observe and understand how user psychology can dramatically affect the realised performance of a given system and how to manage this element accordingly.

This paper has drawn attention to the variables inherent in the implementation of a system which includes biometric authentication technology and has acknowledged the affect that these variables have upon realised performance. Among these variables, one of the most significant is surely that of user psychology and yet this is not often taken into account when either designing or indeed running a system. The author hopes that drawing attention to this issue will promote further investigation into this area, together with a more realistic interpretation of device manufacturers performance claims and how these translate into real world systems performance. In addition, a deeper understanding in this context would do much to tailor expectations of this technology and the benefits that can realistically be delivered.

( The original UPI paper )

When we consider the performance of biometric devices and systems we tend to centre our thinking around the familiar parameters of false accepts, false rejects, and equal error rates as quoted by the biometric device manufacturer. The manufacturer may also quote a figure for the time required to search a database when the system is operating in identification mode (where applicable - only a few products currently work well in identification mode). Interesting though these figures may be, they are not always realised under real world operating conditions for a variety of reasons outside of the manufacturers control. Systems architecture has an important part to play (the subject of a future paper) but perhaps the most significant affect on live performance is that produced as a result of user psychology. This is a complex area worthy of further investigation and consideration if we are to implement successful biometric projects.

In this paper we shall propose the adoption of a User Psychology Index (UPI) to be used in conjunction with published device and system performance characteristics as a method of evaluating project proposals and designing biometric systems. Consider the following scenarios in the context of biometric systems performance.

[1] A device is used in isolation by a laboratory technician in the course of his work. The technician is extremely familiar with the device in question (having been involved with its design) and has a complete understanding of its operation from both a software and hardware perspective. In addition, he is eager to achieve good results with this device.

[2] The same device is used by a technically aware professional class user in a modern office environment, having been comprehensively instructed in its use beforehand and provided with extensive documentation.

[3] The same device is used by a non technical user in a public environment who has had brief instruction at a prior date but has received no documentation or background into the principles of biometric verification.

[4] The same device is used by a technically competent user who has an understanding of biometric principles and is determined to be falsely accepted - i.e. a fraudulent transaction.

If we multiply the instances of the above by 1000 and then analyse the results using our familiar yardsticks of false acceptance and false rejection we shall probably find some quite interesting performance swings across the four profiles. Adjusting the matching threshold of the device (where applicable) will tend to shift the peaks of false accepts and false rejects backwards and forwards across our landscape, however, we shall still experience variances in realised performance among the different user profiles.

In this example we have highlighted one aspect of user psychology, namely, familiarity with the operation of the system coupled to an understanding of how the system works. We may conclude that this one factor is enough to impact the realised performance of the system beyond the manufacturers claims, and yet there are several additional factors which can affect user psychology.

Imagine that the user in example three has had to queue to use the system and is already late for his rendezvous. Impatiently, he offers his biometric sample for verification and does so incorrectly. The system rejects him and there is a short delay before another sample can be accepted, but not knowing this the user repeatedly offers his biometric sample and gets out of synchronisation with the verification process, causing further delays and so on. Multiply this by a number of concurrent users on a given network and the systems architecture aspect starts to look important. There is another type of user who, fundamentally opposed to the concept of biometric verification has every expectation that the system will fail and subconsciously (or consciously) does is best to fulfil the prophecy. Indeed, if he does experience a valid acceptance he will probably conclude that the device is faulty. Yet another type is inherently inconsistent in the way he uses the device, no matter how well it is explained beforehand and of course some individuals simply have physical difficulty in using any such device. Any of these factors, either singly or in combination can affect the attitude of the user at the time and point of verification. This in turn will affect the way he interfaces with the front end biometric device and the probability of an accurate transaction at the first attempt.

The situation is in fact considerably more complex than the above examples illustrate and represents an area where further independent research would no doubt be beneficial. In any event it is a factor which should be taken into account by those considering the implementation of biometric verification within a given process, especially in a potentially large scale installation where the cost of such an exercise would need to be carefully considered in relation to the expected benefits.

OK, but where do we start? In the short term, the author proposes an index of typical user/situation profiles to be considered alongside device and overall system specifications. This index could be of variable granularity according to how far the concept were to be developed in line with the general requirement. A coarse (small) index may be enough to provide a rule of thumb guide, whilst a fine (larger) index will provide for a more scientific approach, given suitably accurate data. Initially, the table reproduced below may suffice to test the theory.

To use the User Psychology Index (UPI) one would identify the appropriate category of user as closely as possible and then multiply the quoted equal error rate of the device in question by the UPI index value. Thus, if we consider a major proportion of our users can be categorised as profile 7 in the example UPI, and the device in question has an equal error rate of 0.5 % we would then multiply the quoted equal error rate by 3 to give an expected actual error rate of 1.5% for this group of users with this device. Using the same device, users aligning with profile 12 can be expected to produce equal error rates of 7.5% or more. The same (profile 12) users using a different device with a quoted equal error rate of 1.5% can be expected to produce actual equal error rates of 22.5% or more - not at all unreasonable in the experience of the author.

It is acknowledged that this concept of the UPI may be contentious in some quarters, especially among manufacturers who naturally wish to portray the best possible performance figures for their particular device. However, it is suggested that in order for the biometric industry to move forward and the concept of biometric verification to be generally accepted on a broader scale, we must more accurately align actual systems performance with user expectations. This is not intended as a criticism of the manner in which manufacturers quote biometric performance characteristics, but as an acknowledgement of just how difficult it is to quote meaningful figures for a diverse range of applications and situations. It is for this reason that the UPI is offered as a practical working tool for those considering biometric projects. The UPI illustration offered here is in draft form and comments are invited from both device manufacturers and users in order that this may be refined and subsequently published for general use.

Julian Ashbourn