Dias 1		
	
Dias 2		
	
Dias 3		language engineering products or even components of such products do not exist in a vacuum- they are designed to carry out specific tasks or solve problems for users- in order to evaluate how good they are we need to define who they are good for and in what contexts of use.   
The EAGLES work on requirements analysis and user profiling was done by Shona Douglas.  As we will see it involves looking at the problem as a number of more and more specific descriptions.
As a starting point work in requirements analysis in software engineering - here I will give a little background - to get the overall picture and terminology clear.
	
Dias 4		As you can see requirements statements are used both in the design/implementation stage and then as a something to evaluate against.
The main purpose of requirements statements in s/w engineering is to support design and implementation whereas our purpose is slightly different in that the focus is evaluation - nevertheless there is a lot to be learnt from requirements engineering.
	
Dias 5		Requirements (or specifications in the case of design) can be seen as an intersection between the attributes of the problem domain (i.e. the problem that the s/w should solve) and the attributes of the machine which will solve the problem.
At this level of analysis we are not looking at specific systems and user groups
	
Dias 6		Requirements are usually broken down into 2 types..... Functional requirements refer to the basic functionality characteristic whilst the non-functional requirements can be seen as “adverbial” to this and are represented by the other characteristics such as reliability, efficiency etc.
In EAGLES the idea is to begin with a task-based functional requirements statement as a skeleton on which to build a full requirements statement with all the non-functional requirements.
	
Dias 7		So far I haven’t said anything about the end-user of the product or system, but only the most general requirements statement. User profiling can be seen as a parameterisation of general requirements.  This not covers human factors such as the language to be used, but also the overall context in which the system is to be used.  The “setup” in K Sparc Jones’s terminology.  I will go into more detail about this in the next few slides where the different steps in requirements analysis and user profiling in Language Engineering are outlined.
	
Dias 8		Although these are presented as steps to be taken they can not entirely be ordered, but could be seen as somewhat cyclical.  In particular the first 4 steps will all feed into the definition of the functional and non-functional requirements.  
Now I will go on to look at each of these steps in turn and in a little more detail, in the following slides I follow in EAGLES in taking the example of a spelling checker.
	
Dias 9		Developing task models can be useful in supporting the model of top-level tasks.This task model which we used as the basis for both spelling and grammar checkers 
identifies the data flows and agents in a proofing task
different roles can be fulfilled by the same person
abstract - says nothing at this stage about how the checking is done - checker could either be a human or a machine supporting the human editor.  NB the introduction of a machine will alter what happens before and after the editing takes place.
Some very basic functionality requirements can be gleaned from this model.  A text with spelling errors is transformed to one without errors.
	
Dias 10		Relevant setups provide a picture on the context in which the system or product will be used for example will the end-user or operator of the system be working in a stand alone mode, or are they part of an organisation where the product will be accessed by several co-workers? this can affect the requirements on the system.
Where do the inputs come from? are we talking about  a human author who writes directly in electronic form or is the input to the proofing process created by OCR reading of printed work?
Where does the text go to? a human reader? a parser? this could also affect the requirements on the system.
It is quite possible that the set of setups will result in contradictory requirements. 
	
Dias 11		Stake holders or actors must be identified in each setup inorder to discover  the different requirements for the setup - however it can well be that different stake-holders will also provide different requirements.  For example the major requirements for management may be increasing throughput of documents and reducing running costs, whilst the end user may be more concerned with user-friendliness and avoiding a situation where the introduction of technology makes their working life more boring....
In LE (and particularly in text transformation) texts themselves are central to the definition of requirements in a setup. typical texts need to be found and analysed by experts or old research so longas it is reliable can be used.
	
Dias 12		As well as the sort of organisational issues such as workflow or hardware considerations (does the system have to fit into an existing hardware platform?), there are also features of the actors in the roles which will form parameters for the user profile.
eg what sort of texts do they write? what is their level of competence in the language? etc.

Once all these aspects have been identified and defined as requirements the next step is to turn them into attributes of the system which can be reliably measured and reported to the customer of the evaluation.
	
Dias 13		So far our requirements analysis has been pretty much top-down describing ever more specific requirements statements.  But when it comes to actually transforming these requirements into attributes of systems which can be measured, we need to take into account the nature of the systems themselves.  How they actually treat different aspects of the overall task.  
Often when trying to evaluate different systems in a class we must be prepared for a number of system specific tests especially when looking at components?
	
Dias 14		Another important aspect of requirements analysis and the subsequent transformation into reportable attributes is that of being able to describe the results of testing in such a way that customers of evaluations can understand the evaluation and make their decisions.
The linguistic functionality of the LE tools is not generally well-known outside the field so that the specific measures created must be re-described in terms of the users needs.
	
Dias 15		One of the most important aspects of requirements analysis which hasn’t been mentioned yet - is how do you get the information for the analysis.  This rather depends on the situation.
Currently we recommend looking at the standard methods from social sciences and also the more formal methods from knowledge acquisition for KBs
In addition corpus analysis is essential to LE evaluation but cannot replace other knowledge acquisition activities
However, I feel that we need to strengthen this aspect of the handbook.
	
Dias 16		requirements analysis is no doubt time consuming and costly however, the validity of the whole evaluation depends on the validity of the requirements statements.
This can be seen as an open-ended process where the quality of the requirements increases as more evaluations are carried out.  Also the various steps outlined feed back into each other.
	
Dias 17