Monthly Archives: June 2011

Thoughts about initial system evaluations

PlanningInterviews to find out what stakeholders feel about Synote, as it is at the moment, are well underway.   Initially we were going to record all the sessions, but as these seem to be extending to around 2 hours at times, I have been using Microsoft OneNote and a scoring system for each web page encountered of 1-6 as to how easy it is to use.  I would like to ask everyone to send me a short audio file that sums up the main comments made during the interview so I can add them to Synote for future comparisons.

In the meantime we have also introduced the System Usability Scale as a ‘quick and dirty’ way of scoring feelings about Synote.  I have set up an iSurvey questionnaire so everyone can use the online version if I have not asked you to fill in a paper based version.

I would like to take this opportunity for thanking those who have already taken part in early interviews and look forward to the remaining few.   I will be back to constantly update all aspects of the development of Synote as a research tool so we can story board the next stage.

Meeting Minutes – 15/05/2011

Adaptable and learnable USer Interface for Analysing Recordings (ALUIAR)

synote guide1st meeting – 15/05/2011 Room 3073 Building 32 (Access Grid Room)

Attendees

Mike Wald, Lisa Harris, Mary Gobbi, Gary Wills, Sebastian Skuse, Yunjia Li, Lisa Roberts and E.A. Draffan (Apologies received from Lester Gilbert for his absence).

Minutes from the meeting

Welcome and Introductions from members of the team

Work packages were discussed and Mike gave an overview of the project with the features that were mentioned in the project plan and others that might be included in the design.

It was decided that there would need to be some API changes and storyboarding of possible interfaces for various functions (Yuniji and Seb).

Mary Gobbi suggested the idea of a ‘Decision Tree’ and methodological framework for the types of interviews undertaken by researchers and the type of coding, annotations etc needed for different types of research. It was felt this would help many people and also act as a guide when deciding which features could be added to Synote and which were left to other types of software supporting research and speech transcription such as NVIVO and Transcriber.

Co-Design – taking the diagram below as a guide to the process being undertaken it was decided that short interviews with a series of stakeholders would be noted and some recorded and uploaded to Synote as part of the shared understanding and show and tell aspect of the process.  (action EA with team members plus other researchers)

co-design diagram

Millard, D., Faulds, S., Gilbert, L., Howard, Y., Sparks, D., Wills, G. and Zhang, P. (2008) Co-design for conceptual spaces: an agile design methodology for m-learning. In: IADIS International Conference Mobile Learning 2008.

Website and future communication choice – The results of this work will be visible on Synote and linked to a blog on the ALUIAR project website – the team will have a mailing list and drop box account.  (action EA and Seb)

The next team meeting will be 11th July, 09.30 – 10.30 Access Grid Room, Building 32 Level 3.

Budget

Here are the details about the project budget. Some information has been blurred out due to the public nature of this blog.

Directly  Incurred  Staff August  10– July 11 August  11– July 12 TOTAL £
(b) blanked out 3219 5611 8830
(c) blanked out 3377 6427 9804
(d) blanked out 1688 2856 4544
Total Directly  Incurred  Staff (A) 8284 14894 23178
Non-Staff
Travel and expenses 430 1771 2201
Hardware/software
Dissemination 500 500
Evaluation 500 500
Total Directly  Incurred  Non-Staff
(B)
430 2771 3201
Directly  Incurred  Total (C) = A+B 8714 17665 26379
Directly  Allocated
Staff 9357 16248 25605
Estates 6131 10556 16687
Infrastructure Technicians 539 928 1467
Directly  Allocated Total (D) 16027 27732 43759
Indirect  Costs (E) 13540 23312 36852
Total Project Cost (C+D+E) 38281 68709 106990
Amount  Requested from JISC 14312 25688 40000
Institutional Contributions 23969 43021 66990
% Contributions over project  life JISC= 37.4% Partners=62.6% Total=100%
No. FTEs used to calculate  indirectand estates charges, and staff included No FTEs

0.99

Which Staff:

All staff Listed in Section 3.10

Directly Incurred Staff

August 10– July 11

August 11– July 12

TOTAL £

(b) Project manager SP35 0.2 FTE

3219

5611

8830

(c) Lead developer SP27 0.29 FTE

3377

6427

9804

(d) Developer SP27 0.13 FTE

1688

2856

4544

Total Directly Incurred Staff (A)

8284

14894

23178

NonStaff

Travel and expenses

430

1771

2201

Hardware/software

Dissemination

500

500

Evaluation

500

500

Total Directly Incurred NonStaff

(B)

430

2771

3201

Directly Incurred Total (C) = A+B

8714

17665

26379

Directly Allocated

Staff

9357

16248

25605

Estates

6131

10556

16687

Infrastructure Technicians

539

928

1467

Directly Allocated Total (D)

16027

27732

43759

Indirect Costs (E)

13540

23312

36852

Total Project Cost (C+D+E)

38281

68709

106990

Amount Requested from JISC

14312

25688

40000

Institutional Contributions

23969

43021

66990

% Contributions over project life

JISC= 37.4%

Partners=62.6%

Total=100%

No. FTEs used to calculate indirect

and estates charges, and staff included

No FTEs

0.99

Which Staff:

All staff Listed in Section 3.10

Risk Analysis

The following shows the main risks identified for this project, as well as contingency actions for each risk.

Title Liklihood x Impact = Priority Action to Prevent/Manage Risk
Loss of personnel 2 x 4 = 8 ECS has many experienced staff and members of the team have the knowledge, experience and ability to support staff and to take over each other’s roles at critical moments if required.
Interest of users 2 x 8 = 16 User interest has been confirmed among a number of schools who will be involved in the co-design and evaluation and will be maintained as the project will support users in existing research and provide a more learnable and adaptable tool that is currently available to them.
Keeping to planned timetable 2 x 8 = 16 An initial study has been conducted to demonstrate the feasibility of the proposed ALUIAR system. The team are very experienced at successfully completing funded projects to time and have the required technical and managerial expertise and support of the institution and will meet regularly to ensure the project runs to plan.
Addressing boy Usability and Accessibility Issues 2 x 8 = 16 Synote was designed with both accessibility and usability in mind (and was closely linked with the development of UDIM though the users and innovations programme) and the ALUIAR project team includes experts on addressing both accessibility and usability issues and if required also has easy access to other HCI experts within ECS and beyond.

Likelihood scored as Very Low = 1, Low = 2, Medium = 3, High = 4, Very High = 5
Impact scored as: Very Low = 1, Low = 2, Medium = 4, High = 8, Very High = 16

Timeline and Work Packages

Project Overview

The project will be led by Dr Mike Wald of Southampton’s School of Electronics and Computer Science (ECS) and his team will use an agile interactive approach involving end users in regular co-design, and review meetings. There will also be regular management meetings and discussions with the Stakeholder Group involving researchers from a wide range of schools/faculties (including Psychology, Health sciences, Nursing, Sociology and Social Policy, Management, Education).

The project will begin with an initial project face-to-face meeting.  Team meetings will occur at monthly intervals to monitor progress against objectives. Minutes of these meetings will be published on this project blog and will be available to the public. Financial reports will be supplied by ECS financial management and a Final Report will be produced at the end of the Project as specified. There will be a final project closure meeting. Each of the work packages will require formal review and sign-off meetings. There will be weekly technical meetings of the core project staff.

Gantt Chart

Project Gantt Chart

ALUIAR Project Gantt Chart

Workpackage 1: Co-design: Team meeting and requirements and specification agreement with our panel of stakeholders from different schools identified in sections 2.1 & 3.1 who will be involved in the co-design and evaluation processes. These users have already identified the following developments required to make the tool more adaptable and learnable and any further issues will also be identified:

  1. Greater flexibility of movement backwards or forwards through a recording (e.g. by typing in new time) as at present can only move in 5 second ‘nudges’ or move time slider or change speed if recording format and player allow.
  2. A drop-down box listing frequently used tags (e.g. for coding name of speaker and category code)
  3. Foot pedal control of player
  4. When manually transcribing a recording it is possible to also annotate this with the start time of the clip entered automatically but the end time needs to be manually entered. Synote allows a section of a created transcript to be selected and the annotation to be linked to that section with the start and end times of those sections to be automatically entered. It would make the system easier to use if it was possible to also do this without having first to save the transcript.
  5. Facility to download the annotation data (e.g. to Microsoft Excel for statistical analysis and charts and graphs or for a report or into other annotation tools). At the moment the information requires copying and pasting
  6. Making it harder to exit without saving and so losing changes made.
  7. Allowing the user to control the recording playback when annotating by providing media player controls in the annotation window. (at present a user can annotate a recording and the annotation can automatically read the time of the recording but the user cannot easily replay a section of the recording while writing the annotation)
  8. Use of text font, colour, size and style for coding
  9. Individual profiles provide a graduated learning curve to suit the preferences of each particular researcher allowing fine-grained control over a wide range of settings (e.g. whether the replay of the video should pause automatically while an annotation is being created).
  10. Redesign of interface to improve learnability
  11. Organise recordings into groups and categories to make them easier to find and manage
  12. Ability to replay just the video clip from a search (at present plays from the start time and manually have to pause at the end time of the clip)

Output: Requirements/functional specifications will be expressed using use cases, personas and scenarios (developed further from the three scenarios outlined in section 2) and will be revisited in WP3 to ensure an agile approach. The different ALUIAR demonstrators themselves will also be used to support the ALUIAR co-design design and evaluation process through the analysis of recorded project evaluation interviews and observations in a fashion similar to that described in scenario 3 in section 2, thus demonstrating our confidence in our own software (an approach colloquially referred to as ‘eating your own dog food’).

Workpackage 2: Dissemination and working with and reporting to JISC

Output: Project website, Blog syndicated, Project plan, reports to and liaison with JISC, attendance at JISC programme level activities (e.g. programme meetings and relevant special interest groups) for at least 10 days per year, extensive involvement with JISC community, OSS Watch, workshops, relevant special interest groups, conferences presentations, mailing lists, published papers and articles.

Workpackage 3: Software Development & Evaluation
The software development and evaluation will be agile and iterative to ensure responsiveness to changes with three phases/iterations and demonstrators and a final version at the end of the project. Analysis and design and development of demonstrators at the different phases will involve usability studies including iterative evaluation of interaction design with users to finalise improved adaptability and learnability for deployment. Improvements in usability of the demonstrators will be measured and evaluated using quantitative and qualitative evaluations, tasks, tests and observations based on deployment by users on the stakeholder panel in their research studies. A wide range of approaches will be used at appropriate stages of pre-development, development, and deployment, including, low and hi fidelity prototypes, cognitive Walkthrough, Heuristic Evaluations, interviews, Adaptable and Learnable User Interface for Analysing Recordings (ALUIAR) Page 4, Adaptable and Learnable User Interface for Analysing Recordings (ALUIAR) Page 5, Focus Groups, and Empirical testing. The different ALUIAR demonstrators themselves will also be used to support the software design and evaluation process through the analysis of recorded ALUIAR project evaluation interviews and observations. Beta testing reports include testing against requirements and acceptance, compatibility, integration, unit and load testing and identifying problems to be addressed, including integration or interoperability issues. Demonstrators will be also made available to the JISC research community for evaluation and feedback including through stakeholders’ subject networks and contacts.

Outputs:
Phase 1 output will be a prototype demonstrator 1 based on requirements identified in 3.1
Phase 2 outputs will be an evaluation of demonstrator 1 and development of demonstrator 2 which will have improvements based on the phase 2 evaluation
Phase 3 outputs will be:

  • An evaluation of demonstrator 2 and development of demonstrator 3 and finalisation of the release version of software
  • Evaluation report of the use and acceptance of the software by researchers
  • A detailed briefing paper that describes the approach taken and the lessons learned in terms of learnability and usability design
  • Documentation with examples of use of the software by researchers
  • A new improved interface to Synote e-research tool that significantly improves its learnability

Workpackage 4: Project Evaluation

Output: Evaluation report of project by independent evaluator using project documentation created throughout the project and with the success of the project measured by the acceptance and take up by users.

Workpackage 5: Sustainability We will consult with OSS watch on sustainability and access to the code and documentation through a best fit ‘OSI approved licence’ and a suitable governance model. ALUIAR will also liaise with the Software Sustainability Institute. Best practice in source code development and management will be followed and Quality factors built in to the workpackages will ensure successful Open Source life through achievement of a good OSMM rating, community engagement, and community stated need. All reports, tools, and code from the project will remain on the project server for a minimum period of 2 years and will be archived in the institutional repository (E-Prints) and appropriate JISC repository, for instance Jorum.

Output: Acceptance by users and a sustainability strategy for the application.

The Team

face view of E. A. DraffanE.A. Draffan, the ALUIAR project manager is Research Fellow at the University of Southampton’s School of Electronics and Computer Science and has extensive experience in usability, accessibility testing and evaluation. She has project managed many JISC funded projects, e.g. LexDis (participatory investigation of elearning tools and strategies), Web2access (evaluation of web application usability and accessibility) , REALISE (open source participatory design), as well as many other government and charity funded projects. She also has extensive experience of using commercial and open source tools to analyse recordings within many previous projects.

face view of Yunjia LiYunjia Li is ALUIAR’s lead developer and researcher, and has been Synote’s lead developer since the JISC MACFOB project in 2008. He has a 1st Class Computer Science Degree and a distinction in MSc Web Technologies, as well as extensive experience of web based user interface design.

face view of Seb SkuseSeb Skuse is an ALUIAR developer/researcher, has a 1st Class Information Technology in Organisations Degree and has been lead developer on many JISC agile open source projects since 2009. He has extensive experience of web based user interface design and has also developed ATbar, the accessibility toolbar for websites.

face view of Mike WaldMike Wald is ALUIAR’s Principal Investigator and a Senior Lecturer at the School of Electronics and Computer Science at the University of Southampton. He has taught on undergraduate and postgraduate HCI/usability courses and led the award winning JISC projects MACFOB (developing Synote), LEXDIS (involving extensive use of NViVO software for interview on ‘Adaptable and Learnable User Interface for Analysing Recordings’ (ALUIAR), Page 7 analysis), has advised HEFCE, BECTA, and JISC on technology and usability issues and was involved in the development of the JISC Users and Innovation Development Model.

Dr Wald will lead the project aided by the following co-investigators:

face view of Gary WillsGary Wills is a Technical Leader and a Senior Lecturer in Computer Science at the University of Southampton who has also taught on undergraduate and postgraduate HCI/usability courses. He has led a number of JISC funded projects involving participatory usability co-design, including projects from the Virtual Research Programmes rounds I & III (CORE, VRIC, IBBRE projects). In addition to the JISC funded work Gary has projects funded by the TSB/DTI, ESRC, MRC and NIHR, and therefore has extensive experience developing usable tools for analysing research in the Engineering, Social Science and Medical domains.

face view of Lester GilbertLester Gilbert is lead on project evaluation, a Senior Lecturer at the School of Electronics and Computer Science and was Principal Investigator on REAQ and EFSCE, technical manager of the VRE CORE, co-investigator on FREMA and R2Q2 and has extensive experience of participatory co-design on JISC projects. He has worked for many years leading commercial teams developing computer based software.

face view of Lisa RobertsLisa Roberts is an experienced researcher and clinical physiotherapist, with experience of leading trials in communication and back pain. In August 2007 she commenced a 5-year post (ARC Senior Lecturer in Physiotherapy), funded by the Arthritis Research Campaign leading a mixed methods research programme. She is currently secretary for the Society for Back Pain Research and a member of the Research Committee of BackCare.

face view of Mary GobbiMary Gobbi is a Senior Lecturer and Award Leader for the Postgraduate Diploma in Nursing with Registration in the School of Nursing and Midwifery and will lead on their involvement in the project. She is responsible for research and evaluation of the HEA and ESRC funded Virtual Interactive Practice (VIP) initiative and has been involved with a range of funded research projects including evaluation of nursing education and training provision. It was Mary who first identified the value of Synote for research and was responsible for its demonstration and dissemination at the “Developing video analysis as a pedagogic and research tool for health care simulation: workshop” in February 2009.

face view of Lisa HarrisLisa Harris is Senior Lecturer in Marketing in the School of Management and Programme Director for the Marketing Analytics MSc. She runs workshops on how to use web 2.0 technologies and is exploring ways of using the technology to enhance research, particularly for international students.

Project Aims and Objectives

ALUIAR interface for the researcherThis project aims to improve the usability and learnability of the user interface of Synote, the open source web based video and audio annotation tool. There is the need support those researchers, non-specialist and novice users who wish to transcribe audio and video recordings with synchronised text.

The key objective is to present users with an interface that is not only easy to use but is designed to suit the needs of researchers when uploading materials rather than the original users of Synote who tend to search and listen to lectures.

None of the main functionality of thee  tool will be changed, but the way in which it will be used will be enhanced, so that it is more intuitive and logical to use. Existing users have already commented on some of the issues that need to be re-examined, such as navigation flexibility, interface learnability and the organisation of recordings and search features etc.

An ethnographic approach will be taken to benchmark Synote’s current performance. A small group of users will be interviewed and ‘walked through’ the tool whilst having their opinions, concerns and behaviour (whilst using the tool) monitored. These results will then be ranked and implemented to help identify the changes that should be made to Synote. These changes will be discussed again with the stakeholders and implemented using an agile interactive approach involving end users in regular co-design, and review meetings.

The success of interventions will be measured by the participants completing a questionnaire based on John Brookes System Usability Scale (SUS) and by quantifying the qualitative results gathered in interviews from the participants.  It is felt that due to time constraints the number of interviews that take place before storyboarding interface designs will be minimal. Nielsen (2000) states that “the best results come from testing no more than 5 users and running as many small tests as you can afford.”  The stated methods will provide quantitative and qualitative results that will inform the need to improve the adaptability, usability and learnability of Synote. Two other methods were considered but discarded due to the size of the project and time of year with many users going on holiday during July and August:

  1. Large Focus Groups – This would give a much richer pool of qualitative data, but as time is short this is an unrealistic data gathering method to undertake without going over schedule.
  2. Mass-questionnaire – These would give a greater pool of data, but work by  Nielsen (2000) also showed that “the ultimate user experience is improved much more by three tests with 5 users than by a single test with 15 users.”