Final Posting with a link to Synote Researcher download.

During the final meeting we discussed some of the lessons learnt when thinking about usability and the development of online applications.  Here Yunjia, who is working on all things Synote, provides his reasons for finding that usability can be improved when there is direct liaison with users rather than working through a project manager or other members of the team. (View via Synote with keyboard access)

Download the latest version of Synote Researcher from SourceForge.

 

Three in One – Recap, Results and Recommendations

It has always been difficult to have a full team meeting and our last meeting was no different, but by using the University of Southampton’s iSurvey system we have managed to get results to some of our questions. Moreover, through the use of face to face interviews, emails, calls and the blog we have gained an insight as to how to develop an Adaptable and Learnable User Interface for Analysing Recordings.  Synote, for the researcher, has been developed into a freely available open source downloadable application that can be used by any individual,  institution or organisation.

We believe that we have succeeded in producing improvements to the ‘existing user interface’ and adapted Synote as an “e-research tool, in order to make [it] easier to learn by non-specialist users.” (JISC Usability and Adaptability of User Interfaces) by using a co-design methodology with a very open approach to the project evaluation.  The final sample video to illustrate its use is being produced and will be made available.

The complete agenda for the last meeting was mentioned in the last blog, but under the main points I have also provided an overview of some of the ideas and comments made throughout the project and during the face to face interviews. There is a final summary available at the end of this blog.

A service becomes a download with a new name

There was much discussion during the lifetime of the project about having to make a new version of Synote whilst leaving the original system intact. Synote was designed for the annotation of educational recordings with notes and transcriptions.  The content has often been drawn from lectures, seminars, videos and audio available online.  Recordings are not usually held on a specific server for Synote users.  Users are required to use their own websites to provide the URL link to the Synote player.  Create recording pageDuring the project we discovered that some users found it hard to make a URL from an uploaded recording and so the new version includes the ability to upload a recording from a desktop computer, to link to YouTube recordings, as well as users’ own websites.

This also led to the decision that the researcher version should be a download rather than held on the University of Southampton server for public use, as is the case with the original version. This decision was made because of the type of sensitive data that might be used by researchers.  By having a download the software can be held on secure servers with increased data protection.

Finally, the fact that the new version had some enhanced features, meant that it needed an addition to its name!   iSurvey was used for voting and although the name ‘Synote Scholar’ was liked by many members of the team – Mike was the final arbiter with the name Synote Researcher winning the day as he felt that the name ‘Synote Scholar’ could equally be seen to apply to the original version of Synote and so didn’t make it clear that the new version was designed for researchers!

Synote researcher logo

Evaluation results related to the new interface design for learnability and usability.

co-design diagram

Millard, D., Faulds, S., Gilbert, L., Howard, Y., Sparks, D., Wills, G. and Zhang, P. (2008) Co-design for conceptual spaces: an agile design methodology for m-learning. In: IADIS International Conference Mobile Learning 2008.

Throughout the project we used the blog to act as part of the evaluation process – publicising storyboards, changes made and slide views of the changing Synote Researcher using the agile co-design methodology mentioned at the outset.  This allowed us to have an on-going conversation with users and other team members.  It resulted in an iterative approach being taken to the development process and was very useful when monitoring changes.

We used John Brooke’s System Usability Scale (SUS)  for a quick analysis of the views of users as to the usability of Synote Researcher. The results showed that there was an increase in the overall scores out of 100 from 48.50 to 67 with 4 out of 5 users agreeing in the second evaluation, that they felt “most people would learn to use this system very quickly” The score that brought the second evaluation down was the need for technical support to set up the system. This is not surprising and the download may need technical support.

More precise evaluations were undertaken via a series of face to face interviews designed to see if the changes made had ensured that Synote was easier to learn and use by researchers wishing to analyse and code multimedia data.   During the interviews users were asked to participate in practical exercises such as logging-in, searching for recordings, creating their own recordings and adding a transcript and synmarks or annotations. They were asked to comment throughout the meeting and offer a score of 1-5 for look and feel where 5 was excellent.

Player pageDuring the discussions background information about the possible use of the demonstrator was requested and how users views about the Synote service may have changed due to the redesign.   The evaluation of the original Synote took place in June 2011 and the final appraisal occurred during November 2011.  The interviews were also analysed using Synote Researcher with annotations and remarks as seen above.

web page scores

Average score for each page - red = original Synote, blue = Synote Researcher

The results for scoring the individual pages were based on the users who completed both interviews although we had an additional three external users who commented on the new version.  As there were some extra interviewees it was decided that an average of each score given for each page discussed would be taken over both interviews.  As can be seen all the scores increased and this was despite ideas for further changes and constant iterations.

As a result of all the requests it was felt necessary to revisit the original work packages to check all the elements requested at the outset of the project had been discussed and included in the design. Only one of the requested changes was not possible to achieve in the time available as this would have involved a redesign of the whole Synote system.

  1. pausing the playerGreater flexibility of movement backwards or forwards through a recording (e.g. by typing in new time) as at present can only move in 5 second ‘nudges’ or move time slider or change speed if recording format and player allow. – Now able to nudge the recording forwards and backwards by 1,5,10 and 20 seconds.
  2. A drop-down box listing frequently used tags (e.g. for coding name of speaker and category code) – These appear as you add a tag to the annotation.
  3. Foot pedal control of player – This has been achieved and tested using a Start-Stop HDP-3S pedal from HTH Engineering Inc.  The software has allowed us to allocate controls as per the original Synote keyboard shortcut accessibility options.synmark tags
  4. When manually transcribing a recording it is possible to also annotate this with the
    start time of the clip entered automatically but the end time needs to be manually entered. Synote allows a section of a created transcript to be selected and the annotation to be linked to that section with the start and end times of those sections to be automatically entered. It would make the system easier to use if it was possible to also do this without having first to save the transcript.. – This has been achieved with + and – keys plus an icon of the clock to indicate the inclusion of time as can be seen in the screen grab on the left.
  5. Facility to download the annotation data (e.g. to Microsoft Excel for statistical analysis and charts and graphs or for a report or into other annotation tools). At the
    moment the information requires copying and pasting – This has been achieve with a csv file with all the synmarks as can be seen in the picture above by the paper icon with the arrow at the top right.save as draft icon
  6. Making it harder to exit without saving and so losing changes made.-  This has been achieved with an automatic save as draft and additional save on exit.play button
  7. Allowing the user to control the recording playback when annotating by providing
    media player controls in the annotation window. (at present a user can annotate a
    recording and the annotation can automatically read the time of the recording but the user cannot easily replay a section of the recording while writing the annotation) – this has been achieved through the use of the right hand mouse button.text colour
  8.  Use of text font, colour, size and style for coding – this has been achieved as shown in the picture.
  9. Individual profiles provide a graduated learning curve to suit the preferences of each
    particular researcher allowing fine-grained control over a wide range of settings (e.g. whether the replay of the video should pause automatically while an annotation is being created). – This has partly been achieved by the viewing of icons and supporting text only when moused over or selected or the use of the right hand mouse menu within the transcript or synmark section.
  10. Redesign of interface to improve learnability – This can be seen when viewing all the attached images and downloading the software as well as the evaluation results.
  11. Organise recordings into groups and categories to make them easier to find and
    manage – This has been achieved through the groups menu and sorting mechanism within the recordings listings. 
  12. Ability to replay just the video clip from a search (at present plays from the start time
    and manually have to pause at the end time of the clip) – it has not been possible to achieve this in the time available as it would require a redesign of the whole Synote system.My synote

tags and a tag cloudIn addition to the changes made to many sections of the original Synote there has been the introduction of ‘MySynote’ – this is an area where the researcher can look at their own recordings, synmarks or annotations and tags used for coding as well as edit their profile.  It allows for the sorting of recordings and the viewing of the most commonly used tags as a tag cloud.

 Overview and discussion about the initial comments from interviews undertaken with the team and external evaluators.

Both Yunjia and I were very heartened to hear the comments made by interviewees including external evaluators who had seen or used the original Synote, and were now being introduced to Synote Reseacher.  Even those who were using the new version commented favourably on the improved usability and learnability.  Dr Sarah Parsons commented ” It feels a lot more intuitive and I think with some of those tweeks we have talked about in terms of labelling clear what a couple of those bits are would strengthen that even further,  it feels very accessible to me….”   Audio clip

homepage commentDebbie said about the home page “Thats easier. (pause) On the old one you had to find information you had to start searching you had to read the whole of the page of other information before you can necessarily find what you need to start the process. So this is much clearer.”

Changes to be made to the final iteration

The final changes as a result of the last few interviews are minor ones listed below

  1. Refine information on front page – Reducing and improving the presentation of the help slides.
  2. MySynote page to just have icons and names to match the list that appears on other pages.
  3. User Profile to contain registration / password information – the captcha is to be removed as this will not be necessary on the download.
  4. Users are going to be able to edit and delete their recordings.
  5. The Groups will work for sharing recordings with chosen users or setting up recordings in different categories.
  6. Legal statements will be added and a simple help file added.

Final Summary
We have been asked to

  • Recap – what you set out to achieve, what you didn’t achieve, why?
  • How successful have you been? What are your metrics?
  • Lessons Learned?

It is hoped that this final blog has covered all these requirements as it was hard to divide the list into three separate blogs without considerable repetition.  We set out to achieve a version of Synote that would suit researchers and be easy to use and learn.  We hope we have been successful as well as providing a unique application that can:

  • be freely used
  • work quickly and easily on any desktop with access to the internet
  • have web based data and multimedia for sharing research.
  • use the data presented for evaluation, appraisal and supervision
  • capture simultaneous conversations with similar timeframes
  • colour code annotations
  • tag themes and categories
  • use titles of annotations for sorting.
  • use a foot pedal to control the media player and other aspects of the interface
  • use keyboard only input and screen reader access (Equality Act 2010)
  • use speech recognition for re-speaking to speed transcriptions.

There is still the challenge of creating annotated clips from the recordings that automatically stop at the end of the clip – this was not possible to achieve in the time available as it requires a significant redesign of the Synote system

Rating our success through the use of various communication methods along with specific surveys plus interviews has provided us with a range of quantitative and qualitative data, despite the low number of users over the period of the project. The Nielsen Norman Group is often quoted as saying “The best results come from testing no more than 5 users and running as many small tests as you can afford.”

When it comes to learning lessons these can be summarised as follows:

  • Allow more time than you think for evaluations – interviews and transcriptions
  • Make sure you have at least one willing volunteer who is not working on the project on a day to day basis but is using it for research (in this case).  We owe Debbie a great deal of thanks as she provided us with several insights that would have been overlooked by those of us who had become over accustomed to the way Synote worked.
  • Have outside evaluators who are willing to comment on the old versions and trial the final iterations with a fresh look – Thank you Lucy, Ring, Sarah and Inga
  • Have some expert researchers (in this case) who are willing to share their requirements early on in the timeline and compare the development ideas with what may be offered by other software programs – Thank you Lisa H, Lisa R and Mary
  • Define your unique selling points so that you can persuade new users to adopt the new program – Thank you Mike, Lester and Gary 
  • Finally but most importantly – you need a good lead developer to change a program that has been developed by someone else, but also one who is willing to be endlessly patient and understanding when translating ideas into reality – Thank you Yunjia

 

Agenda 7th ALUIAR meeting – 16th December 2-3pm Access Grid Room, Building 32, level 3

synote logosHere comes the last of our meetings for the project before we need to finalise the evaluation interviews and set up all the transcripts on our newly named version of Synote for use by researchers!

We still need more votes for the name – Please use the iSurvey site for this purpose

 

Agenda

  1. Apologies already received from Lisa Harris, Lisa Roberts, Mary Gobbi and Lester Gilbert.
  2. Results of the naming exercise for the ALUIAR version of Synote
  3. Results from the SUS evaluation on iSurvey as a result of viewing the new interface design for learnability and usability.
  4. Overview and discussion about the initial comments from interviews undertaken with the team and external evaluators.
  5. Changes to be made to the final iteration
  6. Legal aspects, data protection and downloads.
  7. Final Report Blog and JISC meeting
  8. AOB

 

Minutes for the 6th ALUIAR meeting, 15th November 2011.

Yunjia and Mike at the 6th ALUIAR meeting

Yunjia and Mike at the 6th ALUIAR Meeting

The 6th ALUIAR meeting provided us with the chance to showcase the final iteration of research version of Synote with the minutes as follows.

  • Apologies were received from Lester Gilbert and Lisa Harris, Lisa Roberts and Mary Gobbi was also unable to attend.
  • Outcome of Legal Discussions – we are awaiting some guidance from Diana Galpin as mentioned in a previous blog.
  • Overview of the present web site – overall the new look was well received and individual interviews are now underway – see notes below.
  • Foot-pedal experiments – these have been successful using the stop start foot pedal and software as mentioned in a previous blog.
  • Dates for user testing – three interviews have already been undertaken and transcriptions need to be analysed using the ALUIAR version of Synote.
  • AOB – see below
  • Date for next meeting – it is hoped this will take place on 14/15 or 16th December.

Under any other business Debbie mentioned some other options for coding conversations and data analysis such as AtlasTI.  Lisa Roberts provided us with a very useful link to reviews of all the desktop research tools that could be used for analysing conversations and some multimedia.  CAQDAS is a networking project which also offers training events on research software.

At this point it was felt important to perhaps look at the unique selling points of this version of Synote!   We collected several including the fact that it is possible to:

  • use if for free
  • work quickly and easily on any desktop with access to the internet
  • have web based data and multimedia for sharing research.
  • use the data presented for evaluation, appraisal and supervision
  • capture simultaneous conversations with similar timeframes
  • colour code annotations
  • tag themes and categories
  • use titles of annotations for sorting.
  • use a foot pedal to control the media player and other aspects of the interface
  • use keyboard only input and screen reader access (Equality Act 2010)
  • use speech recognition for re-speaking

There then followed a debate about a name for this version of Synote that will be released on Source Forge as a download for organisations or it could also be on a university server so that downloads can be monitored.

synote logos

Possible logos

Should the name be:

  • Synote Lite
  • Synote Research
  • Synote Scholar
  • SynoteXtra
  • Synote Extra

Please let us know what you think – voting is taking place!

 

So far three interviews have been carried out with researchers and the data will be made available on Synote with annotations – the majority of the comments coming out have been very positive including those from our external evaluator.  However, work still needs to be undertaken on certain icons, tooltips and compatibility with all browsers.

Final usability scores will be collected on the iSurvey version of the SUS evaluation form open to all the team.  Further interviews are planned in the coming weeks with final tweaks to the look and feel of this iteration taking place early in December when the last meeting will be held.

ALUIAR version of Synote undergoes final testing and evaluation

The final phase of the evaluation begins with our outside evaluator Dr Sarah Parsons,  kindly stepping in to help us with the discussion around this demonstrator for researchers and the work that has been undertaken by the team.  Below you can see a silent movie of the latest version of Synote with a transcription being added manually plus synmarks. This version will be available as a download from Source Forge.

We hope to have more samples captured via the resource itself once interviews get underway this week.  In this example you will see the JISC funded REALISE market place for open innovation and assistive technology being discussed – it has just received further sponsorhip from Devices for Dignity

Agenda for 6th ALUIAR Meeting.

white board notes

White board notes

The Agenda for the next meeting is as follows

  • Apologies
  • Outcome of Legal Discussions
  • Overview of the present web site as per requests above
  • Foot-pedal experiments
  • Dates for user testing
  • AOB
  • Date for next meeting.
Yunjia has been working on the ALUIAR version of Synote and we have been adding content to show the new features and discussing how they may impact on the user.
foot pedal

Start-Stop General Purpose Pedal Power Control System

Foot-pedal tests have been undertaken using a Start-Stop HDP-3S pedal from HTH Engineering Inc.  The software has allowed us to allocate controls as per the original Synote keyboard shortcut accessibility options.  This particular software offers the user the chance to set up any combination of keyboard short cuts to control the multimedia player.
So it is also possible just to use the space bar to stop and nudge back as mentioned in the previous meeting.
foot pedal controls
Foot pedal control for the keyboard strokes.
By the time of the meeting we would like to be able to not only continue with the user testing with more researchers but also return to our initial interviews and review reaction related to the changes that have been made to the interface in terms of usability and learnability.

Terms and Conditions, Ethics, Data Protection, Licensing, IP – Legal issues

Help - advice neededAt previous meetings we have rather skirted around all the legal issues but having looked at the  JISC Legal website and chatting with Lisa and Mary it was time to ask for help that might address the issues specifically arising at the University of Southampton  … Diana Galpin very kindly came to our rescue.

The original version of Synote has some Terms and Conditions that were designed to cover all eventualities related to the use of the service as a way of listening to lectures or replaying videos and audio discussions.  The audio and video files were held on other people’s servers and Synote just linked to the output.  The ALUIAR version of Synote rather changes the scene, although perhaps not altogether.

Scenario One

Imagine, as is the case with some members of the team, that a researcher in the Health Professions and Rehabilitation Sciences Faculty at University of Southampton is interviewing patients and wishes to record the interview. The video has to be saved in a secure space but the annotations and transcription of that video can be viewed publicly or just shared with the team or kept completely private.  upload page

The settings in both the original and new version of Synote allow for this to occur and with the new upload system in ALUIAR the video can be hosted on a Southampton server that is within the faculty’s control.  All the legal aspects that surround the video need to be dealt with by the University and the ethics dealt with by the faculty involved.

The ALUIAR team need to ensure that there is a way of allowing those who download the code to be able to make changes to contact names and to ensure alterations can be made to the Terms and Conditions to suit individual faculty needs.

Scenario Two

A researcher in another institution wishes to use the ALUIAR version of Synote – they can download the code from Source Forge but the team need to add a caveat to the repository pointing out in a sentence similar to this one … “this version of Synote is not controlled or hosted by the University Southampton, who accepts no liability for the content and materials held herein.”

In the discussions with Diana Galpin it was felt that it would be helpful to have two downloads available, one for those based in the University of Southampton and one for other institutions.   It was also felt that it would be useful to make a guide regarding what can be uploaded, including limitations as well as a usage reminder – “you are using your own recordings” etc…

source forge screen grabFinally, IP management and source code tracking was discussed with the importance of acknowledging others’ code as part of the tracking process available in the open source repository.

We owe many thanks to Diana, as she has also offered to guide us as to the suitable wording that goes onto the Synote web pages and well as in Source Forge.

Minutes from the meeting held on 13th October, 2011 – 5th ALUIAR Meeting

Meeting group

ALUIAR meeting October 13th

Apologies

Received from Mary Gobbi, Lisa Harris and Lester Gilbert.

Minutes from the Meeting.

E.A provided feedback from Torsten with an overview of the final stages of the project and reporting.  It was confirmed that reports would come in the form of blogs and more information can be found on the blog about at projects “Usability Workshop – Emerging Themes from jiscUX The email that followed mentioned “a few of the headline requirements for final reports:

  • 3 final blog posts:
    • Recap – what you set out to achieve, what you didn’t achieve, why?
    • How successful have you been? What are your metrics?
    • Lessons Learned?
    • Edited version of your posts
    • Send us your final budget and complete the JISC completion survey (we will send you these documents near to the end of your project)

This was followed by a discussion around the Qualitative Data Analysis Steps (Adapted from Creswell and Burnard) that had been provided by Mary earlier in the week – see previous blog.  

It was decided that the points made in the slide should be related to our use of the ALUIAR version of Synote.   Debbie noted that the IBM transcription was not very accurate with a group,  unless each one has a microphone with clear speech –  this makes the speech recognition system rather hard to use – Most researchers are having to transcribe manually. The transcription service is only available at University of Southampton unless others wish to have a license.  The difference between working in the transcript section as opposed to the synmark area was mentioned and where coding occurs etc.

It was felt that it was important to note that the ALUIAR version of Synote will work for qualitative analysis but not in depth Conversational analysis due to the multitude of extra coding features required for a full transcription.   Conversational analysis is best done with specialist software.  However, Synote can be used with other languages (there are some problems with texts that go from right to left), it is possible to share and collaborate over open transcriptions and research projects, transcriptions can be exported along with their annotations and tags can be sorted and saved as a way of generating themes or categories.

The second half of the slide mentions more about the research side of transcriptions and Lisa said that “it is important to note that the researcher must still do the analysis and the system can only manage the data”

Yunjia then introduced the new interface for the system and there were several design features where decisions had to be made – these were captured in the photograph below.  Some buttons and actions in the list of recordings needed moving, a rich text editor with a chance to colour code annotations was requested and Footpedal control only needed for stop and nudge back.  The ability to upload images was felt to be important and the possibility of being able to use the play controls wherever one is in the transcription or synmarks (annotations)

white board notes

White board notes

A walk through of the new interface was completed with a promise to provide the URL in the next two weeks so that we can begin the interviews for the final phase of evaluations with at least 5 users.  Each interviewee will be provided with a consent form so that we can share comments and a chance to try the new version of Synote before the meeting.

Synote screen grabSynote Guide as the public version
ALUIAR Synote screen shotALUIAR Synote with a recording 

 

 

 

 

 

 

There will be a meeting with Diana Galpin as the University’s Legal Adviser to cover all aspects of ethics, privacy and other legal issues involved with the change of use of Synote although as in the past Synote itself will not be hosting the recordings – these are held on the users’ chosen server, whether this is a private one, belongs to a faculty with security settings or is public but offers a URL link as in the case of YouTube.  All transcriptions and annotations have chosen permission levels and once again Synote can be held on a secure server for each Faculty or University.

The meeting was wrapped up with final thanks to Yunjia for all his work on the project and a decision to have the interviews with researchers in the early weeks of November once new features and further testing of the new interface has been completed. The URL for the new version will be sent out with suggested dates for interviews.

Next meeting – to be advised on Doodle.