Category Archives: Minutes

Three in One – Recap, Results and Recommendations

It has always been difficult to have a full team meeting and our last meeting was no different, but by using the University of Southampton’s iSurvey system we have managed to get results to some of our questions. Moreover, through the use of face to face interviews, emails, calls and the blog we have gained an insight as to how to develop an Adaptable and Learnable User Interface for Analysing Recordings.  Synote, for the researcher, has been developed into a freely available open source downloadable application that can be used by any individual,  institution or organisation.

We believe that we have succeeded in producing improvements to the ‘existing user interface’ and adapted Synote as an “e-research tool, in order to make [it] easier to learn by non-specialist users.” (JISC Usability and Adaptability of User Interfaces) by using a co-design methodology with a very open approach to the project evaluation.  The final sample video to illustrate its use is being produced and will be made available.

The complete agenda for the last meeting was mentioned in the last blog, but under the main points I have also provided an overview of some of the ideas and comments made throughout the project and during the face to face interviews. There is a final summary available at the end of this blog.

A service becomes a download with a new name

There was much discussion during the lifetime of the project about having to make a new version of Synote whilst leaving the original system intact. Synote was designed for the annotation of educational recordings with notes and transcriptions.  The content has often been drawn from lectures, seminars, videos and audio available online.  Recordings are not usually held on a specific server for Synote users.  Users are required to use their own websites to provide the URL link to the Synote player.  Create recording pageDuring the project we discovered that some users found it hard to make a URL from an uploaded recording and so the new version includes the ability to upload a recording from a desktop computer, to link to YouTube recordings, as well as users’ own websites.

This also led to the decision that the researcher version should be a download rather than held on the University of Southampton server for public use, as is the case with the original version. This decision was made because of the type of sensitive data that might be used by researchers.  By having a download the software can be held on secure servers with increased data protection.

Finally, the fact that the new version had some enhanced features, meant that it needed an addition to its name!   iSurvey was used for voting and although the name ‘Synote Scholar’ was liked by many members of the team – Mike was the final arbiter with the name Synote Researcher winning the day as he felt that the name ‘Synote Scholar’ could equally be seen to apply to the original version of Synote and so didn’t make it clear that the new version was designed for researchers!

Synote researcher logo

Evaluation results related to the new interface design for learnability and usability.

co-design diagram

Millard, D., Faulds, S., Gilbert, L., Howard, Y., Sparks, D., Wills, G. and Zhang, P. (2008) Co-design for conceptual spaces: an agile design methodology for m-learning. In: IADIS International Conference Mobile Learning 2008.

Throughout the project we used the blog to act as part of the evaluation process – publicising storyboards, changes made and slide views of the changing Synote Researcher using the agile co-design methodology mentioned at the outset.  This allowed us to have an on-going conversation with users and other team members.  It resulted in an iterative approach being taken to the development process and was very useful when monitoring changes.

We used John Brooke’s System Usability Scale (SUS)  for a quick analysis of the views of users as to the usability of Synote Researcher. The results showed that there was an increase in the overall scores out of 100 from 48.50 to 67 with 4 out of 5 users agreeing in the second evaluation, that they felt “most people would learn to use this system very quickly” The score that brought the second evaluation down was the need for technical support to set up the system. This is not surprising and the download may need technical support.

More precise evaluations were undertaken via a series of face to face interviews designed to see if the changes made had ensured that Synote was easier to learn and use by researchers wishing to analyse and code multimedia data.   During the interviews users were asked to participate in practical exercises such as logging-in, searching for recordings, creating their own recordings and adding a transcript and synmarks or annotations. They were asked to comment throughout the meeting and offer a score of 1-5 for look and feel where 5 was excellent.

Player pageDuring the discussions background information about the possible use of the demonstrator was requested and how users views about the Synote service may have changed due to the redesign.   The evaluation of the original Synote took place in June 2011 and the final appraisal occurred during November 2011.  The interviews were also analysed using Synote Researcher with annotations and remarks as seen above.

web page scores

Average score for each page - red = original Synote, blue = Synote Researcher

The results for scoring the individual pages were based on the users who completed both interviews although we had an additional three external users who commented on the new version.  As there were some extra interviewees it was decided that an average of each score given for each page discussed would be taken over both interviews.  As can be seen all the scores increased and this was despite ideas for further changes and constant iterations.

As a result of all the requests it was felt necessary to revisit the original work packages to check all the elements requested at the outset of the project had been discussed and included in the design. Only one of the requested changes was not possible to achieve in the time available as this would have involved a redesign of the whole Synote system.

  1. pausing the playerGreater flexibility of movement backwards or forwards through a recording (e.g. by typing in new time) as at present can only move in 5 second ‘nudges’ or move time slider or change speed if recording format and player allow. – Now able to nudge the recording forwards and backwards by 1,5,10 and 20 seconds.
  2. A drop-down box listing frequently used tags (e.g. for coding name of speaker and category code) – These appear as you add a tag to the annotation.
  3. Foot pedal control of player – This has been achieved and tested using a Start-Stop HDP-3S pedal from HTH Engineering Inc.  The software has allowed us to allocate controls as per the original Synote keyboard shortcut accessibility options.synmark tags
  4. When manually transcribing a recording it is possible to also annotate this with the
    start time of the clip entered automatically but the end time needs to be manually entered. Synote allows a section of a created transcript to be selected and the annotation to be linked to that section with the start and end times of those sections to be automatically entered. It would make the system easier to use if it was possible to also do this without having first to save the transcript.. – This has been achieved with + and – keys plus an icon of the clock to indicate the inclusion of time as can be seen in the screen grab on the left.
  5. Facility to download the annotation data (e.g. to Microsoft Excel for statistical analysis and charts and graphs or for a report or into other annotation tools). At the
    moment the information requires copying and pasting – This has been achieve with a csv file with all the synmarks as can be seen in the picture above by the paper icon with the arrow at the top right.save as draft icon
  6. Making it harder to exit without saving and so losing changes made.-  This has been achieved with an automatic save as draft and additional save on exit.play button
  7. Allowing the user to control the recording playback when annotating by providing
    media player controls in the annotation window. (at present a user can annotate a
    recording and the annotation can automatically read the time of the recording but the user cannot easily replay a section of the recording while writing the annotation) – this has been achieved through the use of the right hand mouse button.text colour
  8.  Use of text font, colour, size and style for coding – this has been achieved as shown in the picture.
  9. Individual profiles provide a graduated learning curve to suit the preferences of each
    particular researcher allowing fine-grained control over a wide range of settings (e.g. whether the replay of the video should pause automatically while an annotation is being created). – This has partly been achieved by the viewing of icons and supporting text only when moused over or selected or the use of the right hand mouse menu within the transcript or synmark section.
  10. Redesign of interface to improve learnability – This can be seen when viewing all the attached images and downloading the software as well as the evaluation results.
  11. Organise recordings into groups and categories to make them easier to find and
    manage – This has been achieved through the groups menu and sorting mechanism within the recordings listings. 
  12. Ability to replay just the video clip from a search (at present plays from the start time
    and manually have to pause at the end time of the clip) – it has not been possible to achieve this in the time available as it would require a redesign of the whole Synote system.My synote

tags and a tag cloudIn addition to the changes made to many sections of the original Synote there has been the introduction of ‘MySynote’ – this is an area where the researcher can look at their own recordings, synmarks or annotations and tags used for coding as well as edit their profile.  It allows for the sorting of recordings and the viewing of the most commonly used tags as a tag cloud.

 Overview and discussion about the initial comments from interviews undertaken with the team and external evaluators.

Both Yunjia and I were very heartened to hear the comments made by interviewees including external evaluators who had seen or used the original Synote, and were now being introduced to Synote Reseacher.  Even those who were using the new version commented favourably on the improved usability and learnability.  Dr Sarah Parsons commented ” It feels a lot more intuitive and I think with some of those tweeks we have talked about in terms of labelling clear what a couple of those bits are would strengthen that even further,  it feels very accessible to me….”   Audio clip

homepage commentDebbie said about the home page “Thats easier. (pause) On the old one you had to find information you had to start searching you had to read the whole of the page of other information before you can necessarily find what you need to start the process. So this is much clearer.”

Changes to be made to the final iteration

The final changes as a result of the last few interviews are minor ones listed below

  1. Refine information on front page – Reducing and improving the presentation of the help slides.
  2. MySynote page to just have icons and names to match the list that appears on other pages.
  3. User Profile to contain registration / password information – the captcha is to be removed as this will not be necessary on the download.
  4. Users are going to be able to edit and delete their recordings.
  5. The Groups will work for sharing recordings with chosen users or setting up recordings in different categories.
  6. Legal statements will be added and a simple help file added.

Final Summary
We have been asked to

  • Recap – what you set out to achieve, what you didn’t achieve, why?
  • How successful have you been? What are your metrics?
  • Lessons Learned?

It is hoped that this final blog has covered all these requirements as it was hard to divide the list into three separate blogs without considerable repetition.  We set out to achieve a version of Synote that would suit researchers and be easy to use and learn.  We hope we have been successful as well as providing a unique application that can:

  • be freely used
  • work quickly and easily on any desktop with access to the internet
  • have web based data and multimedia for sharing research.
  • use the data presented for evaluation, appraisal and supervision
  • capture simultaneous conversations with similar timeframes
  • colour code annotations
  • tag themes and categories
  • use titles of annotations for sorting.
  • use a foot pedal to control the media player and other aspects of the interface
  • use keyboard only input and screen reader access (Equality Act 2010)
  • use speech recognition for re-speaking to speed transcriptions.

There is still the challenge of creating annotated clips from the recordings that automatically stop at the end of the clip – this was not possible to achieve in the time available as it requires a significant redesign of the Synote system

Rating our success through the use of various communication methods along with specific surveys plus interviews has provided us with a range of quantitative and qualitative data, despite the low number of users over the period of the project. The Nielsen Norman Group is often quoted as saying “The best results come from testing no more than 5 users and running as many small tests as you can afford.”

When it comes to learning lessons these can be summarised as follows:

  • Allow more time than you think for evaluations – interviews and transcriptions
  • Make sure you have at least one willing volunteer who is not working on the project on a day to day basis but is using it for research (in this case).  We owe Debbie a great deal of thanks as she provided us with several insights that would have been overlooked by those of us who had become over accustomed to the way Synote worked.
  • Have outside evaluators who are willing to comment on the old versions and trial the final iterations with a fresh look – Thank you Lucy, Ring, Sarah and Inga
  • Have some expert researchers (in this case) who are willing to share their requirements early on in the timeline and compare the development ideas with what may be offered by other software programs – Thank you Lisa H, Lisa R and Mary
  • Define your unique selling points so that you can persuade new users to adopt the new program – Thank you Mike, Lester and Gary 
  • Finally but most importantly – you need a good lead developer to change a program that has been developed by someone else, but also one who is willing to be endlessly patient and understanding when translating ideas into reality – Thank you Yunjia

 

Minutes from the meeting held on 13th October, 2011 – 5th ALUIAR Meeting

Meeting group

ALUIAR meeting October 13th

Apologies

Received from Mary Gobbi, Lisa Harris and Lester Gilbert.

Minutes from the Meeting.

E.A provided feedback from Torsten with an overview of the final stages of the project and reporting.  It was confirmed that reports would come in the form of blogs and more information can be found on the blog about at projects “Usability Workshop – Emerging Themes from jiscUX The email that followed mentioned “a few of the headline requirements for final reports:

  • 3 final blog posts:
    • Recap – what you set out to achieve, what you didn’t achieve, why?
    • How successful have you been? What are your metrics?
    • Lessons Learned?
    • Edited version of your posts
    • Send us your final budget and complete the JISC completion survey (we will send you these documents near to the end of your project)

This was followed by a discussion around the Qualitative Data Analysis Steps (Adapted from Creswell and Burnard) that had been provided by Mary earlier in the week – see previous blog.  

It was decided that the points made in the slide should be related to our use of the ALUIAR version of Synote.   Debbie noted that the IBM transcription was not very accurate with a group,  unless each one has a microphone with clear speech –  this makes the speech recognition system rather hard to use – Most researchers are having to transcribe manually. The transcription service is only available at University of Southampton unless others wish to have a license.  The difference between working in the transcript section as opposed to the synmark area was mentioned and where coding occurs etc.

It was felt that it was important to note that the ALUIAR version of Synote will work for qualitative analysis but not in depth Conversational analysis due to the multitude of extra coding features required for a full transcription.   Conversational analysis is best done with specialist software.  However, Synote can be used with other languages (there are some problems with texts that go from right to left), it is possible to share and collaborate over open transcriptions and research projects, transcriptions can be exported along with their annotations and tags can be sorted and saved as a way of generating themes or categories.

The second half of the slide mentions more about the research side of transcriptions and Lisa said that “it is important to note that the researcher must still do the analysis and the system can only manage the data”

Yunjia then introduced the new interface for the system and there were several design features where decisions had to be made – these were captured in the photograph below.  Some buttons and actions in the list of recordings needed moving, a rich text editor with a chance to colour code annotations was requested and Footpedal control only needed for stop and nudge back.  The ability to upload images was felt to be important and the possibility of being able to use the play controls wherever one is in the transcription or synmarks (annotations)

white board notes

White board notes

A walk through of the new interface was completed with a promise to provide the URL in the next two weeks so that we can begin the interviews for the final phase of evaluations with at least 5 users.  Each interviewee will be provided with a consent form so that we can share comments and a chance to try the new version of Synote before the meeting.

Synote screen grabSynote Guide as the public version
ALUIAR Synote screen shotALUIAR Synote with a recording 

 

 

 

 

 

 

There will be a meeting with Diana Galpin as the University’s Legal Adviser to cover all aspects of ethics, privacy and other legal issues involved with the change of use of Synote although as in the past Synote itself will not be hosting the recordings – these are held on the users’ chosen server, whether this is a private one, belongs to a faculty with security settings or is public but offers a URL link as in the case of YouTube.  All transcriptions and annotations have chosen permission levels and once again Synote can be held on a secure server for each Faculty or University.

The meeting was wrapped up with final thanks to Yunjia for all his work on the project and a decision to have the interviews with researchers in the early weeks of November once new features and further testing of the new interface has been completed. The URL for the new version will be sent out with suggested dates for interviews.

Next meeting – to be advised on Doodle.

Suggested Solutions to Synote ALUIAR Issues – 4th Meeting

The 4th meeting of the ALUIAR team was set up to finalise the storyboarding of the ideas suggested in previous meetings and to present the outcomes from the data gathering plus options for some functional solutions.

Those attending were Mike Wald(MW), Garry Wills (GW), Seb Skuse (SS), Yunjia Li (YL), Mary Gobbi (MG), Lisa Roberts (LR),  and E.A. Draffan (EA)

Apologies

Apologies were received from Lester Gilbert,  Lisa Harris and Debbie Thackray.

Mike opened the meeting with a discussion document related to the functionality issues discussed at the outset of the project and comments collected from initial interviews.   Accepted ideas are in red. 

  1. Greater flexibility of movement backwards or forwards through a recording (e.g. by typing in new time) as at present can only move in 5 second  ‘nudges’ or move time slider or change speed if recording format and player allow.

Possible Solution(s)

a) Enter time into time entry box and player will move to that time

b) Change ‘nudge’ time from 5 seconds to 1 second

c) Add additional ‘nudge’ time of 1 second as well as existing 5 seconds

d) If in editor and transcript text is selected for the editor text box then move player time automatically to the start Synpoint time

  1. A drop-down box listing frequently used tags (e.g. for coding name of speaker and category code)

Possible Solution:

Implement drop-down box listing frequently used tags. E.g.

a)    tags they have used on this recording

b)    tags anyone has used on  this recording

c)    in alphabetic order

  1. foot pedal control of player

Possible Solution:

Find available foot pedal that works or allows pedals to be assigned to keyboard shortcuts – Research the issues – EA to contact Hagger about suitable foot pedals

  1. When manually transcribing a recording it is possible to also annotate this with the start time of the clip entered automatically but the end time needs to be manually entered. Synote allows a section of a created transcript to be selected and the annotation to be linked to that section with the start and end times of those sections to be automatically entered. It would make the system easier to use if it was possible to also do this without having first to save the transcript.

Possible Solution:

If there is text in the editor text box then when selecting create, automatically enter both the start and end Synpoint times into the Synmark start and end times

  1. Facility to download the annotation data (e.g. to Microsoft Excel for statistical analysis and charts and graphs or for a report or into other annotation tools). At the moment the information requires copying and pasting

Possible Solution:

Add csv export for Synmarks and Transcript to print preview

  1. Making it harder to exit without saving and so losing changes made.

 Solution: Already done this in the current version

  1. Allowing the user to control the recording playback when annotating by providing media player controls in the annotation window. (at present a user can annotate a recording and the annotation can automatically read the time of the recording but the user cannot easily replay a section of the recording while writing the annotation)

Possible Solution:

Add the javascript player controls to the Synmark panel

  1. Redesign of interface to improve learnability

Possible Solution:

This is related to the current interface work and can be seen in the PowerPoint slide show below. 

  1. Organise recordings into groups and categories to make them easier to find and manage

Possible Solution:

Add tags to the title field

If categories were to be used they would have to be hard coded and not all the categories would be suitable.

  1. Ability to replay just the video clip from a search (at present plays from the start time and manually have to pause at the end time of the clip)

Possible Solution:

Using linked multimedia fragments – not feasible in the time scale

Additional Issues NOT in original Proposal

xiii Users find it difficult to understand how to store and link to their recordings in their own web space

Possible Solution: (Yunjia is currently investigating this) 

Allowing recordings to be uploaded into database rather than only being linked to in user’s own web based storage area

There then followed a presentation by Yunjia to show the work already carried out on the uploading of videos and audio recordings as well as changes that are happening to the interface. A discussion followed and the ideas were accepted.  The website is not public at present but below are a series of slides to show how the system is changing.

There was no other business and possible dates for the next meeting have been added to the Doodle Calendar for October.

 

2nd ALUIAR meeting, 11th July 2011

Those attending were Mike Wald(MW) Lester Gilbert (LG), Garry Wills (GW), Seb Skuse (SS), Yunjia Li (YL), Mary Gobbi (MG), Lisa Roberts (LR), Debbie Thackray (DT) and E.A. Draffan (EA)

Apologies

Apologies were received from Lisa Harris

Review of work to date

In the interim EA has met up with 5 Synote users (3 from the team) and collected the scores mentioned in the previous blog along with issues that particularly worried most researchers. More interviews are planned.

Synote page scores

Synote page scores 1-6 (excellent) completed by 4 users

There was a lively debate around these issues and it was agreed that the key problem was the uploading of media such as video and audio files ready for transcription. DT had had a particularly difficult time with her research material that needed to be secure as it related to patient data and setting aside an area which allowed DT to make a URL to go into the Synote ‘create recording’ area had proved to be quite difficult. YL agreed to look into this and help Debbie as well as plan a way of solving the problem – initially for Southampton researchers.

Uploading a file to a server and Synote

Uploading a file to a server and Synote

It is also possible to install Synote on an organisation’s server to solve the problem of sensitive data – MG has agreed to check the possibility of using a Virtual Private Network to access the department’s secure server. There was also the debate about using the ePrints API.

There needs to be guidance visible on the Synote interface such as clear steps for file uploads and better error and login feedback notification.

 

EA will contact Ed Fynn to help with easy guidance for uploading media files.

Transcriptions – Conventions

transcription conventions

Transcription conventions

MG provided the team with three documents to explain why it would be helpful to have some conventions available within Synote as well as the ability to colour code transcriptions.

Different colours for the various speakers may be difficult if required automatically.  Colours options for various themes was another request.
Synote can achieve the pausing in the annotations and comments

 

 

Terms and conditions – need to have different permissions for public and private data – check box.

Persona

miifigureThis persona (KT) has been built up from the comments gained from the interviews.

KT wishes to cut down the time spent on transcriptions when using Synote and finds the present design is hampering progress. Relatively accurate transcriptions  are required and a way of exporting  text and audio sections to make them accessible to all.  Ideally there needs to be a way of seeing who is speaking with colour coding and to be able to save annotations with some conventions to a different file format.   The annotations need to be tagged for the various themes and linked in some way to other interviews so that when a search is undertaken the various themes appear as a collection.  KT admitted finding the uploading of files difficult and felt that each time the system was visited there was some learning to be done!

Storyboarding new ideas for Synote

MG to storyboard qualitative methods and send them to us for plan views.
SS and YL to to storyboard initial interface ideas.

AOB

Holiday times discussed

Next Meeting

Doodle URL to be sent to the team.

Meeting Minutes – 15/05/2011

Adaptable and learnable USer Interface for Analysing Recordings (ALUIAR)

synote guide1st meeting – 15/05/2011 Room 3073 Building 32 (Access Grid Room)

Attendees

Mike Wald, Lisa Harris, Mary Gobbi, Gary Wills, Sebastian Skuse, Yunjia Li, Lisa Roberts and E.A. Draffan (Apologies received from Lester Gilbert for his absence).

Minutes from the meeting

Welcome and Introductions from members of the team

Work packages were discussed and Mike gave an overview of the project with the features that were mentioned in the project plan and others that might be included in the design.

It was decided that there would need to be some API changes and storyboarding of possible interfaces for various functions (Yuniji and Seb).

Mary Gobbi suggested the idea of a ‘Decision Tree’ and methodological framework for the types of interviews undertaken by researchers and the type of coding, annotations etc needed for different types of research. It was felt this would help many people and also act as a guide when deciding which features could be added to Synote and which were left to other types of software supporting research and speech transcription such as NVIVO and Transcriber.

Co-Design – taking the diagram below as a guide to the process being undertaken it was decided that short interviews with a series of stakeholders would be noted and some recorded and uploaded to Synote as part of the shared understanding and show and tell aspect of the process.  (action EA with team members plus other researchers)

co-design diagram

Millard, D., Faulds, S., Gilbert, L., Howard, Y., Sparks, D., Wills, G. and Zhang, P. (2008) Co-design for conceptual spaces: an agile design methodology for m-learning. In: IADIS International Conference Mobile Learning 2008.

Website and future communication choice – The results of this work will be visible on Synote and linked to a blog on the ALUIAR project website – the team will have a mailing list and drop box account.  (action EA and Seb)

The next team meeting will be 11th July, 09.30 – 10.30 Access Grid Room, Building 32 Level 3.