Category Archives: Meeting

Last Meeting – Outputs and Lessons Learnt

Decisions around what has been developed and the best options for the future of Synote Mobile.

white board choices

Synote mobile has been developed as a mobile phone web app as mentioned in the previous blog.  It allowed us to play videos, read transcripts and annotations etc. BUT we discovered that much depends on the phone operating system, browser and player as to how much access can be achieved especially  in terms of reaching the video player controls and captioning within the web page.

The main comment was that It is easy to navigate around a specifically designed mobile web site and most features can be used in the phones that offer the access as of Sept 2012.

video on mobile top view

video on a mobile - top view

However, testing also showed it is possible to work with the videos available on the Synote linkeddata site on a smaller screen – 3.5 inches. It may take three scrolls to reach the bottom and require some horizontal scrolling depending on whether one is viewing in portrait or landscape mode.

When it comes to outputs – It is possible to add a transcript and annotations – small fingers and delicate typing required!

The main comment was that the site we have at present could be improved with further work – more responsive designing required.

 

So…

video on the mobile

video on the mobile - second scroll

Should we offer two versions – one for desktop/tablet and then a separate version for the mobile?  In other words two websites  linking to one media collection – this would allow for an improved user experience on the mobile but mean that two versions need to be maintained.

Or …

video on the mobile - third scrollOffer one website that is developed in a responsive manner – all the interactions would be available but some may be harder to achieve on a small screen.

 Decision…

If time was not an issue it would be preferable to develop two sites as a final output – one specifically for the mobile phone.   However, with all the changes occurring with aspects of online video streaming and HTML5 standards it would appear wiser to spend time on making a version of Synote mobile  that is as responsive as possible – working on all platforms whatever the size of screen.

Just as a final point and an illustration of the constantly changing scene…

Recent updates on video streaming would now allow us to use an Apple streaming video server that would appear to allow for captioning on the iPhone whereas at present this is not possible with YouTube videos.

Evaluating Synote mobile

evaluation group

The group meet to evaluate Synote Mobile

A small group of potential Synote mobile users met over lunch bringing with them a collection of devices. The Synote Linkddata home page for the application was provided with an introduction by Yunjia.  The desktop and tablet version of Synote mobile work from these pages and the specific version for smaller screen (those around 4 inch diagonal) is provided by a link from the Mobile Access paragraph. Everyone was happy to comment and agreed to some pictures being taken for the blog. We knew there were several issues that still had to be overcome and explained the barriers to full use due to browser and mobile device problems with video downloads and HTML5.

iPhone play video

Just play the video for a few seconds

On all the devices it was essential to stress that there was a need to buffer the video for a few seconds before any of the menus would work – this is because mobile phones are set to avoid the download large amounts of data and no optimisation is possible. 

Comments were taken from the users – headings in the table below show the devices used and the browsers chosen.  A few areas were covered but it was not possible to track all the comments made during the lively exchanges!

 

 

The overall opinion of the website was good and the main purpose was to discover what further issues were arising and whether users had any other requests from the point of view of the look and feel as well as usability and accessibilty.

iPhone Safari

iPhone Opera mini

iPad Safari

iPad Chrome

Android Galaxy native browser

Android HTC built in browser

Android  HTC Opera Mobile

Website homepage

Look and feel

Web page too wide. Needs to be more resposive Looking good Good Clear and easy to use Easy to reach link Easy to reach link

Website homepage

Ease of Use

login needs page width to be responsive – Onscreen keyboard works well. Login in good Login easy to use Log in  good Log in issue need to be in portrait mode otherwise the edit boxes are hidden by onscreen keyboard Login possible but small screen means not easy. Login hampered by onscreen keyboard

Synote mobile get to video

Download and the pause.  Both central button and play button below works does tracking Total failure with video – just presents as a download Tracking, pause and start do not working on player Tracking, pause and start do not working  on player video needs to buffer.  -video needs to buffer. Video plays well from both buttons on the video and below

Play video

Once a video had been played the buttons stop responding .  Needed to refresh the page Main play works but Play button below the video is not working. Play button works well play button below video tried to load it. Can’t get out of full screen  mode without pressing back button which takes the video player out of ready mode.  Affects jumping to media fragments Can’t get out of full screen  mode without pressing back button which takes the video player out of ready mode.  Affects jumping to media fragments

Read transcription

That worked well but transcript box could take more text, depending on the screensize – needs to be clicked on to make bigger. Possible to reach trancript and return to video but no point as it did not play Some of the buttons seem to lack response or maybe very slow From video to transcription can be slow  – plays video but blank page shown Went relatively quickly Jumps back to beginning  not to position of that section of transcript.

Add/Edit transcription

N/A N/A Works well in portrait mode for editing Video to transcript section and vice versa works well. N/A N/A

Read annotation

Easy to reach once video buffered. Possible to reach synmakrs Easy to read Should the 7  inch screen be considered as a mobile device rather than a tablet as it looks better in landscape but  a lot of white space Can access synmarks Can access synmarks

Add/Edit annotation

N/A N/A Should always use pause button when working with annotations – causes confusion as the stop button  – then restarts at the beginning. Can annotate and better in portrait It is all possible just takes time and rather temperamental – seems to be linked to issue with J-Query library N/A N/A
Android login

Log in on the Samsung Galaxy - Android OS.

One of the most annoying issues was the Android onscreen keyboard was not as responsive and the iOS version which never fails to stay below edit boxes whereas the Android OS is happy to cover them on the Samsung Galaxy.

 

 

 

Tim Berners Lee media fragment

Anyone willing to annotate this media fragment?

HTML 5 Video and Support in Mobile Browsers – Lessons Learnt

Yunjia Li wrote on his personal blog the following article after our last Synote Mobile meeting with Wei Jing who has been helping us develop the tables you see below.  

“Since proposed by Opera Software in 2007, the HTML5 video tag has been around for a pretty long time. Video tag is a great step forward to make video the first class citizen of the Web. Unfortunately, different browsers understand the citizenships in different ways. The citizenship of AVI is terminated. WMV needs to hold Sliverlight Visa in order to get the entry clearance to browsers other than IE. FLV’s visa application for iOS has been Denied by Apple Empire and Adobe has declared that he’s not going to help FLV to re-apply the visa again. H.264 is like a mafia, who asks for protection fees from browsers. You see, there is no Federal Government in HTML5.

Since the smart phone joined the Web, things are getting worse again. It seems to me that there is no systemetic tests for the video compitablity on different mobile browsers. So the goal of our test is to find out which video format could be played in which browsers, or which major versions of the browsers.

Our Test

We need to emphasis something before going into the test.

  1. All the browsers in the test are on mobile phones. They are not desktop versions of the browsers.
  2. We only tested on android and iphones. Browsers for Windows Phone, Blackberry and Palm WebOS are not tested.
  3. All the tests are carried out in mobile phones. Well, yes, you can make a phone call using 7 inch Samsung Galaxy tablets if you don’t feel it’s too big. But the mobile phones used in this test are 3.0 to 4.8 inches for the screensize.
  4. Not all browsers, as well as a specific version of the browser, could be used in all different mobile devices. For example, there is no Opera Mobile browser on iPhone iOS 5. It is the same case for Firefox. But at the time of testing, we chose the latest version of the browser in that OS.
  5. We suppose that the server which hosts the video file is properly configured in order that the file could be properly delivered.

Finally, we have tried our best to keep the data correct on the phones we have tested. But it’s quite likely that some of the test results are not accurate. If you find anything that is not correct, please feel free to let me know. I would be glad to test it again. OK. Let’s go!

Test 1, Playable

This test shows if a video file with a certain codec could be played in a certain browser on a mobile phone. We test mp4, ogv and webm on Android 2.3, 4.0 and iOS 5. We designed a test page, which embeds mp4, ogg and Webm videos in one page. Then we open this page in different mobile browsers on different mobile OSs. Table 1,2 and 3 shows the results.


Table 1.HTM5 Video Compatibility in Android 2.3
Browser MP4, H.264 Ogg, Theora WebM, VP8
Opera Mobile 12.0 Yes No No
Opera Mini 7.0.29 No No No
Firefox 14.0.1 No Yes* Yes*
Android Native Browser Yes No No

*Unlike other browsers, the video will not enter full screen when playing, and there is no full screen control in the player in Firefox 14.0.1.

Test Phones for Table 1:

  1. HTC Design S, Android 2.3.5, 3.7-incn touch screen
  2. Sony SK17i, Android 2.3.4, 3.0-inch touch screen
  3. Samsung Galaxy S II, Android version 2.3.3, 4.3-inch touch screen
  4. HTC G10, Android 2.3.5, 4.3-incn touch screen

P.S. Google Chrome is not compatible with Android 2.3.


Table 2.HTM5 Video Compatibility in Android 4.0.4
Browser MP4, H.264 Ogg, Theora WebM, VP8
Opera Mobile 12.0 Yes No No
Opera Mini 7.0.29 No No No
Chrome 18.0 Yes No Yes
Firefox 14.0.1 No Yes Yes
Android Native Browser Yes No No

Test Phones for Table 2:

  1. Samsung Galaxy S III, 4.8-inch touch screen

Table 3.HTM5 Video Compatibility in iOS 5.1.1
Browser MP4, H.264 Ogg, Theora WebM, VP8
Safari 5 Yes No No
Chrome 21.0 Yes No No
Opera Mini 7.0.29 No No No

Test Phones for Table 3:

  1. iPhone 4S, 3.5 inches screen
  2. iPhone 4, 3.5 inches screen

As a summary, here are some interesting findings:

  1. There is no “killer” video codecing format for mobile browsers currently. But MP4 seems compatible with most “default” browsers, i.e. Android Native browser in Android phons and Safari in iPhones.
  2. Opera Mini is pretty disappointing as no video could be played.
  3. Different from desktop browsers (see this table), WebM is not well supported by the native players on mobile browsers. Interestingly, WebM is developed (or more actually, sponsored) by Google, but the Android Native Browser, which is also developed by Google, can’t play WebM
  4. Most mobile phones play the video in full screen mode by default and users cannot quit the full screen unless stopping the video. But Firefox is an exception. What’s more, if the screen size of the mobile phone is larger than 4 inches, The full screen, if screensize is bigger than 4.3 inches, sometimes, video won’t be played in full screen mode automatically.

WebVTT support in Mobile Browsers

HTML5 video tag not only brings video playing natively, but also adds some cool features alongside the video. One of them is the track tag for subtitles. Subtitles are very important for the accessiblity of videos on the Web. Many standards and de facto standards have been around for a long time. SubRip (SRT), Timed Text Markup Language and .sub, to name a few. I personally think the most promising one is Web Video Text Track (WebVTT). WebVTT is specially tailored for videos on the Web and you can also use it for audio description, chapter navigation, etc. Silvia Pfeiffer has given an very good presentation about WebVTT.

WebVTT is still a moving target and there are many things which need to be nailed before the first release. Except for the WebVTT specification itself, as developers, we are really care about the technical support of WebVTT in different browsers and are there any tools out there we can easily use? Silvia summarised the current situation of WebVTT support in different desktop browsers.

A further question we ask is that is there any mobile browsers which support WebVTT natively? After our test, the answer is NO! Table 4 is our test result. “N/A” means a certain format of the video file cannot be played in that browser, so it is meaning less to say if the WebVTT can be displayed or not.

Table 4.WebVTT Support in Different Mobile Browsers
OS Browser MP4, H.264 Ogg, Theora WebM, VP8
Android 2.3 Opera Mobile 12.0 No N/A N/A
Opera Mini 7.0.29 N/A N/A N/A
Firefox 14.0.1 N/A No No
Android Native Browser No N/A N/A
Android 4.0.4 Opera Mobile 12.0 No N/A N/A
Opera Mini 7.0.29 N/A N/A N/A
Chrome 18.0 No N/A No
Firefox 14.0.1 N/A No No
Android Native Browser No N/A N/A
iPhone iOS 5 Safari 5 No N/A N/A
Chrome 21.0 No N/A N/A
Opera Mini 7.0.29 N/A N/A N/A

 

Of course, there are many HTML5 video players on the Web, VideoJS, PopcornJS, MediaEleemntJS, JWPlayer, Kaltura Player, etc. Some of indeed support WebVTT on desktop browsers. But they are all “polyfills”, which means the support is not native and they use javascript and CSS to somehow present WebVTT together with the video. We did test MediaElementJS on iphones, but unfortunately, it still failed to display the subtitle. However, MediaElementJS can sucessfuly display WebVTT on iPad.

Conclusions and Future Directions

The test for HTML5 video tag and WebVTT on mobile browsers is important. Developers should be clear what video could be played in the target device. On the server side, the web applications need to host the best format that compitable with the target device. Video sharing services, such as YouTube, must be adaptive enough to deliver videos to different devices and browsers. Subtitles are always important for videos, especially on mobile platforms, where the support for subtitle is not thorough yet.

With each updates or release of new versions of browsers, new codecs might be supported and old formats might be abandoned. WebVTT is currently is not well supported on mobile browsers and we are still waiting for some break-through.

The tests tell us the HTML5 video support is on the way. In the tests above, we are trying to keep things easy, i.e. we didn’t bring in a lot of variables and we just used the most popular phones and most popular browsers. However, the real world is not that simple. Between mobile phones and desktop machine, we have got tablets, 7 inches to 10 inches. The screensize of mobile phones are getting bigger and bigger. I am very curious on how different browser kernels choose the display mode for these screensizes. Is there a borderline or threshold? So in the future, it might be necessary to compare the HTML5 video tag in a single browser across different size of devices (phones and tablets). We are also expecting more documents could be released or discussed regarding this aspect.

Again, if you find anything wrong or anything new, please feel free to leave your comments.”

Feedback on prototype Synote Mobile

comparing Galaxy and iPad presentation of Synote mobile

This meeting provided Yunjia with the first chance for a small group of users to provide feedback on the prototype of Synote Mobile using 7 and 9.7 inch tablets with Safari, Opera and Chrome browsers.

Yunjia is using CSS3 media query and viewing port size to decide when you are working with multimedia materials and how they will be displayed on devices – so it is hoped the user automatically experiences a comfortable presentation. In terms of Synote Mobile this means that when the viewing window is below 768 pixels the two columns that hold the transcript and synmarks (annotations) collapse into one. When a video or audio is played the synmarks will be on one tab with the closed captions allowing you to see the transcript across the window. The user can also choose just to read the transcript via that tab.

One issue that has been discovered by taking screen size as a method for changing presentation of videos and transcripts is that the CSS3 media query is dependent on screen size and videos are dependent on pixel widths. It is a tricky problem to explain as it appears that CSS media queries result in different behaviours for continuous media  and paged media.  There is also an inconsistency in behaviour across devices and browsers.

The controls are dependent on viewing modes, devices and browsers – so when using landscape mode on the iPad it senses that it is in desktop mode. When playing the video and viewing the transcript the player controls are always seen but when using Portrait mode the CSS media query senses the reduced width and represents the web page without the player controls as you scroll down. Yunjia says it is possible to solve the problem on some devices, but because this does not work for all devices it has been decided that ‘width’ is probably the best method for defining when the controls will be shown – Yunjia is using ‘Bootstrap’ from Twitter at present, which is considered the most consistent environment across all devices when it comes to presenting media data.

Synote Researcher website

When discussing other user issues such as the look and feel, the Synote Researcher website used as the basis for viewing recordings was considered to be easy to use and attractive but there were one or two login issues on the iPhone.

One user suggested that the highlights of synmarks (annotations) should be different from the transcript highlights as the video or audio plays – the example provided shows a YouTube video working with Synote Mobile.

Synote Mobile closed captionsIf there is a large chunk in the transcript when viewed on the video the caption will overlay too much of the picture. YouTube have set their automatic speech recognition captioning to always generate 2 lines - if the transcript is put in manually the size can be reduced to two lines for each timed section so it appears in a similar fashion but work is needed in this area to ensure this occurs in Synote mobile. It would be possible to edit the transcripts with an autosync system that divides longer sections into 80 character chunks that display as two lines. This cannot be done on the fly but would need to occur as part of the process in the database where the separate caption record is kept. This would not be necessary for audio as can be seen from the image as the caption can be displayed on a plain background.

The mobile phone version is still being adapted with single column viewing and the transcript appearing as captions – at present it has reached the viewing stage but not editing mode. A further meeting with the developer is planned on August 3rd, 2012

Responsive Design – Minutes from the 3rd team meeting

As mentioned in the previous blog this meeting was attended by Dr Mike Wald, Yunjia Li and E.A.Draffan – Gary Wills sent his apologies.

Responsive Design

responsive designIt all started on the white board where Yunjia explained how he was adapting his design to fit with the problem of 7 inch plus screens versus the average smart phone screen.  He demonstrated the ever narrowing of the viewing area and how the video, annotation and transcription windows could shrink and eventually offer access via tabs.  But when it came to mobile phone users there was a need to rearrange the view to be totally linear.  In other words two versions of Synote mobile but with changes that would happen automatically depending on the metadata received.

Here are four views of Synote mobile on the different sized screens

1200 and above

Screen size 1200 pixels and above

screen size 980 to 1200

screen size 980 to 1200

screen size 768 to 979

screen size 768 to 979

screen size 480 to 767

screen size 480 to 767

It will be very important to gather the metadata from the media so that the correct view and player can be selected. At present there are 3 useful approaches for this demonstrator – ffmpeg and YouTube has an api with metadata such as title, resolution etc  and finally the link to the file format itself offers metadata which can be drawn out by a commercial service and this would need to be re-created if we did not want to pay for the service.  The minimum amount of data needed is the duration of the media, the format and coding and whether it is video or audio.

It just so happened that there was a blog on this subject that fitted the discussion around screen size issues. Titled “What’s next for mobile now that adaptive design has failed?” it seemed like an appropriate article to read before the meeting.

“Desktop web browsers, tablets, and mobile devices are fundamentally different and are used in very different ways. Across our properties at CBS Interactive, we have experimented with a variety of adaptive and direct designs and are learning the hard way that a one-size-fits-all solution delivers a subpar user experience.” Peter Yared (VentureBeat MobileBeat blog)

 

 Annotations / Captions

diag of mobile interface

caption illustrationCaptions can be displayed on a desktop with the video but there appear to be no standards for displaying captions within web pages across all  tablets and mobiles alongside transcriptions and annotations.

At present if someone is deaf they have to read the caption and watch the video and then scroll down to the note taking mode.  It is not possible on the iPhone to display both the transcript window and the video due to the size of the screen.  However, it will be possible to capture an image from the video  and annotate this as part of the note taking process.

3rd Team Meeting – Wednesday 23rd May, 2012

wireframe lightbulbThis month seems to have gone by rather quickly and this team meeting will be a small one with Dr Gary Wills away at the moment.  Dr Mike Wald, Yunjia Li and E.A Draffan will meet to hear about Yunjia’s “responsive design for Synote Player and the pages he will be presenting using wireframework’.

He has set the discussion going with the following statement:

“This project will use basic principles of responsive web design to create a new mobile HTML5 version of Synote, able to replay Synote recordings on any student’s mobile device capable of connecting to the Internet.”

Item 2 will be how this is achieved on the various sizes monitors such as those offered on the iPad, Galaxy Tab and iPhone etc.

Item 3 will include the issues that have arisen with the player and being able to read annotations simultaneously whilst watching videos.

As Dr Mike Wald said in a recent email to the team “One issue with captioning overlayed on the video (as suggested for Synote Mobile)  rather than in a separate panel (as we have currently in Synote) is how can a hearing impaired person read the captions when the video is a thumbnail or not displayed at all?  There needs to be a way the captions can be displayed separately … Matterhorn designers are now thinking about ways to do this  …

 

 

 

2nd Team Meeting – Confirm Requirements & Specification for Synote Mobile – 26th April, 2012

Mike and Yunjia thinking about things.

Mike and Yunjia thinking about things.

The team meeting held on 26th April had a simple agenda of confirming the user requirements and specification for Synote Mobile,  based on discussions with colleagues and students in the Web and Internet Science research group and others who have been helping us to debate specific issues and concerns.

Screen size issues

Synote mobile needs to work on mobile phone screens as well as tablet or slate screens – taking the iPhone and iPad as prime examples of the differences it is clear that two versions will need to be developed.

The program should automatically detect the device and load the corresponding page or style sheet. Here are two useful resources:

Yunjia discussed some useful links to help with these requirements:

Best practices for Device detection with Media Queries 

Adding Mobile Views to your Grails Applications with JQuery Mobile: A Real Life Example 

Twitter bootstrap 

Delivery of the videos

The video resources should be delivered in an adaptive way, i.e. devices with low resolution and bandwidth need to download smaller file sizes.

The iPad and iPhone media player does not allow for integrated note taking or annotations so it is proposed that there will be an HTML player version of Synote.

As is mentioned in a blog that discusses the insertion of a video or audio playing link for these devices - the player will take over the fullscreen when playing,so it will not be possible to have a split screen version of the transcriptions or the annotations. Therefore it is proposed that there will be a thumbnail picture displayed alongside the video with the annotations. It is felt that there is a requirement for the extraction of thumbnail pictures when creating annotations.

audio player in iphone

Taken from Ploem.be blog as an example of an audio file playing.

HTML5 Players

Considerations around the HTML5 player used have to include the best accessibility options, but with browser differences this is not easy to achieve. HTML video.org offer a helpful comparison of players.

Format Compatibility

Different devices cause format compatibility complications and Safari, Google Chrome and mobile IE also have different support mechanisms for HTML video codec. This means the delivery of multimedia resources has to be adaptable.

Touch interface

There needs to be careful consideration given to the type of gesture driven / tap type controls. Users who wish to have VoiceOver controls will affect the gestures that can be used within not only the player but also the webpages.  It has already been stressed that there is a need to keep clutter to a minimum and therefore there must not be too many buttons in the screen.  It was confirmed that the design would aim to abide by the accessible and ease of use design concepts mentioned in the blog “Research – Planning for an easy to use and accessible mobile app.” 

Despite the issues and concerns raised it is intended that the specifications being developed will have the bonus of offering

  • Web browser access – No app store download needed
  • Automatic enhancements – No need to connect to an app store for updates
  • Speed – adaptive performance on most connections
  • Automatic adjustment of presentation – depending on device screen size
  • Use of existing accounts.

Project Plan – Minutes for 1st meeting of the Synote mobile team

Minutes

Attending – Mike Wald (MW), Gary Wills (GW) Yunjia Li (YL), E.A. Draffan (EA)

• Design and development of the Synote web app suitable for mobile technologies

There was the initial discussion that led to the following points being made:

  • Not initially designed to work on mobiles – 4 years ago people were not playing videos on mobiles or tablets.
  • Browser issues on mobile – media players at present not playing synchronised audio and videos with notes
  • Devices will use different resolutions this will affect the ability to port the videos

Action
Initial rapid prototyping – download media on a mobile with a suitable Synote mobile interface – New UI so that you can see what Synote will look like on a mobile. (YL)

• Conversions of permitted recordings to suit the mobile platform

  • Find video – we do not store – it would be possible to set up a server and cache the videos
  • If the file is owned by the user it can be added to YouTube and then streamed onto the device within Synote mobile.
  • Adaptive video streaming – single file to many different devices – have to transcode it into several files.

Research
Different formats, adaptive streaming video, investigate YouTube or static file download cached. (EA, MW and YL)

 Issues related to media types, transcriptions and annotations.

What is WebM?

YouTube video – “John Luther, WebM Product Manager, talks about the open-source WebM video format, the problems that it solves, and what WebM means for the open web. Learn more at http://www.webmproject.org/

Device
Testing on different portable devices – Ipad3, iPhone, Android – delay on Windows tablet. (EA, MW, GW and YL – plus user stakeholders)

Work Plan

  • April: Website Template, Blog, Synote Mobile Requirements and Specifications & research into recording/replay formats
  • May: Rapid Prototyping Phase 1 producing prototype demonstrator on tablets and mobiles to replay recordings from file or streamed from Youtube
  • June: User testing and evaluation by Southampton and external collegaues to inform Co-Design & rapid prototyping phase 2

•  OER 3 Programme Meeting being held on 26th March 2012 at Dexter House, London (MW and EA)

• AOB – Discussion around take outs from long videos to produce clips that could be annotated etc.
• Next meeting date. Skype meeting after Easter

1st Team Meeting – 22nd March 9-10am, 4th floor demo room/FPAS board room 32/4073

This will be the first meeting of the team to discuss:

  • Design and development of the Synote web app suitable for mobile technologies
  • Conversions of permitted recordings to suit the mobile platform
  • Issues related to media types, transcriptions and annotations.
  • Preparation for OER 3 Programme Meeting being held on 26th March 2012 at Dexter House, London
  • AOB
  • Next meeting date.