Discussions around embedding media players

There has been much discussion over the last few weeks about embedding media players within web pages that will automatically adapt to the user’s chosen device. The responsive design needs to be more than just responsive to one or two issues as we have discovered – there needs to be fall forward and fall back options! As Yunjia has said

The challenge for HTML5 video/audio in Synote is we need to embed different players not only based on the media type, but also the platform. Obviously, Flash is not well supported on mobile platform. So we must consider the html5 native player and control it through javascript.

MediaElement.js is a “fallforward” html player, which means it is based on HTML5 native player. However, if the browser doesn’t support HTML5, mediaelement.js will embed the self-developed flash and silverlight player. Hopefully this option is going to provide us with a solution to the issue!

The website MediaElement.js has provided a comparison of codecing and applicable browsers, including mobile devices.

There is also a large table providing a useful comparison of the HTML5 video player with available features.

Responsive Design – Minutes from the 3rd team meeting

As mentioned in the previous blog this meeting was attended by Dr Mike Wald, Yunjia Li and E.A.Draffan – Gary Wills sent his apologies.

Responsive Design

It all started on the white board where Yunjia explained how he was adapting his design to fit with the problem of 7 inch plus screens versus the average smart phone screen. He demonstrated the ever narrowing of the viewing area and how the video, annotation and transcription windows could shrink and eventually offer access via tabs. But when it came to mobile phone users there was a need to rearrange the view to be totally linear. In other words two versions of Synote mobile but with changes that would happen automatically depending on the metadata received.

Here are four views of Synote mobile on the different sized screens

Screen size 1200 pixels and above

screen size 980 to 1200

screen size 768 to 979

screen size 480 to 767

It will be very important to gather the metadata from the media so that the correct view and player can be selected. At present there are 3 useful approaches for this demonstrator – ffmpeg and YouTube has an api with metadata such as title, resolution etc and finally the link to the file format itself offers metadata which can be drawn out by a commercial service and this would need to be re-created if we did not want to pay for the service. The minimum amount of data needed is the duration of the media, the format and coding and whether it is video or audio.

It just so happened that there was a blog on this subject that fitted the discussion around screen size issues. Titled “What’s next for mobile now that adaptive design has failed?” it seemed like an appropriate article to read before the meeting.

“Desktop web browsers, tablets, and mobile devices are fundamentally different and are used in very different ways. Across our properties at CBS Interactive, we have experimented with a variety of adaptive and direct designs and are learning the hard way that a one-size-fits-all solution delivers a subpar user experience.” Peter Yared (VentureBeat MobileBeat blog)

Annotations / Captions

Captions can be displayed on a desktop with the video but there appear to be no standards for displaying captions within web pages across all tablets and mobiles alongside transcriptions and annotations.

At present if someone is deaf they have to read the caption and watch the video and then scroll down to the note taking mode. It is not possible on the iPhone to display both the transcript window and the video due to the size of the screen. However, it will be possible to capture an image from the video and annotate this as part of the note taking process.

3rd Team Meeting – Wednesday 23rd May, 2012

This month seems to have gone by rather quickly and this team meeting will be a small one with Dr Gary Wills away at the moment. Dr Mike Wald, Yunjia Li and E.A Draffan will meet to hear about Yunjia’s “responsive design for Synote Player and the pages he will be presenting using wireframework’.

He has set the discussion going with the following statement:

“This project will use basic principles of responsive web design to create a new mobile HTML5 version of Synote, able to replay Synote recordings on any student’s mobile device capable of connecting to the Internet.”

Item 2 will be how this is achieved on the various sizes monitors such as those offered on the iPad, Galaxy Tab and iPhone etc.

Item 3 will include the issues that have arisen with the player and being able to read annotations simultaneously whilst watching videos.

As Dr Mike Wald said in a recent email to the team “One issue with captioning overlayed on the video (as suggested for Synote Mobile) rather than in a separate panel (as we have currently in Synote) is how can a hearing impaired person read the captions when the video is a thumbnail or not displayed at all? There needs to be a way the captions can be displayed separately … Matterhorn designers are now thinking about ways to do this …

Beautiful mobile applications, beautiful user experiences Part 3

Whilst exploring issues around the look and feel of Synote Mobile Mike sent me this blog, which is one of three from The Computer Weekly Application Developer Network

“In part three of this guest blog for the Computer Weekly Developer Network, Sybase technical evangelist and mobile evangelist Ian Thain discusses the new mobile application landscape characterised by new and more beautiful user interfaces.

Links to part one and part two of this series.

There are a few mobile design guidelines that should never be far from your thoughts.

To take a few as examples :-

The initial screen should be kept as clear as possible to act as a launch point, because first impressions count – Synote has always gone for the minimalist approach!
Keep the main/primary controls in the thumb ‘hot zones’ at the edge of the screen and keep the most important content at the top, with controls at the bottom – this is particularly important for screen reader users who often track their fingers around the edges of phone screen to catch menu items.
Be generous with the space on the screen, do not crowd and avoid scrolling where you can. Reducing clutter helps everyone but in particular disabled users as does the following point.
Stick to proven navigation models, which can be used in combination, use flat pages for simple applications, and if possible make use of a tab bar that switches between the app’s main functions, and/or a tree structure for drilling down through a hierarchy of content.”

Please read the rest of the actual blog if you are interested.

2nd Team Meeting – Confirm Requirements & Specification for Synote Mobile – 26th April, 2012

Mike and Yunjia thinking about things.

The team meeting held on 26th April had a simple agenda of confirming the user requirements and specification for Synote Mobile, based on discussions with colleagues and students in the Web and Internet Science research group and others who have been helping us to debate specific issues and concerns.

Screen size issues

Synote mobile needs to work on mobile phone screens as well as tablet or slate screens – taking the iPhone and iPad as prime examples of the differences it is clear that two versions will need to be developed.

The program should automatically detect the device and load the corresponding page or style sheet. Here are two useful resources:

Yunjia discussed some useful links to help with these requirements:

Best practices for Device detection with Media Queries

Adding Mobile Views to your Grails Applications with JQuery Mobile: A Real Life Example

Twitter bootstrap

Delivery of the videos

The video resources should be delivered in an adaptive way, i.e. devices with low resolution and bandwidth need to download smaller file sizes.

The iPad and iPhone media player does not allow for integrated note taking or annotations so it is proposed that there will be an HTML player version of Synote.

As is mentioned in a blog that discusses the insertion of a video or audio playing link for these devices – the player will take over the fullscreen when playing,so it will not be possible to have a split screen version of the transcriptions or the annotations. Therefore it is proposed that there will be a thumbnail picture displayed alongside the video with the annotations. It is felt that there is a requirement for the extraction of thumbnail pictures when creating annotations.

Taken from Ploem.be blog as an example of an audio file playing.

HTML5 Players

Considerations around the HTML5 player used have to include the best accessibility options, but with browser differences this is not easy to achieve. HTML video.org offer a helpful comparison of players.

Format Compatibility

Different devices cause format compatibility complications and Safari, Google Chrome and mobile IE also have different support mechanisms for HTML video codec. This means the delivery of multimedia resources has to be adaptable.

Touch interface

There needs to be careful consideration given to the type of gesture driven / tap type controls. Users who wish to have VoiceOver controls will affect the gestures that can be used within not only the player but also the webpages. It has already been stressed that there is a need to keep clutter to a minimum and therefore there must not be too many buttons in the screen. It was confirmed that the design would aim to abide by the accessible and ease of use design concepts mentioned in the blog “Research – Planning for an easy to use and accessible mobile app.”

Despite the issues and concerns raised it is intended that the specifications being developed will have the bonus of offering

Web browser access – No app store download needed
Automatic enhancements – No need to connect to an app store for updates
Speed – adaptive performance on most connections
Automatic adjustment of presentation – depending on device screen size
Use of existing accounts.

Nutshell – Synote Mobile in 140 characters

Synote Mobile provides users with the ability to access, search, manage and exploit video and audio OER whilst on the move.

Synote Mobile meets the need to make web-based video and audio Open Educational Resources easier to access, search, manage, and exploit. The recordings can be viewed and repurposed with annotations and tags. Many UK students carry mobile devices capable of replaying video, but lack the interactive apps that encourage learning, Synote Mobile provides a collaborative experience with additional personalisation.

Research – Streaming videos onto mobile phones

Since the OER meeting we have been exploring the issues around streaming video on mobile phones and tablets. It appears the latter is easier than the former! We have also been exploring all the web pages of those we met up with at the meeting to see how we can collaborate to stream data from these sites into Synote Mobile but more on that later!

The state of HTML 5 and video is well explained by LongTail Video and it looks as if this is the way to go when one wants to deliver a cross platform service. However, there are issues around fullscreen viewing with iPhones as the video is not presented within the webpage so is no longer browser based as can be seen from the BBC news screen grabs taken on an iPhone 4S.

This fullscreen issue makes it impossible to add captions unless they are included when the video is made. Closed captioning is explained by Longtail video.com with the support provided by JWPlayer and it is clear that external files will be unable to be read with the method presently used by iPhones for rendering videos.

There are several pros and cons when researching the different HTML 5 players and the methods they use to support captioning. A recent blog on Salt Websites offers several options, but the author has come to the same conclusion that we have reached namely:

“Now after all this effort to get your videos working on iOS I have to give you the bad news – the iPhone does not show subtitles! They put the video into full-screen mode and ignore the subtitles. ”

The gauntlet has been thrown down …

Research – Planning for an easy to use and accessible mobile app.

Link to Apple accessibility web pages

Mobile apps have huge potential to help and liberate people, including disabled people and the elderly, who face challenges with other methods of communication. But as with other new technologies, there is also the potential to further exclude people who are already at a disadvantage by providing small, hard-to-use, inflexible interfaces to devices and apps that create more problems than they solve. (One Voice – Moving Together)

One of the main problems this project will need to overcome when considering ease of use and accessibility is the multitude of portable devices and operating systems. The use of a common code such as HTML 5 may overcome some of the difficulties rather than choosing to program a device dependent native app.

Jakob Nielsen’s Alertbox, September 26, 2011 provides some research into usability with an update on the subject pointing out the need to be aware of ‘fat finger syndrome’ and limit the number of features available.

When it comes to usability testing, User Centric provide some key pointers that link to co-design and testing prototypes to ensure ease of use. There are answers to the following questions:

Does a fully functional prototype have to be built before user testing?
Should the user’s device be tested or is it better to provide a device?
What devices and user groups should be tested?
Are there differences between iPhone and Android users?
Do both need to be included in a study?
When is lab testing versus remote testing appropriate?

W3C have produced some Mobile Web Best Practices (MWBP) guidelines for developers that are relevant to HTML 5. There are also the British Standard 8878 guidelines as presented by Jonathan Hassell – BS 8878 in 88 seconds – a lightning summary of the Standard (video with captions and transcript) provides a gallop through of the process!

Perhaps the easiest check list comes from the One Voice for Accessible ICT Coalition

The suggested seven steps are:

Learn about accessibility.
Learn how a user with a disability may use your app.
Quick accessibility check.
Get an estimate of how accessible you app is now.
Publish an Accessibility Statement.
Express your intent to be accessible.
Provide a Contact Us function.
Enable users to tell you easily about accessibility issues.
Ensure reading sequence is logical and comprehensible.
Ensure page navigation is simple.
Create a user interface that is easy to understand and operate.
General usability is an underpinning of accessibility.
Ensure text formatting can be altered.
Allow users to read text using a size and theme that meets their requirements.

Link to IDEAL Group's Android Accessibility Project

There is no such thing as full accessibility for everyone, but that should not stop app developers from attempting to maximise accessibility. (One Voice: Moving Together)

Project Plan – Minutes for 1st meeting of the Synote mobile team

Minutes

Attending – Mike Wald (MW), Gary Wills (GW) Yunjia Li (YL), E.A. Draffan (EA)

• Design and development of the Synote web app suitable for mobile technologies

There was the initial discussion that led to the following points being made:

Not initially designed to work on mobiles – 4 years ago people were not playing videos on mobiles or tablets.
Browser issues on mobile – media players at present not playing synchronised audio and videos with notes
Devices will use different resolutions this will affect the ability to port the videos

Action
Initial rapid prototyping – download media on a mobile with a suitable Synote mobile interface – New UI so that you can see what Synote will look like on a mobile. (YL)

• Conversions of permitted recordings to suit the mobile platform

Find video – we do not store – it would be possible to set up a server and cache the videos
If the file is owned by the user it can be added to YouTube and then streamed onto the device within Synote mobile.
Adaptive video streaming – single file to many different devices – have to transcode it into several files.

Research
Different formats, adaptive streaming video, investigate YouTube or static file download cached. (EA, MW and YL)

Issues related to media types, transcriptions and annotations.

HTML 5 video standards have yet to be agreed – Google have developed webM as a standard – used in Chrome and supported by Mini Opera – there has also been advice on mobile Safari for developers of Iphone and iPad web apps with work also being done for mobile Internet Explorer.

What is WebM?

YouTube video – “John Luther, WebM Product Manager, talks about the open-source WebM video format, the problems that it solves, and what WebM means for the open web. Learn more at http://www.webmproject.org/“

Device
Testing on different portable devices – Ipad3, iPhone, Android – delay on Windows tablet. (EA, MW, GW and YL – plus user stakeholders)

Work Plan

April: Website Template, Blog, Synote Mobile Requirements and Specifications & research into recording/replay formats
May: Rapid Prototyping Phase 1 producing prototype demonstrator on tablets and mobiles to replay recordings from file or streamed from Youtube
June: User testing and evaluation by Southampton and external collegaues to inform Co-Design & rapid prototyping phase 2

• OER 3 Programme Meeting being held on 26th March 2012 at Dexter House, London (MW and EA)

• AOB – Discussion around take outs from long videos to produce clips that could be annotated etc.
• Next meeting date. Skype meeting after Easter

1st Team Meeting – 22nd March 9-10am, 4th floor demo room/FPAS board room 32/4073

This will be the first meeting of the team to discuss:

Design and development of the Synote web app suitable for mobile technologies
Conversions of permitted recordings to suit the mobile platform
Issues related to media types, transcriptions and annotations.
Preparation for OER 3 Programme Meeting being held on 26th March 2012 at Dexter House, London
AOB
Next meeting date.