Introduction
This document is an evaluation of four different software packages that have been proposed as the foundation for the REALISE website. Two, Drupal and ATutor, are content management systems (CMS), and two, myExperiment and CloudWorks, are communities based on open-source software that would be available for us to re-use to create our own community.
Drupal
Drupal is an open-source content management system that has been in development for nearly a decade. It runs on PHP and has a large development community who have produced a range of third-party add-on modules.
Drupal appears to score well in accessibility based on the Web2Access test criteria, with minor issues noted in only four of the fifteen test categories (see http://www.web2access.org.uk/product/233/). It is also extensible, with a wide range of third-party modules available enabling it to provide a broad range of features (including all of the requirements of the REALISE project).
Installation is simple for a UNIX administrator – one simply downloads and extracts the files, sets appropriate permissions, creates an empty database and runs Drupal’s self-installation script. It is unlikely that anyone without technical experience would be able to install this system.
A fresh installation of Drupal uses only the most basic of modules required to run the system. Many other “official” modules, including forum and blog support, are inactive by default. However, one of the main drawbacks of Drupal is its immensity; the admin menu system is labyrinthine, and it can take a long time to locate any given option or feature. The system also looks and feels bloated. Having said this, it took only a couple of hours to have a dummy REALISE site (still available at www.realisepotential.org) up and running.
As detailed here, Drupal is known for using large amounts of memory. Fatal errors related to this were encountered whilst creating even this simple dummy site. It also has a reputation for security flaws and poorly written code, which could cause us expensive difficulties further down the road.
Overall, it may be simpler and more effective to create our own content management system using a framework such as Ruby on Rails or Python Django rather than to use a ready-made but buggy solution like Drupal.
ATutor
ATutor is another open-source web-based content management system. It describes itself as a “Learning Content Management System”, and is designed for use in an education setting. Content is built up into units called “Courses”, which are administered by “Instructors” and used by “Students”.
An immediate difficulty which arises with ATutor is the difficulty of mapping the requirements of the REALISE project (e.g. accessibility products) to ATutor modules (i.e. courses). ATutor is designed for a very specific purpose, and would need modification to be suitable for REALISE.
ATutor, despite having been designed with accessibility in mind, does not score as well for accessibility under the Web2Access test criteria as Drupal (see http://www.web2access.org.uk/product/235/). It performs solidly nevertheless, with notable deficiencies in only two areas; its rich text editor is completely inaccessible without a mouse, and the page layout breaks when using the zoom feature on any major web browser. This makes it unlikely to be suitable for visually impaired users.
As with Drupal, ATutor installation was simple enough for a UNIX administrator, but would be difficult if not impossible for an average user.
Being designed for a specific purpose rather than as a generic solution, ATutor is noticeably smaller and easier to navigate than Drupal. The admin menu systems could still be very confusing for a novice user, however. ATutor is also much less widely used than Drupal, and thus has far less third-party support and extensions. A dummy site was not created in ATutor due to the mapping difficulties mentioned above.
Overall, ATutor has few advantages over Drupal and does not appear to be suitable for use in the REALISE project.
myExperiment
myExperiment is a collaborative working environment designed to enable scientists to share their workflows and experiment plans. It was built using the open source Ruby on Rails, and ECS was involved in its development. Once logged in, a user can upload workflows and other files, and create “packs” that tie them together.
As with ATutor, this presents a mapping problem. In this case, the problem is almost certainly insurmountable, as the site does not provide all of the features necessary for the REALISE project; chiefly, it lacks a suitable method for submitting a product idea for review by the community.
Installation of myExperiment is a lengthy process, requiring existing knowledge of Ruby on Rails. The software has a number of dependencies on other packages, and would not be possible for someone without considerable knowledge of UNIX administration.
The site is seems quite easy to use from an end-user perspective, particularly if the user has a scientific background (in line with the site’s target demographic), but conversely, the average person would probably not understand what the term “workflow” means, let alone how to create one.
In terms of accessibility, myExperiment performs fairly well under the Web2Access tests (see http://www.web2access.org.uk/product/234/). The rich text editor is inaccessible with a keyboard, as is often the case, and the large amount of content displayed on each page makes navigation with the keyboard cumbersome, which is particularly evident when the stylesheet is removed. The tag clouds in particular take a long time to tab through.
In conclusion, myExperiment does not seem appropriate for use in the REALISE project.
CloudWorks
CloudWorks is an online community for sharing ideas about learning and teaching. Content is posted in the form of “Clouds”, and can consist of anything that can be marked up in HTML or that can be embedded from another site such as YouTube. Other users can comment on each Cloud, and can add links to other clouds and web pages as well as references to academic papers. Clouds with common themes can be grouped together into sets called “Cloudscapes”. There is a potential mapping between Clouds in CloudWorks and product ideas in REALISE. Other requirements of the REALISE process are also present, including tagging, text search, and discussion (though this is not threaded and cannot be sorted).
At present, ease of installation is unknown. This is because the source code is not yet available publicly. ECS will be able to obtain this in a couple of weeks, but this would slow the project down considerably. We will ideally need to set up our own instance of CloudWorks in order to apply REALISE branding to the project,, rather than simply creating our own Cloudscape on the existing site.
CloudWorks is easy to use on the whole, and the average user would likely be able to grasp how to use it in a fairly short space of time. The interface is very clean, with little clutter on the screen. Online help (including an optional introductory video) and e-mail support is available.
Accessibility according to the Web2Access guidelines is very good, with the only notable issue being the lack of on-screen feedback after content is submitted. It has performed better under these tests then the other three products that we are evaluating (see http://www.web2access.org.uk/product/236/).
In essence, CloudWorks may be the most appropriate solution out of the four services evaluated, but it will be difficult to know for sure until we can acquire the source code.
The LinkedIn API is being considered for use in the REALISE project. This would enable users to log in to the REALISE site with their LinkedIn credentials, and would allow the REALISE site to make use of LinkedIn users’ profiles and networks.
The accessibility of LinkedIn.com is important if we are to require users of REALISE to have a LinkedIn account. LinkedIn was checked against the Web2Access accessibility criteria in August 2009. The results obtained then still apply, though it has been noted that the sign-in form for the site is not located on the home page and must be navigated to, which is unusual and potentially confusing for some users. The only other problems with the site, as noted in the original review, are that it makes use of font sizes that are too small to read comfortably, that form feedback is sometimes incompatible with some screen readers, and that some non-critical page elements can only be used with a mouse.
The LinkedIn API could be a useful addition to the REALISE site as it would enable us to draw on LinkedIn’s social networking features and may encourage LinkedIn users to make accessibility projects a part of their LinkedIn portfolio. However, there is a potential caveat – some users, particularly those of a more casual nature, may be put off by having to make a LinkedIn account in order to use the REALISE site. It may be possible to work around this by making it optional to link one’s REALISE profile to one’s LinkedIn account.
Did you try the rich text editor with a screen reader? It works quite well.
http://wiki.moxiecode.com/index.php/TinyMCE:Accessibility
I’d be interested in the Web2Access results.
Drupal and ATutor are two different classes of software.
Thank you for your comment and the Web2Access evaluation can be found at http://www.web2access.org.uk/product/235 – we were horribly stringent and used the basic version which does not seem to allow access in Chrome and Safari. I have updated the evaluation and I know how hard it is to get these editors to work in all browsers.
I totally agree Drupal and Atutor are two very different applications as are Cloudworks and Myexperiment which is why we are evaluating them all for potential use for this project.
Thanks for this detailed analysis. I have a few comments.
Drupal
It’s interesting and disappointing to hear you experience with Drupal. Many people are using it, for example the Transfer Summit so perhaps you where unlucky? I had expected it to be plain sailing and not so bloating given the default install only has core modules. My impression and backed by Ross is it will certainly allow almost anything we could want in the early mock ups. I’d want to see some concrete evidence of security problems, and they are then an issue if not dealt with quickly (cif Windows). I can’t comment on the code quality either but can only point to the large number of sites based on it and the mature dev/contrib community. The admin complexity is new to me, I understand there is a learning curve but took that to be in the areas of the architecture and abstractions. As you said you got something going despite your problems.
I find it hard to agree a home grown system might be best in the long term. As I think you are saying (no final conclusion) Certainly for the long term, for example we would have to maintain everything ourselves (or at least pay for it). using an established system will move the maintenance burden to the community and that appears to be a strength of Drupal. Perhaps for the initial mock-up and given the time scales but I would want to revisit this choice next iteration.
Atutor + My Experiment
I’ve nothing to add but results are much as expected given their specialisation.
Cloudworks
I have offered to do a trial install and will keep you posted on this. The framework it uses seems OK but we will have to see. The team are responsive so far but small. But still is better that in house system from sustainability/community perspectives. My overall view it has basic functionality that can be found in other systems and there is probably a lot we would need to add, but to mutal benefit. I think it was inappropriately used at the recent JISC event but how much of that was due to a poor feature match and how much due to inherent problems is unclear.
Linked in
I agree with your concerns and also the fact that Linked in is inhabited by a large community which we want (need) to tap into – namely business users. Fro their point of view REALISE could be seen as extending linked in activities into innovations and buisiness ideas. We identified the need to hook into existing socnets and communities so perhaps this falls into that group as you you indicate? My current feeling is that is the correct approach given LinkedIn focus as a meeting place. It’s also not at all clear that storing our ideas/projects directly in Linked in would be a good fit. Finally using the api keeps us independent from LinkedIn. Perhaps in future we could get them to use a REALISE widget on LinkedIn?
So my conclusion are much as yours. Linked in should be part of the solution and we an mock up somehting to explore. However I would want to be convinced that Drupal or Cloudscape are the better long term solution compared to a home brew. We must remember we have little time and want to spend as little as possible on developing the software. Having something that exists and others working on it is a positive if the extant features are a close enough fit with reasonable dev time and the team are responsive/available.
I’m a bit of an outsider to this project as i’m working on other things in the LSL Lab as one of the Access Technologies Team researchers – but I thought i’d share my views on this.
With regards to Drupal, as far as I can tell it’s a Content Management System, not a web application framework. The only places i’ve seen Drupal used are websites, and its used so many users can edit the content easily rather than anything else. If there are any web applications built on Drupal it’d be interesting to see them to see how they stack up compared to traditional systems.
I don’t know how Drupal have done it, but its a -massive- resource hog. Impressively so, for all it does. There are also issues with upgrading – newer versions may break the custom modules which would mean even more work to keep the system current.
Web Apps behave very differently than websites, and Drupal doesn’t really seem to offer anything for this type of system. As far as I can tell REALISE is an Information System, so you’d be building 99% of the functionality from scratch anyway – so why build it on an already seemingly bloated base.
I’d be one to suggest Zend Framework, Django, Rails etc. These are web application frameworks which have one purpose in life – making projects like this work.
@snowman – thanks for comments – keyboard access is a problem with TinyMCE, but we can home it will improve. Yes ATutor as a VLE is not such an obvious fit.
@Sebastian Skuse Thanks for valuable comments. I agree given a free reign I’d choose something like Django, perhaps with Pinax. However we have little resource for web dev so it is possible using a framework is best route. It also appears we need a very simple web app – db with forms create/edit (perhaps full CRUD) plus social features and Drupal has many modules in the category (eg CKK and Organic) . This leads me to think it is good enough for the type of prototype we are producing. I agree creating new custom modules or embedding widgets is likely to be expensive and I would not want to paint ourselves into a corner. A next action would be to ask an experience Drupal dev.
@Steve Lee – strange, keyboard access seems fine in all our tests. One thing you may not have noticed are the accesskeys to jump between the toolbar, editor area and the status bar. When accessing the toolbar with a screen reader, these shortcuts are announced just before tabbing to the first button.
In any case, if you are looking for an accessible WYSIWYG editor for Drupal, TinyMCE is the most accessible of those out there at the moment.
Make sure your browser ketboard access is turned on:
http://www.456bereastreet.com/archive/200906/enabling_keyboard_navigation_in_mac_os_x_web_browsers/
@snowman – thanks for the tips
Thank you so much from me as well, but the problem is that not everyone is using a screen reader and those with dexterity difficulties may not be aware of the announcements, so we somehow have to make these access keys more obvious and ensure they do not clash with a user’s own specialist technology or assistive technology. Perhaps this is something we can achieve with html links as we did in the LexDis project.
With respect to Drupal I don’t think the evaluation here is entirely fair. I asked our Drupal developer to comment as he makes a living out of Drupal (and is equally knowledgable and biased by that fact). I copy his comments below.
From the point of view of security, it is reasonably true that there are frequent security releases from the Drupal community, is that a bad thing? We don’t know for sure. My own experience is that it’s not bad coding that is the problem – it’s lots of users and a responsive community (“many eyes”).
Code quality is variable. Generally core is pretty good, although the current 6.x development is old and as a result stagnating. The upcoming 7.x release will clean up many of the poor decisions made in 6.x (as 6.x did with 5.x etc.)
In terms of resource management I don’t think that it is normal for people to run into resource problems whilst testing it. That is almost certainly a local issue. By default Drupal is resource intensive since all cache features are turned off and all logging is turned on. However, when properly configured for a production environment it runs pretty well (but it is a PHP app). It’s not like it isn’t used on some very high traffic sites.
Seb comments that it is not a web application framework. That’s true, it is not, but then none of the tools evaluated here are webapp frameworks. If you want a webapp framework you should be evaluating them. Drupal could be described as a content application framework and as such is a good candidate for any project that needs to work with content about, lets say people and project (like yours?).
The admin section of Drupal is indeed a labrynth because everything is configurable to enable you to build the content app you want. Why is this a problem? If you can code your own I’m sure you can manage a Drupal install.
As with the performance issues you are showing your inexperience with Drupal here. The admin user sees everything because they are the “superuser”, someone has to have access to everything. If you want other people to go into the admin interface but only see a small number of essential features then create a role with the appropriate permissions, i.e. check a few tick boxes, and your done – all non-essential items are hidden. You could, if you wanted to adopt a simplified theme for admin too.
Seb says he is only aware of websites being run with it, again this appears to be an assumption based on a cursory evaluation. For example, I’ve built the majority of http://www.transfersummit.com for Ross. This is much more than a website despite what you may see. On the back-end is a full blown conference management application with news aggregation and internal worklflow tools for the team. It just happens to output a website for conference attendees. There are also plenty of advanced drupal based sites, e.g. news agregation, social ranking, social networks and ecommerce too.
It seems strange to evaluate tools that you intend to use for short term rapid prototying and then say “nah – we’ll just build our own because none of these are web frameworks”. It seems strange to reject a solution becuase of (I would argue invalid) long term concerns when the intention is only to create a quick and dirty prototype to experiment with a concept.
I wonder how much of this is a “not invented here” problem?
If there are genuine problems with the other solutions you are looking at and you do want to look at a more web framework rather than content frameowrk kind of tool then perhaps Pinax is more up your street. It’s a Django app providing Social Network modules. Of course, you could just re-invent the wheel (again).
—–
Whilst John is perhaps a little unkind towards the end there I do think he has a point.
Do you really want to build something from scratch? I thought the project was about evaluating the processes involved in the innovation process. The technology should be secondary to that, shouldn’t it?
Ross/Dev
These are good points and the most important for me is that we DO want to spend our energies on the connections and process of open innovation and not the app. Surely then using something that may easily provide a usable solution, if not perfect, with the least dev and maintenance time, is a must?
From what I have seen of available options Drupal seems the best match. If we find we need a full web app platform then we can revisit later and we will will not have spent a lot of resource, assuming we don’t spend energy trying to bend Drupal in a way it doesn’t want to go.
So the question I see is: does Drupal provide the facilities we need for the system. The path of least resistance must be to get a prototype together and then see if it can adapt as we refine our ideas and requirements emerge for later phases. If it doesn’t we’ll have a better understanding of the usefulness of one of the leading content platforms. If it does we will have a low cost solution, potentially have contributed to a project and will not have to maintain everything ourselves. If we go the homebrew route the possible outcomes will be some fairly useless code + in house knowledge, or something we need to maintain or support as a project.
I just came across OpenHippel, a Drupal based project to that is creating “an online tool for people to form groups to work on ideas / projects together. The core is a criteria / milestone grid system, and then on the user level, users’ interests, history &
achievements are shown so that project starters can find & recruit
people w/ the right skills. ”
I’m not at all sure what stage they are at, I get the impression that this is ideas and possibly prototypes only. However, they are running a course as part of the P2PU (Peer to Peer University) which is part of the Mozilla Drumbeat project. If I understand correctly they intend to build a fair part of the system in the course. An interesting experiment in open innovation in itself. See http://p2pu.org/webcraft/drupal-social-web-application
I’m a Drupal developer who has been heavily involved in enhancing Drupal’s accessibility over the last two years. I’m not really familiar with this initiative, but from the mission, it’s really hard to see why a build your own solution is even being considered here. If your goal is:
“to identify routes to sustainable innovative accessibility solutions through engaging key researchers, businesses, charities, developers, institutions and and users in exploring open innovation in software (AKA open development or open source).”
A custom solution starting with Ruby makes no more sense than a custom built solution built with any other language.
@Ross’ points are right on. It’s not like any custom built code is ever going to have the level of code review or documentation or security evaluation that Drupal does. Heck, if it’s security is good enough for the White House, I can’t see how or why it’s being raised here. There is a good security process established within the Drupal community. And it is important to remember that all code has potential security issues.
Custom code is very difficult to maintain. Even if collaboration isn’t a core part of your mandate it seems like an awkward choice. Having thousands of developers already using Drupal means that much of the functionality you might want to use has already been developed.
However, since the core mandate seems to be to enhance sustainable accessibility solutions for researchers, businesses, charities, institutions, it seems like using Drupal is a terrific software community to start with. The other solutions aren’t in the same league and really haven’t proven that they have the critical mass required for sustainability. Drupal is already being used by all of your target institutions so any enhancements you propose can be more easily implemented by them.
The Drupal Accessibility Group is quite active and has been useful for continuing to improve the accessibility of this framework. In Drupal 7 there will be a great many more accessibility enhancements over what was tested in Drupal 6 with the web2access survey. I’m not sure what the timeframe is for this project, but it would be great if it could work on Drupal 7.
LinkedIn is just one of many social networking sites you may want to take advantage of. Having integration with Twitter & Facebook are also options and there are modules developed already to integrate with Drupal.
I do hope that this project is able to emulate the innovations in accessibility that you hope your community will be able to sustainably realize.
Mike thank you so much for taking time to comment, especially as you have little to go on and you are in the middle of a post vacation backlog.
The point that the Drupal community can provide further links is really compelling. I had not considered much more than being keen for the Drupal a11y teams to be one of the communities we engage with. Making the most of the wider Drupal community and their ‘customers’ is a powerful idea indeed.
On another note, part of the way OSS Watch evaluate the health of a project is to look at community activity and Drupal scores highly here. In fact I asked someone on twitter why they bother to tweet how happy they were with the Drupal Community. The answer was
That’s full marks for any project, and especially so for our requirements.
Thanks again
A big +1 to Mike Gifford’s comments.
As an assistive technology (screen-reader) user and the Drupal 7 Core accessibility maintainer (disclosure of bias) I would say that Drupal is a great solution for a content driven site / application. Since most of what I would say has been covered I will point you to this Drupalcon 2010 San Francisco session video on performance.
DrupalCon SF 2010: 2.4 million page views per day, 60 M per month, one server!
http://sf2010.drupal.org/conference/sessions/24-million-page-views-day-60-m-month-one-server In this talk, Khalid will talk about a how to scale a Drupal
web site with the following statistics: 61 million page views per month 14.24 million visits per months 132,650 visits per hour peak 539,000 visits on
peak day 2.4 million pages per day peak The web site runs on a single mid range server … We will discuss how we: How to tune the LAMP stack for optimal
performance How to keep things simple an…
Video: http://www.archive.org/search.php?query=DrupalCon%20SF%202010&page=1
@Everett Zufelt – thanks for leaving a comment and suggesting we chat on Skype. I’m really pleased that you are interested in being involved REALISE and look forward to engagement with the Drupal and Fluid communities
Here are a few more points Everett made in our chat
* Drupal *IS* fine for web apps, it not so good at web services (e.g RESTful), in 7 some useful features like CKK move to core
* accessibility is improving in all areas – user and admin. E.g using ARIA in admin to loose modals for good screen reader experience
* Good to wait for 7 if possible to save work as better a11y and some architectural changes, but can backport most features to 6. Mighrt me out in 1 or 2 months, but as with any OSS project will be when it is ready.
* http://www.drupalgardens.com/ is interesting. Is running 7 in the cloud and has some social features.
* modules we are likely to be interested in are CCK + views (just views in 7), Organic (groups), semantic views.
* The features module allows export and so sharing of drupal config – something we need as an Open Source project (and the TransferSummit project is also workign on this.
* We could potnetially create our own Drupal distro for others wanting similar site features to REALISE.
I was also very interested to hear that the community treat A11y bugs as release blockers. That shows a really enlightened view.