From MAUS SSI Component Design - Online Reconstruction:

• Store plots output by reducers "somewhere" so they can be rendered.
• Plots need to be rendered in a web interface so they're visible in the control room and externally.
• If a reducer fails then this should be visualised too.
• Data transfer to browser via HTTP POST or whatever is appropriate.
• CT can provide access to web server.
• Use an automatically-updating web interface constructed in Django Python web framework, https://www.djangoproject.com/.
• Useful for user to be able to tweak time intervals, axes ranges etc.
• David Colling may be able to provide guidance as he's done similar things before.

Depends on #703

See #703 for simple image presentation via Django.

AJAX allows asynchronous, responsive GUI to run in browser in JavaScript. AJAX/Django links of possible use:

• http://www.kelvinism.com/articles/simple-ajax-django/ "Added XML and JSON serialization...users can choose whatever AJAX library we want"
• http://dev.lethain.com/intro-to-unintrusive-javascript-with-django/
• http://jqueryjs.googlecode.com/files/jquery-1.2.6.min.js
• http://www.b-list.org/weblog/2006/jul/31/django-tips-simple-ajax-example-part-1/

See #809 for additional requirements.

• Change env.sh so it also sets MAUS_WEB_MEDIA_RAW directory path.

• Bigger images on main HTML page and display more on the horizontal.
  • Resolution of onlinerec01 (two monitors) = 3840 width, 1080 height. Use 1920x1080 for important plots => 640x540 plots for a 3 plot wide, 2 plot height display.
• Check file updated time/checksum and display red border round images not updated "for a while".
• Support multiple pages and cycle through.
• Review MAUSThirdPartyOnlineMonitoring for other options.
• Consult with Linda/online group as they have final design say.
  Image handler:
• Non-EPS files in media/raw directory shouldn't display.
• Replace "convert" system call with:
  

  easy_install magicwand
  
  from magickwand.image import Image
  i = Image('foo.jpg')
  i.format = 'PNG'
  i.flip()
  i.save('flip.png')
  

• Add OutputPyMongoDB which saves JSON docs to MongoDB.
• Change web server to read these docs and render the embedded images (rather than the intermediate save/load file step)

• Add support for reading and rendering files with tabular data.

• AJAX allows asynchronous, responsive GUI to run in browser in JavaScript. AJAX/Django links of possible use:
• http://www.kelvinism.com/articles/simple-ajax-django/ "Added XML and JSON serialization...users can choose whatever AJAX library we want"
• http://dev.lethain.com/intro-to-unintrusive-javascript-with-django/
• http://jqueryjs.googlecode.com/files/jquery-1.2.6.min.js
• http://www.b-list.org/weblog/2006/jul/31/django-tips-simple-ajax-example-part-1/

Deployment:

• Try deployment in Apache and mod_wsgi web server (rather than Django's own simple one)

Focus on basic extensible solution.

• Keywords associated with each plot.
• Optional sentence explaining it.
• Django search-by-keyword option.
• Page per keyword e.g. TOF, KL.
• Well-documented API.

When understand more about how we want to display plots (ie. during step 4, so in a year), expand the web app.

Steps used to get Django maus-apps working under Apache/mod_wsgi, using,

• https://docs.djangoproject.com/en/1.3/howto/deployment/modwsgi/
• http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango

$ xemacs -nw /usr/local/apache2/conf/httpd.conf

WSGIScriptAlias / /home/michaelj/maus-apps/apache/django.wsgi

• / root URL
• /... location of WSGI file - usually inside project.

Create Django WSGI configuration file

$ mkdir /home/michaelj/maus-apps/apache
$ chown -R michaelj:epcc /home/michaelj/maus-apps/apache/
$ xemacs /home/michaelj/maus-apps/apache/django.wsgi

import os
import sys
path = '/home/michaelj/maus-apps/src'
if path not in sys.path:
    sys.path.append(path)

os.environ['DJANGO_SETTINGS_MODULE'] = 'maus-web.settings'
os.environ['MAUS_WEB_MEDIA'] = '/home/michaelj/maus-apps/media'
os.environ['MAUS_WEB_DB'] = '/home/michaelj/maus-apps/db'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

The directory added to sys.path is the parent of the directory containing the package for the Django site created and which contains settings.py and manage.py.

Update Apache 2.2 configuration,

$ xemacs -nw /usr/local/apache2/conf/httpd.conf

# Location of django.wsgi
<Directory /home/michaelj/maus-apps/apache>
Order deny,allow 
Allow from all 
</Directory>

[Thu Feb 09 10:58:02 2012] [error] [client 129.215.63.2] 
client denied by server configuration: /home/michaelj/maus-apps/apache/django.wsgi

path = '/home/michaelj/maus-apps/src/mausweb'
if path not in sys.path:
    sys.path.append(path)

maus-web/settings.py
 maus -> maus-web.maus
maus-web/urls.py
  maus.urls -> maus-web.urls
maus-web/maus/urls.py
 maus.views -> maus-web.maus.views
maus-web/maus/views.py
 settings -> maus-web.settings

Now can see MAUS web page using web browser pointed at Apache URL or using Django development server.

Trying to view images give errors e.g.

convert: unable to open image 
`/home/michaelj/maus-apps/media/thumbs/sample-eps-thumb.jpg': 
Permission denied @ error/blob.c/OpenBlob/2589.
[Thu Feb 09 04:56:12 2012] [error] [client 129.215.63.2] 
mod_wsgi (pid=31957): Exception occurred processing WSGI script 
'/home/michaelj/maus-apps/apache/django.wsgi'., 
referer: http://maus.epcc.ed.ac.uk:9090/maus/
[Thu Feb 09 04:56:12 2012] [error] [client 129.215.63.2] 
IOError: failed to write data, 
referer: http://maus.epcc.ed.ac.uk:9090/maus/

Visit EPS thumbnail directly, http://localhost:9090/maus/sample-eps.eps/thumb shows page with

IOError at /maus/sample-eps.eps/thumb
[Errno 2] No such file or directory: 
u'/home/michaelj/maus-apps/media/thumbs/sample-eps-thumb.jpg'Request Method: GET 
Request URL: http://maus.epcc.ed.ac.uk:9090/maus/sample-eps.eps/thumb 
Django Version: 1.3.1 
Exception Type: IOError 
Exception Value: [Errno 2] No such file or directory: 
u'/home/michaelj/maus-apps/media/thumbs/sample-eps-thumb.jpg' 

For PNG thumbnail, http://localhost:9090/maus/sample-png.png/thumb error on page is,

IOError at /maus/sample-png.png/thumb
[Errno 13] Permission denied: 
u'/home/michaelj/maus-apps/media/thumbs/sample-png-thumb.jpg'Request Method: GET 
Request URL: http://maus.epcc.ed.ac.uk:9090/maus/sample-png.png/thumb 
Django Version: 1.3.1 
Exception Type: IOError 
Exception Value: [Errno 13] Permission denied:
 u'/home/michaelj/maus-apps/media/thumbs/sample-png-thumb.jpg' 

Google, http://stackoverflow.com/questions/5543641/django-media-upload-errno-13-permission-denied, ah a permissions error. Ass-umed Apache runs as root. What does it run under?

$ ps -aux
...apache...

$ chmod -R go+rwx /home/michaelj/maus-apps/media/

Commited http://bazaar.launchpad.net/~michaelj-h/maus-apps/devel/revision/14, with changes needed to run under Apache 2.2. Need to check this out fresh and validate instructions added to MAUSDjangoApache

To "17": http://bazaar.launchpad.net/~michaelj-h/maus-apps/devel/revision/17 - updated to remove hard-coded paths where possible.

envvars.sh

#
# This file is generated from envvars-std.in
#
LD_LIBRARY_PATH="/usr/local/apache2/lib:$LD_LIBRARY_PATH" 
LD_LIBRARY_PATH="/home/mice/MAUS/maus_release/third_party/install/lib/:$LD_LIBRARY_PATH" 

# Ideally don't need these but they address the 
#
# Could not find platform independent libraries <prefix>
# Could not find platform dependent libraries <exec_prefix>
# Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
#
# errors.

export PYTHONHOME="/home/mice/MAUS/maus_release/third_party/install/" 
export PATH="/home/mice/MAUS/maus_release/third_party/install/bin/:$PATH" 

export LD_LIBRARY_PATH

# Should not need these.
# LD_RUN_PATH="/home/mice/MAUS/maus_release/third_party/install/lib/:$LD_LIBRARY_PATH" 
# export LD_RUN_PATH

# This is needed later.
# source /home/mice/MAUS/maus_release/env.sh

httpd.conf extract

# Ideally dont need this next line.
# WSGIPythonHome /home/mice/MAUS/maus_release/third_party/install/bin/python2.7
# Alternative fix for
#
# Could not find platform independent libraries <prefix>
# Could not find platform dependent libraries <exec_prefix>
# Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
#
# errors.

LoadModule wsgi_module modules/mod_wsgi.so

WSGIScriptAlias /myapp /usr/local/apache2/zonk/myapp.wsgi

# Location of django.wsgi 
<Directory /usr/local/apache2/zonk> 
Order deny,allow  
Allow from all  
</Directory> 

cd /usr/local/apache2
bin/apachectl stop
rm logs/error_log 
bin/apachectl start
more logs/error_log 

su
cd
cd apache-django/mod-wsgi3.3
make distclean
./configure --with-apxs=/usr/local/apache2/bin/apxs --with-python=/home/mice/MAUS/maus_release/third_party/install/bin/python2.7
LD_RUN_PATH=/home/mice/MAUS/maus_release/third_party/install/lib make
make install

Outstanding problem in deploying in control room is

ImportError: No module named site

18

• configure ensures env.sh adds maus-apps/src to PYTHONPATH and LD_LIBRARY_PATH.
• Non-EPS files in media/raw directory shouldn't display - implemented as non-image files won't display.
• Added comments to all files and Pylinted them.

• install.sh easy_installs magickwand
• Created new image_utils.py module for useful functions and pulled out commonality from views.py.

Image formats supported:

• Matplotlib and PyROOT support: ps eps png pdf svg
• PyROOT also supports: gif jpg jpeg
• Matplotlib also supports: rgba raw svgz emf (only with pyemf)

MagickWand convert cannot do:

• raw - Conversion problem...no decode delegate for this image format `matplotlib-raw.raw' @ error/constitute.c/ReadImage/532
• rgba - Conversion problem...must specify image size `matplotlib-rgba.rgba' @ error/rgb.c/ReadRGBImage/143
• svg, svgz look bad. May need convert recompiled e.g. see http://www.imagemagick.org/Usage/draw/#svg

For now went with the formats it does support: ps, eps, png, pdf, gif, jpg, jpeg.

Commits up to: 21

• ConfigurationDefaults sets image_directory to MAUS_WEB_MEDIA_RAW if this is set.
• maus-apps configure script sets up env.sh to set MAUS_WEB_MEDIA_RAW.

Commits up to: 26

• Removed need for install.sh by rewriting configure.
• configure generates two types of env.sh depending on whether MAUS Python is being used or not.
• Replaced print with logging.

Using MagickWand for thumbnails,

from magickwand.image import Image as MagickWandImage
ipng = MagickWandImage("imagetof_xy_2000161.png")
ieps = MagickWandImage("matplotlib-eps.eps")
ipng.resize((100,100))
ieps.resize((100,100))
ipng.format="JPG" 
ieps.format="JPG" 
ipng.save("ipng.jpg")
ieps.save("ieps.jpg")

ipng2 = MagickWandImage("imagetof_hit_x000008.png")
ipng2.format="JPG" 
ipng2.thumbnail(100)
ipng2.save("ipng2.jpg")

Attempt at using image.dump(...) to try and return image data (after http://benjamin-schweizer.de/python-magickwand-or-how-to-work-with-icons.html) in Django HttpResponse rather than having to save then reload thumbnail yields

please register the function size_t

A scalability comment, for 1300 images the web front-end can generate thumbs BUT ...can't show page as it takes longer than 5s to generate them all! Even if showing full size, it takes about 5s to return all to client before page refreshes. Refresh rate in the web pages may need tweaked depending on time taken to generate the thumbs.

An HTML form handling example is at http://www.djangobook.com/en/1.0/chapter07

In terms of image browsing and refreshing, how do others do it, MAUSThirdPartyOnlineMonitoring:

• Vistar (http://op-webtools.web.cern.ch/op-webtools/vistar/vistars.php?usr=SPS1) uses a jQuery JavaScript drop down menu to allow a user to select an image. Only one image can be viewed at a time. jQuery handles the refresh to avoid reloading the whole HTML page,
• MINOS (http://farweb.minos-soudan.org/events/LiveEvent.html) also visualises one image on their page and uses JavaScript to handle the refresh.
• APS
  • http://www.aps.anl.gov/Facility/Status/ renders thumbnails. Clicking shows a full PNG. Data is updated every 5 minutes it states.
  • http://www.aps.anl.gov/asd/operations/gifplots/MCRstatus.html. Renders an image. HTML refresh every 60s.
  • http://www.aps.anl.gov/aod/blops/status/srStatus.html - Image and meta-data - HTML refresh every 60s

Commit 30

• Supports simple search button based on files matching a keyword
• Removed GENERATOR tag. Added JavaScript Pause, Refresh, Resume