Project

General

Profile

Feature #706

Web-based interface for live execution monitoring

Added by Jackson, Mike about 10 years ago. Updated over 9 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Jackson, Mike
Category:
Online reconstruction
Target version:
Start date:
20 September 2011
Due date:
% Done:

100%

Estimated time:
Workflow:
New Issue

Description

Web interface for live presentation of histograms and status information on worker execution.


Files

maus-image-svg.txt (4.42 KB) maus-image-svg.txt Jackson, Mike, 13 December 2011 12:08
#1

Updated by Jackson, Mike about 10 years ago

From MAUS SSI Component Design - Online Reconstruction:

  • Store plots output by reducers "somewhere" so they can be rendered.
  • Plots need to be rendered in a web interface so they're visible in the control room and externally.
  • If a reducer fails then this should be visualised too.
  • Data transfer to browser via HTTP POST or whatever is appropriate.
  • CT can provide access to web server.
  • Use an automatically-updating web interface constructed in Django Python web framework, https://www.djangoproject.com/.
  • Useful for user to be able to tweak time intervals, axes ranges etc.
  • David Colling may be able to provide guidance as he's done similar things before.

Depends on #703

#2

Updated by Jackson, Mike almost 10 years ago

See #703 for simple image presentation via Django.

AJAX allows asynchronous, responsive GUI to run in browser in JavaScript. AJAX/Django links of possible use:

#3

Updated by Jackson, Mike almost 10 years ago

See #809 for additional requirements.

#4

Updated by Jackson, Mike almost 10 years ago

Attached is update to maus-apps with new views.py that uses OS "convert" call to convert from EPS or SVG to JPG to avoid issues in image degradation from using PIL.

Alternative solutions:

  • Run two workflows concurrently, one with the image reducer outputting EPS/SVG and one with it outputting JPG/PNG.
  • Update image reducer that outputs both the EPS/SVG and a JPG/PNG.
#5

Updated by Jackson, Mike almost 10 years ago

Script
  • Change env.sh so it also sets MAUS_WEB_MEDIA_RAW directory path.
Image display:
  • Bigger images on main HTML page and display more on the horizontal.
    • Resolution of onlinerec01 (two monitors) = 3840 width, 1080 height. Use 1920x1080 for important plots => 640x540 plots for a 3 plot wide, 2 plot height display.
  • Check file updated time/checksum and display red border round images not updated "for a while".
  • Support multiple pages and cycle through.
  • Review MAUSThirdPartyOnlineMonitoring for other options.
  • Consult with Linda/online group as they have final design say.
    Image handler:
  • Non-EPS files in media/raw directory shouldn't display.
  • Replace "convert" system call with:
    easy_install magicwand
    
    from magickwand.image import Image
    i = Image('foo.jpg')
    i.format = 'PNG'
    i.flip()
    i.save('flip.png')
    
Image delivery:
  • Add OutputPyMongoDB which saves JSON docs to MongoDB.
  • Change web server to read these docs and render the embedded images (rather than the intermediate save/load file step)
Tables:
  • Add support for reading and rendering files with tabular data.
Ajax:

Deployment:

  • Try deployment in Apache and mod_wsgi web server (rather than Django's own simple one)
#6

Updated by Jackson, Mike over 9 years ago

Focus on basic extensible solution.

  • Keywords associated with each plot.
  • Optional sentence explaining it.
  • Django search-by-keyword option.
  • Page per keyword e.g. TOF, KL.
  • Well-documented API.

When understand more about how we want to display plots (ie. during step 4, so in a year), expand the web app.

#7

Updated by Jackson, Mike over 9 years ago

Steps used to get Django maus-apps working under Apache/mod_wsgi, using,

Edit,
$ xemacs -nw /usr/local/apache2/conf/httpd.conf

Add,
WSGIScriptAlias / /home/michaelj/maus-apps/apache/django.wsgi

where:
  • / root URL
  • /... location of WSGI file - usually inside project.

Create Django WSGI configuration file

$ mkdir /home/michaelj/maus-apps/apache
$ chown -R michaelj:epcc /home/michaelj/maus-apps/apache/
$ xemacs /home/michaelj/maus-apps/apache/django.wsgi

Add content,
import os
import sys
path = '/home/michaelj/maus-apps/src'
if path not in sys.path:
    sys.path.append(path)

os.environ['DJANGO_SETTINGS_MODULE'] = 'maus-web.settings'
os.environ['MAUS_WEB_MEDIA'] = '/home/michaelj/maus-apps/media'
os.environ['MAUS_WEB_DB'] = '/home/michaelj/maus-apps/db'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

The directory added to sys.path is the parent of the directory containing the package for the Django site created and which contains settings.py and manage.py.

Update Apache 2.2 configuration,

$ xemacs -nw /usr/local/apache2/conf/httpd.conf

Add:
# Location of django.wsgi
<Directory /home/michaelj/maus-apps/apache>
Order deny,allow 
Allow from all 
</Directory>

If this is omitted then the logs will show,
[Thu Feb 09 10:58:02 2012] [error] [client 129.215.63.2] 
client denied by server configuration: /home/michaelj/maus-apps/apache/django.wsgi

Problems with importing the MAUS Django packages arose. Attempts to browse web pages give No module named NNNN errors. Ah, http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango says Django development server allows relative module imports. In use in Apache, either, Append to sys.path in django.wsgi e.g.
path = '/home/michaelj/maus-apps/src/mausweb'
if path not in sys.path:
    sys.path.append(path)

Or, use absolute module paths. Prefer absolute paths. So changed,
maus-web/settings.py
 maus -> maus-web.maus
maus-web/urls.py
  maus.urls -> maus-web.urls
maus-web/maus/urls.py
 maus.views -> maus-web.maus.views
maus-web/maus/views.py
 settings -> maus-web.settings

Then got, invalid syntax (views.py, line 10). The maus-web package name is now the issue. So change maus-web to mausweb across maus-apps project.

Now can see MAUS web page using web browser pointed at Apache URL or using Django development server.

Trying to view images give errors e.g.

convert: unable to open image 
`/home/michaelj/maus-apps/media/thumbs/sample-eps-thumb.jpg': 
Permission denied @ error/blob.c/OpenBlob/2589.
[Thu Feb 09 04:56:12 2012] [error] [client 129.215.63.2] 
mod_wsgi (pid=31957): Exception occurred processing WSGI script 
'/home/michaelj/maus-apps/apache/django.wsgi'., 
referer: http://maus.epcc.ed.ac.uk:9090/maus/
[Thu Feb 09 04:56:12 2012] [error] [client 129.215.63.2] 
IOError: failed to write data, 
referer: http://maus.epcc.ed.ac.uk:9090/maus/

Visit EPS thumbnail directly, http://localhost:9090/maus/sample-eps.eps/thumb shows page with

IOError at /maus/sample-eps.eps/thumb
[Errno 2] No such file or directory: 
u'/home/michaelj/maus-apps/media/thumbs/sample-eps-thumb.jpg'Request Method: GET 
Request URL: http://maus.epcc.ed.ac.uk:9090/maus/sample-eps.eps/thumb 
Django Version: 1.3.1 
Exception Type: IOError 
Exception Value: [Errno 2] No such file or directory: 
u'/home/michaelj/maus-apps/media/thumbs/sample-eps-thumb.jpg' 

For PNG thumbnail, http://localhost:9090/maus/sample-png.png/thumb error on page is,

IOError at /maus/sample-png.png/thumb
[Errno 13] Permission denied: 
u'/home/michaelj/maus-apps/media/thumbs/sample-png-thumb.jpg'Request Method: GET 
Request URL: http://maus.epcc.ed.ac.uk:9090/maus/sample-png.png/thumb 
Django Version: 1.3.1 
Exception Type: IOError 
Exception Value: [Errno 13] Permission denied:
 u'/home/michaelj/maus-apps/media/thumbs/sample-png-thumb.jpg' 

For PNG, problem traced to maus/views.py Image.save call.

Google, http://stackoverflow.com/questions/5543641/django-media-upload-errno-13-permission-denied, ah a permissions error. Ass-umed Apache runs as root. What does it run under?

$ ps -aux
...apache...

So, let's be open,
$ chmod -R go+rwx /home/michaelj/maus-apps/media/

#8

Updated by Jackson, Mike over 9 years ago

Commited http://bazaar.launchpad.net/~michaelj-h/maus-apps/devel/revision/14, with changes needed to run under Apache 2.2. Need to check this out fresh and validate instructions added to MAUSDjangoApache

#9

Updated by Jackson, Mike over 9 years ago

To "17": http://bazaar.launchpad.net/~michaelj-h/maus-apps/devel/revision/17 - updated to remove hard-coded paths where possible.

#10

Updated by Jackson, Mike over 9 years ago

  • Priority changed from Normal to Urgent
#11

Updated by Jackson, Mike over 9 years ago

envvars.sh

#
# This file is generated from envvars-std.in
#
LD_LIBRARY_PATH="/usr/local/apache2/lib:$LD_LIBRARY_PATH" 
LD_LIBRARY_PATH="/home/mice/MAUS/maus_release/third_party/install/lib/:$LD_LIBRARY_PATH" 

# Ideally don't need these but they address the 
#
# Could not find platform independent libraries <prefix>
# Could not find platform dependent libraries <exec_prefix>
# Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
#
# errors.

export PYTHONHOME="/home/mice/MAUS/maus_release/third_party/install/" 
export PATH="/home/mice/MAUS/maus_release/third_party/install/bin/:$PATH" 

export LD_LIBRARY_PATH

# Should not need these.
# LD_RUN_PATH="/home/mice/MAUS/maus_release/third_party/install/lib/:$LD_LIBRARY_PATH" 
# export LD_RUN_PATH

# This is needed later.
# source /home/mice/MAUS/maus_release/env.sh

httpd.conf extract

# Ideally dont need this next line.
# WSGIPythonHome /home/mice/MAUS/maus_release/third_party/install/bin/python2.7
# Alternative fix for
#
# Could not find platform independent libraries <prefix>
# Could not find platform dependent libraries <exec_prefix>
# Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
#
# errors.

LoadModule wsgi_module modules/mod_wsgi.so

WSGIScriptAlias /myapp /usr/local/apache2/zonk/myapp.wsgi

# Location of django.wsgi 
<Directory /usr/local/apache2/zonk> 
Order deny,allow  
Allow from all  
</Directory> 

Commands:
cd /usr/local/apache2
bin/apachectl stop
rm logs/error_log 
bin/apachectl start
more logs/error_log 

Build mod_wsgi
su
cd
cd apache-django/mod-wsgi3.3
make distclean
./configure --with-apxs=/usr/local/apache2/bin/apxs --with-python=/home/mice/MAUS/maus_release/third_party/install/bin/python2.7
LD_RUN_PATH=/home/mice/MAUS/maus_release/third_party/install/lib make
make install

Outstanding problem in deploying in control room is

ImportError: No module named site

May be mix of Python versions in /home/mice/MAUS/maus_release ?

#12

Updated by Jackson, Mike over 9 years ago

18

  • configure ensures env.sh adds maus-apps/src to PYTHONPATH and LD_LIBRARY_PATH.
  • Non-EPS files in media/raw directory shouldn't display - implemented as non-image files won't display.
  • Added comments to all files and Pylinted them.
19
  • install.sh easy_installs magickwand
  • Created new image_utils.py module for useful functions and pulled out commonality from views.py.
#13

Updated by Jackson, Mike over 9 years ago

Image formats supported:

  • Matplotlib and PyROOT support: ps eps png pdf svg
  • PyROOT also supports: gif jpg jpeg
  • Matplotlib also supports: rgba raw svgz emf (only with pyemf)

MagickWand convert cannot do:

  • raw - Conversion problem...no decode delegate for this image format `matplotlib-raw.raw' @ error/constitute.c/ReadImage/532
  • rgba - Conversion problem...must specify image size `matplotlib-rgba.rgba' @ error/rgb.c/ReadRGBImage/143
  • svg, svgz look bad. May need convert recompiled e.g. see http://www.imagemagick.org/Usage/draw/#svg

For now went with the formats it does support: ps, eps, png, pdf, gif, jpg, jpeg.

#14

Updated by Jackson, Mike over 9 years ago

Commits up to: 21

  • ConfigurationDefaults sets image_directory to MAUS_WEB_MEDIA_RAW if this is set.
  • maus-apps configure script sets up env.sh to set MAUS_WEB_MEDIA_RAW.
#15

Updated by Jackson, Mike over 9 years ago

Commits up to: 26

  • Removed need for install.sh by rewriting configure.
  • configure generates two types of env.sh depending on whether MAUS Python is being used or not.
  • Replaced print with logging.
#16

Updated by Jackson, Mike over 9 years ago

Using MagickWand for thumbnails,

from magickwand.image import Image as MagickWandImage
ipng = MagickWandImage("imagetof_xy_2000161.png")
ieps = MagickWandImage("matplotlib-eps.eps")
ipng.resize((100,100))
ieps.resize((100,100))
ipng.format="JPG" 
ieps.format="JPG" 
ipng.save("ipng.jpg")
ieps.save("ieps.jpg")

Looks awful.
ipng2 = MagickWandImage("imagetof_hit_x000008.png")
ipng2.format="JPG" 
ipng2.thumbnail(100)
ipng2.save("ipng2.jpg")

Looks fine.

#17

Updated by Jackson, Mike over 9 years ago

Attempt at using image.dump(...) to try and return image data (after http://benjamin-schweizer.de/python-magickwand-or-how-to-work-with-icons.html) in Django HttpResponse rather than having to save then reload thumbnail yields

please register the function size_t
No clue as to how to solve this - noone else has encountered it. Not a blocker, more a would've-made-things-cleaner.

#18

Updated by Jackson, Mike over 9 years ago

A scalability comment, for 1300 images the web front-end can generate thumbs BUT ...can't show page as it takes longer than 5s to generate them all! Even if showing full size, it takes about 5s to return all to client before page refreshes. Refresh rate in the web pages may need tweaked depending on time taken to generate the thumbs.

An HTML form handling example is at http://www.djangobook.com/en/1.0/chapter07

#19

Updated by Jackson, Mike over 9 years ago

In terms of image browsing and refreshing, how do others do it, MAUSThirdPartyOnlineMonitoring:

#20

Updated by Jackson, Mike over 9 years ago

Commit 30

  • Supports simple search button based on files matching a keyword
  • Removed GENERATOR tag. Added JavaScript Pause, Refresh, Resume
#21

Updated by Jackson, Mike over 9 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Done - latest version running in control room.

#22

Updated by Rogers, Chris over 9 years ago

  • Target version changed from Future MAUS release to MAUS-v0.2.2

Also available in: Atom PDF