Mediawiki collection extension installation
Definition
The Mediawiki collection extension allows a user to organize personal selections of pages in a collection. Collections can be:
- edited and structured using chapters
- persisted, loaded and shared
- rendered as PDF (see Extension:PDF_Writer)
- exported as ODF Text Document (see Extension:OpenDocument_Export)
- exported as DocBook XML (see Extension:XML_Bridge)
- ordered as a printed book at http://pediapress.com/
There are two major levels of installation/configuration
- Just install the Mediawiki extension and use PediaPress free rendering service (ok for low volume wikis)
- Install your own rendering services (that requires much more work)
This page includes some centralized help links and installation tips made for our own use - Daniel K. Schneider 16:31, 4 May 2009 (UTC).
See also:
- Wiki book
- Help:Books (for collection authors and users)
- Mediawiki for a short list of useful extensions.
- old version including some Solaris tips that I removed since outdated.
Help pages
Documentation entry page at PediaPress
Help (can also be used for informal bug reports)
- http://groups.google.com/group/mwlib/topics (Google forums)
Information about the collection extension and related server-side software
- http://www.mediawiki.org/wiki/Extension:Collection
- http://www.mediawiki.org/wiki/Extension:PDF_Writer
- http://code.pediapress.com/wiki/wiki (PediaPress Open Source Repository, Wiki and Bug Tracking System)
- http://code.pediapress.com/git/mwlib/raw-file/tip/docs/commands.txt (command line options, important!)
- http://code.pediapress.com/wiki/wiki/Examples
Installing the whole suite requires some installation skills, but should go fairly smoothly on any Unix system and should be easy on a Debian-based Linux.
Collection Mediawiki extension installation and tuning
If you own a small MediaWiki and do not plan to customize PDF rendering, all you need is this extension. If you use this service often and/or you plan to tweak, then you also have to install the rendering servers. See the sections below.
Base installation of the extension
The collection extension installs like any other Mediawiki extensions. Really easy with Mediawiki=> 1.14 (Spring 2009).
- (1) Installing
Get it from Extension:Collection on Mediawiki.org.
You may try the latest version, however sometimes it doesn't work with your MW installation. E.g. it breaks for MW 1.16.4 on April 20 2011. Get it from GIT
cd extensions git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Collection.git
Formerly, the code was available in SVN
svn checkout http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/Collection/
Read:
README.txt
You then can just leave all the defaults and the PDF will be generated by PediaPress. However, if you have a slow server or a high traffic server, you also should install a local render server (read the whole rest of this page). When we first installed this extensions in 2008, the Pediapress server did loose pages due to server overload. By now, this problem should be fixed (not tested ....)
- (2) Tuning
If you have and old and slow server, I suggest changing the file Collection.i18n.php and change the string (adjust to the power of your server)
'coll-rendering_text' => "<p><strong>Please wait while the document is being generated.
Depending on the size of book you may have to wait 5, 10, 15 minutes or longer.
</strong></p>
......"
'coll-save_collection_text' => 'Choose a storage location for your book and enter a name:',
- (3) Permissions (important !)
If you want users to be able to save and to share collections, add these permission to file Localsettings.php:
$wgGroupPermissions['user']['collectionsaveasuserpage'] = true; $wgGroupPermissions['user']['collectionsaveascommunitypage'] = true;
or maybe:
$wgGroupPermissions['user']['collectionsaveasuserpage'] = true; $wgGroupPermissions['sysop']['collectionsaveascommunitypage'] = true;
- (4) Define templates
You should define the following (language-dependant) templates and categories:
- Template:Saved_book (Grab a copy from a wikipedia and modify).
- Category:Books
- Category:Book tool (not really needed)
For conditional inclusion of text within articles you must define these templates (inspired from the french wikipedia, the English uses some parser extension which we probably should also install and test at some point - Daniel K. Schneider)
Template:Hide in print:
<includeonly><span class="noprint">{{{1}}}</span></includeonly>
Template:Only in print:
<includeonly><span class="hidden">{{{1}}}</span></includeonly>
Then, edit MediaWiki:Common.css and add the following CSS rule:
.hidden {display:none}
Tweaking the collection extension
Also read again README.txt !
Add in Localsettings.php (if not already done) the rendering engines you will support. In case you also installed a local rendering server, typing mw-render --list-writers will list the ones you installed.
Example setting for Localsettings.php:
$wgCollectionFormats = array( 'rl' => 'PDF', 'odf' => 'ODT', );
In Localsettings.php, if not already done so, add the name and port of the server (alternatively you also could install a cgi script).
$wgCollectionMWServeURL = "http://xxx.yyy:8899";
Extra stuff:
For the license, make sure to give the correct RAW wiki URL. Or if this doesn't work, remove the line and the user will see a URL.
$wgLicenseURL = "http://edutechwiki.unige.ch/mediawiki/index.php?title=EduTech_Wiki:Copyrights&action=raw";
Limit number of articles that a book can contain, e.g.
$wgCollectionMaxArticles = 150;
mwlib and servers installation on Ubuntu
At the time of writing (dec 2012), the manual at PediaPress is fine and should include everything a system person needs to know. However, since I am not a system administrator, I have to write down what I do, since it helps me understanding and make things go faster for upgrades. In addition, the manual suggest running some kind of virtual Python environment. The solution below is simpler, but probably not as safe.
In the past, Pediapress had a single server called mw-serve. That solution is now deprecated. As of 2012, mwlib refers to a whole series of software that will allow:
- to to extract wikipages to zip files
- do your own PDF/etc rendering from those zip's
- Send the zips to PediaPress for printing
This section describes the easy way, however easy does not mean real easy if you got lot's of older junk in your python library. Be prepared to uninstall and upgrade older libraries. In my Ubuntu they sit here:
/usr/local/lib/python2.6/dist-packages
Prerequisites
Have root access and become root (or use 'sudo ...')
You must have python installed plus the pip packaging manager
apt-get install python apt-get install python-pip
pip is a tool for installing and managing Python packages, such as those found in the Python Package Index. It's a replacement for easy_install that was used in the past. Since the Ubuntu version is outdated, I'd rather take a fresh one:
git clone https://github.com/pypa/pip.git cd pip python setup.py install
At at time of writing, if you do this, it will install an executable called pip-2.6 to /usr/local/bin/. I suggest keeping the old one since Ubuntu may not like you to remove it.
There must be some other dependencies, but since I installed prior versions of mwlib I can't remember. Maybe have a look at the old installation notes for mw-serve below ?
Pediapress suggest the following:
apt-get install -y gcc g++ make python python-dev python-virtualenv \ libjpeg-dev libz-dev libfreetype6-dev liblcms-dev \ libxml2-dev libxslt-dev \ ocaml-nox git-core \ python-imaging python-lxml \ texlive-latex-recommended ploticus dvipng imagemagick \ pdftk
Install
pip install -i http://pypi.pediapress.com/simple/ pil pip install -i http://pypi.pediapress.com/simple/ mwlib pip install -i http://pypi.pediapress.com/simple/ mwlib.rl
This may not work at all, since dependencies are not met. I had a longer list :(
For each missing piece, try to upgrade, e.g. for apipkg:
pip install --upgrade apipkg
In addition, you may have to remove packages, see below.
Uninstalling old/useless and interfering python packages
Only works with a recent version of pip, e.g. pip-2.6 !!
Syntax:
pip-2.6 uninstall package_name
Since the old pip still sticks around, the following will not work (unless your replace it)
pip uninstall package_name
Sometimes, you may just have to remove packages manually (probably older not finished installs ?)
cd /usr/local/lib/python2.6/dist-packages rm -r package_name
.... but this is not recommended IMHO
Test
Get something from Wikipedia (this is the default)
mw-zip -c :en -o test.zip Acdc Number mw-render -c test.zip -o test.pdf -w rl
Look at test.pdf and enjoy ....
xpdf test.pdf
Try it on yours:
mw-zip -c http://edutechwiki.unige.ch/mediawiki/ -o test2.zip Mediawiki_collection_extension_installation mw-render -c test2.zip -o test2.pdf -w rl xpdf test2.pdf
Upgrades
This may be bad voodoo. After doing this, I had to restart, but this was probably due to too much old junk.
pip install -i http://pypi.pediapress.com/simple/ --upgrade mwlib pip install -i http://pypi.pediapress.com/simple/ --upgrade mwlib.rl
Running the render servers
On the machine
The following will launch the combo. Each program has various options. At least for testing, I suggest to accept the defaults. However, define a cache directory for the nslave.
nserve.py & mw-qserve & # create a directory and make it writable to the user that runs the servers nslave.py --cachedir /data/mwcache & postman.py &
In LocalSettings.php of the wiki
You must add the following line, since by default your wiki will use the PediaPress rendering service:
$wgCollectionMWServeURL = "http://machine.you.domain:port";
For example (using the default port):
$wgCollectionMWServeURL = "http://edutechwiki.unige.ch:8899";
Log information output by the render services
All these servers will write back log information to the console. E.g. if you test creating a simple PDF file you might see something like this (slightly formatted and edited):
=== report Wed Dec 12 14:15:13 2012 === have 0 jobs count: 0 all channels idle new-collection 1 'http://edutechwiki.unige.ch/mediawiki' 'rl' 2012-12-12T14:15:14 mwlib.serve.info >> render 9da9af29fbb8b623 rl 129.194.xxx- - [2012-12-12 14:15:14] "POST / HTTP/1.0" 200 200 0.012274 129.194.xxx- - [2012-12-12 14:15:14] "POST / HTTP/1.0" 200 215 0.002743 0 1.42332196236 ['mw-zip', '-o', '/data/mwcache/9d/9da9af29fbb8b623/collection.zip', '-m', '/data/mwcache/9d/9da9af29fbb8b623/metabook.json', '--status', 'qserve://localhost:14311/9da9af29fbb8b623:makezip', '--config', 'http://edutechwiki.unige.ch/mediawiki', '--template-blacklist', 'MediaWiki:PDF Template Blacklist', '--template-exclusion-category', 'Exclude in print', '--print-template-prefix', 'Print', '--print-template-pattern', '$1/Print'] finish: 9da9af29fbb8b623:makezip: None 0 2.253221035 ['mw-render', '-w', 'rl', '-c', '/data/mwcache/9d/9da9af29fbb8b623/collection.zip', '-o', '/data/mwcache/9d/9da9af29fbb8b623/output.rl', '--status', 'qserve://localhost:14311/9da9af29fbb8b623:render-rl', '--template-blacklist', 'MediaWiki:PDF Template Blacklist', '--template-exclusion-category', 'Exclude in print', '--print-template-prefix', 'Print', '--print-template-pattern', '$1/Print', '--language', 'en'] finish: 9da9af29fbb8b623:render-rl: {'url': 'http://129.194.7.75:8898/cache/9d/9da9af29fbb8b623/output.rl', 'suggested_filename': 'Mediawiki collection extension installation', 'size': 125658} watchdog: dropped 0 jobs, marked 2 jobs with a deadline [....] === report Wed Dec 12 14:15:33 2012 === have 2 jobs count: 2 all channels idle [.....] === report Wed Dec 12 14:16:53 2012 === have 2 jobs count: 2 all channels idle 129.194.xxx - - [2012-12-12 14:16:59] "POST / HTTP/1.0" 200 565 0.039982 129.194.xxx - - [2012-12-12 14:17:01] "POST / HTTP/1.0" 200 565 0.002620 129.194.xxx - - [2012-12-12 14:17:01] "GET /cache/9d/9da9af29fbb8b623/output.rl HTTP/1.0" 200 125919 0.001031
Killing jobs launched from a terminal jobs
[1] Running nserve.py & (wd: /data/portails/mediawiki) [2] Running mw-qserve & (wd: /data/portails/mediawiki) [3]- Running nslave.py --cachedir /data/mwcache & [4]+ Running postman.py &
kill %1 kill %2 etc.
Installing a script
You could imagine redirecting logfiles to a log file, running the processes under a better user than root and also writing a start/stop script in case you boot.
In fact, Pediapress suggests using runit for process supervision. daemontools is similar solution. Another alternative is to use supervisor. The advantage of such a solution is that processes are automatically restarted after a process dies (that can happen for various reasons). Installing [supervisor http://supervisord.org/] is what I personally would attempt since it doesn't seem to interfere much with standard practice from what I could understand and it doesn't look too complicated. Anyhow, in the meantime I made this script (suggestions welcome, I really am not a sysadmin)
Prerequisites:
- Have a user you can run the processes with. E.g. yourself. I got www-data
chown -R /data/mwcache www-data:www-data
#! /bin/bash PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin case "$1" in start) su - www-data -c 'nserve.py >> /data/mwcache/log.txt 2>&1 &' su - www-data -c 'mw-qserve >> /data/mwcache/log.txt 2>&1 &' su - www-data -c 'nslave.py --cachedir /data/mwcache/ >> /data/mwcache/log.txt 2>&1 &' su - www-data -c 'postman.py >> /data/mwcache/log.txt 2>&1 &' ;; stop) mv /data/mwcache/log.txt /data/mwcache/log.old killall nserve.py killall mw-qserve killall nslave.py killall postman.py ;; force-reload|restart) $0 stop $0 start ;; *) echo "Usage: /etc/init.d/mw-serve {start|stop}" exit 1 ;; esac exit 0
Tweaking
- For larger servers you need to multiply .... read the official manual
- For better quality of service, you must run the scripts with a process supervisor software (See the manual).
Manual mwlib installation on Ubuntu
Written in 2010: It concerns the deprecated mw-serve collection and render server.
Again, mwlib is mot needed if you have a fast server and want to use the pediapress server and don't need the lasted tip of the code and don't want to customize.
Below are some installation notes (quick and dirty for now). Install is done by a user with admin rights not root (Ubuntu fashion):
Looking at the versions:
- The repository is here: http://code.pediapress.com/git/
Prerequisites
- Python
- Cython
apt-get install cython
Installing mwlib
Before installing, you might edit mwlib/tagext.py around line 100 a tag to exclude extensions tags used and that can't be handled by mwlib, e.g. at some point I had to exclue manually: 'pageby', 'uml', 'graphviz', 'categorytree', 'summary' (but since sometimes end of 2009 this is fixed).
sudo aptitude install make g++ perl python python-dev python-setuptools python-imaging re2c sudo aptitude install git-core git clone git://github.com/pediapress/mwlib cd mwlib
sudo python setup.py build sudo python setup.py install
Install mwlib.rl
(with a variant in procedure, i.e. we create a new directory where git will put the files)
git clone git://code.pediapress.com/mwlib.rl mwlib.rl.new cd mwlib.rl.new sudo python setup.py build install
Install the source code formatting package from http://pygments.org/ (debian packet: python-pygments)
sudo aptitude install python-pygments
Install extra fonts
sudo aptitude install ttf-indic-fonts ttf-unfonts ttf-farsiweb ttf-arphic-uming ttf-gfs-artemisia ttf-sil-ezra ttf-thai-arundina linux-libertine
In file fontconfig.py
font_paths = [os.path.dirname(mwlib.fonts.__file__), os.path.expanduser('~/mwlibfonts/') ]
This under the assumption that the fonts are in the directory mwlib.rl.hg/mwlib/fonts or ~/mwlibfonts/ of the user that installs or root if you install as root.
Check what it can render
mw-render --list-writers
You then can configure the mediawiki extension accordingly
Install tables of contents for mwlib.rl
apt-get install pdftk
This will produce a visible TOC in the beginning of the book if installed (nothing else to do, mwlib.rl will detect this library). So far this TOC does not have hyperlinks (but users can open the bookmarks to the left in order to navigate)
Configure styling
Customizing the resulting PDFs is possible by adding a custom configuration file. The file needs to named customconfig.py and should reside next to the pdfstyles.py file. Basically you can override anything in the pdfstyles.py file with your custom configuration.
Since I don't talk Python, I just modified /mwlib.rl/mwlib/rl/pdfstyles.py in the source, i.e. added the third line below to make font for codes smaller :)
if mode == 'source' or mode == 'preformatted': style.fontName = mono_font style.fontSize = small_font_size
Then rebuild and reinstall.
Updating mwlib and mwlib.rl
This is one way of doing it:
ssh to your server as root (else add "sudo" to each line ...) cd path-to/src
- consider creating a backup of the old source
mv mwlib mwlib.old mv mwlib.rl mwlib.rl.old
- Get and install mwlib
git clone git://code.pediapress.com/mwlib cd mwlib python setup.py build install cd ..
- Then get and install mwlib.rl (the PDF renderer)
git clone git://code.pediapress.com/mwlib.rl cd mwlib.rl python setup.py build install
Running mw-serv (the old Pediapress rendering server)
(deprecated as of 2012 !!)
mw-serve provided a server interface for the mw-render engine and mw-zip.
- Run the server
- http://code.pediapress.com/git/mwlib/raw-file/tip/docs/commands.txt
- By default this service runs on port 8899
Type in a console:
mw-serve --cache-dir=/data/mwcache/mwlibcache/ --logfile=/data/mwcache/logs/mwserve.log --mwrender-logfile=/data/mwcache/logs/mwrender.log --mwzip-logfile=/data/mwcache/logs/mwzip.log --mwpost-logfile=/data/mwcache/logs/mwpost.log --mwzip-logfile=/data/mwcache/logs/mwzip.log
An init script
If you are happy with your server, you should make it start up automatically, e.g. do the following:
- Create a user, e.g. something like
useradd -u 70002 -g 16100 -s /bin/sh -d /data/mwcache mwserv
- Chown the cache and log directories to this user
chown -R mwserv mwcache/
- /etc/init.d script
see the example at http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/mw-serve/mw-serve.sh Add it to appropriate run-levels with chkconfig (see man chkconfig)
Here is a simple script that can do:
#! /bin/bash
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
DAEMON=/usr/local/bin/mw-serve
test -x $DAEMON || exit 0
case "$1" in
start)
/usr/local/bin/mw-serve --cache-dir=/data/mwcache/mwlibcache/ --logfile=/data/mwcache/logs/mwserve.log --mwrender-logfile=/data/mwcache/logs/mwrender.log --mwzip-logfile=/data/mwcache/logs/mwzip.log --mwpost-logfile=/data/mwcache/logs/mwpost.log --mwzip-logfile=/data/mwcache/logs/mwzip.log&
;;
stop)
killall mw-serve #ou rien
;;
force-reload|restart)
$0 stop
$0 start
;;
*)
echo "Usage: /etc/init.d/mw-serve {start|stop}"
exit 1
;;
esac
exit 0
Tips for creating wiki books
Creating a serious book
It is best just to "hand edit" a stored book. E.g. you may start by adding a category or isolated articles to a book with the collection interface, but once you got most of the articles:
- Save the book
- Then select it from the category:books and then just edit the page in the "normal way". The structure and syntax to respect is demonstrated by the following example:
{{saved_book}} == My Book == === Example === ;Foo :[[First article]] :[[Second article]] :[[Third article|This article renamed in the book]] ;Bar :[[Fourth article]] :[[Fifth article]] :[{{fullurl:Sixth article|oldid=20}}Sixth article version:20] [[Category:Books]]
Install the Mediawiki source extension
If you have articles that include computer source code (XML, ActionScipt or whatever), you also should install the source extension (CodeSyntaxHighlight MediaWiki for formatting computer code.
If you use XML within "pre" tags, the parser may become confused.
Also, the printed Pediapress book will look much prettier. Editing all your wiki pages may represent work, but user reading the on-line pages also will be grateful, colored and indented code is really much more readable !
Pictures and drawings
(1) Readjust some image sizes
(I'll have to be more precise about this, but I'll first need to analyze both a wiki book I got and the generated PDF...)
- Don't use large pictures when smaller ones are readable
- Create screendumps from smallest possible areas if you plan to show hairy details (see also the screen capture tutorial).
(2) Color Printed PDF is mostly grey (Pediapress books certainly are). E.g. when you create drawings or annotate images with text, you should make sure that shades of grey still allow to identify critical elements. A related very difficult issue is writing the text. Avoid writing about "blue" and "green" arrows and "green" or "yellow" dots ...
- To select colors that show on gray printers, you may use a color scheme designer such as colorschemedesigner.com and simulate color vision deficiency with "full colorblindness". Else print your drawing, before you import to wiki.
- A radical solution is to use gray images already upfront in the wiki. This way you are sure to get it somewhat right. Note: "grey" is spelled "gray" in CSS and X11.
Stuff that the parser doesn't handle well
As of mid May 2009 (so this may change)
- Some extensions (like graphwiz) are not supported, see above. There is no solution for this, except not using these.
- Some extensions that are not supported wont' matter, e.g. pageby. But you will have to modify the source code for filtering if you own your own server (see above) or file a request.
- prints "as is", therefore avoid! Use ":" and "::" etc. to indent lines for example
- Source code (either within "pre" or "source" tags that follow a picture will be printed over the picture. Probable reason. The renderer will reduce a picture and try to wrap text around it. Source code cannot be wrapped or not as well. Workaround: Move the image either 20 lines above or after the code.
Conditional inclusion/exclusion
Warning: This wasn't tested with Pediapress. If you plan to buy a book, use these on the very first pages, then have a look at the preview.
1) Exclude templates
You may exclude any template from the PDF generation you wish by adding them to the Category:Exclude in print. Use with care, since this will filter for all users !
2) Exclude certain specific content
By using Template:Hide in print, certain specific content, such as a few words or an image, can be excluded from printing.
This content will be printed.{{Hide in print|This content will not be printed.}}This content will be printed.
Alternative solution: use the class="noprint" within a div or span tag.
3) Include certain specific content only in print versions The Template:Only in print can be used to insert content that shall only be visible in offline versions.
Example: print this and display it in the browser {{Only in print|this is only in PDFs or printed books visible, not with the browser}} this is visible in the browser and in print as well.
Alternative solution: use the class="hidden" within a div or span tag.
4) Substitute templates
You can create a print version of a template under the name "TEMPLATENAME/Print" with TEMPLATENAME being the name of the original template.
(more to come ....) - Daniel K. Schneider 12:54, 20 May 2009 (UTC)