Mediawiki collection extension installation

Category:MediaWiki extension Mediawiki collection extension installation
Extension name	Collection
About this article / disclaimer
Logo
Screenshot
Location of the main author	Germany
Coordinates of the main authors	51.163818, 10.447831
Developers	PediaPress.com, Siebrand Mazeland, Marcin Cieślak
Licences	GPL
Description	This extension makes it possible to collect a number of pages. Collections can be edited, persisted and optionally retrieved as PDF, ODF or DocBook (XML)
Mediawiki requirements	Version 1.17 and later
Dependencies
Related extensions (documented here)
Related extensions	[[Is related to information::PDF Writer, Open Document Export]]
Discussion	There are two ways of using this extension. Easy: Install the extension and use the PediaPress collection and rendering services Install your own collection and rendering services
Language support	Many
Status	stable
First release date
Last release date (as of !)
Last version number
Programming language	PHP, Python
Alternatives
Website	home page
Publications
Support websites	web site, web site
Example websites
Last edited

[UPDATE: Please also read the discussion page ! Both Debian/Ubuntu upgrades and MW 1.27 upgrades do require changes that are not documented here] - Daniel K. Schneider (talk) 21:54, 13 September 2016 (CEST)

Introduction

The Mediawiki collection extension allows a user to organize personal selections of pages in a collection. For an educational or academic wiki, this is a must-have extension. E.g. you could use to create print manuals and textbooks or also to valorize student productions at the end of year (hey "daddy" we made a book...)

Collections can be:

• edited and structured using chapters
• persisted, loaded and shared
• rendered as PDF (see Extension:PDF_Writer)
• exported as ODF Text Document (see Extension:OpenDocument_Export)
• exported as DocBook XML (see Extension:XML_Bridge)
• ordered as a printed book at http://pediapress.com/

There are two major levels of installation/configuration

• Just install the Mediawiki extension and use PediaPress free rendering service (ok for low volume wikis)
• Install your own rendering services (that requires much more work)

This page includes some centralized help links and installation tips made for our own use - Daniel K. Schneider 16:31, 4 May 2009 (UTC).

See also:

• Wiki book
• Help:Books (for collection authors and users)
• Mediawiki for a short list of useful extensions.
• old version including some Solaris tips that I removed since outdated.

Help pages

Documentation entry page at PediaPress

• Pediapress.com/code

Help (can also be used for informal bug reports)

• http://groups.google.com/group/mwlib/topics (Google forums)

Information about the collection extension and related server-side software

• http://www.mediawiki.org/wiki/Extension:Collection
• http://www.mediawiki.org/wiki/Extension:PDF_Writer
• http://code.pediapress.com/wiki/wiki (PediaPress Open Source Repository, Wiki and Bug Tracking System)
• http://code.pediapress.com/git/mwlib/raw-file/tip/docs/commands.txt (command line options, important!)
• http://code.pediapress.com/wiki/wiki/Examples

Installing the whole suite requires some installation skills, but should go fairly smoothly on any Unix system and should be easy on a Debian-based Linux.

Collection Mediawiki extension installation and tuning

If you own a small MediaWiki and do not plan to customize PDF rendering, all you need is this extension. If you use this service often and/or you plan to tweak, then you also have to install the rendering servers. See the sections below.

Base installation of the extension

The collection extension installs like any other Mediawiki extensions. Really easy with Mediawiki=> 1.14 (Spring 2009).

(1) Installing

Get it from Extension:Collection on Mediawiki.org.

You may try the latest version, however sometimes it doesn't work with your MW installation. E.g. it breaks for MW 1.16.4 on April 20 2011. Get it from GIT

cd extensions
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Collection.git
git branch -r
git checkout REL1_25

Read:

README.txt

You then can just leave all the defaults and the PDF will be generated by PediaPress. However, if you have a slow server or a high traffic server, you also should install a local render server (read the whole rest of this page). When we first installed this extensions in 2008, the Pediapress server did loose pages due to server overload. By now, this problem should be fixed (not tested ....)

(2) Tuning messages

If you have and old and slow server, I suggest changing the file Collection.i18n.php and change the string (adjust to the power of your server)

'coll-rendering_text' => "<p><strong>Please wait while the document is being generated.
Depending on the size of book you may have to wait 5, 10, 15 minutes or longer.
</strong></p> 
 ......"

'coll-save_collection_text'       => 'Choose a storage location for your book and enter a name:',

(3) Permissions (important !)

If you want users to be able to save and to share collections, add these permission to file Localsettings.php:

$wgGroupPermissions['user']['collectionsaveasuserpage'] = true;
$wgGroupPermissions['user']['collectionsaveascommunitypage'] = true;

or maybe a more restrictive alternative:

$wgGroupPermissions['user']['collectionsaveasuserpage'] = true;
$wgGroupPermissions['sysop']['collectionsaveascommunitypage'] = true;

(4) Templates and categories for "books" pages

You should define the following (language-dependant) templates and categories:

• Template:Saved_book (Grab a copy from a wikipedia and modify).
• Category:Books
• Category:Book tool (not really needed)

(5) Conditional inclusion (after June 2013 version)

For content to be excluded from the PDF, use the CSS class "noprint":

 <div class="noprint">blub noprint</div>

For content to be included in PDFs and excluded in the wiki page, use the following CSS

<div class="onlyinprint">blob only in print</div>

This css class is ignored by the PDF renderer and thus displayed normally in the PDF. Modify your MediaWiki:Common.css file to hide the content in the wiki page.

.onlyinprint {display: none}

Of course, you then could create simple templates, e.g. a user could write

There is a {{hide in print|unprintable}} {{only in print|printable}} word.

Template:Hide in print:

 <includeonly><span class="noprint">{{{1}}}</span></includeonly>

Template:Only in print:

 <includeonly><span class="onlyinprint">{{{1}}}</span></includeonly>

Tweaking the collection extension

Read again

• Collection manual at PediaPress
• Extension:Collection

Add in Localsettings.php (if not already done) the rendering engines you will support. In case you also installed a local rendering server, typing mw-render --list-writers will list the ones you installed.

Example setting for Localsettings.php:

$wgCollectionFormats = array(
          'rl' => 'PDF',
          'odf' => 'ODT',
      );

In Localsettings.php, if not already done so, add the name and port of the mwlib server.

$wgCollectionMWServeURL = "http://xxx.yyy:8899";

Extra stuff:

For the license, make sure to give the correct RAW wiki URL. Or if this doesn't work, remove the line and the user will see a URL.

$wgLicenseURL = "http://edutechwiki.unige.ch/mediawiki/index.php?title=EduTech_Wiki:Copyrights&action=raw";

Limit number of articles that a book can contain, e.g.

$wgCollectionMaxArticles = 150;

mwlib and servers installation on Ubuntu

Tested in this server, using Ubuntu 10.04.4 LTS, Ubuntu 12.04.2 LTS and Ubuntu 14.04 LTS(server editions). Instructions below are only valid for Ubuntu 14 LTS. I just tested this again on a fresh server - Daniel K. Schneider (talk)

Notice: mwlib now has been replaced by OCG on Wikipedias (at least that's what they say). However, OCG seems to be less stable so far and maybe requires even more installation efforts. That being said, I should at some point migrate and find time for testing - Daniel K. Schneider (talk) 12:30, 13 April 2015 (CEST)

Since my first install unil the time of writing (dec 2012 / august 2013 / April 2015), the manual at PediaPress is fine and should include everything a system person needs to know in order to understand and configure the service. However, the installation instructions are not good enough. The pip installer will find Python packages but cannot add missing Ubuntu libraries.

• In aug 2013, someone made a Setup a render server on Ubuntu 12.04 LTS article on mediawiki.org]. Read that article first, it looks better than mine :) However, since I am not a system administrator, I have to write down what I do, since it helps me understanding and make things go faster for upgrades. In addition, the manual suggest running some kind of virtual Python environment. The solution below is simpler, but probably not as safe.

mwlib refers to a whole series of software that will allow:

• to extract wikipages to zip files
• to do your own PDF/etc rendering from those zip's
• to send the zips to PediaPress for printing

This section describes the easy way, however easy does not mean real easy. Depending on your Ubuntu version and the state of your server, you could suffer:

• if you don't have a fresh Ubuntu machine, you may have lot's of older junk in your python library. Be prepared to uninstall and upgrade older libraries. In my Ubuntu they sit here:

/usr/local/lib/python2.7/dist-packages (formerly, for 10.04, python 2.6)

• As mentioned above, you will have to install a whole bunch of packages, but on a fresh Ubuntu machine this should be really easy. The 14LTS install on April 10 2015 went quickly and rather smoothly.

Prerequisites

Have root access and become root (or use 'sudo ...')

Install/check python

Firstly, you must have python 2.7.* installed (usually already done) plus the pip packaging manager (not done on a fresh machine), plus a lot of graphics and text libraries.

apt-get install python
apt-get install python-pip

pip is a tool for installing and managing Python packages, such as those found in the Python Package Index. It's a replacement for easy_install that was used in the past.

In case pip is installed, make sure that you got the latest version

pip install --upgrade pip

In case the Ubuntu pip version doesn't work, you could install a fresh one (this happened to me sometimes in the past)

git clone https://github.com/pypa/pip.git
cd pip
python setup.py install

Dependencies

(this is not complete, see the Ubuntu package list below)

• Perl => 5
• g++
• Latex
  • Also compile texvc in mediawiki/math directory
• Blahtexml

• Python => 2.7
  • Python setuptools http://pypi.python.org/pypi/setuptools
  • python imaging library (PIL) http://www.pythonware.com/products/pil/

• rec2c
  • http://re2c.org/

• ocaml
  • http://caml.inria.fr/

• Pygments
  • http://pygments.org/

• Fribidi - both a library and the Python bindings
  • http://pyfribidi.sourceforge.net/index.html

Install Ubuntu packages

Below is a complete list of the Ubuntu packages I installed. Some is already on the system, but that's neither a problem for apt-get nor for your server. If you get error messages when you install mwlib (see below), then it is likely that some packages are missing....

sudo apt-get install -y gcc \
dvipng\
g++\
git-core\
imagemagick\
libevent-dev\
libfreetype6-dev\
libjpeg-dev\
liblcms-dev\
libxml2-dev\
libxslt-dev\
libz-dev\
make\
ocaml-nox\
pdftk\
ploticus\
python-all-dev\
python-dev\
python-imaging\
python-lxml\
python-pip\
python-virtualenv\
python\
tcl\
texlive-latex-recommended\
tk
webp\

Below some extra python stuff that seems to be needed.

pip install greenlet
pip install gevent

Pillow, (the PIL Python imaging library) seems to be installed from PediaPress.

Install

(for Ubuntu 14)

pip install -i http://pypi.pediapress.com/simple/ mwlib
: This will take some time, since files have to be compiled
pip install -i http://pypi.pediapress.com/simple/ mwlib.rl

For each missing piece, try to upgrade, e.g. for apipkg:

 pip install --upgrade apipkg

In addition, you may have to remove packages, see below. You also may have to remove/reinstall mwlib and mwlib.rl if you did something wrong ...

After fixing missing dependencies, e.g. a missing image library, you may have to force upgrade. Successfully installed python packages that will break during runtime, will not upgrade without --force-reinstall.

pip install --upgrade --force-reinstall -i http://pypi.pediapress.com/simple/ pil
pip install --upgrade --force-reinstall -i http://pypi.pediapress.com/simple/ mwlib
pip install --upgrade --force-reinstall -i http://pypi.pediapress.com/simple/ mwlib.rl

Upgrade to Ubuntu 16 / Debian 8 and MW 1.27

(thanx to

Step 1 - reinstall and upgrade "pil"

   pip install --trusted-host pypi.pediapress.com --upgrade --force-reinstall -i http://pypi.pediapress.com/simple/ pilil

Step 2 - upgrade "mwlib"

   pip install --trusted-host pypi.pediapress.com -i http://pypi.pediapress.com/simple/ --upgrade mwlib

Step 3 - upgrade "mwlib.rl"

   pip install --trusted-host pypi.pediapress.com -i http://pypi.pediapress.com/simple/ --upgrade mwlib.rl

Step 4 - update "Mwlib server daemon"

   per suggestion of the example below

Step 5 - create user "mw-serve" since "www-data" will no longer work

   adduser mw-serve

Step 6 - change permissions of the log directory

   chown mw-serve:mw-serve -R /var/log/mediawiki/

Step 7 - start render servers

   /etc/init.d/mw-serve start

Step 8 - enjoy

Uninstalling old/useless and interfering python packages

Only works with a recent version of pip, e.g. pip-2.6 or better !!

Syntax:

pip uninstall package_name

Sometimes, you may just have to remove packages manually (probably older not finished installs ?)

cd /usr/local/lib/python2.6/dist-packages
rm -r package_name

.... but this is not recommended IMHO

Test

Get something from Wikipedia (this is the default)

mw-zip -c :en -o test.zip Acdc Number
mw-render -c test.zip -o test.pdf -w rl

Look at test.pdf and enjoy ....

xpdf test.pdf

Try it on yours:

mw-zip -c http://edutechwiki.unige.ch/mediawiki/ -o test2.zip Mediawiki_collection_extension_installation
mw-render -c test2.zip -o test2.pdf -w rl
xpdf test2.pdf

Upgrades

pip install -i http://pypi.pediapress.com/simple/ --upgrade mwlib
pip install -i http://pypi.pediapress.com/simple/ --upgrade mwlib.rl

Running the render servers

On the machine

Firstly, create a directory where files can be collected. Then, make them writable for the process that will run mwlib. For example:

mkdir /data/mwcache
chown www-data:www-data mwcache/

The following will launch the combo. Each program has various options. At least for testing, I suggest to accept the defaults. However, define a cache directory for the nslave.

nserve &
mw-qserve &
# directory mwcache must be writable to the user that runs the servers
nslave --cachedir /data/mwcache &
postman --cachedir /data/mwcache &

Note about older versions

• In earlier versions (e.g. the ones that installed in 2012), the scripts were called nserve.py, nslave.py, etc. as opposed to nserve, nslave, etc.
• After upgrading, make damn sure to fix the script !! (I lost like 2 hours with this ....)

In LocalSettings.php of the wiki

You must add the following line, since by default your wiki will use the PediaPress rendering service:

 $wgCollectionMWServeURL = "http://machine.you.domain:port";

For example (using the default port):

$wgCollectionMWServeURL = "http://edutechwiki.unige.ch:8899";

Log information output by the render services

All these servers will write back log information to the console. E.g. if you test creating a simple PDF file you might see something like this (slightly formatted and edited, version with the old file names for scripts.):

=== report Wed Dec 12 14:15:13 2012 ===
have 0 jobs
count: 0
all channels idle

new-collection 1 'http://edutechwiki.unige.ch/mediawiki'	'rl'
2012-12-12T14:15:14 mwlib.serve.info >> render 9da9af29fbb8b623 rl
129.194.xxx- - [2012-12-12 14:15:14] "POST / HTTP/1.0" 200 200 0.012274
129.194.xxx- - [2012-12-12 14:15:14] "POST / HTTP/1.0" 200 215 0.002743
0 1.42332196236 ['mw-zip', '-o', '/data/mwcache/9d/9da9af29fbb8b623/collection.zip', 
 '-m', '/data/mwcache/9d/9da9af29fbb8b623/metabook.json', 
 '--status', 'qserve://localhost:14311/9da9af29fbb8b623:makezip', 
 '--config', 'http://edutechwiki.unige.ch/mediawiki', 
 '--template-blacklist', 'MediaWiki:PDF Template Blacklist', 
 '--template-exclusion-category', 'Exclude in print', 
 '--print-template-prefix', 'Print',
 '--print-template-pattern', '$1/Print']
finish: 9da9af29fbb8b623:makezip: None
0 2.253221035 ['mw-render', '-w', 'rl', '-c', '/data/mwcache/9d/9da9af29fbb8b623/collection.zip', 
 '-o', '/data/mwcache/9d/9da9af29fbb8b623/output.rl', 
 '--status', 'qserve://localhost:14311/9da9af29fbb8b623:render-rl', 
 '--template-blacklist', 'MediaWiki:PDF Template Blacklist', 
 '--template-exclusion-category', 'Exclude in print', 
 '--print-template-prefix', 'Print', '--print-template-pattern', '$1/Print', 
 '--language', 'en']
finish: 9da9af29fbb8b623:render-rl: 
 {'url': 'http://129.194.7.75:8898/cache/9d/9da9af29fbb8b623/output.rl', 
  'suggested_filename': 'Mediawiki collection extension installation',
  'size': 125658}

watchdog: dropped 0 jobs, marked 2 jobs with a deadline
[....]
=== report Wed Dec 12 14:15:33 2012 ===
have 2 jobs
count: 2
all channels idle
[.....]
=== report Wed Dec 12 14:16:53 2012 ===
have 2 jobs
count: 2
all channels idle
129.194.xxx - - [2012-12-12 14:16:59] "POST / HTTP/1.0" 200 565 0.039982
129.194.xxx - - [2012-12-12 14:17:01] "POST / HTTP/1.0" 200 565 0.002620
129.194.xxx - - [2012-12-12 14:17:01] "GET /cache/9d/9da9af29fbb8b623/output.rl HTTP/1.0" 200 125919 0.001031

Killing jobs launched from a terminal jobs

[1]   Running                 nserve.py &  (wd: /data/portails/mediawiki)
[2]   Running                 mw-qserve &  (wd: /data/portails/mediawiki)
[3]-  Running                 nslave.py --cachedir /data/mwcache &
[4]+  Running                 postman.py --cachedir /data/mwcache &

kill %1 kill %2 etc.

Installing a script

(not yet done, weekend is coming up - Daniel K. Schneider (talk) 19:21, 10 April 2015 (CEST)

You could imagine redirecting logfiles to a log file, running the processes under a better user than root and also writing a start/stop script in case you boot.

In fact, Pediapress suggests using runit for process supervision. daemontools is similar solution. Another alternative is to use supervisor. The advantage of such a solution is that processes are automatically restarted after a process dies (that can happen for various reasons). Installing supervisor is what I personally would attempt since it doesn't seem to interfere much with standard practice from what I could understand and it doesn't look too complicated. Anyhow, in the meantime I made this script (suggestions welcome, I really am not a sysadmin)

Prerequisites:

• Have a user you can run the processes with. E.g. yourself. I just used www-data ^{See this note for Debian 8}
• Make sure that the log file directory is writable by this user

chown -R /data/mwcache www-data:www-data

File /etc/init.d/mw-serve (yes I just recycled the old mw-serve file)

#! /bin/bash

PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin

case "$1" in
    start)
	su - www-data -c 'nserve >> /data/mwcache/log.txt 2>&1  &'
	su - www-data -c 'mw-qserve >> /data/mwcache/log.txt 2>&1 &'
	su - www-data -c 'nslave --cachedir /data/mwcache/ >> /data/mwcache/log.txt 2>&1 &'
	su - www-data -c 'postman --cachedir /data/mwcache/ >> /data/mwcache/log.txt 2>&1 &'
    ;;
  stop)
	mv /data/mwcache/log.txt /data/mwcache/log.old
	killall nserve
	killall mw-qserve
        killall nslave
        killall postman
    ;;
  force-reload|restart)
    $0 stop
    $0 start
    ;;
  *)
    echo "Usage: /etc/init.d/mw-serve {start|stop}"
    exit 1
    ;;
esac

exit 0

Make it executable

chmod 755 /etc/init.d/mw-serve

Then link it in the appropriate RC directories, most importantly:

cd /etc/rc1.d
ln -s ../init.d/mw-serve K20mw-serve

cd /etc/rc2.d
ln -s ../init.d/mw-serve S20mw-serve

cd /etc/rc3.d
ln -s ../init.d/mw-serve S20mw-serve

Tweaking

• For larger servers you need to multiply .... read the official manual
• For better quality of service, you must run the scripts with a process supervisor software (See the manual).

Tips for creating wiki books

Creating a serious book

It is best just to "hand edit" a stored book. E.g. you may start by adding a category or isolated articles to a book with the collection interface, but once you got most of the articles:

• Save the book
• Then select it from the category:books and then just edit the page in the "normal way". The structure and syntax to respect is demonstrated by the following example:

{{saved_book}}
== My Book ==
=== Example ===
;Foo
:[[First article]]
:[[Second article]]
:[[Third article|This article renamed in the book]]
;Bar
:[[Fourth article]]
:[[Fifth article]]
:[{{fullurl:Sixth article|oldid=20}}Sixth article version:20]
[[Category:Books]]

Install the Mediawiki source extension

If you have articles that include computer source code (XML, ActionScipt or whatever), you also should install the source extension (CodeSyntaxHighlight MediaWiki for formatting computer code.

If you use XML within "pre" tags, the parser may become confused.

Also, the printed Pediapress book will look much prettier. Editing all your wiki pages may represent work, but user reading the on-line pages also will be grateful, colored and indented code is really much more readable !

Pictures and drawings

(1) Readjust some image sizes

(I'll have to be more precise about this, but I'll first need to analyze both a wiki book I got and the generated PDF...)

• Don't use large pictures when smaller ones are readable
• Create screendumps from smallest possible areas if you plan to show hairy details (see also the screen capture tutorial).

(2) Color Printed PDF is mostly grey (affordable Pediapress books certainly are). E.g. when you create drawings or annotate images with text, you should make sure that shades of grey still allow to identify critical elements. A related very difficult issue is writing the text. Avoid writing about "blue" and "green" arrows and "green" or "yellow" dots ...

• To select colors that show on gray printers, you may use a color scheme designer such as colorschemedesigner.com and simulate color vision deficiency with "full colorblindness". Else print your drawing, before you import to wiki.
• A radical solution is to use gray images already upfront in the wiki. This way you are sure to get it somewhat right. Note: "grey" is spelled "gray" in CSS and X11.

Stuff that the parser doesn't handle well

As of mid May 2009 (so this may change)

• Some extensions (like graphwiz) are not supported, see above. There is no solution for this, except not using these.

• Some extensions that are not supported wont' matter, e.g. pageby. But you will have to modify the source code for filtering if you own your own server (see above) or file a request.

•   prints "as is", therefore avoid! Use ":" and "::" etc. to indent lines for example

• Source code (either within "pre" or "source" tags that follow a picture will be printed over the picture. Probable reason. The renderer will reduce a picture and try to wrap text around it. Source code cannot be wrapped or not as well. Workaround: Move the image either 20 lines above or after the code.

Conditional inclusion/exclusion

This section is outdated, needs to be adjusted to using CSS - Daniel K. Schneider (talk) 12:27, 26 August 2013 (CEST)

Warning: This wasn't tested with Pediapress. If you plan to buy a book, use these on the very first pages, then have a look at the preview.

1) Exclude templates

You may exclude any template from the PDF generation you wish by adding them to the Category:Exclude in print. Use with care, since this will filter for all users !

2) Exclude certain specific content

By using Template:Hide in print, certain specific content, such as a few words or an image, can be excluded from printing.

This content will be printed.{{Hide in print|This content will not be printed.}}This content will be printed.

Alternative solution: use the class="noprint" within a div or span tag.

3) Include certain specific content only in print versions The Template:Only in print can be used to insert content that shall only be visible in offline versions.

Example: print this and display it in the browser {{Only in print|this is only in PDFs or printed books visible, not with the browser}} this is visible in the browser and in print as well.

Alternative solution: use the class="hidden" within a div or span tag.

4) Substitute templates

You can create a print version of a template under the name "TEMPLATENAME/Print" with TEMPLATENAME being the name of the original template.

(more to come ....) - Daniel K. Schneider 12:54, 20 May 2009 (UTC)