Automating LibreOffice using Perl and UNO

I’ve recently taken an interest in macros and automation of office software. While at work I contented myself with writing Excel macros in Visual Basic, but for personal use I was more intrigued at the possibility of combining my favourite programming language, Perl, with my office suite of choice, LibreOffice. Online searching turned up some tutorials about automating LibreOffice with Python and Excel with Perl, and also, more importantly, a Perl module that can indeed interface with LibreOffice. Frustratingly though, the docs for that module seem to consist of a single example and a rather unhelpful suggestion to go read the API docs. Nevertheless, I decided to try it out, and now I hope that by documenting my experience I may be of help to someone else who decides to try this.

I began by getting the module from CPAN, which can be done with a simple:

cpan install OpenOffice::UNO

In order to compile it I needed to have the ODK installed. I have a Linux from Scratch system, so I had compiled and installed LibreOffice as per these instructions. Of course, as noted there, I needed to remove the “–disable-odk” switch to install the LibreOffice SDK! Once that was done, I tried to run the single example. I added the option

"-accept=socket,host=localhost,port=8100;urp;StarOffice.ServiceManager"

to my desktop file for launching LibreOffice so that I wouldn’t have to launch it from the command line. I then saved the rest of the example code as a plx file and ran it… and got an error: “Binary URP bridge disposed during call.”. I thought my explorations had come to an untimely end, but a bit more research revealed that this is a problem that Pythonistas also face, and that the solution is simple: prior to running the script, one needs to set an environment variable:

export URE_BOOTSTRAP=vnd.sun.star.pathname:/opt/libreoffice-5.4.0.3/lib/libreoffice/program/fundamentalrc

Note that “/opt/libreoffice-5.4.0.3” is specific to my installation; for many it may be /usr or /usr/local, or else something else in /opt. To make this easier to remember, I added an alias in my ~/.bashrc:

alias odkurebootstrap='export URE_BOOTSTRAP=vnd.sun.star.pathname:/opt/libreoffice-5.4.0.3/lib/libreoffice/program/fundamentalrc'

So now I just have to type “odkurebootstrap” before running a Perl or Python script for driving LibreOffice. (Better yet, one could simply put the “export” line directly into the bashrc, and then not have to type anything.)

After that, I tweaked the example to create a new document rather than open an existing one. This involved changing the line

$sdoc = $desktop->loadComponentFromURL("file:///home/jrandom/test1.sxw",
                                       "_blank", 0, [$pv]);

to:

$sdoc = $desktop->loadComponentFromURL("private:factory/swriter",
                                       "_blank", 0, [$pv]);

To figure out how to do this, I needed to wade through Java examples on the Apache OpenOffice wiki, even though I’m using LibreOffice, which gives you some idea of how painfully undocumented this whole thing is.

Right, so I was able to open a document and/or create a new one. How to save it? Well, eventually I found someone who had forked this Perl module and filled their Git repository with some more helpful examples. I took a look at savedoc.pl and it seemed all right, except for the “filter” bit:

$pv2->Name("FilterName");
$pv2->Value("swriter: StarOffice XML (Writer)");

Evidently, I’m not working with StarOffice, and I’m using ODT files, not SXWs. At first I tried using the literal name of the filter from the “Save As” dialog in my LibreOffice, which (since I’m using the Irish localization) was “Téacs ODF (.odt)”. As expected, this didn’t work. My next idea was to try recording a macro in LibreOffice in which I saved a document. Bingo! Here’s an excerpt from the macro:

args1(1).Name = "FilterName"
args1(1).Value = "writer8"

dispatcher.executeDispatch(document, ".uno:SaveAs", "", 0, args1())

So apparently the filter in LibreOffice for ODT files is just “writer8”. Fair enough!

This macro excerpt brings up another issue: the use of the “dispatcher”. While writing a Perl script (or indeed a Python script or Java program) one will tend to use directly the objects provided by LibreOffice and their methods. However, these macros recorded in Basic instead use a dispatcher to do everything. The reason for this seems to be that the macro recorder just follows menu clicks and whatnot, which use “dispatches” to trigger the document’s “controller” (i.e. containing window) to do things to the document. It disregards what the controller actually does in response to these dispatches, which makes it much less sophisticated than Microsoft’s macro recorder. Presumably it’s also much easier to maintain, which is understandable. Either way, this means that the LibreOffice macro recorder is considerably less helpful than Microsoft’s would be for suggesting how to go about automating the software. Oh well!

Another thing I noticed was that a lot of functions seemed to take references to arrays of com.sun.star.beans.PropertyValue” objects. The two examples I mentioned build up these arrays “by hand” (as do recorded macros, although it’s slightly simpler in Basic) but I felt that this was a bit of a waste of space. Surely it would make much more sense to pass these property-value pairs as hashes. Therefore, I decided to create a function to convert a hash to one of these array references, and another to reverse this conversion:

sub createUnoArgSet {
        # Takes UNO object, plus a hash (or reference thereto)
        # and returns a reference to an array of PropertyValues.
        my $uno = shift;
        die "createUnoArgSet called without reference to UNO object!"
                unless ref($uno) eq "OpenOffice::UNO";

        my %inputhash;
        my @outputlist;

        if (scalar(@_) == 1) {
                # Got a scalar - must be a reference!
                %inputhash = %{$_[0]};
        } else {
                # Got more than one value - assume a hash.
                %inputhash = @_;
        }

        for my $arg (keys %inputhash) {
                my $pv = $uno->createIdlStruct("com.sun.star.beans.PropertyValue");
                $pv->Name($arg);
                $pv->Value($inputhash{$arg});
                push @outputlist, $pv;
        }

        return \@outputlist;
}

sub unwrapUnoArgSet {
        # Takes a reference to an array of PropertyValues
        # and returns a reference to a hash.
        my @pvset = @{$_[0]};
        my %outputhash;

        for my $pv (@pvset) {
                $outputhash{$pv->Name()} = $pv->Value();
        }

        return \%outputhash;
}

I was concerned that the lack of ordering in Perl hashes could cause problems, but when I tested it it seemed to work fine. The other problem is that “createUnoArgSet” needs to take the $uno object (created at the start of the program) as its first argument, before the hash. The “die” line wasn’t always there, resulting in a lot of head-scratching situations in which I had forgotten to pass this object! Perhaps one could instead get away with creating a new OpenOffice::UNO object each time the function is called, but I didn’t try this.

Anyway, at this point, I felt ready to try actually making something. What I had in mind was a script to produce a pretty “trace report”. That is, something that would take in the output of an strace command, such as

strace -v -f -s300 -y -yy -q -tt -o example.txt someprogram

and tabulate it in a document. I also wanted to intersperse this report with screenshots. That is, it could search in a defined place for image files, compare their creation time with the timestamps in the trace output, and then insert them between tables in the report. So, I wanted a program that could do this:

tracereport.plx TraceReport.odt ~/TraceScreens example.txt

Here “TraceReport.odt” is the name of the output report file, “~/TraceScreens” is an optional directory containing the screenshots, and “example.txt” is the actual output from the strace command given above.

Some quick experimentation revealed that ODT output would not be suitable due to the limitation on text table sizes, so I decided to make a script to produce ODS spreadsheets instead. After days of trawling through the API docs (which at the time were for LibreOffice 5.3, but now appear to have been bumped to LibreOffice 6.0!), as well as the Developer section of the Apache OpenOffice wiki, I came up with the non-image part of the report generator:

#!/usr/bin/perl
#tracereport.plx
# An attempt to take strace output and put it into a spreadsheet using LibreOffice Calc...
use warnings;
use strict;
use OpenOffice::UNO;

# Check if we have a spreadsheet name.
if (scalar(@ARGV) < 1) { die("You need to specify a spreadsheet path (absolute!) as an argument!\n"); } my $docname = shift @ARGV; # connect to the OpenOffice.org server my $uno = OpenOffice::UNO->new;
my $cxt = $uno->createInitialComponentContext;
my $sm  = $cxt->getServiceManager;
my $resolver = $sm->createInstanceWithContext
                ("com.sun.star.bridge.UnoUrlResolver", $cxt);
my $rsm = $resolver->resolve
    ("uno:socket,host=localhost,port=8100;urp;StarOffice.ServiceManager");

# get an instance of the Desktop service
my $rc = $rsm->getPropertyValue("DefaultContext");
my $desktop = $rsm->createInstanceWithContext("com.sun.star.frame.Desktop", $rc);
# and of the Dispatcher service
#my $dispatcher = $rsm->createInstanceWithContext("com.sun.star.frame.DispatchHelper", $rc);

# create a name/value pair to be used in opening the spreadsheet
my $open_args = createUnoArgSet($uno, {Hidden => OpenOffice::UNO::Boolean->new(1)});

# open a spreadsheet
my $sdoc = $desktop->loadComponentFromURL("private:factory/scalc",
                                       "_blank", 0, $open_args);

# Start recording trace info.
my @trace_output_lines;
my @unfinished_output_lines;
my $mostparams = 0;

while(my $line = <>) {
	# First extract the PID and timestamp.
	$line =~ s/^(\d*)\s*([\d:]*)\s*//;
	my $pid = $1;
	my $timestamp = $2;

	# Next figure out if it's unfinished.
	if($line =~ s/\s*$//) {
		# It's an unfinished line. Record it, along with its call and PID.
		$line =~ /^([^\(]*)\(/;
		my $call = $1;
		chomp($line);
		push @unfinished_output_lines,
			{call => $call,
			line => $line,
			pid => $pid};
	} else {
		# Not unfinished. Is it resumed?
		if($line =~ s/^< \.\.\.\s(.*)\sresumed>//) {
			# This is the resumption of an unfinished line.
			# Identify its call and PID and match it to one we recorded earlier.
			my $call = $1;
			# Loop through list of unfinished lines.
			my $index = 0;
			$index++ until (($unfinished_output_lines[$index]->{"pid"} eq $pid and
				$unfinished_output_lines[$index]->{"call"} eq $call) or
				$index >= $#unfinished_output_lines); # Prevent infinities!
			# Concatenate previous line with this one.
			$line = $unfinished_output_lines[$index]->{"line"}.$line;
			# Take it away from the unfinished list, since it's finished now!
			splice(@unfinished_output_lines, $index, 1);
		}

		# Next parse the full line.
		if($line =~ /^([^\(]*)\((.*)\)\s*=\s*(-?[\d\?]+.*?)$/) {
			my $call = $1;
			my $params = $2;
			my $result = $3;
			# Split up params into an array.
			# Treat structures as single params.
			my @pararray = parseparams($params);
			# Push a nice structure onto our list of lines.
			push @trace_output_lines,
				{pid => $pid,
				timestamp => $timestamp,
				call => $call,
				params => [@pararray],
				result => $result};
			# Determine the longest list of parameters we have.
			$mostparams = scalar(@pararray) if scalar(@pararray) > $mostparams;
		} else {
			# Unparseable.
			chomp($line);
			push @trace_output_lines,
				{pid => $pid,
				timestamp => $timestamp,
				rawline => $line};
		}
	}
}

# Add in the header line.
unshift @trace_output_lines,
	{pid => "PID",
	timestamp => "Timestamp",
	call => "Call",
	result => "Result",
	params => [map {"Parameter $_"} (1..$mostparams)]};

# Get access to our controller.
my $controller = $sdoc->getCurrentController();
# Freeze the header row...
$controller->freezeAtPosition(0,1);

# Get access to our first (and only) spreadsheet.
my $sheet = $sdoc->getSheets()->getByIndex(0);
$sheet->setName("Trace Report");

# Go through the rows.
for my $lineindex (0..$#trace_output_lines) {
	my %linehash = %{$trace_output_lines[$lineindex]};
	$sheet->getCellByPosition(0,$lineindex)->setString($linehash{"pid"});
	$sheet->getCellByPosition(1,$lineindex)->setString($linehash{"timestamp"});
	# Now, was the line parseable or not?
	if (defined $linehash{"rawline"}) {
		# TODO: Merge remaining cells.
		$sheet->getCellByPosition(2,$lineindex)->setString($linehash{"rawline"});
	} else {
		$sheet->getCellByPosition(2,$lineindex)->setString($linehash{"call"});
		$sheet->getCellByPosition(3,$lineindex)->setString($linehash{"result"});
		my @pararray = @{$linehash{"params"}};
		for my $parindex (0..$#pararray) {
			if($parindex == 1019 and $#pararray > 1019) {
				# Max column index is 1023!
				$sheet->getCellByPosition(4+$parindex,$lineindex)->setString("etc.");
				last;
			}
			last if $parindex > 1019;
			my $param = '';
			defined $pararray[$parindex] and $param = $pararray[$parindex];
			# Eval any quote blocks!
			for my $quoteblock ($param =~ /("[^"]*")/g) {
				my $evaled = eval $quoteblock;
				defined $evaled and $param =~ s/("[^"]*")/$evaled/;
			}
			#print "Attempting to put '$param' in cell.\n";
			$sheet->getCellByPosition(4+$parindex,$lineindex)->setString($param);
		}
	}
}

# Make the columns optimal-width.
my $rightmostcol = 4 + $mostparams;
$rightmostcol = 1023 if $rightmostcol > 1023;
for my $colindex (0..$rightmostcol) {
	$sheet->getColumns()->getByIndex($colindex)->setPropertyValue("OptimalWidth",OpenOffice::UNO::Boolean->new(1));
}

# save the spreadsheet
my $save_args = createUnoArgSet($uno, {Overwrite => OpenOffice::UNO::Boolean->new(1),
					FilterName => "calc8"});
$sdoc->storeAsURL("file://" . $docname, $save_args);

# close the spreadsheet
$sdoc->dispose();

###############
# SUBROUTINES #
###############
sub createUnoArgSet {
	# Takes UNO object, plus a hash (or reference thereto)
	# and returns a reference to an array of PropertyValues.
	my $uno = shift;

	my %inputhash;
	my @outputlist;

	if (scalar(@_) == 1) {
		# Got a scalar - must be a reference!
		%inputhash = %{$_[0]};
	} else {
		# Got more than one value - assume a hash.
		%inputhash = @_;
	}

	for my $arg (keys %inputhash) {
		my $pv = $uno->createIdlStruct("com.sun.star.beans.PropertyValue");
		$pv->Name($arg);
		$pv->Value($inputhash{$arg});
		push @outputlist, $pv;
	}

	return \@outputlist;
}

sub unwrapUnoArgSet {
	# Takes a reference to an array of PropertyValues
	# and returns a reference to a hash.
	my @pvset = @{$_[0]};
	my %outputhash;

	for my $pv (@pvset) {
		$outputhash{$pv->Name()} = $pv->Value();
	}

	return \%outputhash;
}

sub parseparams {
	# Split parameters from calls, respecting structures 
	# enclosed by brackets/parentheses/braces.
	# Mostly based on https://stackoverflow.com/a/5052668
	my ($string) = @_;
	my @fields;

	my @comma_separated = split(/,\s*/, $string);

	my @to_be_joined;
	my $depth = 0;
	foreach my $field (@comma_separated) {
		my @brackets = $field =~ /(\(|\)|\[|\]|\{|\})/g;
		foreach (@brackets) {
			$depth++ if /\(|\[|\{/;
			$depth-- if /\)|\]|\}/;
		}

		if ($depth == 0) {
			push @fields, join(", ", @to_be_joined, $field);
			@to_be_joined = ();
		} else {
			push @to_be_joined, $field;
		}
	}

	return @fields;
}

Note that this code is unfinished, not just with respect to the insertion of images. There are other problems with it, such as the lack of sanitation of quoteblocks passed to eval in the parsing section. It also can’t deal with timestamps that include microseconds, which are the output given by the example strace -tt command given earlier.

I think this gives the gist though. First it connects to LibreOffice and opens up a new spreadsheet, just to make sure that the setup is functional. The spreadsheet is “Hidden”, so the user can’t see what’s going on. This (I believe) reduces the overhead since stuff doesn’t have to be drawn and updated on the screen. Either way, I had “Hidden” set to false during testing so I could see exactly what was going on.

Anyway, the next step is to start reading trace lines from stdin and parsing them. This isn’t really relevant to LibreOffice/UNO, so I won’t dwell on it. It takes the data and organizes it into an array of hashes. I probably haven’t done it the best way!

Once this is done, the really interesting bit starts. We get an object representing the document’s “controller” which is a(n invisible) window. We use it to freeze the top row so the headers are always visible in the resulting spreadsheet/report.

We then get an object for the actual spreadsheet itself (the single “tab” of the document). We rename the sheet to “Trace Report” (rather than “Sheet1”) and then start filling in its cells, row by row. It’s fairly straightforward really. We use the fact that the spreadsheet object implements XCellRange to access its cells using the getCellByPosition method. Each cell object implements XTextRange, meaning we can use setString to insert text directly into the cell. The aforementioned eval is there to turn octal/hex representations of special characters in the strace output into the characters themselves. As it’s written here though, it’s not really fool-proof. We also make sure we don’t go past the 1024th column, since that is the maximum number of columns in a Calc spreadsheet at the moment; attempting to go past it would crash the script with an ignominous “index out of bounds”.

Once everything is filled in, we go back to the cosmetic, by going through the columns and making sure they’re optimal-width, to improve readability. Again, we make sure not to go past the 1024th#! We use the getColumns method of XColumnRowRange to get an object representing the set of columns in the spreadsheet. This object itself implements XIndexAccess, so we can call getByIndex to access a particular column. The column objects have properties (by virtue of implementing XPropertySet), one of which is OptimalWidth, which we can set to a boolean value in the same way as we set the Hidden property of the document when we opened (created) it.

Finally the spreadsheet document is saved, using the “calc8” filter, and then closed using the dispose method. Not bad! We now have something that takes output from strace and summarizes it in a spreadsheet. (Your mileage may vary as to how useful you think that actually is!)

Right, so there are still a few things missing. It doesn’t do the image thing at all, and it would also be good if the “unparseable” lines that are just dumped into the spreadsheet “as-is” could be put in merged cells.

Some further perusal of the wiki led me to the conclusion that trying to merge the cells using the “proper” method with UNO objects and methods would be way too convoluted, so instead I opted for the “dispatcher” method as used by a recorded macro. I replaced the TODO line with:

$controller->select($sheet->getCellRangeByPosition(2,$lineindex,$rightmostcol,$lineindex));
$dispatcher->executeDispatch($controller,".uno:MergeCells","",0,[]);

In order for this to work, I also had to uncomment the “my $dispatcher” line near the top of the script, and also move the lines defining $rightmostcol higher up in the script, before the “for my $lineindex” loop. You’ll notice that I was able to select a cell range without using a dispatch, with the help of $controller. However, the actual merging is done by a dispatch, which executes a command in LibreOffice that merges the selected cells.

Next I had to get the images working. Naturally, this involved adding a lot of non-LibreOffice-related code, but I won’t concentrate on that for now. Suffice it to say, my array of hashes could now contain “lines” specifying only an image path. I dealt with these by adding a block to the start of the “for my $lineindex” loop, just after the declaration of %linehash:

if (defined $linehash{"image"}) {
	# This "line" is a pic.
	# Go to the cell we want to put it at.
	$controller->select($sheet->getCellByPosition(0,$lineindex));
	# Now execute the dispatch to insert the picture.
	# First it wants to know the filter for whatever reason.
	my $filter;
	if ($linehash{"image"} =~ /png$/) {
		$filter = "PNG - Portable Network Graphic";
	} elsif ($linehash{"image"} =~ /gif$/) {
		$filter = "GIF - Graphics Interchange Format";
	} elsif ($linehash{"image"} =~ /jpg$/) {
		$filter = "JPEG - Joint Photographic Experts Group";
	}
	$dispatcher->executeDispatch($controller->getFrame(),".uno:InsertGraphic","",0,createUnoArgSet($uno, {
				FileName => "file://".$linehash{"image"},
				FilterName => $filter,
				AsLink => OpenOffice::UNO::Boolean->new(0)}));
	# Now, none of the rest of this stuff is relevant.
	next;
}

You’ll notice I used the dispatcher again. This is because inserting an image into a spreadsheet is something for which I just couldn’t seem to find any documentation. I know it’s probably not something people do that often, but there seemed to be absolutely nothing that could help me with it, either in the API docs or on the OpenOffice wiki.

Actually, this particular use of the dispatcher was one of the head-scratching incidents I alluded to earlier. I forgot to pass $uno to createUnoArgSet, so it was getting only one argument (my hash reference) and therefore producing an empty array from an empty hash. The result was that the dispatch wasn’t getting any arguments passed to it, so the “Insert Picture” dialog kept coming up asking me to choose an image! Eventually I figured it out, but that wasted a lot of my time…

This is fine, but it doesn’t adjust the cell height, so the big screenshot covers a load of output lines below it. Adjusting cell height seemed like it should be a straightforward enough task, but no matter what I tried it just kept ignoring me. At first I just wanted to try setting the height to 4.5 cm, roughly ten times the normal, as a proof of concept. This is something like what I had:

# DOESN'T WORK:
my $row = $sheet->getRows()->getByIndex($lineindex);
$row->setPropertyValue("Height",4500); # Dimensions are in hundredths of a millimetre.

These lines of code seemed completely ineffectual, a suspicion confirmed by something along the lines of

# Diagnostic:
print $row->getPropertyValue("Height");

which gave me back the default value. Attempting the same thing with an interactive Python console gave no issues, confirming that the problem was with Perl, or rather with the very-poorly-documented Perl module I use. For the sake of comparing like with like I loaded up an interactive Perl console and tried to mess around with the UNO interface.

I had the feeling that the fact that the integer was being ignored was somehow related to the fact that booleans need to be passed in an “OpenOffice::UNO::Boolean” object. I eventually realized I could put these two fairly simple lines into my console:

use OpenOffice::UNO;
for my $key (keys %OpenOffice::UNO::) { print "$key\n"; }

This gave me a list of symbols available in the OpenOffice::UNO package, which was:

createComponentContext
Struct::
VERSION
dl_load_flags
Interface::
import
Int32::
bootstrap
Any::
createInitialComponentContext
EXPORT
Exception::
DESTROY
Boolean::
ISA
createIdlStruct
Int64::
new

Sure enough, there’s an “Int32” type and an “Int64” type. Now you see what I mean about this Perl module having terrible docs!

To that end, the next thing I tried was to modify the non-functional code above to something like:

# WORKS:
my $row = $sheet->getRows()->getByIndex($lineindex);
$row->setPropertyValue("Height",OpenOffice::UNO::Int32->new(4500)); # Dimensions are in hundredths of a millimetre.

What do you know, it worked! As such, I created a new wrapper function, and modified my createUnoArgSet function to invoke it:

sub createUnoObj {
	# Take a number or boolean and turn it into an appropriate
	# Uno object. Useful for setting properties and argsets.
	my $value = shift;
	# First of all, is it already an object?
	if(ref($value) =~ /^OpenOffice::UNO/) {
		return $value; # Send it on its merry way!
	}

	# Okay, let's see what to do with it...
	unless ($value & ~$value) {
		# Numeric.
		# Is it a float?
		if(int($value) - $value) {
			# Just send it...
			return $value;
		} elsif (abs($value) < 2**16) {
			# It's an int...
			return OpenOffice::UNO::Int32->new($value);
		} else { 
			# Won't fit in an Int32.
			return OpenOffice::UNO::Int64->new($value);
		}
	} else {
		# String...
		# Okay, is it a Boolean?
		if($value eq "True") {
			return OpenOffice::UNO::Boolean->new(1);
		} elsif($value eq "False") {
			return OpenOffice::UNO::Boolean->new(0);
		} else {
			# Okay, it's just a string.
			return $value;
		}
	}
}

sub createUnoArgSet {
	# Takes UNO object, plus a hash (or reference thereto)
	# and returns a reference to an array of PropertyValues.
	my $uno = shift;
	die "createUnoArgSet called without reference to UNO object!"
		unless ref($uno) eq "OpenOffice::UNO";

	my %inputhash;
	my @outputlist;

	if (scalar(@_) == 1) {
		# Got a scalar - must be a reference!
		%inputhash = %{$_[0]};
	} else {
		# Got more than one value - assume a hash.
		%inputhash = @_;
	}

	for my $arg (keys %inputhash) {
		my $pv = $uno->createIdlStruct("com.sun.star.beans.PropertyValue");
		$pv->Name($arg);
		$pv->Value(createUnoObj($inputhash{$arg}));
		push @outputlist, $pv;
	}

	return \@outputlist;
}

I created another new function to wrap setPropertyValue:

sub setUnoProps {
	# Wrapper of setPropertyValue that sends nice UNO objects.
	my $unoObj = shift;
	die "setUnoProps called without reference to an UNO struct/interface/any!"
		unless ref($uno) =~ /^OpenOffice::UNO/;

	my %inputhash;

	if (scalar(@_) == 1) {
		# Got a scalar - must be a reference!
		%inputhash = %{$_[0]};
	} else {
		# Got more than one value - assume a hash.
		%inputhash = @_;
	}

	for my $arg (keys %inputhash) {
		$unoObj->setPropertyValue($arg,createUnoObj($inputhash{$arg}));
	}
}

Finally I needed to marry these two things: I could insert an image at a cell, and change the height of the row containing that cell; how to change the height of the row to the height of the image? The answer came when I somehow realized that each spreadsheet has a LibreOffice Draw page superimposed on it, which contains all the “shapes”, which include drawn shapes, but also images.

Therefore, I added a line to get access to this Draw page, just before the “for my $lineindex” loop:

my $drawpage = $sheet->getDrawPage();

Strangely enough, the Draw page object itself implements XIndexAccess, so a shape (in this case an image) is accessed by calling getByIndex directly on $drawpage. Now, naturally, I want to adjust the height of a row immediately after inserting the image, at which point it is the last object in this list of shapes (because it’s the most recently inserted!). Therefore, the height can be obtained using a one-liner:

my $graphicheight = $drawpage->getByIndex($drawpage->getCount()-1)->getSize()->Height;

Funnily enough, it’s possible to just call Height on the object representing the image’s size, which I have to do because that object doesn’t implement XPropertySet. Setting the row height is then a simple matter of:

my $row = $sheet->getRows()->getByIndex($lineindex);
setUnoProps($row,{Height=>$graphicheight});

I took this for a test drive, running a trace on Rayman 2, which I ran through Wine. The script ran for ages, then crashed because it hit the row limit, which is a “mere” 220! Therefore, I modified the script to truncate the report if it hits this limit. The final script looks like this, with a bit more polish:

#!/usr/bin/perl
#tracereport.plx
# An attempt to take strace output and put it into a spreadsheet using LibreOffice Calc...
use warnings;
use strict;
use OpenOffice::UNO;

# Check if we have a spreadsheet name.
if (scalar(@ARGV) < 1) {
	die("You need to specify a spreadsheet path (absolute!) as an argument!\n");
}
my $docname = shift @ARGV;
# Next look for a directory of images.
my $imgdir;
if (scalar(@ARGV) and -d($ARGV[0])) {
	$imgdir = shift @ARGV;
} else {
	use Cwd;
	$imgdir = cwd();
}

my @pics;
my @pictimestamps;

# Check for images in our directory.
open FILELIST, "ls -gt --time-style=\"+%X.%N\" $imgdir/{*.png,*.gif,*.jpg} |";
while() {
	my ($perms,$num,$group,$size,$timestamp,$filepath) =
		/^([^\s]*)\s*([^\s]*)\s*([^\s]*)\s*([^\s]*)\s*([^\s]*)\s*(.*)$/;
	# ls sorts by newest first, so unshift to get oldest first.
	unshift(@pictimestamps,makeseconds($timestamp));
	unshift(@pics,$filepath);
}
close FILELIST;

# connect to the OpenOffice.org server
my $uno = OpenOffice::UNO->new;
my $cxt = $uno->createInitialComponentContext;
my $sm  = $cxt->getServiceManager;
my $resolver = $sm->createInstanceWithContext("com.sun.star.bridge.UnoUrlResolver", $cxt);
my $rsm = $resolver->resolve("uno:socket,host=localhost,port=8100;urp;StarOffice.ServiceManager");

# get an instance of the Desktop service
my $rc = $rsm->getPropertyValue("DefaultContext");
my $desktop = $rsm->createInstanceWithContext("com.sun.star.frame.Desktop", $rc);
# and of the Dispatcher service
my $dispatcher = $rsm->createInstanceWithContext("com.sun.star.frame.DispatchHelper", $rc);

# create a name/value pair to be used in opening the spreadsheet
my $open_args = createUnoArgSet($uno, {Hidden => "False"});

# open a spreadsheet
my $sdoc = $desktop->loadComponentFromURL("private:factory/scalc","_blank", 0, $open_args);

# Start recording trace info.
my @trace_output_lines;
my @unfinished_output_lines;
my $mostparams = 0;

while(my $line = <>) {
	# First extract the PID and timestamp.
	$line =~ s/^(\d*)\s*([\d:\.]*)\s*//;
	my $pid = $1;
	my $timestamp = $2;

	# Next figure out if it's unfinished.
	if($line =~ s/\s*<unfinished\s\.\.\.>$//) {
		# It's an unfinished line. Record it, along with its call and PID.
		$line =~ /^([^\(]*)\(/;
		my $call = $1;
		chomp($line);
		push @unfinished_output_lines,
			{call => $call,
			line => $line,
			pid => $pid};
	} else {
		# Not unfinished. Is it resumed?
		if($line =~ s/^<\.\.\.\s(.*)\sresumed>//) {
			# This is the resumption of an unfinished line.
			# Identify its call and PID and match it to one we recorded earlier.
			my $call = $1;
			# Loop through list of unfinished lines.
			my $index = 0;
			$index++ until (($unfinished_output_lines[$index]->{"pid"} eq $pid and
				$unfinished_output_lines[$index]->{"call"} eq $call) or
				$index >= $#unfinished_output_lines); # Prevent infinities!
			# Concatenate previous line with this one.
			$line = $unfinished_output_lines[$index]->{"line"}.$line;
			# Take it away from the unfinished list, since it's finished now!
			splice(@unfinished_output_lines, $index, 1);
		}
		
		# Next parse the full line.
		my $parsedline;
		if($line =~ /^([^\(]*)\((.*)\)\s*=\s*(-?[\d\?]+.*?)$/) {
			my $call = $1;
			my $params = $2;
			my $result = $3;
			# Split up params into an array.
			# Treat structures as single params.
			my @pararray = parseparams($params);
			# Put together a nice structure to push onto our list of lines.
			$parsedline =
				{pid => $pid,
				timestamp => $timestamp,
				seconds => makeseconds($timestamp),
				call => $call,
				params => [@pararray],
				result => $result};
			# Determine the longest list of parameters we have.
			$mostparams = scalar(@pararray) if scalar(@pararray) > $mostparams;
		} else {
			# Unparseable.
			chomp($line);
			$parsedline =
				{pid => $pid,
				timestamp => $timestamp,
				seconds => makeseconds($timestamp),
				rawline => $line};
		}
		
		# Next see if there are any images to put in!
		while (scalar @pics and ($pictimestamps[0] < $parsedline->{"seconds"})) {
			push @trace_output_lines,
				{image => $pics[0]};
			# Clean out the arrays as we're going, so we can always check entry zero.
			shift @pics;
			shift @pictimestamps;
		}
		# Now stick in the actual trace output line.
		push @trace_output_lines, $parsedline;
	}
}

# Add in the header line.
unshift @trace_output_lines,
	{pid => "PID",
	timestamp => "Timestamp",
	call => "Call",
	result => "Result",
	params => [map {"Parameter $_"} (1..$mostparams)]};

# Maximum number of rows…
warn "Too many lines in this trace! The report will be truncated!" if (@trace_output_lines > 2**20);

# Get access to our controller.
my $controller = $sdoc->getCurrentController();
# Freeze the header row...
$controller->freezeAtPosition(0,1);

# Get access to our first (and only) spreadsheet.
my $sheet = $sdoc->getSheets()->getByIndex(0);
$sheet->setName("Trace Report");
# Get access to the draw page too, to manage the "shapes" (images).
my $drawpage = $sheet->getDrawPage();

# Go through the rows.
my $rightmostcol = 4 + $mostparams;
$rightmostcol = 1023 if $rightmostcol > 1023;
for my $lineindex (0..$#trace_output_lines) {
	# Have we reached the end of the spreadsheet
	if ($lineindex == (2**20 - 1) and $#trace_output_lines > $lineindex) {
		# Explain what happened…
		$sheet->getCellByPosition(0,$lineindex)->setString("Report truncated!");
		$controller->select($sheet->getCellRangeByPosition(0,$lineindex,$rightmostcol,$lineindex));
		$dispatcher->executeDispatch($controller,".uno:MergeCells","",0,[]);
		# We're done here.
		last;
	}

	# Otherwise tutto va bene.
	my %linehash = %{$trace_output_lines[$lineindex]};
	if (defined $linehash{"image"}) {
		# This "line" is a pic.
		# Go to the cell we want to put it at.
		$controller->select($sheet->getCellByPosition(0,$lineindex));
		# Now execute the dispatch to insert the picture.
		# First it wants to know the filter for whatever reason.
		my $filter;
		if ($linehash{"image"} =~ /png$/) {
			$filter = "PNG - Portable Network Graphic";
		} elsif ($linehash{"image"} =~ /gif$/) {
			$filter = "GIF - Graphics Interchange Format";
		} elsif ($linehash{"image"} =~ /jpg$/) {
			$filter = "JPEG - Joint Photographic Experts Group";
		}
		$dispatcher->executeDispatch($controller->getFrame(),".uno:InsertGraphic","",0,createUnoArgSet($uno, {
					FileName => "file://".$linehash{"image"},
					FilterName => $filter,
					AsLink => "False"}));
		# Grab the newest-inserted graphic.
		my $graphicheight = $drawpage->getByIndex($drawpage->getCount()-1)->getSize()->Height;
		# Now get the row and adjust its height.
		my $row = $sheet->getRows()->getByIndex($lineindex);
		setUnoProps($row,{Height=>$graphicheight});
		# Now, none of the rest of this stuff is relevant.
		next;
	}
	$sheet->getCellByPosition(0,$lineindex)->setString($linehash{"pid"});
	$sheet->getCellByPosition(1,$lineindex)->setString($linehash{"timestamp"});
	# Now, was the line parseable or not?
	if (defined $linehash{"rawline"}) {
		# Just throw in the entire line.
		$sheet->getCellByPosition(2,$lineindex)->setString($linehash{"rawline"});
		# Now merge remaining cells. Need to dispatch this too it seems.
		$controller->select($sheet->getCellRangeByPosition(2,$lineindex,$rightmostcol,$lineindex));
		$dispatcher->executeDispatch($controller,".uno:MergeCells","",0,[]);
	} else {
		$sheet->getCellByPosition(2,$lineindex)->setString($linehash{"call"});
		$sheet->getCellByPosition(3,$lineindex)->setString($linehash{"result"});
		my @pararray = @{$linehash{"params"}};
		for my $parindex (0..$#pararray) {
			if($parindex == 1019 and $#pararray > 1019) {
				# Max column index is 1023!
				$sheet->getCellByPosition(4+$parindex,$lineindex)->setString("etc.");
				last;
			}
			last if $parindex > 1019;
			my $param = '';
			defined $pararray[$parindex] and $param = $pararray[$parindex];
			# Eval any quote blocks!
			for my $quoteblock ($param =~ /("[^"]*")/g) {
				# Get rid of sigils that confuse Perl.
				$quoteblock =~ s/\$/\\\$/g;
				$quoteblock =~ s/\@/\\\@/g;
				$quoteblock =~ s/\%/\\\%/g;
				# Evaluate
				my $evaled = eval $quoteblock;
				$param =~ s/("[^"]*")/$evaled/ if defined $evaled;
			}
			$sheet->getCellByPosition(4+$parindex,$lineindex)->setString($param);
		}
	}
}

# Make the columns optimal-width.
for my $colindex (0..$rightmostcol) {
	setUnoProps($sheet->getColumns()->getByIndex($colindex),{OptimalWidth=>"True"});
}

# save the spreadsheet
my $save_args = createUnoArgSet($uno, {Overwrite => "True",
					FilterName => "calc8"});
$sdoc->storeAsURL("file://" . $docname, $save_args);

# close the spreadsheet
$sdoc->dispose();

###############
# SUBROUTINES #
###############
sub makeseconds {
	my $timestamp = shift;
	my ($hours, $minutes, $floatseconds) = ($timestamp =~ /(\d\d):(\d\d):(.*)/);
	return $hours * 3600 + $minutes * 60 + $floatseconds;
}

sub createUnoObj {
	# Take a number or boolean and turn it into an appropriate
	# Uno object. Useful for setting properties and argsets.
	my $value = shift;
	# First of all, is it already an object?
	if(ref($value) =~ /^OpenOffice::UNO/) {
		return $value; # Send it on its merry way!
	}

	# Okay, let's see what to do with it...
	unless ($value & ~$value) {
		# Numeric.
		# Is it a float?
		if(int($value) - $value) {
			# Just send it...
			return $value;
		} elsif (abs($value) < 2**16) {
			# It's an int...
			return OpenOffice::UNO::Int32->new($value);
		} else { 
			# Won't fit in an Int32.
			return OpenOffice::UNO::Int64->new($value);
		}
	} else {
		# String...
		# Okay, is it a Boolean?
		if($value eq "True") {
			return OpenOffice::UNO::Boolean->new(1);
		} elsif($value eq "False") {
			return OpenOffice::UNO::Boolean->new(0);
		} else {
			# Okay, it's just a string.
			return $value;
		}
	}
}

sub createUnoArgSet {
	# Takes UNO object, plus a hash (or reference thereto)
	# and returns a reference to an array of PropertyValues.
	my $uno = shift;
	die "createUnoArgSet called without reference to UNO object!"
		unless ref($uno) eq "OpenOffice::UNO";

	my %inputhash;
	my @outputlist;

	if (scalar(@_) == 1) {
		# Got a scalar - must be a reference!
		%inputhash = %{$_[0]};
	} else {
		# Got more than one value - assume a hash.
		%inputhash = @_;
	}

	for my $arg (keys %inputhash) {
		my $pv = $uno->createIdlStruct("com.sun.star.beans.PropertyValue");
		$pv->Name($arg);
		$pv->Value(createUnoObj($inputhash{$arg}));
		push @outputlist, $pv;
	}

	return \@outputlist;
}

sub unwrapUnoArgSet {
	# Takes a reference to an array of PropertyValues
	# and returns a reference to a hash.
	my @pvset = @{$_[0]};
	my %outputhash;

	for my $pv (@pvset) {
		$outputhash{$pv->Name()} = $pv->Value();
	}

	return \%outputhash;
}

sub setUnoProps {
	# Wrapper of setPropertyValue that sends nice UNO objects.
	my $unoObj = shift;
	die "setUnoProps called without reference to an UNO struct/interface/any!"
		unless ref($uno) =~ /^OpenOffice::UNO/;

	my %inputhash;

	if (scalar(@_) == 1) {
		# Got a scalar - must be a reference!
		%inputhash = %{$_[0]};
	} else {
		# Got more than one value - assume a hash.
		%inputhash = @_;
	}

	for my $arg (keys %inputhash) {
		$unoObj->setPropertyValue($arg,createUnoObj($inputhash{$arg}));
	}
}

sub parseparams {
	# Split parameters from calls, respecting structures 
	# enclosed by brackets/parentheses/braces.
	# Mostly based on https://stackoverflow.com/a/5052668
	my ($string) = @_;
	my @fields;

	my @comma_separated = split(/,\s*/, $string);

	my @to_be_joined;
	my $depth = 0;
	foreach my $field (@comma_separated) {
		my @brackets = $field =~ /(\(|\)|\[|\]|\{|\})/g;
		foreach (@brackets) {
			$depth++ if /\(|\[|\{/;
			$depth-- if /\)|\]|\}/;
		}

		if ($depth == 0) {
			push @fields, join(", ", @to_be_joined, $field);
			@to_be_joined = ();
		} else {
			push @to_be_joined, $field;
		}
	}

	return @fields;
}

Now, a truncated trace report on Rayman 2 probably isn’t all that useful. To prevent the truncation, I could find some way of sanitizing the trace before running the script, e.g. by grepping it to get just the “open”s, “read”s and “close”s. Either way, the idea’s well over a year old, but I only got around to implementing it recently. In the meantime, other means of reverse-engineering this game have come on leaps and bounds, so any report that this thing generates probably won’t tell me anything that others haven’t found already. Ah well!

The point is, I rather enjoyed making this, even if it was frustratingly head-scratching sometimes. I hope this can be of help to someone else out there too! For those interested in looking at the code without awkwardly copy-pasting it from this post, I’ve uploaded it (GPG signature – my key fingerprint is FAFA F12C 4440 460A 89D0 A67F 8D31 13F7 D36F 833C).

Version 1.1.1 of the Rayman 2 Irish translation

I am pleased to announce a “point release” of my Irish translation of Rayman 2. This fixes a few more cases of dubious wording, but may have a few display issues.

You can get it at https://www.vigovproductions.net/R2Gaeilge_ponc.zip. You can check the digital signature, which is available at https://www.vigovproductions.net/R2Gaeilge_ponc.zip.sig. (The PGP key used is 0x8D3113F7D36F833C – full fingerprint: FAFA F12C 4440 460A 89D0
A67F 8D31 13F7 D36F 833C – and please don’t use short key IDs!)

Also, if you do play the PC version of Rayman 2 on a modern setup, it might interest you to know that other modders have succeeded in patching it to run in widescreen.

As for that post I said I was going to put up about English spelling, well, it’s still in the pipeline. It has been on the backburner for several months though, and I may have to rewrite it from scratch.

Version 1.1 of the Rayman 2 Irish translation

UPDATE: Version 1.1.1 has been released. Please download that version instead.

I’m pleased to announce that I am releasing a new version of my Irish translation of Rayman 2. Having played through the game again recently I realized that such an update was overdue. You can download it from https://www.vigovproductions.net/R2Gaeilge_nua.zip.

What has been changed:

  • Numerous grammatical errors fixed.
  • Numerous sentences made clearer.
  • Poorly-translated race level names changed to actual cultural references.
  • Fixed typo in README.
  • And, most embarrassingly, fixed this little oversight:
Pour te déplacer sur la prune, tire dans le sens opposé à ta direction.
A little oversight!

Status of Subatomic Particle Simulator: PARKED

Well, there have been occasional minor updates on the status of the VIGoV Subatomic Particle simulator since it was exhibited in January 2014, but no real progress, despite optimistic sentiments voiced in the first half of that year. Since then, I have gone on to do first-year physics in university, including a little Quantum Mechanics. This opened my eyes to just how little of an idea I had of what I was doing when I wrote that code. Examining it again, I’m even less sure of what I was trying to do in certain parts.

In short, I don’t believe it to be salvageable. It was a huge project for someone who had never done university-level physics, and, while it was an educational experience, with some interesting (and sometimes impressive!) results, it can’t really be taken any further. The basic ways in which the simulation is carried out appear to be very flawed, but I don’t know how to fix them, despite what I may have told myself in the past. That is why I am suspending development for a couple of years.

Over the next few years, I will be learning about Quantum Mechanics in university. Then I may be able to continue the project, or at least rewrite some sections of it from scratch, on a sounder footing. Until then, I am frankly afraid to touch the code.

So, while I had fun programming it and learned quite a lot, I feel it best to park the Subatomic Particle Simulator indefinitely.

Rayman 2 i nGaeilge!

UPDATE: Version 1.1.1 has been released. Please download that version instead.

Well, I’ve finally got some good old-fashioned game modding to report here. I have finished my translation of Rayman 2 from French into Irish, done with the help of the tool sna_nochar, created by MixerX and distributed on the Rayman Pirate-Community. As a sample of what has been done, here is the first ten minutes or so of the very last β-test of the translation:

I did miss one Lum on purpose, just to test that cutscene (I had previously tested the cutscene where one gets all 5 Lums on the first try). Then, after making this video this morning, I ran through the rest of the game (skipping some optional bits with no dialogue!) to find any other bugs. I am pleased to announce that the translation is now ready, and available for download from this server:

http://www.vigovproductions.net/R2Gaeilge.zip

The installation instructions are fairly simple, they are written in the Readme.txt file in the above ZIP, in both English and Irish.

By the way, I used VMWare to record the video. I realised it was easier to test my translation in a virtual machine, since I could switch back and forth to a text editor to make changes, without crashing the game. But running the game in a virtual machine caused the physics engine to run too fast at times, which made gameplay really awkward…

That said, the camera glitch seen in the video happens even without a VM, and seems to be caused by the camera reaching the island too early in the cutscene. I guess my rig’s just too powerful!

My chimera Debian system

Well, it’s been a long time! Since my last post, I’ve discovered that there should have been a fifth “D’oh!” in there – turns out that the way the PCIe slots are arranged on my motherboard means that it’s impossible to pass through one graphics card without the other. That probably explains the blank screen I got when booting into the hypervisor. It all boils down to the fact that I should have bought a higher-end motherboard.

But none of that matters one jot anymore, I’ve moved onto other matters! The hoops through which I tried, and failed, to jump in the last post, were mainly so I could play Rayman Legends. I did wind up being able to play this via Wine, although it was very slow. For a fast-paced game like this, the atmosphere was destroyed. Still, I made do, playing the daily and weekly challenges occasionally.

This all changed when I discovered that VMWare Player exists in a freeware version, and that it gives very good 3D acceleration, even under Linux hosts. Though I had to install various proprietary drivers* to make this work reliably, I was happy. I got Windows Vista up and running in a VMWare virtual machine, and Rayman Origins and Legends both work perfectly under it. Thank you VMWare!

Anyway, since then, I decided to sod the whole Linux Mint Debian Edition thing, especially since they decided to switch their base from testing to stable. I changed my sources list to point directly to Debian Jessie repositories. Since Jessie was frozen last November, this didn’t cause me any problems – until Jessie went stable last month! I proceeded to point my package manager to the “Stretch” repositories and installed a plethora of updates.

I had already started compiling my own kernels some time ago, in order to be ahead of the curve (I’m on Linux 4.0 at the moment). I had to modify the sources of some of the VMWare modules in order to make them compile against the newer kernel. So when VMWare asked me to update a few weeks ago, I expected that these patches would have gone in upstream. I was wrong. The update actually overwrote my patched source with older versions, causing me to have to search out the patches again (this one and this one, if you’re interested). This annoyed me, since Linux 4.0 is now actually the latest stable kernel, so one would expect professionally-developed software to work with it out of the box.

What annoyed me even more is that VMWare Player started crashing on startup after I pulled in the above-mentioned plethora of updates from the Stretch repositories. It became apparent that there was a certain package which I needed to downgrade to make it work with VMWare. In order to downgrade it, I had to add the oldstable (“Wheezy”) repositories to my sources list, in addition to the “Stretch” ones. It was then a simple matter of heading into the package manager and selecting the older version of libgtkmm. This made VMWare work… for a while.

I got one session of Rayman Legends played, then it started crashing again. More Googling revealed that there were issues with libcurl. At first I couldn’t believe this to be the cause, since that post was from 2013! But after a week of being unable to find any other possible causes, I decided to try also downgrading libcurl to oldstable. In order to do this, I also needed to downgrade a number of GNU R packages that depend on it, which I installed some time ago. For some reason, the package manager couldn’t seem to figure out that it was possible to downgrade them all at once, so I had to mark them for downgrading separately, which was tedious. Still, I did it, and downgraded libcurl. And, surprise surprise, VMWare worked! Seemingly even the Jessie version of Curl is quite old…

So, I now have something of a chimera Debian system:

  • It’s basically a Stretch (i.e. testing) system.
  • But there are 21 packages in it from oldstable; these are libgtkmm, libcurl, and several R packages.
  • I also compile my own kernel, so that is a package that comes from neither the testing nor oldstable repositories.
  • And there are several relics of Mint Debian Edition still lurking in the system files – GRUB identifies the system as “LinuxMint GNU/Linux” for example.

So this is fine, but I am annoyed that I have to use any oldstable packages. VMWare in fact comes with its own versions of these packages, but is unfortunately programmed to prefer system ones, even if they make it crash. Again, the current version was released very recently, and I think it’s reasonable to expect that it would at least make provisions for systems that use packages that are too new for it (e.g. version check, fallback on the libraries with which it ships).

~It’s been ages since I last updated this blog. Several things have happened. Notably I finally removed GSL from the Subatomic Particle Simulator, disentangling the legal issues. Unfortunately the inbuilt C++ version of the Gamma Function, that I now use instead, had not yet been implemented by Microsoft in the version of the Visual Studio compiler preferred by Valve. In other words, the Simulator won’t compile on Windows for the moment. I suddenly find myself with more free time, maybe I should tackle that now…

*I’ve been through a few GPUs since by the way – I wish I’d updated this blog! Suffice it to say I have a GeForce GTX 980 now!

My experiences with XEN VGA passthrough

The following post gets quite technical, and employs a bit of jargon. You can read the whole anecdote if you want, or just scroll down to the bottom if you want some advice related to Linux Mint Debian Edition, XEN and VGA Passthrough.

I recently built myself a new computer, and after jumping through many hoops (involving Intel’s Haswell Refresh I believe – seems trying a “Devil’s Canyon” processor with an 8-series chipset isn’t a good idea – D’oh #1!), I got a machine up and running with Linux Mint Debian Edition 64-bit. There is an nVidia card in one PCIe slot, an AMD card in another PCIe slot, and an integrated GPU on my Core i7 that is disabled in the BIOS. Once I had this all set up, I thought it would be child’s play (well, not really, but quite simple) to follow this tutorial and get Windows Vista up and running, and using my AMD card, while leaving the nVidia card for Mint.

Boy, was I wrong! At first everything seemed to be going smoothly, but I had a niggling feeling that it was going too well – this was, after all, a tutorial for normal Mint, not the Debian Edition. My first stumbling block came at step 17, which didn’t give the output hoped for. However, I assumed that this was because I skipped steps 9 and 10, opting for Synergy instead. I forged on, and ended up unable to pass through any devices to the VM – D’oh #2!

At this point, I had Vista up and running, able to control it via VNC, but with no VGA passthrough. Therefore, I backtracked, and I’m not sure what I did, but I ended up with a “no such file or directory” error upon attempting to start the VM, with the unhelpful question “Is xend running?”. Seemingly I could no longer use the XM toolstack for whatever reason. Therefore, I switched to XL with the command:

sudo sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xl"/' /etc/default/xen

Of course, now problems with QEMU versions started to rear their ugly heads. XEN couldn’t find the right QEMU binaries anymore, and following the instructions wasn’t enough, because I was on Debian rather than Ubuntu. My suspicions were correct after all! I was eventually able to start the VM with the following config file:

firmware_override = '/usr/lib/xen-default/boot/hvmloader'
builder='hvm'
memory = 8192
name = 'vista'
vcpus=4
acpi=1
apic=1
on_xend_stop='shutdown'
vif = [ 'mac=00:16:3e:68:e1:01,bridge=xenbr0' ]
disk = [ 'phy:/dev/mapper/guest-vista,hda,w' , 'file:/home/mgkeyes/Vista.iso,hdc:cdrom,r' ]
#device_model = '/usr/lib/xen-default/bin/qemu-dm'
device_model_version = 'qemu-xen'
device_model_override = '/usr/bin/qemu-system-x86_64'
boot='c'
sdl=0
vnc=1
vncpasswd='vistapwd'
stdvga=0
serial='pty'
tsc_mode=0
viridian=1
usb=1
usbdevice='tablet'
gfx_passthru=0
#pci=[ '02:00.0', '02:00.1' ]
pci=[ '02:00.1' ]
localtime=1
pci_power_mgmt=0

Now, you may notice that I’ve commented out the line passing through both the GPU and its sound chip, in favour of one passing through only the sound chip. This is because the sound chip is passed through successfully, and detected by Windows. The GPU, on the other hand, causes the VM to not boot, with the only way to shut it down being:

sudo xl destroy vista

Unfortunate, especially after I had spent so much time trying to get the Radeon driver blacklisted so the card was even available for passthrough.

I was puzzled. Surely my card should pass through fine. After all, all Radeon HD 7000 are supported, and I had bought a card from that series, albeit a low-end one. Or so I thought. What I had bought was a “Radeon R7 250”, which I misinterpreted as “Radeon 7250”. Turns out it’s part of a new series of cards altogether – D’oh #3! It is with considerable shame that I divulge this.

=====The rest of this isn’t really related to VGA passthrough, but it is interesting, and somewhat related to XEN. You can still scroll down to the bottom to see my lessons.=====

So, having cursed my inability to properly research components for my new build, I decided to wait until, some time down the line, support for these cards is properly programmed. (But in the meantime, I wonder how I will play Rayman Legends…) Having given up on the VM’s graphics, I had a look at the host’s. The “nouveau” open-source drivers were being used, which caused the Cinnamon desktop environment to run in software mode, and when I tried to install and run Steam, it didn’t even start for me. Therefore, I decided to give the proprietary nVidia drivers a try. After all, they had worked well on my old system, with the same card (a GeForce GTX 550Ti by the way).

Bad idea. Cinnamon, instead of running in software mode, crashed upon login, leaving me in “Fallback Mode”, which was very unpleasant. To make matters worse, Firefox also crashed any time it was even slightly over-taxed, making even regular browsing quite a frustrating experience. Also the screen went irrecoverably blank any time I tried to exit to command-line to install an updated version of the driver (obtained from nVidia’s site). Defeated, I returned to the nouveau driver, an action which itself involved jumping through a few hoops.

On the up-side, it turned out that I could get proper OpenGL rendering by installing the libgl1-mesa-dri-experimental package. This got Cinnamon out of software mode, allowed me to run Steam, and even play through Portal! Afterwards, I tried a few other things, like playing recordings in MythTV, but ended up coming across this little kernel bug, which affects the version still being shipped with this distro (3.11). Still, I was happy, and was looking forward to seeing what would happen with Portal 2.

What I saw was really ugly, and unfortunately the system became too unresponsive to take a screenshot. nouveau decided to spam all ttys (accessible by Ctrl+Alt+F(1-6)) with error messages, making it basically impossible to even try to kill the game from the command line. After a hard reboot, I thought it might be the same kernel bug again, so I disabled the option “Wait for Vertical Sync” in the advanced video settings. It seems I was on to something, because this time the game didn’t go ugly, but it did freeze with three dots left in the loading progress bar for the first map.

Having already found that exiting to command line no longer turned the screen blank, I decided to have another shot at installing the updated driver from nVidia’s website. It installed successfully, but a reboot produced – you guessed it – a blank screen! D’oh #4!

At this point I realised that I was still booting into the XEN hypervisor, which wasn’t much good to me when VGA passthrough wasn’t working. Therefore, I decided to try the normal kernel, and what did I get? Cinnamon running perfectly and Portal 2 playable! I do have a few visual glitches in normal apps, but I can overlook them (for now).

Now, I publish this in the hope that others can benefit from my experience, so I’ll try to boil it down to a few little lessons that I have learned, and that I hope others may be able to take away from this:

  • DO YOUR RESEARCH! If you see something like “Devil’s Canyon” on a processor, be sure you know exactly what it means. Don’t assume that “R7 250” means the same thing as “Radeon 7250”. This is probably obvious to most, but I failed to do it, and, as mentioned, I am ashamed of that.
  • Try not to assume that tutorials for normal Mint work with the Debian Edition.
  • device_model_version = 'qemu-xen'
    device_model_override = '/usr/bin/qemu-system-x86_64

    seems to work on Linux Mint Debian Edition with the correct QEMU packages installed. These packages include qemu and qemu-system.

  • AMD R7 250 cards DO NOT WORK with VGA passthrough at this time, and it’s probable that other members of the Rx 200 series don’t either.
  • Proprietary nVidia drivers don’t work well with the version of the Linux kernel used in the XEN hypervisor (at least in Debian, or it might just be a Linux Mint Debian Edition thing).

I hope this has been in some way helpful, or at least amusing! Suggestions for improvement or clarification are more than welcome.

Subatomic Particle Simulator now on GitHub!

Well, I have just spent an entire weekend using Windows, amounting to probably more hours than I had spent with this operating system in the previous six months! Of course, this meant that I had to endure loads of automatic updates, their drain on bandwidth and automatic restarts! It was worth it though to get the Subatomic Particle Simulator up, working just as well as it does on GNU/Linux! It’s nice to know that I’ve made this thing cross-platform.

More good news: In the process of porting the code, I realised that the best thing to do was to fork the Source SDK repository on GitHub, and that way avoid all the problems of contaminating a working copy to compile the simulator. It is here, so you can now clone it in a ready-to-compile state, or even download a ZIP file if you would prefer.

Again, same legal restrictions apply – due to incompatible licences, it is not currently possible to (legally) distribute binaries. Once again, I apologise for this, and hope I can sort it out in the future.

PS. If you would like to express your opinions on how legal restrictions like this should apply in the future, please go to this new campaign website.

Release of Source Code of Subatomic Particle Visualiser

Well, this is kind of embarrassing. In October, I implemented the GNU Scientific Library as part of the science project I mentioned in the previous post. Since I was working towards an actual deadline, I guess I was too hurried to thoroughly check what licence the GSL uses. Apparently, I assumed it was licensed under the LGPL, which many GNU libraries tend to be. However, it is actually under the much more restrictive GPL, which forbids me from distributing it combined with any proprietary programme. The code given by the Source SDK for the server DLL (the only thing I actually modified) counts as proprietary, since it is distributed under a licence that forbids selling. Therefore, while experimenting with the two pieces of software together is fine, if I were to distribute my compiled server library, I would be in breach of the GPL.

Therefore, the long and the short of it is that my only option is to distribute my code and the GNU Scientific Library on their own, and let you, the user, actually compile it. To that end, here is an archive containing my code plus the GSL code, in a directory structure that will let it fit right into a fresh download of the Source SDK, plus instructions for getting it up and running:
http://www.vigovproductions.net/simsource.tar.gz
It is currently designed to compile only on GNU/Linux. A Windows version will be made in the coming days.

Here is a game directory to put in your SteamApps/SourceMods folder:
http://www.vigovproductions.net/simulator_mod.tar.gz
Once you put it there, don’t forget that you need to add in a bin folder, with the libraries compiled from the source code. They can be found in “sp/game/mod_episodic/bin” and “sp/game/vigov_simulator/bin”, under the Source SDK directory structure.

I apologise for not being able to release a compiled game. In order to do so, I would need to either pick a different game engine or a different mathematical library (or write the code myself…). I may do one of those in the future.

My return to the blogosphere

The time has come for me to return to this blog and explain my absence for the last few months. I was spending a lot of time working on a project for a prestigious science and technology exhibition. As such, between that and school work, I had basically no time to blog, or upload anything more than trivial logo videos to my YouTube channel.
Well, the project is now “finished”, and a selection of the fruits of my labour has been uploaded to YouTube.

The files for this game/tool will be uploaded to this site in a few days’ time, when I have had a chance to sort through them.

Here are some of the things I could have mentioned had I been actively blogging:

  • The change in Saorview frequencies, and how it left everyone with eight extra dud channels in their EPG.
  • The introduction of RTÉ One HD, and the stretching of classic programmes that came therewith.
  • The court cases involving the NSA.
  • Halloween
  • Christmas
  • The New Year

So what’s on the cards now? Well, apart from school, there’s:

  • Sony Pictures Television History Mark IV – I know, when does it stop, right? Well, significant info has been discovered since Mark III was made. Hopefully I can reuse lots of animation from previous videos – the thought of starting from scratch really doesn’t appeal to me!
  • Updates to the spreadsheet-based corporate timelines published on this site.
  • That new Aperture Ireland release I promised but never delivered. Speaking of which, I wonder if Valve will ever get around to porting Portal 2 to GNU/Linux…