Structured data in a wiki
Query your organizational knowledge based on attributes and relationships
Based on my career experience, when I think of “traditional” knowledge stores, I think of things like the Microsoft suite of Word, Excel, and PowerPoint files. I also think of how every group or project has its own SharePoint site. Note this is not just including my jobs at NASA, but also the jobs I held throughout school. There were a few instances of basic databases, but that information was not connected in any way to the Office documents or Sharepoint sites. And if we’re being honest, the information in each Office file is not connected to any of the others. The result is an ever-growing pile of disconnected data silos. We rely on humans to make connections and interpret new information from those connections.
Now you might have had a different experience. Maybe you have worked at a place where there were different tools or processes to connect data. If so, I’d love to hear some examples in the comments.
One thing that sets MediaWiki apart from the above example is the ability to combine the flexibility of a wiki with the power of a structured database.
Flexibility of a wiki
First, when I say “flexibility of a wiki”, maybe I should elaborate. When I think of a wiki and how it could be used, I generally associate it with a Word document. It’s definitely not the same, but it’s close enough. You have the ability to enter text in lots of formats (sentences, paragraphs, lists, links, tables, etc) and you can add images throughout the document. With a wiki page you can embed even more things like videos or animated images. The point is that you get to choose the format and it can accommodate many approaches.
One immediate benefit of using a single wiki instead of several separate Word files is that you can now use a single search box to search all the content. Another is that you can now easily embed links from one document to another, all within one “viewer” (your browser).
Power of a structured database
Now let’s talk about the “power of a structured database” statement. With a basic database you can have a list of items. For each item, you can describe the item based on a list of attributes. For example, maybe you have a list of beers. In that list, you might want to describe each beer based on the type of yeast used, the style of beer, the alcohol-by-volume (ABV), and the bitterness as measured in International Bitterness Units (IBU).
Once you have this data table populated, you can then sort by any attribute (give me the beer with the highest IBU) or filter by attributes (only show me stouts with an ABV of 5.0 or higher).
You might also have another table of breweries. You might want to track their location by country and their size (macro to micro or even home-brewers). Now you can tie the two tables together by relationships. For example, you know which beers are made by each brewery. Your database can capture this information to help you look up all Berliner Weisse beers brewed in Iceland.
An example of disconnected data
Let’s say you have a folder of Word files with each file describing a different beer. Maybe the document has a picture of a glass of the beer and it includes some narrative description of the flavor profile. Maybe it talks about the history behind how the brewer came up with the recipe.
It’d sure be nice to be able to integrate the database information directly into this Word file without having to look it up and copy/paste it into the file.
Then what happens if the database was updated? Maybe there was a mistake caught after creating the Word file. Or maybe the brewery slightly changed the ABV of the recipe. Who is going to notice this discrepancy and then follow-up to update the Word file?
The integrated wiki solution
I’m now going to describe how all the above data could be stored in a single place, making a much more powerful, integrated knowledge capture.
In-text annotation
First, let’s talk about the most basic use of Semantic MediaWiki: Setting a property in a wiki page. One way to do this is using in-line syntax. Imagine you create a wiki page named “Rocket Fuel” and the following text is on that page.
[[Name::Rocket Fuel]] is a [[Beer style::Coffee Porter]] brewed by [[Brewed by::8th Wonder]]. It has an ABV of [[ABV::5.0]] and [[IBU::24]] IBU.
This will look like:
Rocket Fuel is a Coffee Porter brewed by 8th Wonder. It has an ABV of 5.0 and 24 IBU.
Besides the creation of a couple sentences, the page “Rocket Fuel” now has some properties set to describe it as an entity.
For the property “Name”, the “Rocket Fuel” page has the value “Rocket Fuel”
For the property “Beer style”, the “Rocket Fuel” page has the value “Coffee Porter”
For the property “Brewed by”, the “Rocket Fuel” page has the value “8th Wonder”
For the property “ABV”, the “Rocket Fuel” page has the value “5.0”
For the property “IBU”, the “Rocket Fuel” page has the value “24”
Just in case that’s not easily understood, let’s try a different way to describe one example. We’re describing “Rocket Fuel” using properties on the page about “Rocket Fuel”. Let’s think of it in terms of [subject] [predicate] [object]. Rocket Fuel (the subject of this conversation) is brewed by (the predicate of this sentence, or the attribute/property being described) 8th Wonder (the object, or the value of the property being described).
Perhaps this is the most natural syntax to a newcomer because it is written in-line within the text, but there are some issues and limitations. For one, you have to manually type all this out for every page. I’ll show you a different approach how to use templates and forms to make this easier and more consistent. You also probably don’t want to link to wiki pages for “5.0” and “24”. There are ways to make these not create links and still use this syntax, but let me instead show you what we use for nearly every property definition in our wikis at NASA.
Using {{#set: property = value }}
Let’s create the same page in the example above, but using a different approach. The following snippet would be at the top of the page to set the page properties.
{{#set: Name=Rocket Fuel}}
{{#set: Beer style=Coffee Porter}}
{{#set: Brewed by=8th Wonder}}
{{#set: ABV=5.0}}
{{#set: IBU=24}}
The above will result in some blank lines before the first text on the page, but it’s easier to understand for this introductory example. Instead, to eliminate those blank lines, one way would be to do the following.
{{#set: Name=Rocket Fuel
| Beer style=Coffee Porter
| Brewed by=8th Wonder
| ABV=5.0
| IBU=24
}}
Note how the pipe (the “|” symbol) is a delimiter between the multiple property values being set within one use of {{#set:.
Lower in the page, after using either method above, you could have the following paragraph with text and values populated from the page properties.
{{#show: Rocket Fuel |? Name}}is a{{#show: Rocket Fuel |? Beer style}}brewed by{{#show: Rocket Fuel |? Brewed by}}. It has an ABV of{{#show: Rocket Fuel |? ABV}}and{{#show: Rocket Fuel |? IBU}}IBU.
This will look like:
Rocket Fuel is a Coffee Porter brewed by 8th Wonder. It has an ABV of 5.0 and 24 IBU.
Is that really worth the overhead?
You’re probably thinking this is a lot of nonsense and not worth my time. I don’t blame you. It was not intuitive to me at all for the first several days when I first discovered Semantic MediaWiki. But eventually it “clicked”. I’m hoping I can accelerate the time it’ll take for you to reach that point.
In the above example, I’m only talking about one page. But I have yet to show how the use of a template can standardize the above introductory paragraph. I also have not yet shown an example of how you could query all your wiki pages for a subset based on values or value ranges for these properties. So please bear with me.
Let’s add a template
In MediaWiki, a template is a way to standardize content that will be repeated in many use cases across the wiki. Let’s create a very simple template to go along with the above example. The template is intended to generate the introductory paragraph for any page about a beer. We’ll call the template “Beer intro”. Since it’s a template page, the name of the page will be “Template:Beer intro”. Here are the contents:
<includeonly>{{#show: {{PAGENAME}} |? Name}}is a{{#show: {{PAGENAME}} |? Beer style}}brewed by{{#show: {{PAGENAME}} |? Brewed by}}. It has an ABV of{{#show: {{PAGENAME}} |? ABV}}and{{#show: {{PAGENAME}} |? IBU}}IBU.</includeonly>
The <includeonly> and </includeonly> tags tell MediaWiki which part of the page to use and which part it should ignore. If you wanted to include instructions or examples of how the template can be used on the template page, you would put that content outside of those tags (ideally within <noinclude></noinclude> tags just to be sure).
The only other difference is that I have changed the text “Rocket Fuel” (the name of the page) to {{PAGENAME}}. This is a “magic word” within MediaWiki that tells the software to replace that snippet with the name of the wiki page.
So why did we go to the trouble of making this template page? Let’s return to the page Rocket Fuel and use the template instead of the full text we entered above. Instead of
{{#show: Rocket Fuel |? Name}}is a{{#show: Rocket Fuel |? Beer style}}brewed by{{#show: Rocket Fuel |? Brewed by}}. It has an ABV of{{#show: Rocket Fuel |? ABV}}and{{#show: Rocket Fuel |? IBU}}IBU.
we can now use the following:
{{Beer intro}}
That’s it. Now instead of copy/pasting that long section of wikitext to possibly thousands of wiki pages, each page just has this template. This way, if you decide to modify that intro sentence you do it in one place instead of thousands.
Where to go from here
This post is both too long and too short. It’s too long because I fear I may have lost some readers before they made it to this point. It has a lot of snippets of wikitext (or “code” as some new wiki users describe it) and can be daunting. But congrats to those of you who made it this far! You might be thinking this post is too short because it’s not obvious where to go from here.
I still haven’t explained how templates and forms can be used to make page creation and editing easier and more standardized. It’ll certainly help with soliciting those property values I described above.
I also haven’t shown how a simple query of the above property values can be used. Okay, here’s a teaser:
{{#ask: [[Category:Beer]] [[Beer style::IPA]] [[ABV::<5]]
|?Brewed by
|?ABV
|?IBU
}}
The above query will generate a table with columns for beer (wiki page name), Brewed by, ABV, and IBU. It will populate this table only with IPAs with an ABV less than 5 (some call this “sessionable”).
Hopefully this grabs your attention to the potential power this could add to your knowledge capture. The power of a structured database integrated with your unstructured knowledge capture in a flexible architecture.
If you want to go further down this rabbit hole with me, hit the subscribe button:
It’s not obvious to me how to sequentially introduce new aspects of this software but I’m doing my best. If you have questions, please post in the comments. If some example I’ve given isn’t clear, please let me know.
SMW resources
If you want to dive into using Semantic MediaWiki, here is a good resource.
Go to semantic-mediawiki.org
Under the menu for “Users”, click on “Manual”.
Once on the “User manual” page, you’ll have to look more to the right side of the page and hover over “User manual” for a list of reference pages.
Some of the documentation here is very good, but sometimes it can be hard to find the right page or the right example. That’s part of what I’m trying to do here. I’d like to supplement with real-world examples and feed off of community feedback.
Hopefully the walk-through above was useful to you. If so, please share this substack to others. If it was not useful, please share how I could do better in the comments. And please subscribe so I have a better sense of interest in this content.