|
|
| (9 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| = From Lorebooks to MediaWiki: A Demonstration with Semantic MediaWiki (SMW) and Scribunto = | | = From Lorebooks to MediaWiki: quick tour = |
|
| |
|
| This article shows, with concrete, runnable examples, how common Lorebook concepts (World Info, PLists, Ali:Chat, keys, placement, recursion, and filters) map cleanly to MediaWiki using Semantic MediaWiki (SMW) and Scribunto (Lua). The goal is a frictionless migration path: you can author knowledge once in wiki pages and expose it to downstream tools, UIs, and even export JSON for front-ends like SillyTavern.
| | '''notice: this page, including related templates and scripts, are generated by LLM and slightly modified manually''' |
| | |
| | This page shows how a SillyTavern-style lorebook can live inside MediaWiki with Semantic MediaWiki (SMW) and Scribunto. |
|
| |
|
| References for context: | | References for context: |
| - World Info & Lorebook concepts: https://rentry.co/world-info-encyclopedia
| | * [https://rentry.co/world-info-encyclopedia World Info Encyclopedia (World Info & Lorebooks)] |
| - Character formatting (PLists + Ali:Chat): https://wikia.schneedc.com/bot-creation/trappu/creation
| | * [https://wikia.schneedc.com/bot-creation/trappu/creation Character writing guide (PLists + Ali:Chat)] |
| | |
| == What we will build ==
| |
| We’ll model a small universe called "Farlandia" with:
| |
| - Places (Environment): town, landmark, etc.
| |
| - Entities (Lore): monsters like "slime" with both PList-like traits and Ali:Chat-like example reactions.
| |
| - Keys and specificity (primary/secondary keys, AND/NOT), trigger rules and placement.
| |
| - Recursive scanning (parent→child activation), constant vs conditional entries.
| |
| - Exporters to JSON for use as a Lorebook, and transclusion for in-wiki pages.
| |
| | |
| Technologies used:
| |
| - Semantic MediaWiki (SMW) for structured data: properties, categories, queries.
| |
| - Scribunto (Lua) for lightweight logic: key matching, AND/NOT filters, recursion, and JSON export.
| |
| | |
| == Data Model (SMW) ==
| |
| We use pages for concepts and SMW properties for fields. Minimal property set:
| |
| - [[Has type::Page]] properties:
| |
| * [[Property:Belongs to world]] — parent world (e.g., Farlandia)
| |
| * [[Property:Parent concept]] — parent concept to support recursion (e.g., Monsters → Slimes)
| |
| - [[Has type::Text]] properties:
| |
| * [[Property:Plist]] — serialized PList-like content (single-line or semicolon-separated)
| |
| * [[Property:AliChat]] — example dialogues (Ali:Chat-style text)
| |
| * [[Property:Primary keys]] — comma-separated keys (e.g., slime,slimes)
| |
| * [[Property:Secondary keys]] — comma-separated keys (e.g., enemy,gelatin)
| |
| * [[Property:Key mode]] — one of: constant | conditional | disabled
| |
| * [[Property:Logic]] — one of: ANY | AND | NOT (maps to Lorebook specificity)
| |
| * [[Property:Placement]] — hints like after-character | before-character | depth:5
| |
| * [[Property:Non-recursable]] — yes/no (to opt out from recursion)
| |
| | |
| Tip: Keep properties predictable, single purpose, and text-based for simple editing. You can always add more typed properties later (e.g., Has type::Boolean for Non-recursable).
| |
| | |
| == Minimal Page Templates ==
| |
| Create simple templates to standardize data entry. Below are wikitext examples; adapt names as you like.
| |
| | |
| === Template:World ===
| |
| Purpose: model a world (like a global Lorebook/WI scope) and group entries.
| |
| | |
| <pre>
| |
| <includeonly>[[Category:World]]
| |
| ; Name: {{{name|{{PAGENAME}}}}}
| |
| ; Description: {{{description|}}}
| |
| </includeonly><noinclude>
| |
| Usage:
| |
| {{World|name=Farlandia|description=A fantasy setting used for examples.}}
| |
| </noinclude>
| |
| </pre>
| |
| | |
| === Template:Concept ===
| |
| Purpose: model an Environment/Lore entry (place, monster, item, etc.).
| |
| | |
| <pre>
| |
| <includeonly>[[Category:Concept]]
| |
| [[Belongs to world::{{{world|}}}]]
| |
| {{#if:{{{parent|}}}|[[Parent concept::{{{parent}}}]]|}}
| |
| {{#if:{{{plist|}}}|[[Plist::{{{plist}}}]]|}}
| |
| {{#if:{{{alichat|}}}|[[AliChat::{{{alichat}}}]]|}}
| |
| {{#if:{{{primary|}}}|[[Primary keys::{{{primary}}}]]|}}
| |
| {{#if:{{{secondary|}}}|[[Secondary keys::{{{secondary}}}]]|}}
| |
| {{#if:{{{logic|}}}|[[Logic::{{{logic}}}]]|}}
| |
| {{#if:{{{mode|}}}|[[Key mode::{{{mode}}}]]|}}
| |
| {{#if:{{{placement|}}}|[[Placement::{{{placement}}}]]|}}
| |
| {{#if:{{{nonrecursable|}}}|[[Non-recursable::{{{nonrecursable}}}]]|}}
| |
| </includeonly><noinclude>
| |
| Usage:
| |
| {{Concept
| |
| | world=Farlandia
| |
| | parent=Farlandia:Monsters
| |
| | plist=[slime: enemy, slimeball, made of gelatin, bounces to move, annoyance]
| |
| | alichat={{user}}: Slime?
| |
| {{char}}: "Oh... Those things." She looks down and slightly blushes.
| |
| | primary=slime,slimes
| |
| | secondary=
| |
| | logic=ANY
| |
| | mode=conditional
| |
| | placement=depth:5
| |
| | nonrecursable=no
| |
| }}
| |
| </noinclude>
| |
| </pre>
| |
| | |
| == Example Pages ==
| |
| | |
| === Farlandia (world) ===
| |
| Create page: "Farlandia" with:
| |
| | |
| <pre>
| |
| {{World|description=A fantasy setting used for examples.}}
| |
| </pre>
| |
| | |
| === Farlandia:Monsters (category concept) ===
| |
| Create page: "Farlandia:Monsters" with:
| |
| | |
| <pre>
| |
| {{Concept
| |
| | world=Farlandia
| |
| | plist=[Farlandia's monsters: slimes, dragons]
| |
| | primary=monsters
| |
| | logic=ANY
| |
| | mode=constant
| |
| | placement=after-character
| |
| | nonrecursable=no
| |
| }}
| |
| </pre>
| |
| | |
| This acts like a parent that mentions child keys (slimes, dragons), enabling recursive discovery.
| |
| | |
| === Slime (concept) ===
| |
| Create page: "Slime" with:
| |
| | |
| <pre>
| |
| {{Concept
| |
| | world=Farlandia
| |
| | parent=Farlandia:Monsters
| |
| | plist=[slime: enemy, slimeball, made of gelatin, bounces to move, annoyance]
| |
| | alichat={{user}}: Slime?
| |
| {{char}}: "Oh... Those things." She looks down and slightly blushes in embarrassment. "I remember first fighting those things. They're so quick that it's hard to get a good strike on them." Shizuru brushes away the memory. "Anyways, let's keep going. There's more places to explore."
| |
| | primary=slime,slimes
| |
| | logic=ANY
| |
| | mode=conditional
| |
| | placement=depth:5
| |
| | nonrecursable=no
| |
| }}
| |
| </pre>
| |
| | |
| === Mossford (place/environment) ===
| |
| Create page: "Mossford" with:
| |
| | |
| <pre>
| |
| {{Concept
| |
| | world=Farlandia
| |
| | plist=[Mossford(The town of Moss): town, mossy buildings, moss used for(magic, power), has(tavern, bank, inn, castle), kind people, wealthy]
| |
| | primary=Mossford,town,moss
| |
| | logic=ANY
| |
| | mode=conditional
| |
| | placement=before-character
| |
| | nonrecursable=yes
| |
| }}
| |
| </pre>
| |
| | |
| Notes:
| |
| - Mode=constant means always injected; conditional means key-triggered; disabled means off.
| |
| - Logic=AND/NOT works with secondary keys:
| |
| - AND example: Primary: house,home; Secondary: your,yours,she,her — both must match.
| |
| - NOT example: Secondary keys negate injection if present.
| |
| - Non-recursable=yes prevents child-follows-parent injection for this concept (mimics Non-Recursable in lorebooks, often used for Ali:Chat-only entries).
| |
| | |
| == Querying with SMW ==
| |
| You can list concepts for a world or find concepts by key. Examples:
| |
| | |
| List all concepts in Farlandia:
| |
| <pre>
| |
| {{#ask: [[Category:Concept]] [[Belongs to world::Farlandia]]
| |
| |?Plist
| |
| |?AliChat
| |
| |?Primary keys
| |
| |?Secondary keys
| |
| |format=table
| |
| |limit=50
| |
| }}
| |
| </pre>
| |
| | |
| Find concepts that mention a particular key string in Primary keys (simple contains):
| |
| <pre>
| |
| {{#ask: [[Category:Concept]] [[Primary keys::~*slime*]] [[Belongs to world::Farlandia]]
| |
| |?Plist
| |
| |?AliChat
| |
| |format=table
| |
| |limit=20
| |
| }}
| |
| </pre>
| |
| | |
| == Scribunto (Lua) Module: Key Matching, Logic, Recursion, Export ==
| |
| We’ll create a Lua module to:
| |
| - Parse a "query context" (the current prompt text) and decide which concepts trigger.
| |
| - Implement Logic ANY/AND/NOT with primary/secondary keys.
| |
| - Walk parent->child recursion unless Non-recursable=yes.
| |
| - Produce a plain text injection block and/or JSON export compatible with Lorebook formats.
| |
| | |
| Create a module page: "Module:Lorebook" with this Lua code:
| |
| | |
| <pre>
| |
| local p = {}
| |
| | |
| local function splitCSV(s)
| |
| if not s or s == '' then return {} end
| |
| local t = {}
| |
| for token in mw.text.gsplit(s, ',', true) do
| |
| token = mw.text.trim(token)
| |
| if token ~= '' then table.insert(t, token:lower()) end
| |
| end
| |
| return t
| |
| end
| |
| | |
| local function containsAny(haystack, needles)
| |
| local s = ' ' .. haystack:lower() .. ' '
| |
| for _, n in ipairs(needles) do
| |
| if n ~= '' and s:find('%f[%w]' .. mw.ustring.gsub(mw.text.nowiki(n), '([^%w])', '%%%1') .. '%f[%w]') then
| |
| return true
| |
| end
| |
| end
| |
| return false
| |
| end
| |
| | |
| local function andMatch(haystack, primary, secondary)
| |
| return containsAny(haystack, primary) and containsAny(haystack, secondary)
| |
| end
| |
| | |
| local function notMatch(haystack, primary, secondary)
| |
| return containsAny(haystack, primary) and not containsAny(haystack, secondary)
| |
| end
| |
| | |
| local function anyMatch(haystack, primary, secondary)
| |
| -- secondary unused for ANY
| |
| return containsAny(haystack, primary)
| |
| end
| |
| | |
| local logicDispatch = {
| |
| ANY = anyMatch,
| |
| AND = andMatch,
| |
| NOT = notMatch,
| |
| }
| |
| | |
| local function getProp(page, prop)
| |
| local data = mw.ext.semwiki.getProperty(page, prop) -- requires Semantic MediaWiki
| |
| if type(data) == 'table' then return data[1] end
| |
| return data
| |
| end
| |
| | |
| local function getAllConcepts(world)
| |
| local query = string.format('[[Category:Concept]][[Belongs to world::%s]]', world)
| |
| local res = mw.ext.semwiki.ask(query, {
| |
| mainlabel = 'page',
| |
| ['?Plist'] = 'plist',
| |
| ['?AliChat'] = 'alichat',
| |
| ['?Primary keys'] = 'primary',
| |
| ['?Secondary keys'] = 'secondary',
| |
| ['?Logic'] = 'logic',
| |
| ['?Key mode'] = 'mode',
| |
| ['?Parent concept'] = 'parent',
| |
| ['?Non-recursable'] = 'nonrec',
| |
| ['?Placement'] = 'placement',
| |
| limit = 999
| |
| })
| |
| return res or {}
| |
| end
| |
| | |
| local function indexByTitle(rows)
| |
| local idx = {}
| |
| for _, r in ipairs(rows) do
| |
| idx[r.page] = r
| |
| end
| |
| return idx
| |
| end
| |
| | |
| local function shouldTrigger(row, haystack)
| |
| local mode = (row.mode or 'conditional'):upper()
| |
| if mode == 'DISABLED' then return false end
| |
| if mode == 'CONSTANT' then return true end
| |
| local logic = (row.logic or 'ANY'):upper()
| |
| local f = logicDispatch[logic] or anyMatch
| |
| local primary = splitCSV(row.primary)
| |
| local secondary = splitCSV(row.secondary)
| |
| return f(haystack, primary, secondary)
| |
| end
| |
| | |
| local function walkRecursion(rows, idx, startPages)
| |
| -- Build children map from Parent concept
| |
| local children = {}
| |
| for _, r in ipairs(rows) do
| |
| local parent = r.parent
| |
| if parent and parent ~= '' then
| |
| children[parent] = children[parent] or {}
| |
| table.insert(children[parent], r.page)
| |
| end
| |
| end
| |
| | |
| local visited = {}
| |
| local stack = {}
| |
| for _, ptitle in ipairs(startPages) do table.insert(stack, ptitle) end
| |
| | |
| local ordered = {}
| |
| while #stack > 0 do
| |
| local cur = table.remove(stack)
| |
| if not visited[cur] then
| |
| visited[cur] = true
| |
| table.insert(ordered, cur)
| |
| local row = idx[cur]
| |
| if row and (row.nonrec or ''):lower() ~= 'yes' then
| |
| for _, c in ipairs(children[cur] or {}) do
| |
| table.insert(stack, c)
| |
| end
| |
| end
| |
| end
| |
| end
| |
| return ordered
| |
| end
| |
| | |
| local function buildInjection(rows, triggered)
| |
| -- Simple: PList first (joined), then Ali:Chat snippets
| |
| local plistChunks, chatChunks = {}, {}
| |
| for _, title in ipairs(triggered) do
| |
| local r
| |
| for _, row in ipairs(rows) do if row.page == title then r = row break end end
| |
| if r then
| |
| if r.plist and r.plist ~= '' then table.insert(plistChunks, r.plist) end
| |
| if r.alichat and r.alichat ~= '' then table.insert(chatChunks, r.alichat) end
| |
| end
| |
| end
| |
| local text = ''
| |
| if #plistChunks > 0 then
| |
| text = text .. table.concat(plistChunks, '\n') .. '\n'
| |
| end
| |
| if #chatChunks > 0 then
| |
| text = text .. table.concat(chatChunks, '\n')
| |
| end
| |
| return text
| |
| end
| |
| | |
| function p.inject(frame)
| |
| local world = frame.args.world or 'Farlandia'
| |
| local haystack = frame.args.context or ''
| |
| local rows = getAllConcepts(world)
| |
| local idx = indexByTitle(rows)
| |
| | |
| local start = {}
| |
| for _, r in ipairs(rows) do
| |
| if shouldTrigger(r, haystack) then table.insert(start, r.page) end
| |
| end
| |
| | |
| local triggered = walkRecursion(rows, idx, start)
| |
| return buildInjection(rows, triggered)
| |
| end
| |
| | |
| function p.export(frame)
| |
| -- Export JSON array resembling a lorebook entries list
| |
| local world = frame.args.world or 'Farlandia'
| |
| local rows = getAllConcepts(world)
| |
| local entries = {}
| |
| for _, r in ipairs(rows) do
| |
| table.insert(entries, {
| |
| title = r.page,
| |
| plist = r.plist or '',
| |
| alichat = r.alichat or '',
| |
| primary = r.primary or '',
| |
| secondary = r.secondary or '',
| |
| logic = (r.logic or 'ANY'),
| |
| mode = (r.mode or 'conditional'),
| |
| parent = r.parent or '',
| |
| nonrecursable = r.nonrec or '',
| |
| placement = r.placement or ''
| |
| })
| |
| end
| |
| return mw.text.jsonEncode(entries)
| |
| end
| |
| | |
| return p
| |
| </pre>
| |
| | |
| Notes:
| |
| - This module uses Semantic MediaWiki querying via mw.ext.semwiki.ask/getProperty. If your SMW exposes a different API wrapper (e.g., mw.ext.smw), adjust the calls accordingly:
| |
| - Replace mw.ext.semwiki.ask with mw.ext.smw.ask
| |
| - Replace mw.ext.semwiki.getProperty with mw.ext.smw.getQueryResult or mw.smw.getProperty per your setup.
| |
| - The matching in containsAny is deliberately lightweight. You can expand to support quoted keys or regex-based matching.
| |
| | |
| == Using the Module ==
| |
| | |
| Inject text for a given world and a chat/context string:
| |
| <pre>
| |
| {{#invoke:Lorebook|inject|world=Farlandia|context=We entered the town of Mossford where slimes attack.}}
| |
| </pre>
| |
| This will collect all triggered concepts (e.g., Mossford and Slime), apply recursion (e.g., parent Farlandia:Monsters), and produce a concatenated block of PList+Ali:Chat snippets.
| |
|
| |
|
| Export the current world as JSON (to save or feed a frontend):
| | == TL;DR demo == |
| <pre> | | # Visit the example world page [[Farlandia]] and a concept such as [[Slime]]. |
| <nowiki>{{#invoke:Lorebook|export|world=Farlandia}}</nowiki>
| | # Run <pre>{{#invoke:Lorebook| inject|world=Farlandia |context=Slime in Mossford}}</ pre> |
| </pre> | | # You get context like: |
| You can put the export on a page like "Farlandia:Export" and copy the JSON.
| | <code>{{#invoke:Lorebook|inject|world=Farlandia|context=Slime in Mossford}}</code> |
| | That output mirrors the original lorebook behaviour while everything remains queryable inside the wiki. |
|
| |
|
| == Mapping Lorebook Concepts → MediaWiki == | | == Architecture snapshot == |
| - World Info vs Lorebook: Use pages and categories. A page like "Farlandia" groups concepts through [[Belongs to world::Farlandia]]. Character-specific lore can use another property [[For character::Name]].
| | * Templates capture lore in structured form (worlds and concepts). |
| - PList: Store as text in [[Plist::...]] exactly as you'd author it in a Lorebook. You may also split categories into separate properties later.
| | * SMW properties index every chunk for queries and recursion. |
| - Ali:Chat: Store in [[AliChat::...]] verbatim (use nowiki or pipes carefully when needed).
| | * The Scribunto module reads those annotations, applies key logic, and emits prompt text or JSON. |
| - Keys: [[Primary keys::...]] and [[Secondary keys::...]] as comma-separated strings. Logic [[Logic::ANY|AND|NOT]].
| |
| - Placement: Record as a hint in [[Placement::...]]. Consumers (Lua or exporters) decide how to map to a UI’s positions.
| |
| - Constant/Conditional/Disabled: [[Key mode::...]] toggles inclusion.
| |
| - Recursion: Use [[Parent concept::...]] edges; Lua decides to follow unless [[Non-recursable::yes]].
| |
| - Stacking and filters: Add [[For character::...]] and [[Exclude character::...]] if you need per-character scoping. Extend Lua to honor them.
| |
|
| |
|
| == Advanced: PList Base World == | | == Why it works == |
| You can implement the "PList base world" pattern by creating bracket border concepts with insertion orders. In wiki, model insertion order as [[Property:Order]] (number) and update the Lua builder to wrap all PList between order 2 and 998. Then omit [] in child PList and end lines with semicolons, matching the guide’s algorithm.
| | === Template === |
| | MediaWiki allows users to create custom templates, which is easily reusable. We created two templates [[Template:World]] and [[Template:Concept]] to mimic actual lorebooks. Here are some example pages using these templates: [[Farlandia]], [[Farlandia:Monsters]], [[Slime]], [[Mossford]]. The custom styles also renders them beautifully. |
|
| |
|
| == Quality and Maintenance == | | === Semantic MediaWiki === |
| - Pages are collaboratively editable and diffable.
| | SMW stores each lore snippet as property triples (world ownership, keys, recursion hints). Queries can list all concepts in a world, filter by keys, or export structured JSON. |
| - SMW queries provide dynamic tables and navigation.
| |
| - Lua module localizes decision logic; exporting keeps compatibility with external tools.
| |
|
| |
|
| == Try it checklist == | | === Scribunto === |
| 1) Create properties (SMW): Belongs to world, Parent concept, Plist, AliChat, Primary keys, Secondary keys, Key mode, Logic, Placement, Non-recursable (and optionally Order).
| | The module mirrors Lorebook matching rules: ANY/AND/NOT key logic, constant/conditional entries, and recursive parent/child traversal. It can output prompt text or JSON for downstream tools. Here is the vibe coded script used in this example [[Module:Lorebook]] |
| 2) Create templates: World and Concept.
| |
| 3) Create pages: Farlandia, Farlandia:Monsters, Slime, Mossford using examples above.
| |
| 4) Create Module:Lorebook with the Lua code and verify {{#invoke:Lorebook|inject}} works.
| |
| 5) Optionally add an Export page calling {{#invoke:Lorebook|export}} and copy JSON into a lorebook-compatible frontend.
| |
|
| |
|
| If you’d like, I can auto-create the pages and a minimal Module:Lorebook on your wiki via the API and save a ready-to-import XML export in this repo.
| | ''We are trying to find more ways to fully utilize the potential of MediaWiki, you can find more in process works [https://github.com/pubwiki here]'' |
