How to add a mapping namespace - Revision history

Jimkont at 07:51, 5 March 2015

2015-03-05T07:51:26Z

← Older revision		Revision as of 09:51, 5 March 2015
Line 92:		Line 92:

	<pre>		<pre>
	sudo /etc/init.d/~~mappings~~-server stop		sudo /etc/init.d/dbpedia-server stop
	</pre>		</pre>

Line 130:		Line 130:

	<pre>		<pre>
	sudo /etc/init.d/~~mappings~~-server start		sudo /etc/init.d/dbpedia-server start
	</pre>		</pre>

Chrisahn: /* Get language code and rank */ Don't use namespace numbers >= 400

2014-12-21T13:50:45Z

Get language code and rank: Don't use namespace numbers >= 400

← Older revision		Revision as of 15:50, 21 December 2014
Line 13:		Line 13:
	'''CAUTION''': If the calculated namespace number already exists for another language (because the ranking has changed) do '''not''' change the existing namespace number. Please find a neighboring or close enough number that works.		'''CAUTION''': If the calculated namespace number already exists for another language (because the ranking has changed) do '''not''' change the existing namespace number. Please find a neighboring or close enough number that works.

	Example: if 288 is in use, we choose some other number that is not used, let's say 298.		Example: If 288 is in use, we choose some other number that is not used, let's say 298.

			'''CAUTION''': Do not use namespace numbers >= 400. Namespaces 400 and above are used by MediaWiki. Please find a number between 200 and 398 that is not yet used.

			Example: If the language rank is 100, the formula above would yield namespace number 400. Instead, we choose some other number that is not used, let's say 398.

	=== Update mappings wiki ===		=== Update mappings wiki ===

Chrisahn: sourceforge -> github correct section name

2014-12-21T13:44:56Z

sourceforge -> github correct section name

← Older revision		Revision as of 15:44, 21 December 2014
Line 59:		Line 59:
	</pre>		</pre>

	==== Edit ~~extract~~.default.properties ====		==== Edit extraction.default.properties ====

	Edit your copy of [https://github.com/dbpedia/extraction-framework/blob/master/dump/extraction.default.properties dump/extraction.default.properties]. Add something like this in the correct alphabetical position:		Edit your copy of [https://github.com/dbpedia/extraction-framework/blob/master/dump/extraction.default.properties dump/extraction.default.properties]. Add something like this in the correct alphabetical position:

Chrisahn: sourceforge -> github correct link

2014-12-21T13:44:19Z

sourceforge -> github correct link

← Older revision		Revision as of 15:44, 21 December 2014
Line 61:		Line 61:
	==== Edit extract.default.properties ====		==== Edit extract.default.properties ====

	Edit your copy of [https://github.com/dbpedia/extraction-framework/blob/master/dump/~~extract~~.default.properties dump/~~extract~~.default.properties]. Add something like this in the correct alphabetical position:		Edit your copy of [https://github.com/dbpedia/extraction-framework/blob/master/dump/extraction.default.properties dump/extraction.default.properties]. Add something like this in the correct alphabetical position:

	<pre>		<pre>

Chrisahn: sourceforge -> github

2014-12-21T13:42:47Z

sourceforge -> github

← Older revision		Revision as of 15:42, 21 December 2014
Line 2:		Line 2:

	'''CAUTION''': several subtle code changes will be needed to accomodate '''language codes that contain a dash''' (e.g. ''roa-rup'' or ''be-x-old''), especially in regard to URLs, file names and other identifiers, also in parts of the code base not listed here. In this case, please update the code and this guide.		'''CAUTION''': several subtle code changes will be needed to accomodate '''language codes that contain a dash''' (e.g. ''roa-rup'' or ''be-x-old''), especially in regard to URLs, file names and other identifiers, also in parts of the code base not listed here. In this case, please update the code and this guide.

	'''TODO''': Many links on this page still point to the Mercurial repository on '''SourceForge''', but the DBpedia source code now lives on '''[https://github.com/dbpedia/extraction-framework GitHub]'''.

	=== Get language code and rank ===		=== Get language code and rank ===
Line 49:		Line 47:
	==== Update [[DBpedia datasets\|datasets overview]] ====		==== Update [[DBpedia datasets\|datasets overview]] ====

	Edit [[DBpedia datasets]]. Add a column for the new language in the correct alphabetical position and update all rows according to the settings in [~~http~~://~~dbpedia.hg.sourceforge~~.~~net/hgweb~~/dbpedia/~~extraction_framework~~/~~file~~/~~dump~~/dump/extraction.default.properties dump/extraction.default.properties]. This is probably the most tedious part...		Edit [[DBpedia datasets]]. Add a column for the new language in the correct alphabetical position and update all rows according to the settings in [https://github.com/dbpedia/extraction-framework/blob/master/dump/extraction.default.properties dump/extraction.default.properties]. This is probably the most tedious part...

	=== Update the extraction framework ===		=== Update the extraction framework ===
Line 55:		Line 53:
	==== Edit Namespace.scala ====		==== Edit Namespace.scala ====

	Edit your copy of [~~http~~://~~dbpedia.hg.sourceforge~~.~~net/hgweb~~/dbpedia/~~extraction_framework~~/~~file~~/~~default~~/core/src/main/scala/org/dbpedia/extraction/wikiparser/Namespace.scala core/src/main/scala/org/dbpedia/extraction/wikiparser/Namespace.scala]. Add something like this in the correct alphabetical position:		Edit your copy of [https://github.com/dbpedia/extraction-framework/blob/master/core/src/main/scala/org/dbpedia/extraction/wikiparser/Namespace.scala core/src/main/scala/org/dbpedia/extraction/wikiparser/Namespace.scala]. Add something like this in the correct alphabetical position:

	<pre>		<pre>
Line 63:		Line 61:
	==== Edit extract.default.properties ====		==== Edit extract.default.properties ====

	Edit your copy of [~~http~~://~~dbpedia.hg.sourceforge~~.~~net/hgweb~~/dbpedia/~~extraction_framework~~/~~file~~/~~default~~/dump/extract.default.properties dump/extract.default.properties]. Add something like this in the correct alphabetical position:		Edit your copy of [https://github.com/dbpedia/extraction-framework/blob/master/dump/extract.default.properties dump/extract.default.properties]. Add something like this in the correct alphabetical position:

	<pre>		<pre>

Jimkont: /* Extract data from Wikipedia dump file */

2014-02-05T06:19:18Z

Extract data from Wikipedia dump file

← Older revision		Revision as of 08:19, 5 February 2014
Line 145:		Line 145:
	<pre>		<pre>
	dump> ../run download config={download-config-file}		dump> ../run download config={download-config-file}
			</pre>

	Run RedirectExtractor, InfoboxExtractor and TemplateParameterExtractor. [https://github.com/dbpedia/extraction-framework/blob/master/dump/extraction.stats.properties dump/extraction.stats.properties] should already contain the correct settings. cd into directory dump/ and execute		Run RedirectExtractor, InfoboxExtractor and TemplateParameterExtractor. [https://github.com/dbpedia/extraction-framework/blob/master/dump/extraction.stats.properties dump/extraction.stats.properties] should already contain the correct settings. cd into directory dump/ and execute

Jimkont: /* Extract data from Wikipedia dump file */

2014-02-05T06:19:00Z

Extract data from Wikipedia dump file

← Older revision		Revision as of 08:19, 5 February 2014
Line 142:		Line 142:
	==== Extract data from Wikipedia dump file ====		==== Extract data from Wikipedia dump file ====

	Download the latest dump for language xx.		Download the latest dump for language xx. [https://github.com/dbpedia/extraction-framework/wiki/Extraction-Instructions see here for details]
			<pre>
	~~Modify the DBpedia source code: set~~ [https://github.com/dbpedia/extraction-framework/~~blob~~/~~master/core/src/main/scala/org/dbpedia/extraction~~/config~~/mappings/InfoboxExtractorConfig.scala InfoboxExtractorConfig.extractTemplateStatistics] to true.~~		dump> ../run download config={download-config-file}

	Run RedirectExtractor, InfoboxExtractor and TemplateParameterExtractor. [~~http~~://~~dbpedia.hg.sourceforge~~.~~net/hgweb~~/dbpedia/~~extraction_framework~~/~~file~~/~~dump~~/dump/extraction.stats.properties dump/extraction.stats.properties] should already contain the correct settings. cd into directory dump/ and execute		Run RedirectExtractor, InfoboxExtractor and TemplateParameterExtractor. [https://github.com/dbpedia/extraction-framework/blob/master/dump/extraction.stats.properties dump/extraction.stats.properties] should already contain the correct settings. cd into directory dump/ and execute

	<pre>		<pre>
	dump> ../run extraction extraction.stats.properties		dump> ../run stats-extraction extraction.stats.properties
	</pre>		</pre>

Chrisahn: /* Extract data from Wikipedia dump file */ InfoboxExtractorConfig.extractTemplateStatistics must be true

2013-03-16T22:55:08Z

Extract data from Wikipedia dump file: InfoboxExtractorConfig.extractTemplateStatistics must be true

← Older revision		Revision as of 00:55, 17 March 2013
Line 143:		Line 143:

	Download the latest dump for language xx.		Download the latest dump for language xx.

			Modify the DBpedia source code: set [https://github.com/dbpedia/extraction-framework/blob/master/core/src/main/scala/org/dbpedia/extraction/config/mappings/InfoboxExtractorConfig.scala InfoboxExtractorConfig.extractTemplateStatistics] to true.

	Run RedirectExtractor, InfoboxExtractor and TemplateParameterExtractor. [http://dbpedia.hg.sourceforge.net/hgweb/dbpedia/extraction_framework/file/dump/dump/extraction.stats.properties dump/extraction.stats.properties] should already contain the correct settings. cd into directory dump/ and execute		Run RedirectExtractor, InfoboxExtractor and TemplateParameterExtractor. [http://dbpedia.hg.sourceforge.net/hgweb/dbpedia/extraction_framework/file/dump/dump/extraction.stats.properties dump/extraction.stats.properties] should already contain the correct settings. cd into directory dump/ and execute

Chrisahn: added TODO about sourceforge links, moved mappings wiki steps to top, added section "Update namespace settings for mappings wiki"

2013-03-12T14:09:38Z

added TODO about sourceforge links, moved mappings wiki steps to top, added section "Update namespace settings for mappings wiki"

@@ Line 2: / Line 2: @@
 '''CAUTION''': several subtle code changes will be needed to accomodate '''language codes that contain a dash''' (e.g. ''roa-rup'' or ''be-x-old''), especially in regard to URLs, file names and other identifiers, also in parts of the code base not listed here. In this case, please update the code and this guide.
 === Get language code and rank ===
@@ Line 14: / Line 16: @@
 Example: if 288 is in use, we choose some other number that is not used, let's say 298.
 === Update the extraction framework ===
@@ Line 34: / Line 70: @@
 You can add more extractors, but make sure that the required configuration exists for the new language.
 ==== Commit changes ====
@@ Line 93: / Line 137: @@
 ../run server &>server-<YYYY>-<MM>-<DD>.01.log &
 </pre>
 === Generate and deploy statistics ===

Chrisahn: /* Update and restart the mapping server */ wikiStats|et -> wikiStats|xx

2013-03-12T13:26:18Z

Update and restart the mapping server: wikiStats|et -> wikiStats|xx

← Older revision		Revision as of 15:26, 12 March 2013
Line 64:		Line 64:

	<pre>		<pre>
	wikiStats\|et		wikiStats\|xx

	redirects\|0		redirects\|0