DBpedia Release Evaluation: Difference between revisions

From Mediawiki1
Jump to navigationJump to search
No edit summary
No edit summary
Line 9: Line 9:


{| class="wikitable"
{| class="wikitable"
|+ Completeness / Recall (fixed mappings from 3.5.1)
|+ Mapped triples (Mappings from 3.5.1 DBpedia release)
|-
|-
! Category
! Category
Line 78: Line 78:


== Evaluation Results ==
== Evaluation Results ==
The table shows the completeness (recall) of the different release versions. To achieve comparability and to clean the results from the effect of more and better mappings, the mapping version of the 3.5.1 release is taken as constant.


{| class="wikitable"
{| class="wikitable"
Line 89: Line 91:
|-
|-
| Total
| Total
| 1504
| 45,7%
| 45,7%
| 60,2%
| 60,2%
Line 95: Line 96:
|-
|-
| Plain Property
| Plain Property
| 514
| 80,5%
| 80,5%
| 83,7%
| 83,7%
Line 101: Line 101:
|-
|-
| Number-Unit
| Number-Unit
| 51
| 68,6%
| 68,6%
| 68,6%
| 68,6%
Line 107: Line 106:
|-
|-
| Coordinate
| Coordinate
| 36
| 100,0%
| 100,0%
| 100,0%
| 100,0%
Line 113: Line 111:
|-  
|-  
| Interval
| Interval
| 22
| 72,7%
| 72,7%
| 68,2%
| 68,2%
Line 119: Line 116:
|-
|-
| List
| List
| 478
| 33,9%
| 33,9%
| 75,7%
| 75,7%
Line 125: Line 121:
|-
|-
| One-Property-Table
| One-Property-Table
| 242
| 5,4%
| 5,4%
| 5,8%
| 5,8%
Line 131: Line 126:
|-
|-
| Multi-Poprty-Table
| Multi-Poprty-Table
| 83
| 0,0%
| 0,0%
| 0,0%
| 0,0%
Line 137: Line 131:
|-
|-
| Open Property
| Open Property
| 13
| 23,1%
| 23,1%
| 23,1%
| 23,1%
Line 143: Line 136:
|-
|-
| Open Property Table
| Open Property Table
| 0
| na
| na
| na
| na
Line 149: Line 141:
|-
|-
| Internal Template
| Internal Template
| 58
| 8,6%
| 8,6%
| 10,3%
| 10,3%
Line 155: Line 146:
|-
|-
| Merged Properties
| Merged Properties
| 7
| 57,1%
| 57,1%
| 57,1%
| 57,1%

Revision as of 16:04, 14 September 2011

DBpedia Release Evaluation

The Quality Assessment Framework (QAF) is developed to document the quality of the knowledge base and furthermore the progress of DBpedia's extraction framework. The main idea of the QAF is a comparison between a manually created best-case dataset (Gold Standard) and the output from DBpedia's ontology based extraction. The QAF estimates the precision of the extraction framework and the completeness (recall) of DBpedia compared to its source Wikipedia.

Sample Data / Gold Standard

For a significant evaluation, only potentially extractable triples are considered. Only if a triple arise from a mapped property it can be extracted. Here this triples are called mapped triples. The following table shows the number of mapped triples, the total number of triples in the Gold Standard and the percentage of mapped triples for each category. The categories result from different patterns in which the information is given in Wikipedia infoboxes. The number of cases differ in a high extent depending on the category. The results for the categories based on small numbers should be handled with care.

Mapped triples (Mappings from 3.5.1 DBpedia release)
Category Mapped Triples Triples %
Total 1504 3221 46.7
Plain Property 514 893 57.6
Number-Unit 51 76 67.1
Coordinate 36 54 66.7
Interval 22 31 71.0
List 478 801 59.7
One-Property-Table 242 447 54.1
Multi-Poprty-Table 83 625 13.3
Open Property 13 139 9.4
Open Property Table 0 26 0.0
Internal Template 58 116 50.0
Merged Properties 7 13 53.8

Evaluation Results

The table shows the completeness (recall) of the different release versions. To achieve comparability and to clean the results from the effect of more and better mappings, the mapping version of the 3.5.1 release is taken as constant.

Completeness / Recall (fixed mappings from 3.5.1)
Category Cases DBpedia 3.5.1 DBpedia 3.6 DBpedia 3.7
Total 45,7% 60,2% 61,8%
Plain Property 80,5% 83,7% 86,0%
Number-Unit 68,6% 68,6% 66,7%
Coordinate 100,0% 100,0% 100,0%
Interval 72,7% 68,2% 72,7%
List 33,9% 75,7% 77,8%
One-Property-Table 5,4% 5,8% 5,8%
Multi-Poprty-Table 0,0% 0,0% 0,0%
Open Property 23,1% 23,1% 30,8%
Open Property Table na na na
Internal Template 8,6% 10,3% 10,3%
Merged Properties 57,1% 57,1% 71,4%
Precision (fixed mappings from 3.5.1)
Category DBpedia 3.5.1 DBpedia 3.6 DBpedia 3.7
Total 91,2% 92,3% 92,4%
Plain Property 96,3% 96,6% 97,4%
Number-Unit 85,4% 85,4% 85,0%
Coordinate 100,0% 100,0% 100,0%
Interval 100,0% 100,0% 88,9%
List 91,5% 92,8% 93,2%
One-Property-Table 32,5% 36,8% 34,1%
Multi-Poprty-Table na na na
Open Property 100,0% 100,0% 100,0%
Open Property Table na na na
Internal Template 83,3% 75,0% 75,0%
Merged Properties 80,0% 80,0% 100,0%
Retrieved from "https://mediawiki1.informatik.uni-mannheim.de/index.php?title=DBpedia_Release_Evaluation&oldid=15137"