Accuracy assessments

A classification accuracy assessment generally includes three basic components: sampling design, response design, and estimation and analysis procedures (Stehman and Czaplewski 1998). Selection of a suitable sampling strategy is a critical step (Congalton 1991). The major components of a sampling strategy include sampling unit (pixels or polygons), sampling design, and sample size (Muller et al. 1998). Possible sampling designs include random, stratified random, systematic, double, and cluster sampling. A detailed description of sampling techniques can be found in previous literature such as Stehman and Czaplewski (1998) and Congalton and Green (1999).

The error matrix approach is the one most widely used in accuracy assessment (Foody 2002). In order to properly generate an error matrix, one must consider the following factors: (1) reference data collection, (2) classification scheme, (3) sampling scheme, (4) spatial autocorrelation, and (5) sample size and sample unit (Congalton and Plourde 2002). After generation of an error matrix, other important accuracy assessment elements, such as overall accuracy, user accuracy, producer accuracy (table 6), and kappa coefficient can be derived. Kappa is the difference between the observed accuracy and the chance agreement divided by one minus that chance agreement (Lillesand and Kiefer 1994).

Подпись: SVM

Подпись: ANN Подпись: DT

image21LUC classes

Bareground Щ Bulrush

Permanent farmland Щ Coniferus [ Decidious Grassland

Щ Irrigated farmland Щ Non-irrigated farmland Sand dunes Щ Settlement ■ Water bodies

Fig. 7. Data dependent LUC classification results using strong training dataset

Error matrix

Agricultur

e

Forest

Water

Total classified pixels

User accuracy

Agriculture

Forest

32

5

2

39

7

34

2

43

1

1

36

38

40

40

40

120

32 / 40 = 80% 34 / 40 = 85%

Water

36 / 40 = 90%

Total ground truth pixels

Producer

32 / 39

34 / 43

36 / 38

accuracy

82%

79%

95%

Overall accuracy

Correct pixels / Total pixel = 32+34+36 / 120 = 85%

Table 6. Error matrix and accuracy calculations

Overall classification accuracies and kappa coefficiencies of each classification using weak (6962 training pixels) and strong (16300 training pixels) training dataset were evaluated (table 7). In addition, each of the LUC user, producer and kappa accuracies were compared using strong training dataset to assess results in detail (table 8). No ancillary data integrated to the classifications, however, it was discussed in section 5.

Подпись: Accuracies (%) Classifier Weak Strong Overall Kappa Overall Kappa MLC 69.5 65.6 85.6 83.8 MB MD 67.6 63.3 72 68.5 LDA 73.5 70 80.5 78 K-means Overall = 57.3 Kappa = 52.1 DD ANN 70.5 66.5 76.9 74.2 DT 73 69.5 82.5 80 SVM 74 70 79.2 76.7 Table 7. Overall and kappa accuracies of model based (MB) and data dependent (DD) classifiers using weak and strong training dataset

image22Overall classification accuracies indicated that MLC was the most accurate model based classifier when the strong training dataset was used. However, LDA with weak training dataset performed accurately because of its distance separation algorithm. On the other hand, unsupervised k-means classifier was the least accurate one due to the fact that no training pixels were used.

Подпись: cb

f

Settlement

Non-irrigated farmland

I__ I Irrigated farmland

I Permanent farmland Water bodies

□ Sand dunes Bareground

0 0.5 1 2 f

Fig. 8. Visual detail of a small subview; (a) Ground truth, (b) MLC, (c) MD, (d) LDA, (e) ANN, (f) DT and (g) SVM results

SVM has a reasonable performance than other data dependent classifiers using weak training dataset. However, the largest accuracy was resulted in DT classifier using strong dataset.

SVM classified forestlands, grassland and permanent farmlands more accurate than other classifiers. There was not significant difference in built up areas among classifiers. The most accurate sand dunes, bulrush and irrigated farmland class accuracies were resulted from DT classifier. DT, LDA, SVM showed reasonably well performance with both weak and strong training data sets (figure 8).

In general, data dependent classifiers performed well with weak training dataset. Especially SVM was successful in vegetative area separation. It is clear that if more detailed classification scheme required (e. g. forest tree species) using weak training dataset, SVM might be first option in terms of classification accuracy. On the other hand, application of SVM is time costly when using standard PC and laptops.

Three accuracy calculation methods were shown in table 8, however, major question is which one should be used? Large number of studies have utilized the kappa coefficiencies as an ideal approache for LUC classification.

A number of criteria were selected for the comparison of both model based and data dependent classifiers as (a) Overall accuracy, (b) classification speed, (c) input parameter handling, (d) hardness in application, (e) accuracy with different training sizes and accuracy difference between each class or classification stability (table 9).

Criteria

MLC

MD

LDA

k-means

ANN

DT

SVM

Overall accuracy

****

**

****

*

***

*****

****

Classification speed

*****

*****

****

****

**

***

*

Input parameter handling

*****

*****

*****

*****

***

***

**

Hardness in application

*****

*****

****

*****

***

**

*

Accuracy with different

****

*

****

No

***

*****

*****

training sizes

training

Classification stability

***

***

****

*

***

*****

***

Table 8. Comparing hard classifiers (***** stars and * star refer the most accurate and the poorest performances respectively).