Link to the paper
A summary on the evaluation of NILM-Eval on certain NILM-Algorithms
The paper was focused on primarily demonstrating the NILM-Eval framework alongside their publicly available ECO Dataset on 2 supervised (semi) and 2 unsupervised algorithms. The analysis included defining configurations, testing relevant dataset portions on each algorithm and determining the efficiency and accuracy of each.
Algorithms criteria and Evaluation methodology
The primary parameters which were considered in NILM-Eval while selecting the algorithms were:
- Data Granularity: ECO Dataset is sampled at 1 Hz. Variations in the algorithms needed to be considered.
- Learning Methods: Supervised or unsupervised. Considerations on clustering algorithms and Hidden Markov models.
- Information Detail: Various parameters data sampled.
Bound setting configuration files (default and customized) with multiple dataset parameters for the algorithms implemented in the Framework. Uses MATLAB with multiple instances to scale over. Some parameters measured are:
- Precision(PRE)(True positives/ All positives)and Recall(REC)(True positives/All true cases) measured.
- Root means square error, deviation for data and switching calculated.
- Calculated F1 which is the harmonic mean of the precision and the deviation observed.
ECO Dataset
- Primary focus on Plug data.
- Publicly available dataset of 6 Swiss households.
- 8 months duration of Smart meter and plug data.
- 1 Hz sampling frequency.
- Measures real and reactive 3-phase power.
- Significant Other portion in dataset owing to the limited plug data sampled.
- Includes occupancy information.
Algorithms
Each algorithm was used having frequency input comparable to that of ECO Dataset viz. 1 Hz.
Parson's Algorithm
- Uses Hidden Markov Models (HMM), which are systems consisting of and monitoring memoryless processes with random variables. In HMMs, the states observed are kept secret.
- Uses the Viterbi algorithm, which finds the hidden states observed in the HMMs.
- Finds most probable state transitions and calculates difference in appliance and average consumption. Repeats for others.
- Makes a training model using these semi-supervised approaches.
- Uses a generic application model for creating the model, unless a specific application model is considered.
- Uses the REDD Dataset.
Baranski's Algorithm
- Extracts and clusters events into separate entities.
- Uses the Genetic algorithm, which implements natural selection in the clusters to give most probable application clusters.
- Unsupervised approach followed to cluster. Final most probable state accepted.
- However for more specific application labeling, cluster classification for at least the chief nodes needs to be manual.
Weiss' Algorithm
- Similar to Hart's algorithm to measure and extract switching events in the dataset.
- Maps the Signature database for predefined application categories defined to the event data.
- Uses real, active and distortion power to present a more 3 dimensional outlook.
- Supervised approach in general.
Kolter's Algorithm
- Uses HMM's like Parson's algorithm but unsupervised still.
- Detects the application cycle for ON OFF.
- Implements k-means clustering in order to find the centroid in a cluster of data points s.t. the distance to maximum points is minimized.
- Extension to implement the AFAMAP algorithm for further analysis.
- Obtains no. of applications and their respective consumption pattern.
- Done on the REDD Dataset.
NILM Results on Algorithms and Conclusions
Parson's Algorithm
- ECO Dataset scaled to 1/60 Hz (requirement of the algorithm).
- 2 primary appliances, Fridge and Microwave.
- Using probable Markov state determination through Viterbi.
- Fridge and Freezer data comparison difficult.
- Household 6 most accurate with max(F1) = 0.91 for Fridge.
- Household 4 and 5 most accurate for Microwave.
- Due to data aggregation, 59/60 portions of data lost, leading to consumption patterns to differ considerably.
Baranski's Algorithm
- Segregation and clustering of dataset based on switching events with similar changes in power consumption.
- Finite State Machines (FSM) consists of many finite states which are interchangeable amongst each other.
- Classification of these clusters into FSMs based on grouping, which defines approximate application boundaries.
- Average classifications reached at. Fridge discarded in the FSM list owing to OFF periods in separate clusters.
- Did not arrive at concrete comparable data owing to different I/O for the algorithm.
- Final application labeling done manually based on the power consumption, number of switching events and variations.
Weiss' Algorithm
- Used supervised approach with 1 Hz dataset.
- Categorized into cooling appliances, high power consumption appliances and remaining others.
- Highly accurate for cooling appliances with F1 = 0.91 to 0.94
- Highly accurate for high power consumption appliances with similar F1. However many False negatives leading to low Recall values.
- Overall successful in application modeling through provided through signature database.
Kolter's Algorithm
- Uses k-means clustering to find centroid in cluster and approximates application entities over the dataset.
- Identifies clusters and data snippets before segregating data.
- However, based on the HMM approach, no individual application model construction close to the given consumption model.
- Future expansion suggested based on the AFAMAP algorithm (no info found on the algorithm).
This proves that Supervised (or semi-supervised ) learning was dominant to obtain accurate
results with high F1, PRC and REC values. Weiss' and Parson's perform better on the ECO Dataset
and were more consistent with the results. With HMM usage, generic and application specific
model, Parson's approach is promising, but needs higher frequency data to obtain finer
results. Weiss potential and performance better than Parson due to same reason.
Some limitations that I saw with the framework was that too many constraints and configuration bindings. This leads to a more artificial simulation than real time load monitoring. Also, as it has been only done on the ECO Dataset, expansion to other sets is favorable.
Comparison with NILMTK in the next post...
Mridul