CodeBalance

A Theorical Introduction to Data Mining

Posted on 2010-12-14 by CB

This article introduces the aim of data mining and explains basic concepts and terms.

Data Mining (i. e. Knowledge discovery from data): Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) patterns or knowledge from huge amount of data.

Data Warehouse : A single, complete and consistent store of data obtained from a variety of different sources made available to end users in a what they can understand and use in a business context. [Barry Devlin] Data warehouses are used for data mining.

Potential Usages : Web information mining, spam filtering, medical data mining, weather data mining, market sale strategies etc.

Data Mining Related Operations

Preprocessing:

Handling Noisy Data : Handling missing, duplicate or errorneous data before data mining. Noisy data can be removed, or corrected by a specific approach (i.e. correlation analysis).

Integration : Combining data from multiple sources.

Normalization : Scaling data to specified range. For example, scaling 750 in [500, 1000] to range [0,1] (the result is 0.5)

Feature Selection : Selecting only useful features (i.e. attributes for record data) of data.

Data Mining:

Classification: Finding a model for a class attribute of data to predict the values of other attributes. (An example class attribute: CustomerBuysProduct (bool))

Different methods can be used for classification:

Decision Trees: Uses decision trees to make model and evaluates new data on the tree.
Rule-Based Classifying: Deduces rules on the data (if X = Y and if Z z T result is W etc.).
Bayes Classifying: Uses previous probabilities to classify.
K-Nearest Neighbor Classifying: Uses distances between previous data to new data, to classify.
...

Clustering: Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups.

Different methods can be used for clustering:

K-means Clustering: Splits data according to a previously known number of clusters.
Hierarchical Clustering: Produces a set of nested clusters organized as a hierarchical tree.
...

Association (Rule) Discovery: Producing dependency rules which will predict occurrence of a feature (i.e. attribute) of data based on occurrences of other features.

Pattern Discovery: Deducing patterns as a result of classification, clustering, Pattern discovery etc.

Postprocessing: Evaluating and selecting interesting patterns, interpreting and visualizing them as an information report.

Posted in Databases and Data Mining | 3 Comments

Software Antipatterns : The Golden Hammer

Posted on 2010-10-29 by CB

Preface

A "software pattern" is a predefined and accepted solution for a specific software problem. Similarly, a "software antipattern" is a predefined and accepted unsuccesful solution. It's known as a bad solution, and should not be used.

The Golden Hammer Antipattern

The Golden Hammer can be defined as an architecture, a solution or a software tool that is believed to be the best solution for every software problem/project.

If a software team had successful experiences with a software architecture, tool or solution they may want to use that in every other problem/project. Especially if that experience had a success in the past, this probability will be higher. Or if that experience had been costly for the company and new experiences will be more costly, managers may not give resource for new techniques.

There can be other causes: The software team either don't want to learn new technologies and techniques, so they want to use their "known" technology/technique in every problem or they are not aware of the growing world of technology and new solutions.

For example, "using well known software patterns in every software project" is a golden hammer antipattern. Because of one or more causes told before, this approach can be used and probably will be harmful for the software. And maybe it will not be corrected in the future because of project calendar.

Eventually, each software problem/project is different is its own context. And must be evaluated in its own cases. Using same approaches for every problem is dangerous.