This project is based on the research done by Rina Dechter and Daniel Frost.

Constraint Satisfaction Problems were in the focus of the theoretical and experimental research over the last two decades [1-5]. As the result of this research there is a verity of different algorithms, which are efficient for solving some classes of CSP. A lot of work is done to provide a theoretical guarantee of the worst case performance. But in practice a lot of problems do not belong to any class of CSP where we have a reasonable guarantee of the worst case performance. Nevertheless even for these cases existing algorithms often are able to provide a solution much faster then the estimated worst case time.

This project provides a statistical analysis of an average case performance of five different CSP algorithms for some classes of CSP problems where the worst case analysis provides guarantees, which are far behind of any computational resources available nowadays.

The project addresses two main questions. Are some instances "naturally" more difficult then other even if they have the same number of variables, constraints, and values in each domain? If there are some "naturally" difficult instances then what makes them difficult?

The behavior of the five search algorithms is investigated in this project (BJ means conflict-directed backjumping, FC means forward checking, AC3X means full arc-consistency checking, DVO means dynamic variable ordering, VMC means maximum cardinality variable ordering, VFA means fixed arbitrary variable ordering):

1. BJ-AC3X-DVO

2. BJ-AC3X-VMC

3. BJ-AC3X-VFA

4. BJ-FC-DVO

5. BJ-FC-VMC

The choice of the set of CSP to test the algorithm performance is really crucial. For one type of CSP problems algorithm A can overperform algorithm B while for another type of CSP A overperforms B. The choice of the testing set of CSP is very subjective. It has been shown [1,3] that there are set of parameters (so called cross-over points) which make CSP very difficult to solve. In this project only sets nearby cross-over points are investigated.

Eight sets of CSP problems are chosen (see table 1). Each set contains 10000 parameterized, randomly generated, binary CSP instances. Four generation parameters determines the each set of CSP: N is the number of variables, D is the size of each variable’s domain, C is the number of constraints, T is an indicator of the tightness of each constraint. The short notation is used everywhere below: N_D_C_T. For example the type 50_3_164_2 means N = 50, D = 3, C = 164, T = 2.

All possible constraints could be enumerated using a simple rule: constraint 1--2 has number 1, constraint 1--3 has number 2, ..., constraint 1--N has number N - 1, ..., constraint (N-1)--N has number N*(N-1)/2. Four sets contain the instances where constraints are uniformly distributed over all variables (these sets denoted as "uniform"). In the uniform case the distribution function of a binary random variable "Constraint x is chosen" is linear, so its derivative (the probability density function) is constant. Four sets contain the instances where constraints are distributed non-uniformly (these sets denoted as "non-uniform"). In the non-uniform case the distribution function is polynomial with randomly chosen power between 1 and 2, so its derivative (the probability density function) is polynomial with randomly chosen power between 0 and 1.

To make the performance estimation simpler only one parameter is used to judge how good a particular algorithms solves a particular CSP instance. This parameter is the number of consistency checks made by the algorithm while solving the instance. The number of consistency checks for all algorithms under investigation is strongly correlated with CPU time spent by these algorithms at some particular computer. And the number of consistence checks is implementation independent and will be the same for computers with different performance.

Eight different graph parameters are used as in advance estimators of the difficulty of some particular instance of CSP. Four of them deal with the original constraint graph and the corresponding set of four graph parameters is calculated for the induced constraint graph. Minimum induced width algorithm is used to build the induced width graph. These parameters for original and induced graphs respectively are width, number of variables in the core, hyperwidth of the core, and relative density of constraints in the core. The definition of the core is the set of variables in the graph (original or induced respectively) with width at least 80% of the width of the corresponding graph. For example, if the induced width is equal to 50 then the core of the induced graph contains all variables with induced width 40 or bigger. The hyperwidth of the core is defined as the sum over widths of all variables in the core. The definition of the relative density of constraints in the core is a little bit cumbersome: it is the ratio of the actual number of constraints between variables in the core (both variables should be in the core) to the expected number of constraints between variables in the core. The actual number of constraints between variables in the core could be easily counted for the particular graph. The expected number of constrains between variables in the core is equal to C*N_C*( N_C -1)/( N*( N -1) ), where C is the number of constraints, N is the number of variables, and N_C is the number of variables in the core.

The average numbers of consistency checks over 10000 instances for each algorithm and each set of generation parameters of CSP is shown in the table 1. The graph 1 represents the same information (results for BJ-FC-VMC are omitted because they are way bigger then other). We can see that the algorithms BJ-AC3X-DVO, BJ-AC3X-VMC and BJ-AC3X-VFA give almost the same performance for simple (less time consuming for all algorithms) types of CSP: 50_3_164_2 and 50_3_92_3. In the case of more difficult types of CSP (50_3_380_1 and 30_6_151_10) the choice of the "right" variable ordering becomes more important so BJ-AC3X-DVO and BJ-AC3X-VMC perform better than BJ-AC3X-VFA. Only the BJ-AC3X-VFA algorithm shows the significant (about three times) difference between performance for "uniform" and "non-uniform" types of CSP. This result seems counterintuitive because one can expect that the algorithm that use "smart" variable ordering will take bigger advantage of the information about non-uniformity of the constraint distribution.

The main focus of this project is the analysis of the set of correlation matrixes. This set is presented in tables 2 – 25. In these tables "all" means correlation matrixes over all instances, "zeros" means correlation matrixes over instances which are proved to have no solution and "ones" means correlation matrixes over instances which have at least one solution. Remember that algorithms stop after the first solution is found or the absence of the solution is proved.

Let’s first address the question about the "natural" difficulty of some instances inside the same type of CSP. There are some interesting trends here.

The first one is that for "difficult" types of CSP (50_3_380_1 and 30_6_151_10) the performance of all algorithms, except BJ-AC3X-VFA, is highly correlated. One should not make the conclusion that for this type of CSP the variable ordering is fully responsible for the performance. We can see that FC algorithms perform significantly worse than AC3X algorithms in these types of CSP. I would suggest the interpretation that for these types of CSP there is a significant amount of instances where the "smart" variable ordering can detect a "good" ordering, which simultaneously improves performance of all nontrivial ordering algorithms.

Another very interesting trend is that the correlation between BJ-AC3X-VFA and other AC3X algorithms is higher in the cases of non-uniformly distributed constraints. For "difficult" types of CSP (50_3_380_1 and 30_6_151_10) the growth of the correlation is minimal. But for "easy" types of CSP (50_3_164_2 and especially 50_3_92_2) the difference is very high. Maybe this type of correlation is connected with the fact that BJ-AC3X-VFA for "difficult" types of CSP performs significantly better when constraints are distributed non-uniformly.

Surprisingly enough, there is not so much difference between the set of correlation matrixes for problems where solution exists only, for problems where the absence of the solution is proven only and for mixture of those problems. One interesting observation can be made here. For difficult types of CSP, instances where no solution is proved are more time consuming for all algorithms (negative correlation between -0.2 and –0.5). But for easy types of CSP the situation is more complicated. FC algorithms perform approximately the same way in solvable instance and in instances where no solution is proved. BJ-AC3X-DVO and BJ-AC3X-VMC perform significantly better in instances where no solution is proved. While BJ-AC3X-VFA performs significantly better in instances where no solution is proved only if constraints are distributed non-uniformly. In the case of uniformly distributed constraints there is no correlation between performance of BJ-AC3X-VFA and the existence of the solution.

The last part of the statistical analysis deals with the correlation between the performance and different graph parameters. It would be really nice to be able to predict how difficult is each particular instance of CSP simply by analyzing the constraint graph. Unfortunately, at least for the chosen set of graph parameters there is no significant correlation between any graph parameter and any algorithm. There is a small correlation between induced width and the performance of algorithms that use VMC (the maximum cardinality variable ordering) at least for some types of CSP. One should expect it because VMC is closely connected with induced graph. But even this correlation is too small (non bigger than 0.3) to be used on practice.

The obvious observation is that for these particular types of CSP the best choice would be BJ_AC3X_VMC.

Less trivial observation is that there are strong dependencies between performance of different algorithms applied to the same CSP instances. And these dependencies change significantly when we change the set of generation parameters.

The set of graph parameters introduced in this project is not suitable for an advance prediction of the difficulty of some particular CSP instance inside the same generation type. Three possible reasons could explain this fact. The first possible reason is that the author of this project is not lucky (or smart) enough to guess the right set of graph parameters. The second explanation is that the pairwise correlation between each particular parameter and performance is not enough and one should statistically analyze the performance vs. the whole set of graph parameters simultaneously. For example, the parity function could not be detected using pairwise correlation matrix. The last possible reason could be that it is impossible in principal to predict the "hardness" of some particular instance inside the same type of CSP before we solve it or prove the absence of the solution by some algorithm.