- ago
Anyone who is interested in getting a better understanding of the variety of classifiers in the ML domain, and an assessment of the capabilities over a large variety of data sets, might want to read the following journal article:
QUOTE:
https://jmlr.csail.mit.edu/papers/volume15/delgado14a/delgado14a.pdf

I believe that this will give you an appreciation for the capabilities of the various families.

Vince
0
541
4 Replies

Reply

Bookmark

Sort
- ago
#1
This is a very nice review article simply because of its breath. Discriminant analysis is the classical linear systems way of doing classification. And this article mentions around 20 methods of discriminant analysis. (I didn't even know there were that many. Typicaly, a multivariant stat class would study only one or two.)

It mentions even more neural network (NN) methods. The edge that neural networks have is that they handle nonlinear systems well, which discriminant analysis does not. The question I have is how well do they handle discontinuous systems such as event-driven simulation (Wealth-Lab)? If someone can cite some papers for me, I would love to read them. I'm sure some NN methods work better than others for this type of discontinuous problem.

Editorial note: I wouldn't give up on discriminant analysis simply because one is trying to model a discontinuous system. It may not be the optimal solution, but it still can provide "some" classification insight. In addition, it will compute a "P" (i.e. probability) for the classification so one can judge "risk" and weight their trades against that risk. NN solutions cannot do that. Or at least we haven't figured out good way of doing that yet with NN solutions and "unseen" data.
0
- ago
#2
superticker,

You ask the most interesting questions! ;)

It is well known that theoretically a 3-layer NN (Input, Hidden, Output) can represent any continuous multivariate function.
QUOTE:
Hecht-Nielsen R., Kolmogorov’s mapping neural network existence theorem, In: Proc. I987 IEEE Int. Conf. on Neural Networks, IEEE Press, New York, 1987, vol. 3, 11–14

It had further been surmised that a 4-layer NN was necessary to represent a discontinuous multivariate function, though a recent paper makes the case that a 3-layer NN can do that also.
QUOTE:
https://arxiv.org/pdf/2012.03016.pdf

But I would like to remind everyone of this:
QUOTE:
In theory there is no difference between theory and practice. In practice there is.
Yogi Berra

The Deep-Learning (DL) community has postulated that the success of DL, where the number of hidden layers can be >10, is partially based on its ability to handle the discontinuous nature of real-life data.

As you know, we are lacking a good theoretical foundation for most of ML, and DL is no exception.

Vince
0
- ago
#3
Thanks for the link. I knew about the first citation (way back in school), and that's why I call the hidden layer the "middle layer"; I see NN as a 3-layered system. But I didn't know about the citation in the following link. Interesting.

Yes, I'm wondering which NN methods are best at modeling discontinuous functions (Yes, like with Wealth-Lab event-driven simulations.). This is not an easy problem.
0
- ago
#4
superticker,

Based on the results of years of Kaggle competitions, I would suggest that ensemble methods are the clear winners, not NNs.
QUOTE:
https://towardsdatascience.com/basic-ensemble-learning-random-forest-adaboost-gradient-boosting-step-by-step-explained-95d49d1e2725

WL does not have a way to handle these, unfortunately.

Vince
1

Reply

Bookmark

Sort