Automatic document classification involves classifying natural language documents according to a set of predefined topics/classes. Sentiment analysis is a well-known example. This problem becomes more difficult when topics are user defined, and therefore not known a priori. We tested several algorithms, from basic ones like Regex matching to more sophisticated machine learning algorithms. The most known available open implementations in Java and Python proven to be not good enough to solve our problem. Therefore, we propose a new multi-level method that combines several existing approaches and may be customized. The results obtained with the proposed method outperform each individual approach that we have tested.