Recently, a lot of studies in the domain of numerical cognition have been published demonstrating a robust association between numerical symbol processing and individual differences in mathematics achievement. Because numerical symbols are so important for mathematics achievement, many researchers want to provide an answer on the 'symbol grounding problem,' i.e., how does a symbol acquires its numerical meaning? The most popular account, the approximate number system (ANS) mapping account, assumes that a symbol acquires its numerical meaning by being mapped on a non-verbal and ANS. Here, we critically evaluate four arguments that are supposed to support this account, i.e., (1) there is an evolutionary system for approximate number processing, (2) non-symbolic and symbolic number processing show the same behavioral effects, (3) non-symbolic and symbolic numbers activate the same brain regions which are also involved in more advanced calculation and (4) non-symbolic comparison is related to the performance on symbolic mathematics achievement tasks. Based on this evaluation, we conclude that all of these arguments and consequently also the mapping account are questionable. Next we explored less popular alternative, where small numerical symbols are initially mapped on a precise representation and then, in combination with increasing knowledge of the counting list result in an independent and exact symbolic system based on order relations between symbols. We evaluate this account by reviewing evidence on order judgment tasks following the same four arguments. Although further research is necessary, the available evidence so far suggests that this symbol-symbol association account should be considered as a worthy alternative of how symbols acquire their meaning.