Abstract: 
A bitdepth optimization engine reduces the hardware cost of a neural network. When training data is applied to a neural network during training routines, accuracy cost and hardware costs are generated. A hardware complexity cost generator generates costs for weights near bitdepth steps where the number of binary bits required to represent a weight decreases, such as from 2N to 2N–1, where one less binary bit is required. Gradients are generated from costs for each weight, and weights near bitdepth steps are easily selected since they have a large gradient, while weights far away from a bitdepth step have nearzero gradients. The selected weights are reduced during optimization. Over many cycles of optimization, a lowbitdepth neural network is generated that uses fewer binary bits per weight, resulting in lower hardware costs when the lowbitdepth neural network is manufactured on an ApplicationSpecific Integrated Circuit (ASIC).
