Most Brain-inspired Visual Object Recognition Models(BVORMs) do not consider local and global reciprocal connections in visual pathway. We addressed this weakness and implemented an attention modulation mechanism based on feedback connections in BVORMs, where feature-selectivity is shaped and modulated by categorization of objects based on their visual features. This modification is inspired by the top-down neuromodulatory signals that make changes in post-synaptic activities of the feature-selective neurons. We also incorporated an implicit memory unit in BVORMs to accumulate recent Hebbian synaptic plasticity’s of the neurons in each task. This mechanism guides the top-down feature-based attention modulation to retrieve the interrelated feature-selectivity pattern for each task.HMax and CNN models were used as two BVORMs and tested on a visual categorization problem: natural versus artificial objects in CALTECH-256. Based on experimental results, our proposed modifications not only increased their biological-plausibility but also significantly improved their categorization accuracies compared to the original models.