This paper presents an approach to design of a neural architecture for both associative (content-addressed) and address-based memories. Several interesting properties of the memory module are mathematically analyzed in detail. When used as an associative memory, the proposed neural memory module supports recall from partial input patterns, (sequential) multiple recalls and fault-tolerance. When used as an address-based memory, the memory module can provide working space for dynamic representations for symbol processing and shared message-passing among neural network modules within an integrated neural network system. It also provides for real-time update of memory contents by one-shot learning without interference with other stored patterns.