I prepared a quick POC to test whether monngo will be sufficient for what I need. In short - I very frequently do a lot of bulk writes to a collection and I very often remove stuff from that collection.
Outcome is that there is pretty constant number of documents in it (10k) but every hour 180k entities "rotate" through that collection (are added and then removed).
I started noticing that indexes size grows despite the fact that I am removing old entities. See attached screenshot from the atlas chart which represents db size (collection documents size is pretty constant and it's 1.5MB)
From what I read this is called index fragmentation. As I understand, mongo needs to rearrange space on the drive in order to free this memory.
Here is the screenshots from indexes view.
What worries me is that it seems that mongo allocates more new, free space faster than it frees the old space.
What I would like to know:
- what triggers this compacting process in free tier?
- is this possible that I will run out of physical space if I keep the same write/delete rate or mongo will go hardcore mode when it detects that more and more space is being used?
- does this behaviour differ in flex tier?
- does this behaviour differ in dedicated tier?
- can I do something about it in general?
I am getting great results from mongo in terms of ease of use, latency, driver and cost but I am worried that this index fragmentation thing will just make this db unusable for this use case.