I have done a test run. New empty drawing saved is 18 kb, imported some blocks from library browser is 100 kb, deleted the visible drawing still 100kb, deleted from Block List is 21 kb, deleted the added layers is 18 kb again.
Have you missed to clear up the Block List?
It depends on your LibreCAD version.
in newer versions you can use CTRL and SHIFT key to select multiple lines in block list and then delete them all together.
Try it by clicking the top row and then click the bottom row with SHIFT key pressed.
When all blocks are selected, you can use the delete button to delete them all.
Don't worry, there will be a confirmation window before they are deleted.
If only one line stays selected, you have to delete them one by one.
There is nothing copied between drawings.
This may be an issue with your work flow or possibly you have replaced the empty drawing template with an other version containing a bunch of blocks.
Blocks are only stored in a drawing when you create or import them.
But when you use an existing drawing with blocks, remove all entities for a clean screen, the blocks remain in the drawing, even when they are not used.