Remove files without rebuilding datasets

[pavb-fr]

Currently the deletion of files is not supported by Ursadb as mentioned in the documentation
https://github.com/CERT-Polska/ursadb/blob/master/docs/datasets.md#add-and-remove-files

But we have some use cases where we want to add a lot of temporary files for a short duration.
We would like to be able to remove them after use without reindexing the database which could be very time consuming.

The tag feature on dataset seems very interesting for such use cases but it is not supported by the solution MalwareDB / MQuery.

• Be able to delete individual files from dataset without rebuild

[pavb-fr]

Maybe it can be linked to this issue on Mwdb
CERT-Polska/mwdb-core#560

If we can have a different ursadb tag by mwdb bucket

[msm-code]

But we have some use cases where we want to add a lot of temporary files for a short duration.

Hi, do I understand correctly that you want to index some files temporarily and later remove them all? You can remove datasets (groups of files) (as you already know from documentation you linked), just not individual files.

The only problem is that it's hard to know which datasets to remove after indexing, but it's actually very simple to fix. You can group files during indexing by adding a specified tag (known as taint in ursadb world), for example:

index "/opt" with taints ["temporary123"]

This will create datasets with tags, similarly to the screen below:

Datasets with one set of tags will never merge with datasets that have different tags. So after your temporary work is done, you can just query for all datasets with your tag temporary123 and issue dataset "xxxx" drop commands.

I think this is even easier than keeping track of which files to remove after processing. And the added benefit is that (if you want to) you may use the tags to scope your mquery queries to for example only query your temporary files. Would this work for you?

[pavb-fr]

Yes I think it should be fine for my use case when the issue CERT-Polska/mwdb-core#560 will be closed. Indeed I need a way to split the sample files by tag to be able to index them correctly.
Having a different s3 bucket or a different root path for the samples should fixed that.

On the other hand, I still have errors in mquery with missing files. But as described in the documentation mquery should ignored them. Maybe the issue should be open on mquery rather than here.

[msm-code]

On the other hand, I still have errors in mquery with missing files. But as described in the documentation mquery should ignored them. Maybe the issue should be open on mquery rather than here.

If you remove the dataset from ursadb it shouldn't be an issue - the files will be gone from mquery as well. But if you just remove files from disk then you'll get errors.

I agree that CERT-Polska/mwdb-core#560 may help with implementing this in practice