Data Structure recommendation

[James]

I am looking for your recommendations on data structures to
consider for use in a program for efficiency, etc. . . . I have tens
of thousands of small text files that are the output of some analysis
algorithms. (Their number may grow significantly in the future.)

Each file name indicates a variety of attributes of the image
files from which they were derived. I need to read in each file and
count a few items for each but in a number of ways, according to
different attributes individually and in combination. Then I need to
write out these counts in many different ways to a CSV output file.

Finally, my question: What data structure(s) would you recommend
that I consider to hold all these different counts by attributes (and
combinations of attributes), so I can efficiently report it out a
bunch of different ways?

For example, if you think of these statistics in a tree structure,
I want to be able to take each branch of the tree and quickly
determine the sum of the counts. I also want to be able to go down to
particular leaves of the tree and sum their counts. etc.

Thanks in advance for any recommendations.

 

[Patricia Shanahan]

James wrote:
....

Each file name indicates a variety of attributes of the image
files from which they were derived. I need to read in each file and
count a few items for each but in a number of ways, according to
different attributes individually and in combination. Then I need to
write out these counts in many different ways to a CSV output file.

....


Have you considered loading the data into a database and querying it?

Patricia