finding most common elements between thousands of multiple arrays.

[Raymond Hettinger]

[Scott David Daniels]

def most_frequent(arr, N):
     '''Return the top N (freq, val) elements in arr'''
     counted = frequency(arr) # get an iterator for freq-val pairs
     heap = []
     # First, just fill up the array with the first N distinct
     for i in range(N):
         try:
             heap.append(counted.next())
         except StopIteration:
             break # If we run out here, no need for a heap
     else:
         # more to go, switch to a min-heap, and replace the least
         # element every time we find something better
         heapq.heapify(heap)
         for pair in counted:
             if pair > heap[0]:
                 heapq.heapreplace(heap, pair)
     return sorted(heap, reverse=True) # put most frequent first.

In Py2.4 and later, see heapq.nlargest().
In Py3.1, see collections.Counter(data).most_common(n)


Raymond