no data exclution and unique combination.

[giuseppe.amatulli]

Hi,
would like to take eliminate a specific number in an array and its correspondent in an other array, and vice-versa.

given

a=np.array([1,2,4,4,5,4,1,4,1,1,2,4])
b=np.array([1,2,3,5,4,4,1,3,2,1,3,4])

no_data_a=1
no_data_b=2

a_clean=array([4,4,5,4,4,4])
b_clean=array([3,5,4,4,3,4])

after i need to calculate unique combination in pairs to count the observations
and obtain
(4,3,2)
(4,5,1)
(5,4,1)
(4,4,2)

For the fist task i did

a_No_data_a = a[a != no_data_a]
b_No_data_a = b[a != no_data_a]

b_clean = b_No_data_a[b_No_data_a != no_data_b]
a_clean = a_No_data_a[a_No_data_a != no_data_b]

but the results are not really stable.

For the second task
The np.unique would solve the problem if it can be apply to a two arrays.

Any idea?
thanks in advance
Giuseppe

 

[MRAB]

Hi,
would like to take eliminate a specific number in an array and its correspondent in an other array, and vice-versa.

given

a=np.array([1,2,4,4,5,4,1,4,1,1,2,4])
b=np.array([1,2,3,5,4,4,1,3,2,1,3,4])

no_data_a=1
no_data_b=2

a_clean=array([4,4,5,4,4,4])
b_clean=array([3,5,4,4,3,4])

after i need to calculate unique combination in pairs to count the observations
and obtain
(4,3,2)
(4,5,1)
(5,4,1)
(4,4,2)

For the fist task i did

a_No_data_a = a[a != no_data_a]
b_No_data_a = b[a != no_data_a]

b_clean = b_No_data_a[b_No_data_a != no_data_b]
a_clean = a_No_data_a[a_No_data_a != no_data_b]

but the results are not really stable.

mask = (a != no_data_a) & (b != no_data_b)
a_clean = a[mask]
b_clean = b[mask]

 

[MRAB]

Hi,
would like to take eliminate a specific number in an array and its correspondent in an other array, and vice-versa.

given

a=np.array([1,2,4,4,5,4,1,4,1,1,2,4])
b=np.array([1,2,3,5,4,4,1,3,2,1,3,4])

no_data_a=1
no_data_b=2

a_clean=array([4,4,5,4,4,4])
b_clean=array([3,5,4,4,3,4])

after i need to calculate unique combination in pairs to count the observations
and obtain
(4,3,2)
(4,5,1)
(5,4,1)
(4,4,2)

For the fist task i did

a_No_data_a = a[a != no_data_a]
b_No_data_a = b[a != no_data_a]

b_clean = b_No_data_a[b_No_data_a != no_data_b]
a_clean = a_No_data_a[a_No_data_a != no_data_b]

but the results are not really stable.

mask = (a != no_data_a) & (b != no_data_b)
a_clean = a[mask]
b_clean = b[mask]

For the second task
The np.unique would solve the problem if it can be apply to a two arrays.

I couldn't figure out how to do the second part in numpy, so:

from collections import Counter
counts = Counter(zip(a_clean, b_clean))
counts = [pair + (count,) for pair, count in counts.items()]

 

[Terry Reedy]

Hi,
would like to take eliminate a specific number in an array and its correspondent in an other array, and vice-versa.

given

a=np.array([1,2,4,4,5,4,1,4,1,1,2,4])
b=np.array([1,2,3,5,4,4,1,3,2,1,3,4])

no_data_a=1
no_data_b=2

a_clean=array([4,4,5,4,4,4])
b_clean=array([3,5,4,4,3,4])

As I discovered when running the solution before, your test data are
wrong, leaving out 2,3 before the last pair (4,4). Anyway, for those
interested in a plain Python solution, without numpy:

a=[1,2,4,4,5,4,1,4,1,1,2,4]
b=[1,2,3,5,4,4,1,3,2,1,3,4]

no_data_a=1
no_data_b=2

a_clean=(4,4,5,4,4,2,4)
b_clean=(3,5,4,4,3,3,4)

cleaned = list(zip(*(pair for pair in zip(a,b)
if pair[0] != no_data_a and pair[1] != no_data_b)))
print(cleaned, cleaned == [a_clean, b_clean])
#
[(4, 4, 5, 4, 4, 2, 4), (3, 5, 4, 4, 3, 3, 4)] True

 

[giuseppe.amatulli]

Terry and MRAB,
thanks for yours suggestions,
in the end i found this solution


mask=( a != 0 ) & ( b != 0 )

a_mask=a[mask]
b_mask=b[mask]

array2D = np.array(zip(a_mask,b_mask))

unique=dict()
for row in array2D :
row = tuple(row)
if row in unique:
unique[row] += 1
else:
unique[row] = 1

print unique
{(4, 5): 1, (5, 4): 1, (4, 4): 2, (2, 3): 1, (4, 3): 2}

I choose this solution because i could not install "from collections import Counter".
Anyway how i can print to a file the unique results without the brackets and obtain something like this?
4 5 1
5 4 1
4 4 2
2 3 1
4 3 2

Thanks in advance
Best regards.
Giuseppe

 

[Dave Angel]

<SNIP>

print unique
{(4, 5): 1, (5, 4): 1, (4, 4): 2, (2, 3): 1, (4, 3): 2}

I choose this solution because i could not install "from collections import Counter".

Nothing to install, at least for Python 2.7. collections is in the
standard library, and Counter is in collections (new in version 2.7)

Anyway how i can print to a file the unique results without the brackets and obtain something like this?
4 5 1
5 4 1
4 4 2
2 3 1
4 3 2

To print out a dict in an explicit format, you want a loop.

for key, val in unique.items():
print key[0], key[1], val

Note that you cannot guarantee the order they will print out, once
stored in a dict. You may want to sort them, or otherwise order them if
it matters.

<SNIP>

Note I added my responses after the parts of your message that I was
quoting. To put the response first is called top-posting, and against
policy here.

 

[Terry Reedy]

Terry and MRAB,
thanks for yours suggestions,
in the end i found this solution


mask=( a != 0 ) & ( b != 0 )

a_mask=a[mask]
b_mask=b[mask]

array2D = np.array(zip(a_mask,b_mask))

unique=dict()
for row in array2D :
row = tuple(row)
if row in unique:
unique[row] += 1
else:
unique[row] = 1

I believe the 4 lines above are equivalent to
unique[row] = unique.get(row, 0) + 1

 

[Hans Mulder]

On 8/9/2012 4:06 PM, (e-mail address removed) wrote: [...]

unique=dict()
for row in array2D :
row = tuple(row)
if row in unique:
unique[row] += 1
else:
unique[row] = 1

I believe the 4 lines above are equivalent to
unique[row] = unique.get(row, 0) + 1

I would write that bit as:

from collections import defaultdict

unique = defaultdict(int)
for row in array2D:
unique[row] += 1


Hope this helps,

-- HansM