There is a list of storage registers holding a number.
Each storage register has an address, itself a number.
The addresses are sequential: e.g. 0, 1, 2, 3, ..., N.
The contents of a storage register can be read, and its
contents interpreted as a machine instruction and
executed.
The contents of a storage register can be interpreted
as an address, A, to fetch the contents of the storage
register at address A.
That's it. That is the only difficulty. The rest is
details and thinking straight---a chronic difficulty
for some, but not an inherent difficulty of pointers.
Pointer values are more than just the addresses of things; they are a
higher level construct than that, and most of the problems associated
with them occur at the higher level. The most important high-level
feature of pointers, which is not explicitly covered by your
description, is the fact that they can be invalid. They can become
invalid for some purposes while remaining valid for others:
1. comparison of two pointer values for equality
2. comparison of two pointer values for relative order
3. reading through the pointer value
4. writing through the pointer value
Your description of hardware addresses incorrectly implies that
comparison of addresses for equality should always be safe, and that
comparison of addresses for relative order should always produce
consistent results; there are real-world fully-conforming
implementations of C for which that's not the case for C pointer values.
Uninitialized pointer objects, and pointer objects whose representation
has been manipulated by means other than assignment through an lvalue of
a compatible type, may have trap representations. The non-existent value
of such pointer objects, and pointer values that used to point at
objects whose lifetime has since ended, have undefined behavior for all
four uses.
Null pointer values can only be compared for equality with other pointer
values; the other three uses all have undefined behavior.
Comparing pointer values for relative order that do not point within (or
one past the end of) the same array object (with single objects being
treated as 1-element arrays) has undefined behavior.
Attempting to write through a pointer value that points at an object
defined as 'const' has undefined behavior; the same is true of any
pointer value that points into the array corresponding to a string literal.
Likely we are thinking of different notions when using
the word "difficulty."
Possibly; in my experience, most of the difficulty with C pointers comes
from the necessity of avoiding the undefined behavior referred to above,
an issue not even hinted at by your description of hardware addresses.
With care, it's possible to avoid most such problems; but care is indeed
required, and it's not always easy. To say it's "no difficulty" is to
imply that such care is unnecessary.