HTTPClient Suddenly Fails - Cannot Verify Certificate File

[Mike Papper]

Today (Tues. Aug 24 2010) using HTTPClient for a SSL connection fails.
It was working fine previously. The issue is not with the server at the
other end but with httpclient reading its local cert. file. the error
is:

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3
read server certificate B: certificate verify failed

We tried directly pointing to the installed httpclient 2.1.5.2 cert
file:

client = HTTPClient.new(nil, "WSDL4R")
client.ssl_config.set_trust_ca('/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/cacert.p7s')

but get the same error. We have also tried "other" .pem files such as
this:
client.ssl_config.set_trust_ca('/etc/openldap/cacerts')
but get the same error.

NOTE: last year, we had the same error and fixed it by upgrading to
2.1.5.2 of the httpclient gem (which presumably had a cert. that was not
expired).

NOTE: we ran openssl verify on both the 2.1.5.2 AND the 2.1.4 gem
version of lib/httpclient/cacert_sha1.p7s and that program reported 'OK'
for both files. Thus, I do not trust that the program verify is useful
in this case.

Any ideas here??

Here is a stack trace when the call to httpclient fails:
==================Stack Trace==================
loading wsdl: https://secure.more.com/ws/profile/authenticate?WSDL
at depth 0 - 20: unable to get local issuer certificate
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3
read server certificate B: certificate verify failed
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient/session.rb:247:in `connect'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient/session.rb:247:in `ssl_connect'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient/session.rb:639:in `connect'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient/timeout.rb:128:in `timeout'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient/session.rb:631:in `connect'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient/session.rb:522:in `query'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient/session.rb:147:in `query'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient.rb:953:in `do_get_block'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient.rb:765:in `do_request'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient.rb:848:in `protect_keep_alive_disconnected'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient.rb:764:in `do_request'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient.rb:833:in `follow_redirect'
from /home/rgm/.gem/ruby/1.8/gems/httpclient-2.1.5.2/lib/
httpclient.rb:519:in `get_content'
from /u/apps/more/releases/20100803213846/lib/
registration_service.rb:47:in `fetch'
from /usr/local/lib/ruby/1.8/wsdl/xmlSchema/importer.rb:30:in
`import'
from /usr/local/lib/ruby/1.8/wsdl/importer.rb:18:in `import'
from /usr/local/lib/ruby/1.8/soap/wsdlDriver.rb:124:in `import'
from /usr/local/lib/ruby/1.8/soap/wsdlDriver.rb:28:in
`initialize'
from /u/apps/more/releases/20100803213846/lib/
registration_service.rb:337:in `new'
from /u/apps/more/releases/20100803213846/lib/
registration_service.rb:337:in `create_driver'
from /u/apps/more/releases/20100803213846/lib/
registration_service.rb:297:in `authenticate_driver'
from /u/apps/more/releases/20100803213846/lib/
registration_service.rb:190:in `authenticate'

Any ideas are appreciated, thanks,

Mike

 

[Richard Conroy]

[Note: parts of this message were removed to make it a legal post.]

Today (Tues. Aug 24 2010) using HTTPClient for a SSL connection fails.
It was working fine previously. The issue is not with the server at the
other end but with httpclient reading its local cert. file. the error
is:

Dumb question (you did ask for *any* advice)

Have there been any changes in the certificate status of the site
you are connecting to? Any expiries for instance?

Are you certain the problem is at the local (HttpClient) end?

 

[Mike Papper]

Dumb question (you did ask for *any* advice)

Have there been any changes in the certificate status of the site
you are connecting to? Any expiries for instance?

Are you certain the problem is at the local (HttpClient) end?

Hi, I have tried to connect to other sites using the rails console to
call HTTPClient.new and connecting. Offhand I cannot remember the url (I
may be able to look it up tomorrow).

My understanding is that this is an issue with httpclient reading the
local 'cert' file and having issues with it. At the point of error, I
believe that the server has not been contacted yet. But my
'understanding' is definitely theory.

Mike

 

[Mike Papper]

Heres is a test I ran from the rails console to verify that the issue is
from my local machine, not the endpoint:

--------------------------

require 'httpclient'
uri='https://www.rsa.com'
=> "https://www.rsa.com"=> #<HTTPClient:0x2b2ff0882a48
@www_auth=#<HTTPClient::WWWAuth:0x2b2ff08829a8
@authenticator=[#<HTTPClient::NegotiateAuth:0x2b2ff0882728
@scheme="Negotiate", @challenge={}, @ntlm_opt={:ntlmv2=>true},
@auth_default=nil, @auth={}>,
#<HTTPClient::SSPINegotiateAuth:0x2b2ff0882688 @scheme="Negotiate",
@challenge={}>, #<HTTPClient:igestAuth:0x2b2ff08827c8
@scheme="Digest", @challenge={}, @nonce_count=0, @auth={}>,
#<HTTPClient::BasicAuth:0x2b2ff08827f0 @scheme="Basic", @cred=nil,
@challengeable={}, @auth={}>],
@negotiate_auth=#<HTTPClient::NegotiateAuth:0x2b2ff0882728
@scheme="Negotiate", @challenge={}, @ntlm_opt={:ntlmv2=>true},
@auth_default=nil, @auth={}>,
@basic_auth=#<HTTPClient::BasicAuth:0x2b2ff08827f0 @scheme="Basic",
@cred=nil, @challengeable={}, @auth={}>,
@sspi_negotiate_auth=#<HTTPClient::SSPINegotiateAuth:0x2b2ff0882688
@scheme="Negotiate", @challenge={}>,
@digest_auth=#<HTTPClient:igestAuth:0x2b2ff08827c8 @scheme="Digest",
@challenge={}, @nonce_count=0, @auth={}>>,
@ssl_config=#<HTTPClient::SSLConfig:0x2b2ff0881a08
@ciphers="ALL:!ADH:!LOW:!EXP:!MD5:+SSLv2STRENGTH",
@cert_store=#<OpenSSL::X509::Store:0x2b2ff08814e0>, @verify_depth=nil,
@timeout=nil, @client_key=nil, @options=16781303,
@client=#<HTTPClient:0x2b2ff0882a48 ...>, @dest=nil, @client_cert=nil,
@verify_callback=nil, @client_ca=nil, @verify_mode=3>,
@session_manager=#<HTTPClient::SessionManager:0x2b2ff0881a58
@connect_retry=1, @sess_pool_mutex=#<Mutex:0x2b2ff08815f8>,
@protocol_version=nil, @from=nil, @read_block_size=16384,
@ssl_config=#<HTTPClient::SSLConfig:0x2b2ff0881a08
@ciphers="ALL:!ADH:!LOW:!EXP:!MD5:+SSLv2STRENGTH",
@cert_store=#<OpenSSL::X509::Store:0x2b2ff08814e0>, @verify_depth=nil,
@timeout=nil, @client_key=nil, @options=16781303,
@client=#<HTTPClient:0x2b2ff0882a48 ...>, @dest=nil, @client_cert=nil,
@verify_callback=nil, @client_ca=nil, @verify_mode=3>,
@connect_timeout=60, @sess_pool=[], @agent_name=nil,
@receive_timeout=60, @client=#<HTTPClient:0x2b2ff0882a48 ...>,
@chunk_size=4096, @proxy=nil, @test_loopback_http_response=[],
@debug_dev=nil, @send_timeout=120, @socket_sync=true,
@protocol_retry_count=5>, @no_proxy=nil,
@redirect_uri_callback=#<Method:
HTTPClient#default_redirect_uri_callback>,
@proxy_auth=#<HTTPClient:roxyAuth:0x2b2ff0882840
@authenticator=[#<HTTPClient::NegotiateAuth:0x2b2ff0882318
@scheme="Negotiate", @challenge={}, @ntlm_opt={:ntlmv2=>true},
@auth_default=nil, @auth={}>,
#<HTTPClient::SSPINegotiateAuth:0x2b2ff0881f08 @scheme="Negotiate",
@challenge={}>, #<HTTPClient::BasicAuth:0x2b2ff0882340 @scheme="Basic",
@cred=nil, @challengeable={}, @auth={}>],
@negotiate_auth=#<HTTPClient::NegotiateAuth:0x2b2ff0882318
@scheme="Negotiate", @challenge={}, @ntlm_opt={:ntlmv2=>true},
@auth_default=nil, @auth={}>,
@basic_auth=#<HTTPClient::BasicAuth:0x2b2ff0882340 @scheme="Basic",
@cred=nil, @challengeable={}, @auth={}>,
@sspi_negotiate_auth=#<HTTPClient::SSPINegotiateAuth:0x2b2ff0881f08
@scheme="Negotiate", @challenge={}>>, @proxy=nil, @debug_dev=nil,
@cookie_manager=#<WebAgent::CookieManager:0x2b2ff08815d0 @cookies=[],
@accept_domains=[], @reject_domains=[], @is_saved=true,
@netscape_rule=false, @cookies_file=nil>, @test_loopback_response=[],
@follow_redirect_count=10,
@request_filter=[#<HTTPClient:roxyAuth:0x2b2ff0882840
@authenticator=[#<HTTPClient::NegotiateAuth:0x2b2ff0882318
@scheme="Negotiate", @challenge={}, @ntlm_opt={:ntlmv2=>true},
@auth_default=nil, @auth={}>,
#<HTTPClient::SSPINegotiateAuth:0x2b2ff0881f08 @scheme="Negotiate",
@challenge={}>, #<HTTPClient::BasicAuth:0x2b2ff0882340 @scheme="Basic",
@cred=nil, @challengeable={}, @auth={}>],
@negotiate_auth=#<HTTPClient::NegotiateAuth:0x2b2ff0882318
@scheme="Negotiate", @challenge={}, @ntlm_opt={:ntlmv2=>true},
@auth_default=nil, @auth={}>,
@basic_auth=#<HTTPClient::BasicAuth:0x2b2ff0882340 @scheme="Basic",
@cred=nil, @challengeable={}, @auth={}>,
@sspi_negotiate_auth=#<HTTPClient::SSPINegotiateAuth:0x2b2ff0881f08
@scheme="Negotiate", @challenge={}>>,
#<HTTPClient::WWWAuth:0x2b2ff08829a8
@authenticator=[#<HTTPClient::NegotiateAuth:0x2b2ff0882728
@scheme="Negotiate", @challenge={}, @ntlm_opt={:ntlmv2=>true},
@auth_default=nil, @auth={}>,
#<HTTPClient::SSPINegotiateAuth:0x2b2ff0882688 @scheme="Negotiate",
@challenge={}>, #<HTTPClient:igestAuth:0x2b2ff08827c8
@scheme="Digest", @challenge={}, @nonce_count=0, @auth={}>,
#<HTTPClient::BasicAuth:0x2b2ff08827f0 @scheme="Basic", @cred=nil,
@challengeable={}, @auth={}>],
@negotiate_auth=#<HTTPClient::NegotiateAuth:0x2b2ff0882728
@scheme="Negotiate", @challenge={}, @ntlm_opt={:ntlmv2=>true},
@auth_default=nil, @auth={}>,
@basic_auth=#<HTTPClient::BasicAuth:0x2b2ff08827f0 @scheme="Basic",
@cred=nil, @challengeable={}, @auth={}>,
@sspi_negotiate_auth=#<HTTPClient::SSPINegotiateAuth:0x2b2ff0882688
@scheme="Negotiate", @challenge={}>,
@digest_auth=#<HTTPClient:igestAuth:0x2b2ff08827c8 @scheme="Digest",
@challenge={}, @nonce_count=0, @auth={}>>]>at depth 3 - 20: unable to get local issuer certificate
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read
server certificate B: certificate verify failed
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:247:in
`connect'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:247:in
`ssl_connect'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:639:in
`connect'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/timeout.rb:128:in
`timeout'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:631:in
`connect'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:522:in
`query'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:147:in
`query'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:953:in
`do_get_block'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:765:in
`do_request'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:848:in
`protect_keep_alive_disconnected'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:764:in
`do_request'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:666:in
`request'
from
/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:591:in
`get'
from (irb):7

The last error response mentions "local issuer certificate" which leads
me to believe the issue is opening the cacerts.p7s file.

Incidently, adding in:
client.ssl_config.set_trust_ca('/usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/cacert.p7s')

has no effect. I also tried calling client.ssl_config.set_trust_ca and
pointing to "some other .pem file that sitting on the linux box
somewhere" but that made no difference either.

Mike